You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-09 - 23:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue Jun 09, 2026 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
228602189104,78sleep10-21swapper/107:09:041
228692173118,48sleep00-21swapper/007:09:100
227932161106,27sleep30-21swapper/307:08:143
226702159105,28sleep20-21swapper/207:06:352
164202770,2sleep10-21swapper/111:15:181
213602730,2sleep30-21swapper/310:05:583
213602730,2sleep30-21swapper/310:05:573
11492650,9sleep334-21ksoftirqd/311:04:173
296362590,2sleep10-21swapper/111:25:131
22612590,4sleep00-21swapper/011:05:090
252142570,2sleep20-21swapper/212:35:002
297042550,1sleep10-21swapper/111:00:241
41622540,1sleep30-21swapper/311:05:323
231632530,1sleep10-21swapper/112:09:031
96602520,5sleep20-21swapper/211:58:482
235932520,2sleep10-21swapper/112:33:351
146622520,2sleep30-21swapper/312:26:183
190702510,2sleep20-21swapper/211:40:442
323852500,1sleep20-21swapper/211:50:432
10212500,1sleep00-21swapper/010:15:170
45712490,1sleep20-21swapper/210:18:042
284432490,1sleep30-21swapper/310:35:263
241922490,2sleep024193-21sendmail_mailqu08:25:210
196402490,2sleep30-21swapper/312:05:293
163372480,1sleep20-21swapper/209:38:392
285072390,1sleep00-21swapper/009:23:290
285072390,1sleep00-21swapper/009:23:280
159712390,2chrt0-21swapper/212:27:382
233482380,2sleep30-21swapper/309:43:573
23206993710,12cyclictest9947-21diskmemload10:25:240
45782320,4sleep2271rcuc/211:30:202
2320699324,15cyclictest2044-21ssh09:27:430
23207993126,3cyclictest53050irq/142-enp3s0-09:18:081
2320699315,11cyclictest18322-21munin-run12:30:000
2320699310,16cyclictest0-21swapper/011:02:350
23209993023,5cyclictest16918-21ssh09:15:023
2320799301,3cyclictest30872-21ssh09:25:151
2320799300,6cyclictest6383-21ls12:20:201
2320699303,16cyclictest996-21sh11:51:450
2320699303,12cyclictest9947-21diskmemload12:10:290
2320699302,27cyclictest0-21swapper/009:18:070
2320699302,14cyclictest0-21swapper/012:31:430
23209992924,3cyclictest0-21swapper/310:35:013
2320999292,24cyclictest25486-21ssh12:35:143
2320899291,4cyclictest28549-21perl11:00:152
23207992924,3cyclictest27197-21ssh10:59:361
2320699296,8cyclictest10300-21ssh11:35:100
2320699295,8cyclictest9947-21diskmemload09:35:250
2320699293,15cyclictest10689-21sh09:58:420
2320699293,15cyclictest10025-21sh11:59:110
2320899281,3cyclictest31295-21cut12:15:142
2320899281,3cyclictest31295-21cut12:15:132
2320799281,25cyclictest16539-21ssh12:28:141
2320699281,15cyclictest0-21swapper/011:13:040
2320699281,14cyclictest4559-21ssh10:42:060
51062270,4sleep30-21swapper/311:30:243
277752270,4sleep0131rcuc/007:20:130
2320999270,3cyclictest21024-21ssh12:31:013
2320899271,3cyclictest0-21swapper/210:55:002
2320899271,24cyclictest14450-21ssh09:12:472
2320899271,23cyclictest9947-21diskmemload09:25:162
23207992720,5cyclictest30565-21ssh10:37:231
2320799271,7cyclictest16610-21latency_hist11:40:011
2320799271,1cyclictest0-21swapper/109:14:181
2320699275,9cyclictest31426-21/usr/sbin/munin09:50:230
2320699275,9cyclictest19992-21rm11:17:580
2320699273,13cyclictest14461-21ssh09:12:490
2320699273,10cyclictest636-21runrttasks12:05:530
23206992724,2cyclictest0-21swapper/011:20:210
2320699271,2cyclictest0-21swapper/012:35:190
2320699271,25cyclictest0-21swapper/012:15:550
2320699271,25cyclictest0-21swapper/012:15:540
2320699271,25cyclictest0-21swapper/010:59:350
23209992621,3cyclictest9947-21diskmemload09:45:163
2320899261,2cyclictest5415-21/usr/sbin/munin12:20:232
2320899261,2cyclictest0-21swapper/212:35:222
2320899261,23cyclictest9947-21diskmemload09:15:332
2320899261,23cyclictest0-21swapper/212:00:182
2320799264,6cyclictest21316-21/usr/sbin/munin08:20:151
2320799262,7cyclictest22447-21sh11:20:111
23207992624,1cyclictest0-21swapper/112:15:541
23207992624,1cyclictest0-21swapper/112:15:541
23207992617,8cyclictest211rcuc/111:59:001
2320799261,1cyclictest0-21swapper/109:56:181
2320799260,24cyclictest0-21swapper/112:35:211
2320699263,9cyclictest9947-21diskmemload10:05:290
2320699263,9cyclictest9947-21diskmemload10:05:280
2320699262,9cyclictest9947-21diskmemload12:03:410
23206992624,1cyclictest0-21swapper/012:20:350
2320699261,8cyclictest0-21swapper/009:44:060
23206992611,9cyclictest0-21swapper/010:00:110
2320699260,24cyclictest0-21swapper/011:25:190
23209992522,2cyclictest0-21swapper/312:20:143
23209992519,4cyclictest30894-21ssh11:50:153
23209992519,3cyclictest31293-21tr12:15:143
23209992519,3cyclictest31293-21tr12:15:133
2320999251,23cyclictest0-21swapper/310:55:023
2320999251,23cyclictest0-21swapper/309:20:193
2320999251,23cyclictest0-21swapper/309:20:183
2320999251,23cyclictest0-21swapper/309:12:123
2320999251,1cyclictest0-21swapper/311:20:263
23208992522,2cyclictest0-21swapper/210:58:492
23208992516,6cyclictest793-21grep07:30:252
2320899251,23cyclictest0-21swapper/211:25:232
2320899251,1cyclictest0-21swapper/211:23:182
2320799251,8cyclictest13121-21sh10:49:011
23207992511,8cyclictest12436-21apt-get10:00:111
2320799250,12cyclictest249672ps09:20:231
2320799250,12cyclictest249672ps09:20:221
2320699253,9cyclictest12967-21ssh10:48:520
2320699253,8cyclictest27099-21ssh11:47:060
2320699253,7cyclictest0-21swapper/009:49:270
2320699252,10cyclictest7574-21ssh10:20:180
23206992510,13cyclictest4322-21cron09:34:550
186952250,2chrt0-21swapper/310:28:523
317372240,1sleep00-21swapper/007:30:000
299642240,3sleep30-21swapper/308:40:133
23209992413,8cyclictest19225-21tune2fs08:15:163
23208992422,1cyclictest0-21swapper/209:45:152
23208992421,2cyclictest0-21swapper/209:20:122
23208992421,2cyclictest0-21swapper/209:20:122
2320799244,6cyclictest29945-21sh10:12:531
2320799243,7cyclictest20144-21sh11:41:481
2320799243,6cyclictest22-21ksoftirqd/110:32:121
2320799243,6cyclictest22-21ksoftirqd/110:08:431
2320799243,6cyclictest22-21ksoftirqd/110:08:431
2320799242,8cyclictest28042-21sh12:12:151
2320799241,8cyclictest6317-21sh11:31:201
2320799241,8cyclictest12577-21sh11:12:151
23207992416,5cyclictest24739-21ssh11:45:211
2320799240,9cyclictest16179-21sh10:26:191
2320699243,8cyclictest26882-21ssh10:35:110
2320699243,8cyclictest19412-21ssh10:53:400
23209992321,1cyclictest0-21swapper/311:25:113
23209992318,4cyclictest34-21ksoftirqd/307:55:003
23209992318,3cyclictest20987-21latency_hist10:55:013
23209992311,6cyclictest34-21ksoftirqd/311:17:363
23208992312,8cyclictest13419-21netstat10:00:212
2320799233,8cyclictest6841-21ntpq09:00:191
2320799233,6cyclictest22-21ksoftirqd/110:43:281
2320799233,6cyclictest22-21ksoftirqd/109:44:331
2320799232,6cyclictest22-21ksoftirqd/109:35:541
2320799232,6cyclictest2070-21sh09:51:431
2320799232,6cyclictest15734-21sh12:03:121
2320799232,6cyclictest10851-21sh09:34:521
23207992320,2cyclictest211rcuc/109:45:011
2320699233,7cyclictest19242-21ssh11:40:530
2320699232,6cyclictest0-21swapper/011:32:390
2320699231,8cyclictest12-21ksoftirqd/008:05:200
2320699231,7cyclictest12-21ksoftirqd/010:11:370
2320699230,8cyclictest12-21ksoftirqd/008:45:200
256262220,1sleep00-21swapper/008:30:150
2320999222,17cyclictest27529-21cut08:35:123
2320999221,6cyclictest5123-21diskstats09:30:143
23209992213,7cyclictest560-21dbus-daemon09:09:593
23209992213,7cyclictest18750-21cat08:15:013
23208992214,5cyclictest52950irq/141-enp3s0-10:20:132
23208992211,7cyclictest52950irq/141-enp3s0-11:35:112
2320799225,9cyclictest22-21ksoftirqd/107:35:231
2320799223,6cyclictest31306-21seq08:40:501
2320799222,7cyclictest1230-21seq08:48:331
23207992219,2cyclictest211rcuc/111:06:531
2320799221,6cyclictest9393-21sh10:21:111
2320799221,6cyclictest2838-21sh10:16:201
23207992214,6cyclictest30756-21diskstats11:50:141
2320799220,7cyclictest16722-21sh10:50:551
2320699222,6cyclictest12-21ksoftirqd/009:00:440
23206992210,9cyclictest21252-21awk10:30:200
294062210,2sleep30-21swapper/311:49:273
23209992113,6cyclictest13860-21ls10:00:243
23209992112,7cyclictest1560-21ssh09:51:123
23209992112,6cyclictest6377-21fw_packets09:00:143
23209992112,6cyclictest11665-21python311:35:233
2320899218,9cyclictest560-21dbus-daemon09:59:592
2320899217,8cyclictest19175-21ssh10:05:142
2320899217,8cyclictest19175-21ssh10:05:142
2320899213,13cyclictest1222-21ssh10:40:132
2320799212,12cyclictest31827-21apt-get07:30:121
23207992113,6cyclictest19266-21fw_packets08:15:171
23207992112,6cyclictest11599-21cat07:55:251
2320699211,7cyclictest12-21ksoftirqd/008:54:290
2320699211,7cyclictest12-21ksoftirqd/008:23:320
2320699211,7cyclictest12-21ksoftirqd/007:45:130
2320699211,6cyclictest12-21ksoftirqd/008:15:010
2320699211,6cyclictest12-21ksoftirqd/008:13:320
2320699211,6cyclictest12-21ksoftirqd/007:30:440
2320699210,7cyclictest12-21ksoftirqd/008:42:230
2320699210,7cyclictest12-21ksoftirqd/008:35:130
2320699210,7cyclictest12-21ksoftirqd/007:55:190
2320699210,7cyclictest12-21ksoftirqd/007:35:040
2320699210,7cyclictest12-21ksoftirqd/007:15:140
249912200,3sleep20-21swapper/212:10:172
2320999202,11cyclictest9947-21diskmemload10:40:493
23209992012,6cyclictest23509-21cat07:10:123
23209992012,6cyclictest17838-21ls11:40:153
23209992011,7cyclictest27393-21cat07:20:013
2320999201,12cyclictest9947-21diskmemload10:10:203
2320899206,3cyclictest141rcu_preempt11:15:122
2320899205,12cyclictest18734-21missed_timers12:05:172
2320899204,4cyclictest28-21ksoftirqd/209:50:382
2320899203,12cyclictest640-21snmpd10:10:102
2320899201,4cyclictest0-21swapper/207:55:232
23208992011,6cyclictest25227-21wc11:45:252
2320799204,5cyclictest22-21ksoftirqd/107:53:551
2320799202,5cyclictest31361-21seq07:27:271
2320799202,12cyclictest3816-21apt-get07:40:121
23207992013,5cyclictest25685-21perl08:30:151
23207992013,4cyclictest29432-21sh07:24:591
23207992012,6cyclictest2852-21threads08:50:231
23207992011,6cyclictest15686-21python308:05:231
23207992010,7cyclictest6877-21sh07:45:211
2320699209,8cyclictest9409-21wc07:50:250
2320699209,8cyclictest4344-21sed07:40:160
2320699207,10cyclictest23399-21tr07:10:120
2320699207,10cyclictest12890-21grep08:00:150
2320699201,7cyclictest12-21ksoftirqd/009:05:450
23206992011,6cyclictest4517-21taskset08:55:190
2320999199,7cyclictest28373-21date07:20:183
2320999199,5cyclictest0-21swapper/309:35:123
2320999191,8cyclictest9947-21diskmemload09:27:033
2320999191,8cyclictest5196-21/usr/sbin/munin11:55:193
23209991914,3cyclictest361-21cstates10:15:113
23209991911,6cyclictest6932-21apt10:20:133
23209991911,6cyclictest560-21dbus-daemon07:34:593
23209991911,6cyclictest4453-21cat08:55:183
23209991910,6cyclictest17656-21cut12:05:003
2320899198,9cyclictest0-21swapper/210:37:162
2320899197,9cyclictest560-21dbus-daemon07:55:012
2320899197,8cyclictest10711-21grep11:10:242
2320899192,12cyclictest14529-21apt-get08:05:132
23208991912,5cyclictest29032-21awk07:20:262
23208991911,6cyclictest3664-21date08:55:012
2320799191,7cyclictest22-21ksoftirqd/108:35:011
2320799191,6cyclictest24735-21seq08:26:511
2320799191,6cyclictest18335-21seq08:12:251
23207991912,5cyclictest25735-21forks07:15:151
23207991911,6cyclictest4685-21open_files08:55:221
23207991911,5cyclictest13119-21cat08:00:171
23207991910,6cyclictest9038-21awk09:05:211
2320999189,6cyclictest52750irq/139-enp3s0-09:55:133
2320999189,6cyclictest24011-21ntpq08:25:193
2320999188,7cyclictest21307-21cat08:20:143
2320999182,11cyclictest75-21kswapd010:45:103
23209991813,3cyclictest12091-21sh11:11:463
23209991812,4cyclictest636-21runrttasks12:10:143
23209991811,5cyclictest6774-21date07:45:203
23209991811,5cyclictest14696-21cat08:05:153
23209991811,5cyclictest10473-21expr07:55:153
23209991810,6cyclictest1667-21cut07:35:133
2320899189,7cyclictest2267-21grep07:35:172
2320899189,7cyclictest21929-21cut08:20:222
2320899189,7cyclictest13163-21sed08:00:192
2320899189,6cyclictest18273-21grep09:40:132
2320899187,8cyclictest7828-21apt-get10:45:122
2320899182,11cyclictest8054-21apt-get09:05:112
2320899182,11cyclictest640-21snmpd08:55:122
2320899182,11cyclictest13557-21apt-get10:25:112
23208991812,4cyclictest8109-21sh11:09:302
23208991812,4cyclictest29731-21/usr/sbin/munin08:40:192
23208991812,3cyclictest25222-21/usr/sbin/munin08:30:242
23208991811,5cyclictest31130-21awk07:25:232
23208991811,4cyclictest20691-21idleruntime-cro08:19:592
23208991810,6cyclictest917-21awk08:45:262
23208991810,6cyclictest24583-21grep08:25:242
23208991810,4cyclictest636-21runrttasks10:33:592
2320799182,5cyclictest22-21ksoftirqd/107:10:161
2320999179,6cyclictest25371-21cut08:30:133
23209991711,4cyclictest29935-21expr07:25:133
23209991710,5cyclictest4078-21expr07:40:143
2320899179,6cyclictest5344-21head09:30:152
2320899179,5cyclictest5117-21sed07:40:252
23208991712,3cyclictest27363-21/usr/sbin/munin08:35:242
23208991711,4cyclictest6874-21ntp_states07:45:212
23208991710,5cyclictest26753-21sed07:15:242
2320999169,5cyclictest2348-21cat08:50:173
2320999169,5cyclictest12654-21expr08:00:123
2320999167,6cyclictest31948-21awk08:45:133
2320899169,5cyclictest28-21ksoftirqd/209:05:012
2320899169,4cyclictest17971-21timerwakeupswit08:10:222
2320899159,4cyclictest24595-21cat07:10:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional