You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-24 - 05:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Sun May 24, 2026 00:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
104832179157,15sleep30-21swapper/319:07:023
105552173101,28sleep10-21swapper/119:07:561
104572161104,27sleep20-21swapper/219:06:412
821121600,1sleep10-21swapper/122:52:591
103682160100,53sleep00-21swapper/019:05:310
1840621420,1sleep10-21swapper/119:25:211
43522770,2sleep00-21swapper/023:14:370
190092750,1sleep10-21swapper/120:40:241
98182690,1chrt0-21swapper/000:05:380
288762650,1sleep00-21swapper/023:55:430
187652630,8sleep012-21ksoftirqd/019:25:240
314972620,7sleep331499-21grep21:10:163
29852560,1sleep20-21swapper/200:00:312
173172560,1sleep20-21swapper/221:23:562
173172560,1sleep20-21swapper/221:23:552
118732530,7sleep012-21ksoftirqd/022:07:230
118732530,7sleep012-21ksoftirqd/022:07:230
215862490,2sleep00-21swapper/021:26:260
163722480,1sleep10-21swapper/122:59:251
160562470,2sleep20-21swapper/222:59:062
189892460,2sleep10-21swapper/122:36:341
10989994614,18cyclictest17415-21sh00:11:322
10989994614,18cyclictest17415-21sh00:11:322
10989994210,18cyclictest27118-21ls23:55:132
1098999409,16cyclictest30249-21diskmemload00:36:572
17082390,2sleep20-21swapper/223:11:582
1098999379,15cyclictest30249-21diskmemload22:48:462
1098999379,15cyclictest30249-21diskmemload22:48:462
10989993510,17cyclictest30249-21diskmemload23:23:432
1099099341,8cyclictest32434-21munin-run20:00:003
1098999341,27cyclictest6247-21ntp_states23:15:222
1098999330,9cyclictest30424-21ssh23:33:412
1098999330,6cyclictest17762-21expr23:00:132
210692320,1chrt0-21swapper/322:14:533
1098999322,28cyclictest21468-21cpuspeed_turbos00:15:152
1098999321,4cyclictest0-21swapper/220:30:102
1098999320,6cyclictest6667-21/usr/sbin/munin23:40:172
1098999318,20cyclictest9221-21kworker/u8:022:40:222
1098999318,15cyclictest30249-21diskmemload22:21:432
1098999313,23cyclictest0-21swapper/223:50:122
1098999310,3cyclictest31232-21ssh21:10:132
1098999301,5cyclictest12300-21timerandwakeup19:10:232
10989993014,14cyclictest31186-21sh21:34:172
1098999301,25cyclictest19761-21/usr/sbin/munin21:25:202
10989993012,16cyclictest24092-21/usr/sbin/munin23:05:252
1098999300,3cyclictest30249-21diskmemload00:28:532
1098999300,3cyclictest2462-21awk22:00:212
1099099290,27cyclictest13270-21processes21:20:213
1099099290,27cyclictest13270-21processes21:20:213
1098999296,19cyclictest648-21ntpd21:40:122
1098999293,23cyclictest19571-21sort20:45:002
1098999291,25cyclictest16128-21ssh22:35:122
1098999291,23cyclictest219252chrt19:35:012
10989992910,16cyclictest19301-21gltestperf23:25:162
10989992910,16cyclictest18284-21sed19:25:192
1098999290,7cyclictest30635-21cut19:55:132
1098999290,3cyclictest3066-21ssh21:36:372
1098899291,26cyclictest10552-21sh21:18:571
215892280,3sleep122-21ksoftirqd/122:15:121
130682280,2sleep00-21swapper/021:44:530
1099099284,7cyclictest22400-21ssh21:27:143
1099099283,8cyclictest29880-21cat22:45:163
1099099283,8cyclictest29880-21cat22:45:163
1098999287,18cyclictest27701-21unixbench_singl21:00:242
1098999286,18cyclictest30895-1kworker/u9:121:15:172
1098999282,21cyclictest0-21swapper/221:05:122
1098999282,21cyclictest0-21swapper/220:10:102
1098999281,6cyclictest21199-21latency_hist22:15:002
1098999281,4cyclictest8950-21cron22:30:002
1098999281,24cyclictest0-21swapper/222:50:142
10989992810,16cyclictest636-21runrttasks20:00:352
1098999280,4cyclictest11357-21ssh00:31:062
1098999280,3cyclictest22090-21vmstat21:50:252
1098999280,3cyclictest14069-21df_inode21:45:172
1098999280,3cyclictest13779-21ssh22:33:052
126552270,1sleep3331rcuc/322:55:433
1099099272,10cyclictest20955-21cat00:15:003
1099099272,10cyclictest20955-21cat00:15:003
1098999277,18cyclictest624-21gdbus20:55:002
1098999277,17cyclictest0-21swapper/219:25:002
1098999273,5cyclictest13281-21grep20:30:002
1098999273,19cyclictest27106-21apt-get21:55:122
1098999271,3cyclictest0-21swapper/222:15:162
1098999271,23cyclictest11721-21ssh22:07:142
1098999271,23cyclictest11721-21ssh22:07:142
1098999271,11cyclictest8904-21cat20:20:002
1098999270,4cyclictest3159-21sed20:05:172
1098999270,3cyclictest8675-21sed00:05:172
10988992723,2cyclictest0-21swapper/121:20:031
10988992723,2cyclictest0-21swapper/121:20:031
151592260,1chrt8547-21perf00:34:550
1099099266,6cyclictest34-21ksoftirqd/300:09:283
1099099263,9cyclictest7575-21diskmemload22:52:223
1099099262,6cyclictest13280-21ssh22:08:483
1099099262,6cyclictest13280-21ssh22:08:483
1099099260,12cyclictest34-21ksoftirqd/323:45:513
1098999264,3cyclictest3828-1kworker/u9:223:39:112
1098999261,9cyclictest28-21ksoftirqd/219:40:132
1098999260,3cyclictest28648-21hwlatdetect00:20:172
1098799263,16cyclictest2299-21ssh00:25:120
1098799261,24cyclictest0-21swapper/021:21:010
1098799261,24cyclictest0-21swapper/021:21:010
1099099253,7cyclictest31417-21ssh21:58:013
1099099253,7cyclictest29513-21ssh23:32:473
1099099253,6cyclictest30249-21diskmemload23:03:273
1099099253,6cyclictest30249-21diskmemload00:27:433
1099099251,8cyclictest27142-21sh23:06:443
1099099251,8cyclictest14928-21sh00:34:413
1099099251,5cyclictest0-21swapper/300:00:013
10990992510,9cyclictest34-21ksoftirqd/323:36:093
1099099250,13cyclictest34-21ksoftirqd/323:19:163
1098999259,13cyclictest9677-21if_enp3s020:20:182
1098999258,14cyclictest30286-21latency_hist19:55:002
1098999254,18cyclictest26411-21cpuspeed_turbos19:45:142
1098999251,7cyclictest21502-21munin-plugin-st19:35:002
1098999251,4cyclictest13578-21grep23:45:152
1098999251,3cyclictest0-21swapper/220:45:112
10988992519,4cyclictest6508-21/usr/sbin/munin23:15:241
10987992519,4cyclictest12-21ksoftirqd/000:39:590
10987992515,6cyclictest6539-21sh23:40:030
233892240,2sleep30-21swapper/320:50:263
143182240,2sleep314281-21ssh21:45:193
1099099243,7cyclictest20751-21ssh00:38:353
1099099243,6cyclictest34-21ksoftirqd/321:43:273
1099099243,6cyclictest30249-21diskmemload23:51:553
1099099243,6cyclictest30249-21diskmemload22:34:013
1099099243,6cyclictest28869-21sh21:31:593
1099099243,6cyclictest24764-21sh23:29:463
1099099243,5cyclictest27567-21if_enp3s023:55:173
1099099242,7cyclictest26219-21sh00:18:413
1099099242,6cyclictest30249-21diskmemload22:44:273
10990992421,2cyclictest331rcuc/322:36:153
1099099240,7cyclictest34-21ksoftirqd/322:17:443
1098999241,8cyclictest15052-21cron19:20:002
1098999241,6cyclictest24428-21fschecks_count20:55:142
1098999241,6cyclictest16559-21uname20:35:212
1098799242,7cyclictest18694-21apt-get23:25:120
1098799242,16cyclictest30249-21diskmemload23:15:150
1098799242,15cyclictest11955-21ssh23:20:120
235302230,3sleep10-21swapper/100:15:581
1099099233,6cyclictest21100-21irqstats21:50:173
10990992321,1cyclictest0-21swapper/321:19:333
1099099231,8cyclictest34-21ksoftirqd/322:27:173
1099099230,8cyclictest5923-21sh21:39:293
1099099230,7cyclictest34-21ksoftirqd/323:41:363
1099099230,13cyclictest34-21ksoftirqd/320:10:203
10988992315,6cyclictest25219-21sensors_temp20:55:221
10988992313,7cyclictest15658-21df19:20:131
10988992313,7cyclictest10944-21cron20:25:011
1098799233,17cyclictest26001-21cron19:45:000
1098799233,14cyclictest28012-21ssh00:20:140
10987992321,1cyclictest0-21swapper/021:15:220
10987992317,4cyclictest12-21ksoftirqd/022:55:000
1099099223,7cyclictest34-21ksoftirqd/323:11:573
1099099222,8cyclictest24025-21date19:40:003
1099099221,6cyclictest648-21ntpd00:21:563
1099099221,6cyclictest34-21ksoftirqd/320:35:163
1099099221,4cyclictest0-21swapper/322:00:263
10990992213,6cyclictest18632-21sensors19:25:233
1099099220,7cyclictest34-21ksoftirqd/323:22:573
10988992214,4cyclictest8344-21tr22:05:141
10988992214,4cyclictest8344-21tr22:05:141
10988992213,6cyclictest31384-21tune2fs21:10:151
10987992216,4cyclictest12-21ksoftirqd/022:34:590
10987992216,4cyclictest12-21ksoftirqd/021:50:000
10987992212,7cyclictest4841-21cron21:15:000
10987992210,3cyclictest12-21ksoftirqd/022:26:380
327202210,1sleep012-21ksoftirqd/023:35:160
324502210,3chrt0-21swapper/122:46:341
324502210,3chrt0-21swapper/122:46:331
1099099215,8cyclictest34-21ksoftirqd/319:20:573
1099099213,6cyclictest34-21ksoftirqd/322:21:303
1099099211,6cyclictest34-21ksoftirqd/319:43:073
1099099211,6cyclictest34-21ksoftirqd/319:30:243
1099099211,6cyclictest13611-21cut19:15:163
10990992113,5cyclictest278-21systemd-journal21:10:003
1099099210,6cyclictest26199-21/usr/sbin/munin21:00:213
10988992116,4cyclictest22-21ksoftirqd/123:15:001
10988992115,4cyclictest22-21ksoftirqd/121:35:001
10988992113,6cyclictest5388-21if_err_enp3s020:10:161
10988992113,6cyclictest26526-21seq21:54:511
10988992113,6cyclictest14069-21seq23:20:481
10988992113,6cyclictest13911-21hddtemp_smartct20:30:151
10988992112,7cyclictest14715-21ntpq21:45:231
10988992110,3cyclictest141rcu_preempt22:02:311
1098799211,6cyclictest0-21swapper/022:47:410
1098799211,6cyclictest0-21swapper/022:47:400
10987992112,6cyclictest16818-21awk19:20:250
10987992111,6cyclictest52750irq/139-enp3s0-23:45:140
10987992110,8cyclictest24126-21apt-get23:05:110
44552200,2sleep10-21swapper/100:25:541
248962200,4sleep1221ksoftirqd/121:29:441
159152200,2chrt0-21swapper/100:35:121
1099099203,4cyclictest12129-21ntp_states20:25:203
10990992012,6cyclictest7245-21diskstats20:15:163
10990992010,7cyclictest608-21cat20:00:163
1098899209,9cyclictest30759-21ssh23:34:011
10988992015,3cyclictest30249-21diskmemload23:28:351
10988992014,5cyclictest22-21ksoftirqd/119:50:001
10988992014,4cyclictest26185-21sh23:54:451
10988992013,4cyclictest15676-21latency_hist22:35:001
10988992012,6cyclictest3121-21seq23:37:041
10988992012,6cyclictest28674-21cat22:20:141
10988992012,6cyclictest28189-21python323:55:241
10988992012,6cyclictest23265-21forks22:40:141
10988992011,6cyclictest15142-21fw_packets00:10:171
10988992011,6cyclictest15142-21fw_packets00:10:171
1098799203,13cyclictest75-21kswapd023:30:130
1098799201,8cyclictest28411-21ssh22:20:130
1098799201,6cyclictest0-21swapper/023:04:390
10987992014,4cyclictest12-21ksoftirqd/023:55:010
10987992013,4cyclictest30249-21diskmemload22:18:410
10987992011,7cyclictest24517-21seq00:16:570
10987992011,6cyclictest13749-21tune2fs20:30:140
260182190,1sleep30-21swapper/321:00:013
10990991912,5cyclictest10144-21gzip20:20:233
10990991911,6cyclictest3494-21rt-features20:05:223
10990991911,6cyclictest27715-21unixbench_singl19:45:263
10990991910,6cyclictest20871-21gzip20:45:223
1099099190,7cyclictest34-21ksoftirqd/320:35:003
1098899197,3cyclictest141rcu_preempt21:10:011
1098899196,3cyclictest7793-21missed_timers21:40:191
1098899191,6cyclictest17388-21/usr/sbin/munin23:00:181
10988991914,3cyclictest15781-21sh23:46:021
10988991913,4cyclictest6942-21sh22:28:001
10988991913,4cyclictest20379-21idleruntime20:45:171
10988991912,5cyclictest9166-21mailstats00:05:221
10988991912,5cyclictest17514-21sed20:40:001
10988991911,6cyclictest3796-21ls20:05:241
10988991911,6cyclictest32279-21seq19:58:181
10988991911,6cyclictest25760-21seq19:41:571
10988991911,6cyclictest2498-21readlink00:00:231
10988991911,6cyclictest14989-21seq19:19:141
10988991911,6cyclictest1456-21python321:35:231
10988991911,6cyclictest11284-21cat20:25:131
10988991910,7cyclictest27994-21apt-get00:20:121
10988991910,6cyclictest8940-21find00:30:111
1098799193,10cyclictest26226-21apt-get21:00:100
1098799192,11cyclictest9070-21apt-get20:20:140
1098799191,5cyclictest0-21swapper/021:33:390
10987991914,4cyclictest12-21ksoftirqd/022:45:000
10987991913,4cyclictest7697-21missed_timers20:15:220
10987991912,5cyclictest16373-21ntp_kernel_err20:35:190
10987991911,6cyclictest2406-21needreboot22:00:210
10987991911,6cyclictest11944-21run-parts22:55:230
10987991911,6cyclictest11461-21expr20:25:130
1098799191,15cyclictest3116-21ls20:05:170
10987991911,5cyclictest16482-21df22:35:140
10987991911,5cyclictest1430-21sendmail-mta22:12:100
1099099189,6cyclictest17977-21df20:40:143
10990991810,6cyclictest28983-21idleruntime19:50:173
1098899189,6cyclictest26921-21cat21:00:161
1098899182,14cyclictest28915-21grep19:50:171
1098899181,7cyclictest26042-21ssh23:05:361
1098899181,4cyclictest0-21swapper/121:56:291
10988991813,3cyclictest8974-21sh23:40:581
10988991813,3cyclictest12915-21latency_hist19:15:001
10988991811,5cyclictest23492-21seq20:51:031
10988991811,5cyclictest21010-21grep19:30:241
1098899181,12cyclictest32700-21apt-get20:00:131
1098899181,12cyclictest21967-21apt-get19:35:131
10988991810,6cyclictest7404-21grep20:15:171
1098799189,6cyclictest560-21dbus-daemon19:40:000
1098799183,5cyclictest3674-21ssh00:01:120
1098799182,5cyclictest0-21swapper/021:56:140
1098799181,4cyclictest0-21swapper/021:50:460
10987991812,4cyclictest12-21ksoftirqd/020:05:010
10987991811,5cyclictest27105-21grep19:45:200
10987991811,5cyclictest24307-21expr20:55:120
10987991811,4cyclictest110342chrt19:10:110
1098799180,2cyclictest141rcu_preempt21:38:010
1099099179,6cyclictest11135-21cut19:10:123
10988991712,3cyclictest17307-21sh22:11:051
1098799179,6cyclictest21685-21ls20:50:000
1098799178,6cyclictest20508-21grep19:30:190
1098799176,9cyclictest21917-21kworker/u8:200:10:020
1098799176,9cyclictest21917-21kworker/u8:200:10:020
1098799175,11cyclictest12-21ksoftirqd/020:50:230
10987991712,3cyclictest30244-21sh21:10:000
10987991712,3cyclictest14201-21processes19:15:220
10987991711,4cyclictest12-21ksoftirqd/020:15:010
10987991711,4cyclictest12-21ksoftirqd/020:00:000
1098799161,11cyclictest28404-21apt-get19:50:140
10987991610,4cyclictest18914-21sed20:40:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional