You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-14 - 07:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Tue Jul 14, 2026 00:43:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1535521650,5sleep2168299cyclictest21:12:532
2178921060,5sleep0168099cyclictest00:35:140
16809910535,53cyclictest0-21swapper/022:05:310
16809910535,53cyclictest0-21swapper/022:05:310
16809910435,44cyclictest14369-21sshd19:45:300
168099996,79cyclictest12375-21diskmemload22:10:310
1680999920,30cyclictest0-21swapper/000:05:330
168199982,31cyclictest29205-21fschecks_count23:25:371
1680999718,54cyclictest0-21swapper/023:55:290
1680999522,58cyclictest27881-21apt22:45:170
257222920,5sleep3168599cyclictest22:02:503
168199923,24cyclictest27472-21ssh22:05:291
168199923,24cyclictest27472-21ssh22:05:291
1680999120,65cyclictest0-21swapper/023:25:350
1680999016,61cyclictest16536-21wget23:10:310
1680998635,25cyclictest4552-21unixbench_multi22:15:470
1680998622,25cyclictest0-21swapper/020:25:310
1680998614,55cyclictest0-21swapper/019:36:470
1682998425,12cyclictest4563-21grep22:15:472
168099846,24cyclictest20638-21ssh23:15:310
1680998421,13cyclictest12359-21cron21:10:170
1680998313,51cyclictest0-21swapper/020:20:400
1680998015,54cyclictest1622-21latency_hist21:35:180
1680998014,49cyclictest10932-21grep00:20:440
1680998014,49cyclictest10932-21grep00:20:440
1680997922,54cyclictest11664-21wget22:25:310
1680997871,5cyclictest0-21swapper/022:56:120
168099783,39cyclictest125-21systemd-journal22:20:470
1680997816,48cyclictest16165-21awk19:50:290
1680997816,48cyclictest16165-21awk19:50:290
1680997814,49cyclictest0-21swapper/020:04:080
168099775,15cyclictest0-21swapper/019:10:300
1680997712,53cyclictest27635-21ssh21:27:010
1680997620,37cyclictest3628-21wget20:45:320
1680997514,41cyclictest646-21aten_rbpower_en23:30:310
1680997414,40cyclictest12737-21taskset21:10:220
28312730,2sleep30-21swapper/321:35:443
1680997321,30cyclictest27129-21rm00:01:060
1680997317,31cyclictest18871-21ssh22:33:590
1680997316,43cyclictest16865-21sh23:50:180
1682997213,8cyclictest7850irq/56-dwc_otg22:05:292
1682997213,8cyclictest7850irq/56-dwc_otg22:05:292
1680997213,45cyclictest13715-21taskset21:49:130
1680997211,41cyclictest9445-21modprobe00:20:180
1680997015,40cyclictest8707-21sh19:30:170
1680997011,44cyclictest0-21swapper/023:38:380
1680996920,45cyclictest17114-21ssh00:29:140
1680996914,44cyclictest7439-21taskset21:40:520
1680996913,42cyclictest0-21swapper/000:12:190
1680996912,48cyclictest0-21swapper/023:50:190
1680996912,41cyclictest0-21swapper/019:20:440
1680996910,46cyclictest0-21swapper/019:45:000
168099689,40cyclictest0-21swapper/022:39:370
168099689,40cyclictest0-21swapper/022:39:370
1680996820,31cyclictest1866-21aten_rbpower_po20:40:320
1680996820,28cyclictest26100-21latency_hist00:40:180
1680996820,27cyclictest0-21swapper/021:00:290
1680996812,42cyclictest0-21swapper/021:21:310
1680996811,42cyclictest0-21swapper/023:05:410
1681996710,43cyclictest0-21swapper/122:15:471
1680996712,43cyclictest0-21swapper/022:00:270
1680996712,36cyclictest12122-21ssh23:44:190
168199663,27cyclictest0-21swapper/122:20:491
168099668,44cyclictest4434-21ssh21:38:040
168099667,45cyclictest0-21swapper/021:15:480
1680996612,38cyclictest9141-21/usr/sbin/munin23:00:430
1680996612,38cyclictest9141-21/usr/sbin/munin23:00:430
1680996610,46cyclictest0-21swapper/021:54:520
1680996511,42cyclictest0-21swapper/023:24:540
1680996510,46cyclictest0-21swapper/022:50:310
1680996418,5cyclictest0-21swapper/020:55:290
1680996410,40cyclictest0-21swapper/021:57:370
168099639,46cyclictest23070-21taskset20:09:390
168099639,40cyclictest0-21swapper/020:37:130
1680996316,37cyclictest28574-21fschecks_time22:45:340
1680996312,37cyclictest25364-21cpuspeed20:15:330
1680996311,14cyclictest0-21swapper/019:30:310
168099629,16cyclictest0-21swapper/020:30:290
168099629,16cyclictest0-21swapper/020:30:290
1682996139,16cyclictest29154-21rm23:25:342
168099619,42cyclictest0-21swapper/019:17:410
212472600,1sleep20-21swapper/223:55:182
1680995911,11cyclictest19387-21seq19:58:400
168599573,14cyclictest4609-21head22:15:483
1685995713,29cyclictest0-21swapper/322:20:483
221352560,2sleep20-21swapper/223:16:092
1680995513,34cyclictest6824-21taskset20:53:470
73722540,2sleep20-21swapper/200:16:362
1680995413,25cyclictest0-21swapper/020:12:250
109172540,4sleep310734-21/usr/sbin/munin19:35:323
168199531,30cyclictest0-21swapper/100:05:331
1685995112,17cyclictest7850irq/56-dwc_otg23:25:353
168299502,26cyclictest0-21swapper/222:20:482
1685994911,8cyclictest8050irq/56-dwc_otg_22:05:303
1685994911,8cyclictest8050irq/56-dwc_otg_22:05:303
1681994914,23cyclictest7950irq/56-dwc_otg_21:45:231
1681994910,8cyclictest7850irq/56-dwc_otg23:45:341
12852483,11sleep00-21swapper/019:08:370
12852483,11sleep00-21swapper/019:08:370
1682994611,4cyclictest0-21swapper/223:10:432
168199468,24cyclictest0-21swapper/120:50:321
168299454,5cyclictest7950irq/56-dwc_otg_20:10:332
168199457,25cyclictest0-21swapper/123:00:311
168199457,25cyclictest0-21swapper/123:00:311
168199449,22cyclictest0-21swapper/119:35:311
168199447,24cyclictest0-21swapper/123:20:311
168199445,8cyclictest7950irq/56-dwc_otg_20:10:281
1685994321,4cyclictest2000450wget22:35:313
1685994321,4cyclictest2000450wget22:35:313
168199431,3cyclictest0-21swapper/122:33:161
28302410,2sleep30-21swapper/322:15:023
168199411,2cyclictest0-21swapper/123:30:451
38472400,5sleep3168599cyclictest00:12:423
1685994018,10cyclictest8050irq/56-dwc_otg_00:23:463
1685994018,10cyclictest8050irq/56-dwc_otg_00:23:463
1682994015,4cyclictest17688-21latency21:15:402
1682994012,6cyclictest29801-21latency_hist21:30:192
1681994015,6cyclictest15691-21switchtime21:50:451
109872400,10sleep339-21ksoftirqd/323:03:253
109872400,10sleep339-21ksoftirqd/323:03:253
99542391,3sleep10-21swapper/121:45:181
168599391,16cyclictest0-21swapper/300:35:393
1682993921,8cyclictest6313-21threads20:50:452
1682993913,5cyclictest15923-21idleruntime-cro19:50:182
168199395,25cyclictest29970-21ssh21:30:291
168199392,7cyclictest8050irq/56-dwc_otg_19:30:301
168599386,22cyclictest26818-21ls20:20:183
1682993818,5cyclictest7990-21processes20:55:422
1682993814,4cyclictest1409-21sh22:12:282
1682993812,4cyclictest0-21swapper/222:45:272
168199386,22cyclictest25552-21ssh00:00:231
168199386,22cyclictest0-21swapper/100:35:311
168199381,9cyclictest20205-21fschecks_count22:35:341
168199381,9cyclictest20205-21fschecks_count22:35:341
1681993812,5cyclictest21-21rcuc/121:20:181
160302380,1sleep30-21swapper/323:48:553
168299378,4cyclictest7950irq/56-dwc_otg_22:30:292
1682993718,4cyclictest23765-21idleruntime-cro22:40:182
1682993718,4cyclictest23765-21idleruntime-cro22:40:182
1682993715,5cyclictest6460-21unixbench_multi19:20:462
1682993714,5cyclictest301ktimersoftd/220:05:322
1682993713,5cyclictest568-21missed_timers22:50:392
1682993712,9cyclictest750-21snmpd21:35:382
1682993711,4cyclictest26768-21timerwakeupswit00:00:462
168199377,16cyclictest8050irq/56-dwc_otg_00:00:171
1681993716,5cyclictest9081-21sh23:40:321
168599361,23cyclictest31022-21ls21:30:433
168299369,9cyclictest8050irq/56-dwc_otg_23:45:462
168299367,11cyclictest7950irq/56-dwc_otg_20:45:172
168199367,19cyclictest0-21swapper/123:50:231
1681993618,5cyclictest5729-21idleruntime-cro21:40:171
168199361,26cyclictest0-21swapper/120:35:311
168199361,17cyclictest7850irq/56-dwc_otg20:45:461
268372350,3sleep326889-21ssh00:00:483
1685993518,6cyclictest17517-21fschecks_count23:50:343
168599351,22cyclictest15524-21ls22:30:183
168299356,14cyclictest8050irq/56-dwc_otg_23:40:422
1682993525,7cyclictest301ktimersoftd/220:15:302
1682993515,13cyclictest14665-21/usr/sbin/munin21:50:402
1682993513,5cyclictest96952chrt21:00:412
1682993513,4cyclictest14241-21fschecks_time00:25:362
1682993510,7cyclictest13574-21unixbench_multi23:05:462
168199357,6cyclictest0-21swapper/121:59:171
1681993523,9cyclictest8050irq/56-dwc_otg_21:25:301
168199351,25cyclictest0-21swapper/123:14:071
286602341,4sleep323237-1kworker/u9:021:28:493
168599345,7cyclictest8050irq/56-dwc_otg_21:40:393
168599344,14cyclictest7950irq/56-dwc_otg_00:30:373
168599342,6cyclictest30625-21ssh00:05:453
168599342,3cyclictest0-21swapper/322:40:283
1685993413,12cyclictest381ktimersoftd/300:29:483
1685993412,8cyclictest25440-21cat00:00:183
168299346,13cyclictest7950irq/56-dwc_otg_22:27:372
168299341,15cyclictest24519-21unixbench_singl22:00:472
168299341,15cyclictest0-21swapper/223:02:202
168299341,15cyclictest0-21swapper/223:02:202
1682993410,5cyclictest7950irq/56-dwc_otg_00:20:332
1682993410,5cyclictest7950irq/56-dwc_otg_00:20:332
168199348,6cyclictest7950irq/56-dwc_otg_21:20:351
168199345,5cyclictest7850irq/56-dwc_otg22:40:321
168199345,5cyclictest7850irq/56-dwc_otg00:25:321
168199345,20cyclictest5611-21ssh00:15:301
168199344,15cyclictest8050irq/56-dwc_otg_20:30:181
168199341,8cyclictest14302-21latency_hist19:45:181
1681993418,5cyclictest1817-21/usr/sbin/munin20:40:321
1681993414,12cyclictest0-21swapper/100:10:331
1681993412,5cyclictest21584-21aten_rbpower_cu20:05:321
271582330,2sleep10-21swapper/122:05:171
168599335,8cyclictest0-21swapper/320:45:343
168599335,14cyclictest8050irq/56-dwc_otg_19:50:183
168599335,12cyclictest8050irq/56-dwc_otg_19:20:463
1685993314,12cyclictest198082chrt21:18:133
168599331,29cyclictest9849-21ls23:40:443
168599331,21cyclictest0-21swapper/323:39:083
168599331,20cyclictest21922-21ls20:05:373
168299339,4cyclictest0-21swapper/200:12:092
168299337,13cyclictest0-21swapper/221:20:462
168299334,7cyclictest7950irq/56-dwc_otg_21:45:422
168299332,6cyclictest0-21swapper/220:05:012
1682993313,4cyclictest29927-21/usr/sbin/munin21:30:352
1682993310,5cyclictest9055-21aten_rbpower_vo19:30:322
168199338,12cyclictest18723-21ps19:55:441
168199337,5cyclictest7850irq/56-dwc_otg19:20:401
168199332,16cyclictest0-21swapper/122:25:351
168199331,7cyclictest4866-21ssh23:35:321
168199331,21cyclictest18258-21ls00:30:321
1681993310,5cyclictest23-21ksoftirqd/123:00:181
1681993310,15cyclictest32172-21apt20:35:181
1681993310,15cyclictest32172-21apt20:35:181
168599329,11cyclictest8050irq/56-dwc_otg_20:50:473
168599323,12cyclictest0-21swapper/319:15:173
168599321,25cyclictest0-21swapper/321:23:463
1685993211,8cyclictest14666-21acpi21:50:293
168599321,17cyclictest8050irq/56-dwc_otg_21:13:293
168599321,16cyclictest0-21swapper/321:50:073
1685993211,5cyclictest19353-21sh23:13:363
168299329,4cyclictest0-21swapper/200:38:232
168299327,4cyclictest8050irq/56-dwc_otg_21:55:442
168299326,13cyclictest8050irq/56-dwc_otg_19:55:472
168299325,6cyclictest7850irq/56-dwc_otg23:23:142
1682993225,5cyclictest12501-21latency_hist21:10:182
168299321,2cyclictest0-21swapper/222:55:302
1682993211,4cyclictest7950irq/56-dwc_otg_19:35:302
1682993211,15cyclictest24162-21ssh22:40:312
168199329,5cyclictest0-21swapper/122:10:471
168199328,16cyclictest0-21swapper/123:07:241
168199327,15cyclictest0-21swapper/122:50:311
168199323,6cyclictest7850irq/56-dwc_otg19:50:471
168199323,6cyclictest7850irq/56-dwc_otg19:50:471
168199323,20cyclictest0-21swapper/120:02:471
1681993216,5cyclictest3631-21basename19:15:321
168199321,4cyclictest0-21swapper/123:15:291
168199321,12cyclictest7850irq/56-dwc_otg22:45:311
168599317,4cyclictest0-21swapper/322:55:343
168599312,17cyclictest727-21runrttasks23:19:353
168599311,24cyclictest0-21swapper/319:52:153
168599311,24cyclictest0-21swapper/319:52:153
168599311,21cyclictest0-21swapper/319:20:103
1685993111,12cyclictest0-21swapper/323:09:143
1685993110,5cyclictest31285-21ntp_offset20:30:413
1685993110,5cyclictest31285-21ntp_offset20:30:413
1685993110,5cyclictest28087-21grep20:20:463
168299317,4cyclictest7950irq/56-dwc_otg_21:40:392
168299317,12cyclictest8050irq/56-dwc_otg_20:25:472
168299315,4cyclictest0-21swapper/223:37:082
168299313,6cyclictest8050irq/56-dwc_otg_23:30:342
1682993115,4cyclictest13706-21grep19:40:462
168199316,10cyclictest9598-21missed_timers21:00:391
168199312,6cyclictest7850irq/56-dwc_otg20:55:421
168199312,19cyclictest0-21swapper/121:10:371
168199311,9cyclictest7468-21fschecks_time19:25:351
168199311,5cyclictest0-21swapper/121:05:381
13512313,18sleep10-21swapper/119:09:311
13512313,18sleep10-21swapper/119:09:311
168599302,17cyclictest8050irq/56-dwc_otg_21:55:313
168599301,6cyclictest8050irq/56-dwc_otg_19:30:393
168599301,14cyclictest0-21swapper/320:11:363
168599301,13cyclictest8050irq/56-dwc_otg_23:20:473
168299308,5cyclictest7950irq/56-dwc_otg_20:20:282
168299307,9cyclictest8050irq/56-dwc_otg_20:30:452
168299307,9cyclictest8050irq/56-dwc_otg_20:30:452
168299307,4cyclictest0-21swapper/223:57:432
168299306,12cyclictest0-21swapper/200:31:412
1682993011,9cyclictest680-21open_inodes20:35:402
1682993010,5cyclictest3899-21fschecks_time20:45:362
168199306,14cyclictest0-21swapper/100:20:291
168199306,14cyclictest0-21swapper/100:20:291
168599295,11cyclictest8050irq/56-dwc_otg_22:45:343
168599294,14cyclictest8050irq/56-dwc_otg_22:50:393
168599293,4cyclictest0-21swapper/300:15:293
168599291,17cyclictest0-21swapper/323:31:283
168599291,16cyclictest8050irq/56-dwc_otg_21:00:453
168599291,12cyclictest0-21swapper/320:35:393
168299299,11cyclictest301ktimersoftd/200:05:442
168299291,5cyclictest0-21swapper/219:10:442
1681992923,4cyclictest27225-21fschecks_count20:20:341
1681992915,4cyclictest1924-21sh19:10:321
14092296,16sleep21438-21munin-run19:10:172
14092296,16sleep21438-21munin-run19:10:172
12152293,19sleep30-21swapper/319:07:433
12152293,19sleep30-21swapper/319:07:433
168599289,3cyclictest0-21swapper/322:32:273
168599289,12cyclictest0-21swapper/320:40:373
168599285,13cyclictest1676-21cyclictest21:05:423
168599281,16cyclictest7255-21aten_rbpower_en20:55:313
168599281,11cyclictest17962-21/usr/sbin/munin19:55:333
168299281,14cyclictest8050irq/56-dwc_otg_19:25:312
168299281,14cyclictest7850irq/56-dwc_otg19:15:512
1685992719,6cyclictest21328-21sh20:05:173
168599271,6cyclictest8050irq/56-dwc_otg_19:25:463
168599271,18cyclictest0-21swapper/320:25:253
1685992710,11cyclictest13482-21ls19:40:453
168199271,5cyclictest125-21systemd-journal20:15:211
1681992617,6cyclictest14664-21fschecks_time19:45:351
168299241,21cyclictest16429-21cut19:50:352
168299241,21cyclictest16429-21cut19:50:352
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional