You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-19 - 11:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Sep 19, 2026 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304450550,0irq/25-eth10-21swapper/318:42:103
2035223526,0sleep20-21swapper/218:39:082
2066223423,0sleep00-21swapper/018:42:470
2054723422,0sleep10-21swapper/118:41:231
207989983,0cyclictest8852-21awk23:18:430
207989983,0cyclictest20175-21awk21:03:440
207989983,0cyclictest19964-21awk21:23:440
207989983,0cyclictest16352-21awk23:43:440
207989982,0cyclictest23944-21irqrtprio00:08:480
207989975,0cyclictest0-21swapper/021:57:330
207989973,0cyclictest3556-21lldpd20:51:430
207989973,0cyclictest3556-21lldpd19:08:400
207989973,0cyclictest29874-21ps19:13:460
207989972,0cyclictest9535-21munin-node19:23:430
207989972,0cyclictest7776-21awk23:58:450
207989972,0cyclictest2820-21awk18:53:440
207989972,0cyclictest2053-21cut23:33:460
207989972,0cyclictest17893-21awk22:43:450
207989971,0cyclictest19054-21awk20:18:460
207989971,0cyclictest16638-21unixbench_singl21:38:540
207989971,0cyclictest0-21swapper/022:13:450
207989971,0cyclictest0-21swapper/018:52:540
207989970,0cyclictest9635-21sh22:18:400
207989970,0cyclictest7799-21sh22:35:460
207989970,0cyclictest7799-21sh22:35:460
207989970,0cyclictest7647-21kthreadcore21:13:480
207989970,0cyclictest5933-21kthreadcore19:18:470
207989970,0cyclictest4826-21munin-node20:08:440
207989970,0cyclictest4779-21sendmail_mailqu20:28:520
207989970,0cyclictest3872-21munin-node19:43:540
207989970,0cyclictest32323-21rm23:13:360
207989970,0cyclictest29591-21munin-node19:38:490
207989970,0cyclictest29356-21munin-node19:58:490
207989970,0cyclictest27802-21df_abs21:08:440
207989970,0cyclictest25196-21sh22:07:120
207989970,0cyclictest16828-21diskstats23:03:420
207989970,0cyclictest11561-21diskstats20:58:410
207989970,0cyclictest0-21swapper/020:53:460
339960,0migration/35513-21sh23:55:183
339960,0migration/319435-21kthreadcore22:23:453
339960,0migration/317064-21sh21:40:023
29725060,0irq/27-eth0-tx-11069-21munin-node21:18:502
29715060,0irq/26-eth0-rx-30196-21python23:08:533
29715060,0irq/26-eth0-rx-0-21swapper/321:18:463
29715060,0irq/26-eth0-rx-0-21swapper/320:23:473
207999966,0cyclictest18928-21diskmemload21:58:161
207989961,0cyclictest30653-21awk20:03:410
207989960,0cyclictest9969-21sed21:58:300
207989960,0cyclictest8945-21rm22:38:310
207989960,0cyclictest8455-21ssh22:58:360
207989960,0cyclictest7401-21sh23:38:330
207989960,0cyclictest32738-21wc22:53:300
207989960,0cyclictest31192-21date23:53:290
207989960,0cyclictest26842-21cat21:28:300
207989960,0cyclictest2649-21aten2.4-expect21:33:410
207989960,0cyclictest26341-21ls21:48:300
207989960,0cyclictest26049-21aten_r1power_vo22:08:410
207989960,0cyclictest25514-21grep20:23:430
207989960,0cyclictest25326-21sh22:28:360
207989960,0cyclictest24845-21sh22:48:320
207989960,0cyclictest24558-21aten_r1power_vo23:08:430
207989960,0cyclictest24144-21munin-node23:28:420
207989960,0cyclictest23623-21tr23:48:420
207989960,0cyclictest22876-21grep19:33:300
207989960,0cyclictest21140-21aten_r1power_vo18:43:420
207989960,0cyclictest19170-21munin-node20:43:420
207989960,0cyclictest18580-21aten_r1power_fr21:43:420
207989960,0cyclictest17624-21munin-node19:53:420
207989960,0cyclictest17497-21aten_r1power_po22:23:400
207989960,0cyclictest16257-21cut19:28:400
207989960,0cyclictest15978-21munin-node23:23:410
207989960,0cyclictest15294-21aten_r1power_cu19:03:400
207989960,0cyclictest14800-21sh00:03:400
207989960,0cyclictest12491-21latency_hist20:38:300
207989960,0cyclictest11451-21grep20:13:300
207989960,0cyclictest11137-21munin-node21:18:410
207989960,0cyclictest10781-21egrep19:48:420
207989960,0cyclictest0-21swapper/020:33:300
207989960,0cyclictest0-21swapper/018:58:430
8215050,0irq/24-0000:00:26193-21idleruntime-cro21:48:302
339950,0migration/3885-21kthreadcore23:33:453
339950,0migration/38574-21sh22:38:053
339950,0migration/38574-21sh22:38:053
339950,0migration/38231-21kthreadcore19:43:483
339950,0migration/38202-21taskset18:56:023
339950,0migration/3616-21munin-node22:53:433
339950,0migration/34288-21munin-node21:13:463
339950,0migration/331969-21kthreadcore21:08:493
339950,0migration/331654-21taskset22:29:243
339950,0migration/331462-21latency_hist20:28:303
339950,0migration/330322-21kthreadcore21:28:463
339950,0migration/328963-21munin-node19:13:423
339950,0migration/328117-21taskset19:10:593
339950,0migration/327905-21gltestperf20:48:433
339950,0migration/324034-21aten_r1power_vo23:28:413
339950,0migration/322772-21aten_r1power_cu00:08:423
339950,0migration/320884-21kthreadcore21:43:473
339950,0migration/318993-21latency_hist21:23:303
339950,0migration/318414-21munin-node20:18:433
339950,0migration/317837-21kthreadcore23:03:463
339950,0migration/316909-21munin-node23:23:453
339950,0migration/31678-21runrttasks21:53:263
339950,0migration/314010-21needreboot18:58:493
339950,0migration/312873-21kthreadcore20:13:463
339950,0migration/311416-21kthreadcore19:23:453
339950,0migration/311338-21munin-node20:33:493
29725050,0irq/27-eth0-tx-7986-21munin-node23:18:442
29725050,0irq/27-eth0-tx-7712-21ssh22:35:302
29725050,0irq/27-eth0-tx-7712-21ssh22:35:302
29725050,0irq/27-eth0-tx-28299-21open_files00:08:522
29725050,0irq/27-eth0-tx-26966-21munin-node21:28:522
29725050,0irq/27-eth0-tx-24884-21munin-node22:48:402
29725050,0irq/27-eth0-tx-24672-21ssh22:27:192
29725050,0irq/27-eth0-tx-0-21swapper/223:55:102
29725050,0irq/27-eth0-tx-0-21swapper/222:39:052
29725050,0irq/27-eth0-tx-0-21swapper/222:21:562
29725050,0irq/27-eth0-tx-0-21swapper/222:04:342
29725050,0irq/27-eth0-tx-0-21swapper/221:00:472
29725050,0irq/27-eth0-tx-0-21swapper/220:55:062
29715050,0irq/26-eth0-rx-9536-21kthreadcore23:18:453
29715050,0irq/26-eth0-rx-9491-21iostat22:58:453
29715050,0irq/26-eth0-rx-7752-21ssh22:14:083
29715050,0irq/26-eth0-rx-18957-21grep22:43:473
29715050,0irq/26-eth0-rx-15098-21munin-node00:03:413
29715050,0irq/26-eth0-rx-14428-21kthreadcore20:58:453
29715050,0irq/26-eth0-rx-1051-21irqcore23:13:463
29715050,0irq/26-eth0-rx-10297-21kthreadcore23:58:473
29715050,0irq/26-eth0-rx-0-21swapper/323:48:483
29715050,0irq/26-eth0-rx-0-21swapper/323:43:463
29715050,0irq/26-eth0-rx-0-21swapper/322:49:223
29715050,0irq/26-eth0-rx-0-21swapper/322:22:333
29715050,0irq/26-eth0-rx-0-21swapper/322:04:583
29715050,0irq/26-eth0-rx-0-21swapper/321:04:373
29715050,0irq/26-eth0-rx-0-21swapper/319:58:463
29715050,0irq/26-eth0-rx-0-21swapper/319:03:403
27150,0ktimersoftd/20-21swapper/221:13:492
27150,0ktimersoftd/20-21swapper/221:12:152
259950,0migration/232531-21unixbench_singl21:48:532
259950,0migration/231132-21kthreadcore18:48:472
259950,0migration/227508-21kthreadcore23:48:482
208009955,0cyclictest0-21swapper/220:48:262
207999955,0cyclictest0-21swapper/123:47:121
207999955,0cyclictest0-21swapper/123:40:261
207999955,0cyclictest0-21swapper/122:35:001
207999955,0cyclictest0-21swapper/122:35:001
207999955,0cyclictest0-21swapper/122:02:311
207999955,0cyclictest0-21swapper/121:40:551
207999955,0cyclictest0-21swapper/121:13:481
207999954,0cyclictest3514-21snmpd20:48:591
207999950,0cyclictest0-21swapper/122:58:101
207999950,0cyclictest0-21swapper/122:44:271
207999950,0cyclictest0-21swapper/121:47:521
207999950,0cyclictest0-21swapper/121:28:091
179950,0migration/1671-21hddtemp_smartct23:13:431
179950,0migration/130743-21timerandwakeup22:48:501
179950,0migration/127517-21aten_r1power_en21:08:421
179950,0migration/12700-21kthreadcore23:33:461
179950,0migration/125582-21kthreadcore23:28:461
179950,0migration/124345-21taskset20:20:551
179950,0migration/117113-21kthreadcore00:03:461
179950,0migration/116869-21fschecks_count23:03:431
169950,0watchdog/13398-21if_err_eth121:33:441
169950,0watchdog/125989-21kthreadcore23:08:481
8215040,0irq/24-0000:00:28508-21cat19:13:302
8215040,0irq/24-0000:00:0-21swapper/219:03:302
339940,0migration/37008-21kthreadcore20:08:473
339940,0migration/34114-21irqrtprio19:18:463
339940,0migration/329154-21runrttasks19:36:113
339940,0migration/327901-21kthreadcore22:08:473
339940,0migration/325271-21kthreadcore18:43:473
339940,0migration/316157-21kthreadcore20:38:473
29725040,0irq/27-eth0-tx-7423-21ssh23:38:372
29725040,0irq/27-eth0-tx-5684-21ssh23:34:122
29725040,0irq/27-eth0-tx-15817-21ssh23:02:322
29725040,0irq/27-eth0-tx-1516-21kthreadcore23:13:462
29725040,0irq/27-eth0-tx-0-21swapper/223:43:332
29725040,0irq/27-eth0-tx-0-21swapper/223:04:142
29725040,0irq/27-eth0-tx-0-21swapper/222:53:382
29725040,0irq/27-eth0-tx-0-21swapper/222:43:372
29725040,0irq/27-eth0-tx-0-21swapper/222:13:452
29725040,0irq/27-eth0-tx-0-21swapper/221:58:432
29725040,0irq/27-eth0-tx-0-21swapper/221:38:432
29715040,0irq/26-eth0-rx-0-21swapper/323:38:463
29715040,0irq/26-eth0-rx-0-21swapper/322:38:573
29715040,0irq/26-eth0-rx-0-21swapper/321:58:553
29715040,0irq/26-eth0-rx-0-21swapper/321:55:283
29715040,0irq/26-eth0-rx-0-21swapper/321:36:223
29715040,0irq/26-eth0-rx-0-21swapper/320:53:453
29715040,0irq/26-eth0-rx-0-21swapper/320:43:473
29715040,0irq/26-eth0-rx-0-21swapper/320:03:473
259940,0migration/23333-21fschecks_count19:18:432
259940,0migration/232644-21irqcore20:28:472
259940,0migration/230989-21kthreadcore19:38:462
259940,0migration/230979-21kthreadcore22:08:482
259940,0migration/22928-21irqcore18:53:452
259940,0migration/226249-21kthreadcore19:33:472
259940,0migration/222149-21kthreadcore21:23:462
259940,0migration/219558-21cpuspeed_turbos21:03:412
259940,0migration/218595-21kthreadcore23:23:472
259940,0migration/218480-21kthreadcore19:28:462
249940,0watchdog/225990-21kthreadcore23:08:482
249940,0watchdog/223197-21unixbench_singl19:53:532
249940,0watchdog/210812-21kernelversion19:23:452
249940,0watchdog/210507-21python20:08:532
249940,0watchdog/20-21swapper/222:28:482
249940,0watchdog/20-21swapper/220:18:482
208009943,0cyclictest7849-21awk23:58:452
208009943,0cyclictest3556-21lldpd20:48:432
208009943,0cyclictest3514-21snmpd00:03:442
208009943,0cyclictest3416-21awk21:33:442
208009943,0cyclictest31479-21awk20:03:452
208009943,0cyclictest30296-21sadc20:03:302
208009943,0cyclictest24764-21awk23:28:442
208009943,0cyclictest22876-21awk19:08:472
208009943,0cyclictest13428-21awk20:38:452
208009943,0cyclictest11417-21awk19:48:442
208009940,0cyclictest0-21swapper/221:53:522
207999944,0cyclictest0-21swapper/123:54:461
207999943,0cyclictest9533-21awk18:58:461
207999943,0cyclictest3514-21snmpd19:03:431
207999943,0cyclictest3514-21snmpd18:50:131
207999943,0cyclictest3514-21snmpd18:47:431
207999943,0cyclictest31455-21awk20:03:451
207999943,0cyclictest29549-21awk19:13:451
207999943,0cyclictest27844-21awk21:28:431
207999943,0cyclictest2764-21awk18:53:441
207999943,0cyclictest26216-21awk22:28:441
207999943,0cyclictest10330-21sed19:23:431
179940,0migration/15192-21kthreadcore19:18:471
179940,0migration/12802-21kthreadcore22:13:481
179940,0migration/127532-21kthreadcore00:08:491
179940,0migration/126082-21kthreadcore19:58:461
179940,0migration/124065-21sh22:25:291
179940,0migration/123264-21sh23:27:271
179940,0migration/122695-21turbostat19:33:301
179940,0migration/119953-21fschecks_time21:03:421
179940,0migration/114986-21ntp_offset22:18:481
179940,0migration/113213-21kthreadcore19:48:471
179940,0migration/111944-21kthreadcore22:58:471
169940,0watchdog/127340-21cut23:48:481
169940,0watchdog/123198-21unixbench_singl19:53:531
169940,0watchdog/118219-21sh21:01:241
169940,0watchdog/114390-21ssh00:03:121
169940,0watchdog/10-21swapper/121:48:481
169940,0watchdog/10-21swapper/121:18:441
169940,0watchdog/10-21swapper/120:13:451
8215030,0irq/24-0000:00:0-21swapper/220:23:522
339930,0migration/328057-21munin-node18:48:423
29715030,0irq/26-eth0-rx-0-21swapper/319:49:223
29715030,0irq/26-eth0-rx-0-21swapper/319:28:413
249930,0watchdog/28651-21munin-node18:58:492
249930,0watchdog/212443-21if_eth1.220:13:452
208009933,0cyclictest4780-21awk19:43:462
208009932,0cyclictest6698-21awk20:33:442
208009932,0cyclictest3514-21snmpd18:44:132
207999932,0cyclictest20030-21awk20:43:441
207999932,0cyclictest18831-21awk22:03:441
207999932,0cyclictest13554-21sed20:38:461
207999930,0cyclictest8630-21perl23:18:421
179930,0migration/19335-21diskmemload20:54:241
179930,0migration/121915-21runrttasks19:08:391
169930,0watchdog/118928-21diskmemload22:38:321
169930,0watchdog/10-21swapper/122:08:481
169930,0watchdog/10-21swapper/120:11:011
169930,0watchdog/10-21swapper/119:33:451
329920,0watchdog/30-21swapper/319:54:413
29715020,0irq/26-eth0-rx-0-21swapper/319:38:323
207999921,0cyclictest0-21swapper/120:28:421
207999921,0cyclictest0-21swapper/120:23:301
207999920,0cyclictest0-21swapper/119:43:301
169920,0watchdog/10-21swapper/120:33:321
169920,0watchdog/10-21swapper/119:38:331
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional