You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-19 - 06:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Fri Dec 19, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
141223727,0sleep30-21swapper/319:08:173
130123526,0sleep20-21swapper/219:06:562
124123524,0sleep00-21swapper/019:06:180
122223524,0sleep10-21swapper/119:06:031
339960,0migration/34165-21kthreadcore21:29:443
339960,0migration/329673-21sh22:01:363
339960,0migration/32356-21munin-node22:59:483
339960,0migration/321169-21sh00:27:253
339960,0migration/315592-21sh21:16:343
29725060,0irq/27-eth0-tx-10457-21munin-node20:04:430
29715060,0irq/26-eth0-rx-6733-21python23:39:493
29715060,0irq/26-eth0-rx-6719-21ssh00:38:493
29715060,0irq/26-eth0-rx-23067-21proc_pri21:19:483
29715060,0irq/26-eth0-rx-15426-21cut20:54:453
29715060,0irq/26-eth0-rx-15070-21ssh22:12:483
19160,0ktimersoftd/10-21swapper/121:57:431
15839960,0cyclictest32080-21diskmemload21:26:122
15839960,0cyclictest0-21swapper/223:29:482
15839960,0cyclictest0-21swapper/200:17:542
15819966,0cyclictest3895-21ssh21:09:450
15819965,0cyclictest23551-21ps23:14:420
149960,0migration/09314-21hwlatdetect21:14:410
149960,0migration/09245-21dump-pmu-power23:44:260
149960,0migration/031177-21aten_r1power_fr23:19:390
149960,0migration/02165-21missed_timers21:44:460
35150,0ktimersoftd/30-21swapper/319:59:563
339950,0migration/35825-21fschecks_time23:04:403
339950,0migration/35700-21kthreadcore19:34:443
339950,0migration/332301-21kthreadcore19:54:453
339950,0migration/325800-21munin-node23:34:443
339950,0migration/324811-21sh23:51:033
339950,0migration/324382-21kthreadcore00:09:453
339950,0migration/322191-21taskset23:29:533
339950,0migration/321982-21aten_r1power_po19:24:383
339950,0migration/321578-21kthreadcore20:34:453
339950,0migration/316480-21munin-node19:44:393
29725050,0irq/27-eth0-tx-5267-21munin-node23:04:452
29725050,0irq/27-eth0-tx-27724-21iostat23:54:412
29725050,0irq/27-eth0-tx-22270-21kthreadcore20:34:460
29725050,0irq/27-eth0-tx-0-21swapper/222:59:442
29725050,0irq/27-eth0-tx-0-21swapper/200:20:472
29725050,0irq/27-eth0-tx-0-21swapper/021:04:390
29725050,0irq/27-eth0-tx-0-21swapper/020:42:270
29725050,0irq/27-eth0-tx-0-21swapper/020:29:390
29725050,0irq/27-eth0-tx-0-21swapper/019:37:270
29725050,0irq/27-eth0-tx-0-21swapper/019:29:340
29725050,0irq/27-eth0-tx-0-21swapper/019:25:270
29715050,0irq/26-eth0-rx-9781-21cut00:19:453
29715050,0irq/26-eth0-rx-8199-21ssh22:27:023
29715050,0irq/26-eth0-rx-5724-21iostat_ios20:24:433
29715050,0irq/26-eth0-rx-3439-21ssh21:45:583
29715050,0irq/26-eth0-rx-32162-21irqcore23:19:453
29715050,0irq/26-eth0-rx-31408-21aten2.4-expect22:04:393
29715050,0irq/26-eth0-rx-22452-21sshd23:14:323
29715050,0irq/26-eth0-rx-2215-21ssh23:58:343
29715050,0irq/26-eth0-rx-20583-21ssh21:55:353
29715050,0irq/26-eth0-rx-17963-21ssh00:05:083
29715050,0irq/26-eth0-rx-15769-21ssh22:29:523
29715050,0irq/26-eth0-rx-10725-21ssh00:03:033
29715050,0irq/26-eth0-rx-0-21swapper/323:10:353
29715050,0irq/26-eth0-rx-0-21swapper/322:54:383
29715050,0irq/26-eth0-rx-0-21swapper/322:51:193
29715050,0irq/26-eth0-rx-0-21swapper/322:44:303
29715050,0irq/26-eth0-rx-0-21swapper/322:41:303
29715050,0irq/26-eth0-rx-0-21swapper/322:35:473
29715050,0irq/26-eth0-rx-0-21swapper/321:49:393
29715050,0irq/26-eth0-rx-0-21swapper/321:10:033
29715050,0irq/26-eth0-rx-0-21swapper/320:41:553
29715050,0irq/26-eth0-rx-0-21swapper/320:14:393
29715050,0irq/26-eth0-rx-0-21swapper/300:29:443
29715050,0irq/26-eth0-rx-0-21swapper/300:29:443
259950,0migration/225082-210anacron21:00:272
259950,0migration/213907-21sh23:09:372
249950,0watchdog/20-21swapper/223:51:322
19150,0ktimersoftd/10-21swapper/123:29:371
19150,0ktimersoftd/10-21swapper/122:09:041
19150,0ktimersoftd/10-21swapper/119:59:561
179950,0migration/19009-21sshd19:36:331
179950,0migration/13155-21kthreadcore23:39:441
179950,0migration/131254-21kthreadcore21:44:441
179950,0migration/127794-21latency00:29:451
179950,0migration/127794-21latency00:29:451
179950,0migration/127147-21kthreadcore20:14:451
179950,0migration/123490-21munin-node23:49:451
179950,0migration/11879-21munin-node19:09:391
179950,0migration/11291-21kthreadcore00:34:431
169950,0watchdog/120795-21kthreadcore21:19:451
169950,0watchdog/117674-21sort20:09:411
15839955,0cyclictest0-21swapper/222:54:302
15839955,0cyclictest0-21swapper/222:02:002
15839955,0cyclictest0-21swapper/221:58:132
15839955,0cyclictest0-21swapper/221:54:262
15839955,0cyclictest0-21swapper/200:37:172
15839950,0cyclictest0-21swapper/223:39:542
15839950,0cyclictest0-21swapper/222:21:332
15839950,0cyclictest0-21swapper/200:30:492
15839950,0cyclictest0-21swapper/200:30:492
15829955,0cyclictest0-21swapper/122:40:071
15829955,0cyclictest0-21swapper/100:28:451
15829955,0cyclictest0-21swapper/100:09:021
15829955,0cyclictest0-21swapper/100:04:091
15829950,0cyclictest0-21swapper/122:56:591
15829950,0cyclictest0-21swapper/122:27:021
15829950,0cyclictest0-21swapper/122:19:561
15819955,0cyclictest0-21swapper/023:31:550
15819955,0cyclictest0-21swapper/022:00:000
15819955,0cyclictest0-21swapper/021:40:460
15819955,0cyclictest0-21swapper/021:30:590
15819955,0cyclictest0-21swapper/000:14:190
15819950,0cyclictest0-21swapper/023:10:090
149950,0migration/09035-21kthreadcore23:24:430
149950,0migration/08076-21sh22:26:470
149950,0migration/06455-21sh22:07:430
149950,0migration/04580-21kthreadcore23:59:450
149950,0migration/04550-21munin-node20:44:470
149950,0migration/032538-21kthreadcore00:14:440
149950,0migration/030904-21kthreadcore22:59:440
149950,0migration/02882-21kthreadcore00:34:430
149950,0migration/026172-21sh21:24:340
149950,0migration/025057-21taskset21:00:150
149950,0migration/024414-21fschecks_time20:14:420
149950,0migration/02337-21missed_timers19:54:460
149950,0migration/022172-21sh21:59:000
149950,0migration/021085-21fschecks_time22:54:410
149950,0migration/020844-21kthreadcore19:44:450
149950,0migration/017508-21aten_r1power_po21:19:390
149950,0migration/017110-21runrttasks23:47:540
149950,0migration/015563-21taskset19:40:250
149950,0migration/015114-21latency_hist19:19:270
149950,0migration/013562-21latency_hist00:24:260
149950,0migration/012044-21munin-node21:34:400
149950,0migration/010461-21kthreadcore22:09:440
149950,0migration/010034-21df_abs22:29:400
8215040,0irq/24-0000:00:0-21swapper/219:34:272
339940,0migration/37523-21sh23:24:383
339940,0migration/332489-21kthreadcore00:14:443
339940,0migration/329672-21irqrtprio19:29:423
339940,0migration/327034-21ksmtuned21:40:383
339940,0migration/319196-21kthreadcore20:09:453
339940,0migration/314615-21kthreadcore19:39:453
29725040,0irq/27-eth0-tx-3202-21ssh23:59:402
29725040,0irq/27-eth0-tx-1503-21cut23:19:462
29725040,0irq/27-eth0-tx-0-21swapper/223:44:362
29725040,0irq/27-eth0-tx-0-21swapper/223:24:442
29725040,0irq/27-eth0-tx-0-21swapper/223:14:452
29725040,0irq/27-eth0-tx-0-21swapper/222:44:392
29725040,0irq/27-eth0-tx-0-21swapper/222:34:382
29725040,0irq/27-eth0-tx-0-21swapper/222:29:342
29725040,0irq/27-eth0-tx-0-21swapper/222:04:292
29725040,0irq/27-eth0-tx-0-21swapper/221:44:442
29725040,0irq/27-eth0-tx-0-21swapper/221:39:292
29725040,0irq/27-eth0-tx-0-21swapper/221:14:312
29725040,0irq/27-eth0-tx-0-21swapper/200:24:292
29725040,0irq/27-eth0-tx-0-21swapper/200:10:452
29725040,0irq/27-eth0-tx-0-21swapper/200:04:472
29725040,0irq/27-eth0-tx-0-21swapper/020:24:450
29725040,0irq/27-eth0-tx-0-21swapper/020:19:500
29725040,0irq/27-eth0-tx-0-21swapper/019:59:420
29725040,0irq/27-eth0-tx-0-21swapper/019:14:370
29715040,0irq/26-eth0-rx-5967-21kthreadcore19:09:453
29715040,0irq/26-eth0-rx-0-21swapper/323:44:373
29715040,0irq/26-eth0-rx-0-21swapper/322:19:383
29715040,0irq/26-eth0-rx-0-21swapper/322:16:013
29715040,0irq/26-eth0-rx-0-21swapper/321:34:423
29715040,0irq/26-eth0-rx-0-21swapper/321:24:543
29715040,0irq/26-eth0-rx-0-21swapper/320:49:403
29715040,0irq/26-eth0-rx-0-21swapper/320:44:443
259940,0migration/27486-21proc_pri19:09:482
259940,0migration/26000-21munin-node20:49:412
259940,0migration/232389-21kthreadcore19:54:452
259940,0migration/225499-21hwlatdetect20:39:422
259940,0migration/224563-21kthreadcore19:24:432
259940,0migration/214815-21missed_timers19:39:452
249940,0watchdog/232297-21ssh21:09:292
249940,0watchdog/212999-21perl20:54:412
249940,0watchdog/20-21swapper/223:34:412
249940,0watchdog/20-21swapper/222:09:452
249940,0watchdog/20-21swapper/220:09:412
179940,0migration/19454-21kthreadcore23:04:461
179940,0migration/15957-21fschecks_time21:49:401
179940,0migration/15118-21munin-node20:24:401
179940,0migration/129123-21aten_r1power_vo22:59:391
179940,0migration/126589-21df_abs21:24:401
179940,0migration/126116-21fschecks_count21:04:411
179940,0migration/124692-21taskset20:39:301
179940,0migration/124654-21kthreadcore19:24:431
179940,0migration/120361-21latency_hist21:39:271
179940,0migration/110652-21kthreadcore19:14:431
179940,0migration/110379-21idleruntime23:44:401
179940,0migration/110279-21kthreadcore20:49:471
169940,0watchdog/13097-21kthreadcore21:09:451
169940,0watchdog/125924-21cut23:34:451
169940,0watchdog/121410-21ssh23:12:451
169940,0watchdog/11451-21rm23:56:561
169940,0watchdog/10-21swapper/122:29:571
169940,0watchdog/10-21swapper/122:09:451
169940,0watchdog/10-21swapper/121:34:451
169940,0watchdog/10-21swapper/120:19:451
169940,0watchdog/10-21swapper/119:54:451
15839943,0cyclictest3556-21lldpd22:24:572
15839943,0cyclictest3556-21lldpd21:29:272
15839943,0cyclictest3556-21lldpd19:48:272
15839943,0cyclictest3514-21snmpd22:49:392
15839943,0cyclictest3514-21snmpd22:39:392
15839943,0cyclictest3514-21snmpd20:46:242
15839943,0cyclictest3514-21snmpd20:35:392
15839943,0cyclictest3514-21snmpd20:33:392
15839943,0cyclictest3514-21snmpd20:25:392
15839943,0cyclictest3514-21snmpd20:19:392
15839943,0cyclictest3514-21snmpd20:02:392
15839943,0cyclictest3514-21snmpd19:23:092
15839943,0cyclictest26660-21awk21:04:442
15839943,0cyclictest24690-21awk20:14:432
15839943,0cyclictest23718-21awk19:49:432
15839943,0cyclictest18355-21awk21:19:432
15839943,0cyclictest17082-21awk22:14:412
15839943,0cyclictest12724-21awk21:34:432
15839943,0cyclictest11324-21awk20:04:422
15829943,0cyclictest6126-21awk00:19:421
15829943,0cyclictest3556-21lldpd23:24:591
15829943,0cyclictest3556-21lldpd20:44:571
15829943,0cyclictest3556-21lldpd19:50:271
15829943,0cyclictest3556-21lldpd19:44:271
15829943,0cyclictest3556-21lldpd00:11:321
15829943,0cyclictest3514-21snmpd23:22:391
15829943,0cyclictest3514-21snmpd23:16:391
15829943,0cyclictest3514-21snmpd21:14:391
15829943,0cyclictest3241-21irqbalance22:53:451
15829943,0cyclictest3241-21irqbalance21:30:151
15829943,0cyclictest29374-21awk19:29:411
15829943,0cyclictest23540-21awk21:59:411
15829943,0cyclictest19109-21awk22:34:421
15829943,0cyclictest13295-21awk20:54:431
15829940,0cyclictest0-21swapper/121:03:381
15819944,0cyclictest0-21swapper/000:33:430
15819944,0cyclictest0-21swapper/000:33:430
15819944,0cyclictest0-21swapper/000:27:150
15819944,0cyclictest0-21swapper/000:04:330
15819943,0cyclictest6320-21awk21:49:420
15819943,0cyclictest3514-21snmpd23:07:240
15819943,0cyclictest17108-21awk22:14:420
149940,0migration/026240-21runrttasks22:38:500
149940,0migration/018710-21kthreadcore20:09:450
149940,0migration/01502-21runrttasks23:57:030
29715030,0irq/26-eth0-rx-0-21swapper/321:07:403
29715030,0irq/26-eth0-rx-0-21swapper/320:59:443
29715030,0irq/26-eth0-rx-0-21swapper/320:19:423
29715030,0irq/26-eth0-rx-0-21swapper/320:07:023
29715030,0irq/26-eth0-rx-0-21swapper/319:49:403
29715030,0irq/26-eth0-rx-0-21swapper/319:19:373
29715030,0irq/26-eth0-rx-0-21swapper/319:15:063
249930,0watchdog/20-21swapper/219:29:492
169930,0watchdog/10-21swapper/122:16:531
169930,0watchdog/10-21swapper/120:34:451
169930,0watchdog/10-21swapper/120:29:371
169930,0watchdog/10-21swapper/119:19:421
169930,0watchdog/10-21swapper/100:14:281
15839932,0cyclictest3514-21snmpd19:17:392
15829932,0cyclictest4051-21awk22:44:421
15829932,0cyclictest3241-21irqbalance19:44:461
15819933,0cyclictest8963-21kworker/0:222:19:280
15819933,0cyclictest8963-21kworker/0:220:54:300
15819933,0cyclictest8963-21kworker/0:220:49:300
15819933,0cyclictest8963-21kworker/0:219:49:330
15819933,0cyclictest19022-21kworker/0:023:34:300
15819933,0cyclictest19022-21kworker/0:022:49:320
15819933,0cyclictest19022-21kworker/0:022:44:280
15819933,0cyclictest19022-21kworker/0:022:39:320
15819933,0cyclictest10337-21kworker/0:023:49:280
15819932,0cyclictest3514-21snmpd19:09:310
29715020,0irq/26-eth0-rx-0-21swapper/320:29:383
169920,0watchdog/10-21swapper/120:04:291
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional