You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-29 - 03:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Mon Dec 29, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304450650,0irq/25-eth10-21swapper/319:03:383
554523926,0sleep10-21swapper/119:03:571
561223524,0sleep00-21swapper/019:04:440
572423223,0sleep20-21swapper/219:06:022
60089983,0cyclictest29068-21awk22:58:490
29715070,0irq/26-eth0-rx-3483-21munin-node21:23:533
60099960,0cyclictest0-21swapper/121:12:401
60089960,0cyclictest0-21swapper/022:53:520
339960,0migration/325469-21sh22:18:003
339960,0migration/325253-21sh00:33:233
339960,0migration/316559-21sh00:27:403
339960,0migration/316343-21sh22:11:033
339960,0migration/315083-21sh21:33:393
339960,0migration/313613-21missed_timers21:48:503
29725060,0irq/27-eth0-tx-17206-21munin-node00:28:552
29715060,0irq/26-eth0-rx-5032-21latency21:43:513
29715060,0irq/26-eth0-rx-11877-21load21:28:493
8215050,0irq/24-0000:00:6166-21latency_hist21:28:302
8215050,0irq/24-0000:00:24762-21cat21:58:302
8215050,0irq/24-0000:00:15559-21munin-run20:28:302
60109955,0cyclictest0-21swapper/223:53:302
60109955,0cyclictest0-21swapper/223:06:412
60109954,0cyclictest4008-21awk22:43:462
60109950,0cyclictest0-21swapper/223:37:442
60099950,0cyclictest0-21swapper/122:25:591
60099950,0cyclictest0-21swapper/121:21:301
60099950,0cyclictest0-21swapper/121:17:051
60099950,0cyclictest0-21swapper/100:01:031
60089955,0cyclictest0-21swapper/022:49:270
60089955,0cyclictest0-21swapper/022:44:150
60089955,0cyclictest0-21swapper/022:35:540
60089955,0cyclictest0-21swapper/020:48:410
60089955,0cyclictest0-21swapper/000:10:290
60089950,0cyclictest0-21swapper/023:40:230
60089950,0cyclictest0-21swapper/023:35:580
60089950,0cyclictest0-21swapper/023:25:330
60089950,0cyclictest0-21swapper/021:09:580
60089950,0cyclictest0-21swapper/000:34:080
4150,0ktimersoftd/00-21swapper/000:33:210
4150,0ktimersoftd/00-21swapper/000:19:190
339950,0migration/39181-21kthreadcore21:08:503
339950,0migration/37358-21sh00:20:373
339950,0migration/35452-21kthreadcore19:53:503
339950,0migration/329470-21sh21:38:513
339950,0migration/329230-21sh00:33:503
339950,0migration/329000-21latency_hist20:38:303
339950,0migration/328561-21sh21:20:553
339950,0migration/327148-21kthreadcore23:53:483
339950,0migration/32603-21munin-node20:18:463
339950,0migration/321945-21munin-node23:13:463
339950,0migration/32184-21kthreadcore19:28:473
339950,0migration/320336-21kthreadcore20:28:483
339950,0migration/31977-21sh22:41:413
339950,0migration/319249-21kthreadcore21:53:473
339950,0migration/318065-21kernelversion20:53:483
339950,0migration/316759-21kthreadcore19:38:503
339950,0migration/316664-21kthreadcore23:48:493
339950,0migration/315606-21pmu-power20:48:503
339950,0migration/315176-21taskset20:24:173
339950,0migration/312826-21kthreadcore00:03:473
29725050,0irq/27-eth0-tx-8847-21munin-node00:23:452
29725050,0irq/27-eth0-tx-8163-21munin-node19:58:472
29725050,0irq/27-eth0-tx-8041-21df_abs00:03:422
29725050,0irq/27-eth0-tx-4448-21munin-node21:08:532
29725050,0irq/27-eth0-tx-23697-21aten_r1power_cu21:38:412
29725050,0irq/27-eth0-tx-21483-21munin-node20:08:452
29725050,0irq/27-eth0-tx-18744-21munin-node22:33:522
29725050,0irq/27-eth0-tx-17675-21munin-node22:13:542
29725050,0irq/27-eth0-tx-10056-21munin-node20:48:512
29725050,0irq/27-eth0-tx-0-21swapper/222:51:352
29725050,0irq/27-eth0-tx-0-21swapper/222:23:502
29725050,0irq/27-eth0-tx-0-21swapper/221:22:142
29725050,0irq/27-eth0-tx-0-21swapper/200:36:072
29715050,0irq/26-eth0-rx-9593-21ssh22:27:223
29715050,0irq/26-eth0-rx-5883-21fschecks_time23:23:443
29715050,0irq/26-eth0-rx-32404-21idleruntime23:58:443
29715050,0irq/26-eth0-rx-30764-21tr23:38:433
29715050,0irq/26-eth0-rx-29814-21ssh23:37:433
29715050,0irq/26-eth0-rx-27316-21kthreadcore22:18:473
29715050,0irq/26-eth0-rx-25251-21diskstats21:58:443
29715050,0irq/26-eth0-rx-21363-21ssh23:32:313
29715050,0irq/26-eth0-rx-20268-21munin-node22:53:483
29715050,0irq/26-eth0-rx-16856-21memory22:48:463
29715050,0irq/26-eth0-rx-10371-21kthreadcore23:43:503
29715050,0irq/26-eth0-rx-0-21swapper/323:18:493
29715050,0irq/26-eth0-rx-0-21swapper/322:47:233
29715050,0irq/26-eth0-rx-0-21swapper/322:33:423
29715050,0irq/26-eth0-rx-0-21swapper/322:29:583
29715050,0irq/26-eth0-rx-0-21swapper/322:05:583
29715050,0irq/26-eth0-rx-0-21swapper/321:14:573
29715050,0irq/26-eth0-rx-0-21swapper/300:15:583
259950,0migration/27252-21kthreadcore20:43:492
259950,0migration/26280-21munin-node23:23:452
259950,0migration/26151-21latency_hist19:08:312
259950,0migration/2417-21hwlatdetect21:43:472
259950,0migration/232438-21kthreadcore23:18:482
259950,0migration/231743-21sendmail_mailqu22:18:512
259950,0migration/227733-21sh22:58:112
259950,0migration/22408-21kthreadcore00:18:482
259950,0migration/223505-21kthreadcore23:13:472
259950,0migration/22313-21kthreadcore19:28:472
259950,0migration/222151-21runrttasks23:51:382
259950,0migration/21765-21sh22:41:112
259950,0migration/217183-21sh22:30:272
259950,0migration/211875-21kthreadcore21:48:492
249950,0watchdog/225407-21kthreadcore20:33:492
249950,0watchdog/213455-21fschecks_time21:13:452
179950,0migration/1923-21taskset19:51:051
179950,0migration/132022-21kthreadcore21:03:511
179950,0migration/131870-21kthreadcore23:18:481
179950,0migration/130604-21aten2.4-expect23:38:421
179950,0migration/129621-21taskset21:00:421
179950,0migration/123235-21latency_hist20:58:301
179950,0migration/117247-21kthreadcore19:38:501
179950,0migration/116189-21kthreadcore20:03:481
179950,0migration/114311-21fschecks_count23:28:471
179950,0migration/110725-21kthreadcore20:23:471
169950,0watchdog/13794-21perl22:43:441
169950,0watchdog/118840-21kthreadcore00:08:481
8215040,0irq/24-0000:00:0-21swapper/221:03:302
60109943,0cyclictest3363-21awk20:18:492
60109943,0cyclictest3241-21irqbalance23:28:582
60109943,0cyclictest29968-21awk20:38:452
60109943,0cyclictest25427-21awk00:13:452
60109943,0cyclictest2290-21awk19:53:472
60109943,0cyclictest17407-21awk21:53:452
60109943,0cyclictest14794-21awk19:38:482
60109943,0cyclictest10311-21awk22:08:482
60099943,0cyclictest9827-21awk00:23:461
60099943,0cyclictest8551-21awk00:03:441
60099943,0cyclictest7386-21awk21:28:461
60099943,0cyclictest3556-21lldpd22:08:451
60099943,0cyclictest3556-21lldpd21:33:441
60099943,0cyclictest3556-21lldpd19:15:281
60099943,0cyclictest3514-21snmpd22:17:451
60099943,0cyclictest3514-21snmpd21:50:451
60099943,0cyclictest3514-21snmpd20:21:461
60099943,0cyclictest3514-21snmpd20:17:461
60099943,0cyclictest2815-21tail21:43:501
60099943,0cyclictest21851-21awk23:13:451
60099943,0cyclictest20427-21awk19:18:451
60099943,0cyclictest19677-21awk22:33:461
60099943,0cyclictest18215-21awk00:28:471
60099943,0cyclictest1808-21awk22:03:461
60099943,0cyclictest12396-21awk22:48:431
60099943,0cyclictest1231-21awk00:18:461
60099940,0cyclictest0-21swapper/120:49:541
60089944,0cyclictest0-21swapper/023:20:210
60089944,0cyclictest0-21swapper/022:13:310
60089944,0cyclictest0-21swapper/021:15:110
60089943,0cyclictest5075-21awk23:03:450
60089943,0cyclictest3556-21lldpd23:29:160
60089943,0cyclictest3556-21lldpd21:43:450
60089943,0cyclictest3514-21snmpd23:53:300
60089943,0cyclictest3514-21snmpd22:18:300
60089943,0cyclictest3514-21snmpd22:04:150
60089943,0cyclictest3514-21snmpd20:43:460
60089943,0cyclictest3514-21snmpd20:38:160
60089943,0cyclictest3255-21awk20:18:480
60089943,0cyclictest32543-21awk23:58:450
60089943,0cyclictest3241-21irqbalance23:13:580
60089943,0cyclictest3241-21irqbalance22:40:490
60089943,0cyclictest3241-21irqbalance21:48:390
60089943,0cyclictest3241-21irqbalance21:42:290
60089943,0cyclictest3241-21irqbalance21:30:090
60089943,0cyclictest3241-21irqbalance21:20:390
60089943,0cyclictest3241-21irqbalance19:59:290
60089943,0cyclictest3241-21irqbalance19:53:590
60089943,0cyclictest3241-21irqbalance19:48:390
60089943,0cyclictest3241-21irqbalance19:43:390
60089943,0cyclictest3241-21irqbalance19:38:390
60089943,0cyclictest3241-21irqbalance19:34:190
60089943,0cyclictest3241-21irqbalance19:31:090
60089943,0cyclictest25353-21awk00:13:450
60089943,0cyclictest17506-21awk21:53:460
60089943,0cyclictest16634-21awk20:28:450
60089943,0cyclictest16293-21kworker/0:122:24:180
60089943,0cyclictest13577-21awk23:08:460
60089940,0cyclictest0-21swapper/021:06:210
339940,0migration/39764-21kthreadcore19:33:463
339940,0migration/38743-21fschecks_time19:58:453
339940,0migration/38627-21kthreadcore19:08:483
29725040,0irq/27-eth0-tx-28642-21fschecks_count22:58:472
29725040,0irq/27-eth0-tx-28159-21munin-node20:13:492
29725040,0irq/27-eth0-tx-14826-21munin-node20:03:542
29725040,0irq/27-eth0-tx-0-21swapper/223:58:312
29725040,0irq/27-eth0-tx-0-21swapper/223:43:502
29725040,0irq/27-eth0-tx-0-21swapper/223:38:422
29725040,0irq/27-eth0-tx-0-21swapper/221:33:352
29725040,0irq/27-eth0-tx-0-21swapper/221:23:442
29725040,0irq/27-eth0-tx-0-21swapper/220:28:462
29725040,0irq/27-eth0-tx-0-21swapper/219:33:422
29725040,0irq/27-eth0-tx-0-21swapper/200:09:442
29715040,0irq/26-eth0-rx-22048-21kthreadcore19:18:473
29715040,0irq/26-eth0-rx-0-21swapper/323:08:423
29715040,0irq/26-eth0-rx-0-21swapper/323:03:513
29715040,0irq/26-eth0-rx-0-21swapper/322:59:573
29715040,0irq/26-eth0-rx-0-21swapper/320:03:423
29715040,0irq/26-eth0-rx-0-21swapper/300:09:073
259940,0migration/232760-21kthreadcore21:03:512
259940,0migration/228424-21kthreadcore19:23:482
259940,0migration/224213-21kthreadcore19:18:482
259940,0migration/221131-21kthreadcore20:53:492
259940,0migration/213846-21irqrtprio23:08:472
249940,0watchdog/24186-21diskmemload22:03:332
249940,0watchdog/221209-21perl19:43:452
179940,0migration/17643-21kthreadcore23:03:481
179940,0migration/17245-21kthreadcore23:23:471
179940,0migration/16873-21fschecks_time19:08:441
179940,0migration/13508-21aten_r1power_vo20:43:441
179940,0migration/130970-21hwlatdetect21:23:471
179940,0migration/129408-21irqrtprio22:58:501
179940,0migration/12914-21taskset20:41:211
179940,0migration/127264-21kthreadcore23:33:501
179940,0migration/127115-21aten_r1power_en22:38:411
179940,0migration/125720-21kthreadcore21:38:471
179940,0migration/123254-21sh21:55:241
179940,0migration/122137-21kthreadcore19:43:481
179940,0migration/119149-21runrttasks23:09:511
179940,0migration/117879-21kthreadcore20:28:471
179940,0migration/117493-21taskset22:31:131
179940,0migration/117295-21kthreadcore23:48:491
179940,0migration/112045-21kthreadcore19:58:481
179940,0migration/111665-21kthreadcore19:33:471
169940,0watchdog/14186-21diskmemload00:33:481
169940,0watchdog/13543-21kthreadcore19:53:491
169940,0watchdog/128705-21cut21:58:491
169940,0watchdog/125406-21kthreadcore20:33:491
169940,0watchdog/11389-21ssh22:22:561
169940,0watchdog/10-21swapper/123:53:481
169940,0watchdog/10-21swapper/122:53:481
169940,0watchdog/10-21swapper/120:08:491
60109930,0cyclictest0-21swapper/219:48:402
60109930,0cyclictest0-21swapper/219:13:442
60099932,0cyclictest6979-21cat19:33:301
60089933,0cyclictest16293-21kworker/0:122:01:120
60089933,0cyclictest16293-21kworker/0:121:23:360
60089933,0cyclictest16293-21kworker/0:120:13:480
60089933,0cyclictest16293-21kworker/0:120:03:520
60089933,0cyclictest16293-21kworker/0:119:08:440
60089933,0cyclictest16293-21kworker/0:100:24:050
60089932,0cyclictest9248-21tr23:43:500
60089932,0cyclictest8668-21awk00:03:450
60089932,0cyclictest3241-21irqbalance21:33:390
60089932,0cyclictest32097-21sed19:23:540
60089932,0cyclictest32062-21kthreadcore20:38:470
60089932,0cyclictest27321-21expr20:08:550
60089932,0cyclictest24256-21awk20:58:480
60089932,0cyclictest20442-21awk19:18:450
60089932,0cyclictest17645-21awk20:53:470
60089932,0cyclictest15881-21kthreadcore19:13:490
60089932,0cyclictest15744-21awk23:48:470
60089932,0cyclictest14495-21grep20:23:490
60089932,0cyclictest11359-21awk22:28:450
60089932,0cyclictest10164-21awk22:08:470
35130,0ktimersoftd/30-21swapper/321:03:333
35130,0ktimersoftd/30-21swapper/320:46:213
29715030,0irq/26-eth0-rx-0-21swapper/320:58:443
29715030,0irq/26-eth0-rx-0-21swapper/320:33:433
29715030,0irq/26-eth0-rx-0-21swapper/320:13:453
29715030,0irq/26-eth0-rx-0-21swapper/319:48:423
29715030,0irq/26-eth0-rx-0-21swapper/319:43:473
29715030,0irq/26-eth0-rx-0-21swapper/319:23:423
29715030,0irq/26-eth0-rx-0-21swapper/319:13:433
169930,0watchdog/10-21swapper/123:44:521
169930,0watchdog/10-21swapper/119:23:491
169930,0watchdog/10-21swapper/100:14:241
329920,0watchdog/30-21swapper/320:08:463
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional