You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-30 - 04:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Tue Dec 30, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304450640,0irq/25-eth10-21swapper/319:04:313
1980723724,0sleep20-21swapper/219:05:412
2001323624,0sleep00-21swapper/019:08:070
1970323627,0sleep10-21swapper/119:04:281
339960,0migration/39203-21munin-node22:18:423
339960,0migration/320254-21sh21:28:173
339960,0migration/313487-21sh00:14:283
339960,0migration/312365-21sh23:37:333
29715060,0irq/26-eth0-rx-15248-21iostat_ios23:58:413
29715060,0irq/26-eth0-rx-13711-21grep22:58:413
201089960,0cyclictest0-21swapper/223:43:302
201069960,0cyclictest25724-21ssh22:47:570
19160,0ktimersoftd/10-21swapper/123:45:181
4150,0ktimersoftd/00-21swapper/021:43:230
339950,0migration/37926-21rt-features19:43:453
339950,0migration/35639-21taskset21:38:343
339950,0migration/34080-21munin-node21:53:453
339950,0migration/33357-21runrttasks23:13:333
339950,0migration/332425-21cpuspeed_turbos00:28:383
339950,0migration/329558-21runrttasks00:05:343
339950,0migration/328682-21irqrtprio23:08:413
339950,0migration/328134-21munin-node19:13:433
339950,0migration/327079-21sh21:13:373
339950,0migration/324169-21kthreadcore23:43:413
339950,0migration/320464-21kthreadcore20:43:443
339950,0migration/319925-21munin-node23:03:393
339950,0migration/319110-21munin-node21:08:383
339950,0migration/318498-21kthreadcore20:18:423
339950,0migration/317542-21kthreadcore19:53:423
339950,0migration/316445-21kthreadcore23:38:423
339950,0migration/315147-21kthreadcore20:38:443
339950,0migration/310104-21irqcore19:48:413
339950,0migration/310080-21runrttasks22:57:433
29725050,0irq/27-eth0-tx-760-21ssh22:50:482
29725050,0irq/27-eth0-tx-6016-21ssh21:57:012
29725050,0irq/27-eth0-tx-30623-21if_err_eth123:48:392
29725050,0irq/27-eth0-tx-2942-21ssh23:30:132
29725050,0irq/27-eth0-tx-27730-21gltestperf21:13:402
29725050,0irq/27-eth0-tx-23816-21munin-node00:23:402
29725050,0irq/27-eth0-tx-20607-21munin-node21:28:412
29725050,0irq/27-eth0-tx-14660-21munin-node19:28:482
29725050,0irq/27-eth0-tx-13873-21ssh22:00:332
29725050,0irq/27-eth0-tx-11797-21munin-node23:18:492
29725050,0irq/27-eth0-tx-0-21swapper/223:41:352
29725050,0irq/27-eth0-tx-0-21swapper/223:08:282
29725050,0irq/27-eth0-tx-0-21swapper/223:03:412
29725050,0irq/27-eth0-tx-0-21swapper/222:38:282
29725050,0irq/27-eth0-tx-0-21swapper/222:20:482
29725050,0irq/27-eth0-tx-0-21swapper/221:45:222
29725050,0irq/27-eth0-tx-0-21swapper/221:42:552
29725050,0irq/27-eth0-tx-0-21swapper/221:28:032
29725050,0irq/27-eth0-tx-0-21swapper/200:33:262
29725050,0irq/27-eth0-tx-0-21swapper/200:08:432
29715050,0irq/26-eth0-rx-7948-21kthreadcore21:18:423
29715050,0irq/26-eth0-rx-6056-21kthreadcore20:08:443
29715050,0irq/26-eth0-rx-5968-21ssh22:13:463
29715050,0irq/26-eth0-rx-5124-21ssh21:37:583
29715050,0irq/26-eth0-rx-3403-21kthreadcore00:08:433
29715050,0irq/26-eth0-rx-29251-21grep22:48:423
29715050,0irq/26-eth0-rx-28566-21kthreadcore00:23:423
29715050,0irq/26-eth0-rx-27827-21ssh23:27:173
29715050,0irq/26-eth0-rx-25105-21kthreadcore21:48:413
29715050,0irq/26-eth0-rx-19643-21kthreadcore22:03:443
29715050,0irq/26-eth0-rx-17596-21kthreadcore21:43:403
29715050,0irq/26-eth0-rx-16356-21if_eth000:18:413
29715050,0irq/26-eth0-rx-12422-21kthreadcore19:23:443
29715050,0irq/26-eth0-rx-12049-21kthreadcore20:13:433
29715050,0irq/26-eth0-rx-0-21swapper/323:51:133
29715050,0irq/26-eth0-rx-0-21swapper/323:28:443
29715050,0irq/26-eth0-rx-0-21swapper/323:19:593
29715050,0irq/26-eth0-rx-0-21swapper/322:46:493
29715050,0irq/26-eth0-rx-0-21swapper/322:39:273
29715050,0irq/26-eth0-rx-0-21swapper/322:23:423
29715050,0irq/26-eth0-rx-0-21swapper/322:00:283
29715050,0irq/26-eth0-rx-0-21swapper/321:29:433
29715050,0irq/26-eth0-rx-0-21swapper/319:58:433
259950,0migration/27585-21kthreadcore22:53:432
259950,0migration/27402-21kthreadcore21:18:422
259950,0migration/23841-21munin-node23:13:392
259950,0migration/232057-21taskset22:31:002
259950,0migration/230692-21sh00:08:182
259950,0migration/225275-21sh21:10:262
259950,0migration/22440-21munin-node21:33:462
259950,0migration/218452-21kthreadcore20:18:422
259950,0migration/210606-21kthreadcore19:23:442
249950,0watchdog/25510-21cut20:08:432
201089955,0cyclictest0-21swapper/222:06:232
201089954,0cyclictest23366-21awk21:48:392
201089950,0cyclictest0-21swapper/222:44:042
201079955,0cyclictest18447-21diskmemload23:16:461
201079955,0cyclictest0-21swapper/123:09:121
201079955,0cyclictest0-21swapper/119:18:431
201079955,0cyclictest0-21swapper/100:12:541
201079950,0cyclictest0-21swapper/123:42:471
201079950,0cyclictest0-21swapper/121:23:441
201079950,0cyclictest0-21swapper/121:15:131
201079950,0cyclictest0-21swapper/100:24:341
201069955,0cyclictest0-21swapper/023:22:380
201069955,0cyclictest0-21swapper/022:00:390
201069955,0cyclictest0-21swapper/000:09:370
201069950,0cyclictest0-21swapper/023:50:420
201069950,0cyclictest0-21swapper/022:40:230
201069950,0cyclictest0-21swapper/021:08:280
201069950,0cyclictest0-21swapper/000:28:320
19150,0ktimersoftd/10-21swapper/122:57:231
179950,0migration/128777-21sh21:49:051
179950,0migration/127945-21kthreadcore22:28:431
169950,0watchdog/110030-21kthreadcore21:38:431
149950,0migration/09563-21taskset20:11:090
149950,0migration/08120-21aten_r1power_fr19:23:380
149950,0migration/029825-21kthreadcore19:38:410
149950,0migration/028996-21sh00:04:160
149950,0migration/023050-21sh22:24:270
149950,0migration/022797-21sh21:28:420
149950,0migration/022587-21sh00:21:130
149950,0migration/020891-21sh23:59:550
149950,0migration/01765-21kthreadcore21:33:450
149950,0migration/011507-21sh21:23:060
149950,0migration/011472-21taskset20:59:320
149950,0migration/010609-21kthreadcore23:53:440
339940,0migration/37623-21kthreadcore20:58:453
339940,0migration/36470-21fschecks_time23:53:403
339940,0migration/327886-21kthreadcore20:23:443
339940,0migration/325155-21diskmemload22:28:373
29725040,0irq/27-eth0-tx-5322-21munin-node20:58:402
29725040,0irq/27-eth0-tx-0-21swapper/223:58:262
29725040,0irq/27-eth0-tx-0-21swapper/223:54:182
29725040,0irq/27-eth0-tx-0-21swapper/222:23:312
29725040,0irq/27-eth0-tx-0-21swapper/222:13:352
29725040,0irq/27-eth0-tx-0-21swapper/200:18:322
29725040,0irq/27-eth0-tx-0-21swapper/200:13:292
29715040,0irq/26-eth0-rx-3307-21runrttasks20:28:453
29715040,0irq/26-eth0-rx-2412-21munin-node19:18:443
29715040,0irq/26-eth0-rx-0-21swapper/322:33:303
29715040,0irq/26-eth0-rx-0-21swapper/322:08:493
29715040,0irq/26-eth0-rx-0-21swapper/320:49:503
29715040,0irq/26-eth0-rx-0-21swapper/300:33:443
259940,0migration/229753-21kthreadcore19:38:412
259940,0migration/228221-21irqrtprio19:13:432
259940,0migration/223174-21munin-node19:58:402
259940,0migration/220960-21gltestperf23:23:362
259940,0migration/22049-21irqcore22:33:412
259940,0migration/217176-21kthreadcore21:03:452
249940,0watchdog/26055-21ssh00:28:542
249940,0watchdog/213848-21kthreadcore20:38:432
249940,0watchdog/20-21swapper/222:08:392
249940,0watchdog/20-21swapper/219:08:442
201089943,0cyclictest3556-21lldpd23:33:312
201089943,0cyclictest3556-21lldpd20:15:302
201089943,0cyclictest3556-21lldpd19:55:592
201089943,0cyclictest3556-21lldpd19:18:292
201089943,0cyclictest32014-21awk20:53:412
201089943,0cyclictest29853-21awk20:03:402
201089943,0cyclictest11569-21awk22:58:392
201079944,0cyclictest18447-21diskmemload22:20:201
201079943,0cyclictest696-21awk22:13:381
201079943,0cyclictest4149-21awk21:18:381
201079943,0cyclictest3556-21lldpd23:37:011
201079943,0cyclictest3556-21lldpd23:25:321
201079943,0cyclictest3556-21lldpd22:48:311
201079943,0cyclictest3556-21lldpd20:16:301
201079943,0cyclictest3556-21lldpd20:00:301
201079943,0cyclictest3556-21lldpd00:37:041
201079943,0cyclictest3514-21snmpd20:47:521
201079943,0cyclictest3514-21snmpd20:40:221
201079943,0cyclictest3514-21snmpd20:23:371
201079943,0cyclictest3514-21snmpd20:20:371
201079943,0cyclictest3514-21snmpd19:25:211
201079943,0cyclictest3241-21irqbalance00:01:031
201079943,0cyclictest29822-21awk20:03:401
201079943,0cyclictest29489-21awk23:28:411
201079943,0cyclictest28835-21awk19:38:391
201079943,0cyclictest27931-21awk19:13:421
201079943,0cyclictest25243-21awk20:48:401
201079943,0cyclictest22250-21awk19:33:421
201079943,0cyclictest16429-21awk00:18:421
201079943,0cyclictest12709-21awk23:18:401
201069944,0cyclictest0-21swapper/022:48:440
201069944,0cyclictest0-21swapper/021:47:050
201069944,0cyclictest0-21swapper/000:35:470
201069943,0cyclictest5580-21awk23:33:400
201069943,0cyclictest5295-21awk20:33:420
201069943,0cyclictest3241-21irqbalance23:29:230
201069943,0cyclictest3241-21irqbalance23:24:430
201069943,0cyclictest3241-21irqbalance22:18:330
201069943,0cyclictest3241-21irqbalance22:14:330
201069943,0cyclictest3241-21irqbalance21:03:430
201069943,0cyclictest3241-21irqbalance20:49:030
201069943,0cyclictest3241-21irqbalance20:43:530
201069943,0cyclictest3241-21irqbalance20:18:340
201069943,0cyclictest3241-21irqbalance20:03:340
201069943,0cyclictest3241-21irqbalance19:54:340
201069943,0cyclictest3241-21irqbalance00:14:030
201069943,0cyclictest24798-21awk00:23:400
201069943,0cyclictest24245-21awk20:23:420
201069943,0cyclictest21179-21awk19:08:410
201069943,0cyclictest15592-21awk19:28:410
179940,0migration/19114-21kthreadcore00:13:431
179940,0migration/1681-21kthreadcore20:28:401
179940,0migration/14821-21kthreadcore20:08:431
179940,0migration/130097-21df_abs23:48:371
179940,0migration/12831-21kthreadcore22:33:421
179940,0migration/126813-21munin-node21:28:481
179940,0migration/124648-21kthreadcore00:03:411
179940,0migration/122093-21missed_timers22:23:441
179940,0migration/121807-21kthreadcore22:43:421
179940,0migration/118719-21aten_r1power_cu21:08:361
179940,0migration/117905-21kthreadcore19:53:431
179940,0migration/117898-21kthreadcore22:03:441
179940,0migration/11476-21kthreadcore20:53:421
179940,0migration/111774-21timerandwakeup23:53:491
179940,0migration/110660-21taskset22:58:341
169940,0watchdog/16741-21cut20:33:431
169940,0watchdog/132191-21munin-node00:28:421
169940,0watchdog/124613-21cut22:08:391
169940,0watchdog/12269-21stty19:43:351
169940,0watchdog/117519-21pmu-power21:03:471
169940,0watchdog/113794-21grep22:38:431
169940,0watchdog/10-21swapper/121:58:431
169940,0watchdog/10-21swapper/119:28:401
169940,0watchdog/10-21swapper/119:10:201
149940,0migration/0731-21kthreadcore20:53:420
149940,0migration/04425-21munin-node23:13:410
149940,0migration/03960-21missed_timers21:53:450
149940,0migration/03943-21taskset20:31:380
149940,0migration/03247-21kthreadcore19:18:440
149940,0migration/029701-21kthreadcore19:13:440
149940,0migration/028569-21irqcore23:08:410
149940,0migration/027662-21kthreadcore21:48:420
149940,0migration/02487-21kthreadcore22:33:420
149940,0migration/023096-21hddtemp_smartct19:58:390
149940,0migration/022675-21irqstats23:43:400
149940,0migration/021040-21kthreadcore23:03:410
149940,0migration/020006-21kthreadcore22:03:450
149940,0migration/015503-21kthreadcore20:38:440
149940,0migration/015472-21kthreadcore21:23:450
149940,0migration/014703-21kthreadcore23:38:420
149940,0migration/013380-21kthreadcore19:48:420
149940,0migration/011909-21kthreadcore20:13:430
29725030,0irq/27-eth0-tx-30033-21aten2.4-expect20:28:342
29715030,0irq/26-eth0-rx-0-21swapper/321:04:263
29715030,0irq/26-eth0-rx-0-21swapper/320:53:483
29715030,0irq/26-eth0-rx-0-21swapper/320:03:483
29715030,0irq/26-eth0-rx-0-21swapper/319:33:383
29715030,0irq/26-eth0-rx-0-21swapper/319:28:403
29715030,0irq/26-eth0-rx-0-21swapper/319:08:433
27130,0ktimersoftd/20-21swapper/220:43:332
201089932,0cyclictest4340-21munin-node20:33:412
201089932,0cyclictest2974-21grep19:43:382
201089930,0cyclictest0-21swapper/219:49:532
201089930,0cyclictest0-21swapper/219:33:412
201079933,0cyclictest3556-21lldpd21:33:301
201079932,0cyclictest6239-21awk20:58:441
201079932,0cyclictest20846-21kthreadcore23:03:411
201079930,0cyclictest0-21swapper/119:49:531
201069933,0cyclictest0-21swapper/021:14:560
201069932,0cyclictest8026-21cat19:43:470
201069932,0cyclictest3514-21snmpd22:28:290
201069932,0cyclictest3514-21snmpd22:09:220
201069932,0cyclictest26970-21sensors19:33:480
201069932,0cyclictest17145-21ssh22:59:300
169930,0watchdog/10-21swapper/121:53:431
169930,0watchdog/10-21swapper/121:43:431
339920,0migration/34737-21diskstats20:33:393
29715020,0irq/26-eth0-rx-0-21swapper/319:38:363
201089921,0cyclictest3200-17sedispatch20:48:252
201089921,0cyclictest0-21swapper/220:23:252
201069920,0cyclictest0-21swapper/022:53:330
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional