You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-18 - 12:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Thu Dec 18, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
446325141,0sleep20-21swapper/219:09:302
435623724,0sleep00-21swapper/019:08:200
422723531,0sleep30-21swapper/319:06:473
408523526,0sleep10-21swapper/119:05:081
4543992222,0cyclictest14700-21kworker/2:023:44:442
4543991717,0cyclictest14700-21kworker/2:023:28:382
45439973,0cyclictest24066-21awk00:09:472
45439973,0cyclictest23140-21sadc21:39:322
45439972,0cyclictest516-21awk22:59:472
45439972,0cyclictest2715-21awk23:19:482
45439970,0cyclictest23538-21grep22:14:492
45419977,0cyclictest14285-21sensors20:49:540
45439963,0cyclictest7173-21awk22:44:472
45439963,0cyclictest3556-21lldpd21:19:532
45439962,0cyclictest5585-21awk19:09:432
45439962,0cyclictest4074-21basename22:24:432
45439962,0cyclictest3556-21lldpd00:37:542
45439962,0cyclictest32128-21basename21:44:432
45439962,0cyclictest20926-21awk20:09:452
45439962,0cyclictest13585-21awk22:29:462
45439961,0cyclictest24549-21awk22:54:472
45439960,0cyclictest8261-21kthreadcore23:59:482
45439960,0cyclictest3851-21munin-node23:39:462
45439960,0cyclictest31882-21cpu00:14:442
45439960,0cyclictest26170-21iostat_ios00:29:472
45439960,0cyclictest18835-21latency_hist19:44:332
45439960,0cyclictest18382-21kthreadcore22:49:492
45439960,0cyclictest0-21swapper/223:55:072
45439960,0cyclictest0-21swapper/223:52:152
45439960,0cyclictest0-21swapper/222:38:372
45439960,0cyclictest0-21swapper/222:20:412
45439960,0cyclictest0-21swapper/222:08:012
45439960,0cyclictest0-21swapper/221:34:192
45439960,0cyclictest0-21swapper/219:50:302
339960,0migration/37910-21sh00:35:373
339960,0migration/37282-21sh23:02:443
339960,0migration/3558-21munin-node23:34:513
339960,0migration/34182-21irqrtprio21:09:483
339960,0migration/323727-21sh23:11:083
29725060,0irq/27-eth0-tx-1894-21munin-node20:44:420
29715060,0irq/26-eth0-rx-9309-21sh22:07:353
29715060,0irq/26-eth0-rx-26694-21iostat19:49:493
29715060,0irq/26-eth0-rx-25956-21diskstats21:59:453
29715060,0irq/26-eth0-rx-22374-21ssh00:24:553
149960,0migration/06891-21sh21:47:010
149960,0migration/0407-21sh23:17:190
45439954,0cyclictest3514-21snmpd20:07:082
45439952,0cyclictest8829-21sensors21:09:542
45439952,0cyclictest3556-21lldpd21:07:532
45439952,0cyclictest21911-21awk20:34:492
45439951,0cyclictest30250-21awk21:24:472
45439950,0cyclictest9427-21if_eth1.323:04:472
45439950,0cyclictest8454-21aten_r1power_po21:49:422
45439950,0cyclictest4164-21ssh22:40:432
45439950,0cyclictest28398-21kthreadcore23:14:502
45439950,0cyclictest23682-21ssh21:39:432
45439950,0cyclictest22901-21ssh00:25:182
45439950,0cyclictest16904-21latency_hist23:09:322
45439950,0cyclictest14140-21ssh00:20:022
45439950,0cyclictest0-21swapper/223:34:502
45439950,0cyclictest0-21swapper/223:32:042
45439950,0cyclictest0-21swapper/222:10:492
45439950,0cyclictest0-21swapper/222:00:132
45439950,0cyclictest0-21swapper/221:54:512
45439950,0cyclictest0-21swapper/221:15:052
45439950,0cyclictest0-21swapper/220:54:322
45439950,0cyclictest0-21swapper/219:24:482
45439950,0cyclictest0-21swapper/200:04:502
45429955,0cyclictest0-21swapper/123:45:401
45429950,0cyclictest0-21swapper/123:03:251
45429950,0cyclictest0-21swapper/119:14:581
45419955,0cyclictest0-21swapper/000:20:250
45419950,0cyclictest0-21swapper/021:41:580
45419950,0cyclictest0-21swapper/021:23:310
45419950,0cyclictest0-21swapper/019:39:470
4150,0ktimersoftd/00-21swapper/022:52:170
339950,0migration/39664-21munin-node22:24:523
339950,0migration/38990-21kthreadcore19:34:503
339950,0migration/37472-21munin-node23:19:533
339950,0migration/37121-21kthreadcore20:44:503
339950,0migration/36724-21fschecks_count22:44:453
339950,0migration/332134-21gltestperf19:29:463
339950,0migration/331810-21munin-node23:14:583
339950,0migration/331617-21sh00:30:393
339950,0migration/330842-21taskset19:26:013
339950,0migration/328045-21kthreadcore20:14:503
339950,0migration/327457-21kthreadcore22:54:503
339950,0migration/327423-21runrttasks23:50:253
339950,0migration/321538-21fschecks_count20:34:473
339950,0migration/32117-21kthreadcore23:54:493
339950,0migration/318963-21latency_hist22:14:323
339950,0migration/318840-21gltestperf19:19:463
339950,0migration/318761-21taskset23:45:093
339950,0migration/31407-21taskset20:41:383
339950,0migration/311962-21kthreadcore23:04:493
339950,0migration/311549-21taskset21:14:363
339950,0migration/311323-21latency21:29:493
339950,0migration/310502-21diskmemload22:09:393
29725050,0irq/27-eth0-tx-6559-21runrttasks19:14:550
29725050,0irq/27-eth0-tx-2948-21munin-node21:09:460
29725050,0irq/27-eth0-tx-13395-21munin-node20:04:510
29725050,0irq/27-eth0-tx-0-21swapper/019:13:120
29715050,0irq/26-eth0-rx-3495-21ssh21:25:353
29715050,0irq/26-eth0-rx-25567-21ssh23:30:433
29715050,0irq/26-eth0-rx-24333-21grep19:44:503
29715050,0irq/26-eth0-rx-20202-21sed20:29:553
29715050,0irq/26-eth0-rx-19287-21grep22:49:493
29715050,0irq/26-eth0-rx-18322-21kthreadcore20:54:493
29715050,0irq/26-eth0-rx-17826-21ssh23:27:353
29715050,0irq/26-eth0-rx-16908-21ssh21:54:363
29715050,0irq/26-eth0-rx-12456-21kthreadcore21:49:483
29715050,0irq/26-eth0-rx-10307-21kthreadcore23:59:493
29715050,0irq/26-eth0-rx-0-21swapper/322:34:493
29715050,0irq/26-eth0-rx-0-21swapper/321:44:333
29715050,0irq/26-eth0-rx-0-21swapper/321:42:573
29715050,0irq/26-eth0-rx-0-21swapper/321:35:023
29715050,0irq/26-eth0-rx-0-21swapper/321:24:253
29715050,0irq/26-eth0-rx-0-21swapper/300:23:333
29715050,0irq/26-eth0-rx-0-21swapper/300:17:023
28803250,0sleep030771-21kthreadcore20:39:500
179950,0migration/130969-21kthreadcore21:04:501
179950,0migration/127769-21aten_r1power_en22:19:431
179950,0migration/126692-21munin-node21:19:561
179950,0migration/125646-21kthreadcore23:49:481
179950,0migration/125504-21sh21:59:371
179950,0migration/124580-21latency_hist19:24:331
179950,0migration/114366-21latency_hist00:04:321
149950,0migration/09584-21kthreadcore19:34:500
149950,0migration/06077-21kthreadcore22:04:500
149950,0migration/06073-21cpu23:59:440
149950,0migration/04980-21munin-node22:24:470
149950,0migration/03954-21kthreadcore20:19:500
149950,0migration/031553-21sh22:58:590
149950,0migration/031477-21timerandwakeup19:49:560
149950,0migration/028573-21idleruntime22:19:460
149950,0migration/027194-21kthreadcore00:29:480
149950,0migration/02664-21idleruntime23:19:480
149950,0migration/025312-21runrttasks22:15:010
149950,0migration/023455-21taskset21:55:420
149950,0migration/020957-21sendmail_mailtr20:54:540
149950,0migration/019183-21kthreadcore00:24:490
149950,0migration/018896-21hwlatdetect19:19:470
149950,0migration/017890-21runrttasks22:12:360
149950,0migration/017772-21taskset23:27:280
149950,0migration/016248-21kthreadcore23:44:480
149950,0migration/015056-21taskset21:51:060
149950,0migration/015044-21cpuspeed_turbos21:34:450
149950,0migration/014059-21taskset20:28:420
149950,0migration/012725-21aten_r1power_en22:29:420
149950,0migration/012290-21munin-node19:59:550
149950,0migration/012095-21munin-node22:44:540
149950,0migration/010489-21sh23:40:500
149950,0migration/010246-21kthreadcore21:29:490
45439943,0cyclictest3556-21lldpd19:49:222
45439943,0cyclictest3514-21snmpd20:42:392
45439943,0cyclictest3241-21irqbalance20:30:012
45439943,0cyclictest3241-21irqbalance20:21:212
45439943,0cyclictest3241-21irqbalance20:14:512
45439943,0cyclictest3241-21irqbalance20:02:512
45439943,0cyclictest13671-21head20:24:562
45439943,0cyclictest12366-21awk19:14:472
45439942,0cyclictest18521-21grep20:54:492
45439941,0cyclictest915-21awk19:54:462
45429943,0cyclictest9283-21awk21:49:461
45429943,0cyclictest8655-21awk20:24:471
45429943,0cyclictest601-21awk21:44:471
45429943,0cyclictest3514-21snmpd22:50:231
45429943,0cyclictest3514-21snmpd22:39:531
45429943,0cyclictest30590-21awk23:54:471
45429943,0cyclictest28515-21awk20:39:481
45429943,0cyclictest27631-21awk20:14:481
45429943,0cyclictest22205-21awk22:34:471
45429943,0cyclictest20950-21awk20:09:451
45429943,0cyclictest14277-21awk20:04:481
45429943,0cyclictest12478-21awk21:14:461
45419944,0cyclictest0-21swapper/022:02:370
45419943,0cyclictest7622-21sadc00:19:320
45419943,0cyclictest3556-21lldpd21:00:530
45419943,0cyclictest3514-21snmpd00:36:390
45419943,0cyclictest3241-21irqbalance00:09:400
45419943,0cyclictest30492-21awk23:54:460
45419943,0cyclictest30191-21awk21:24:470
45419943,0cyclictest25692-21awk19:24:480
45419943,0cyclictest22785-21sadc00:09:320
45419943,0cyclictest22252-21awk22:34:470
339940,0migration/38932-21kthreadcore19:09:473
339940,0migration/3485-21kthreadcore21:04:513
339940,0migration/331004-21kthreadcore22:19:493
339940,0migration/32988-21kthreadcore19:54:483
339940,0migration/320834-21idleruntime20:09:453
339940,0migration/315444-21kthreadcore20:04:503
339940,0migration/313693-21kernelversion19:39:493
339940,0migration/313303-21kthreadcore19:14:493
339940,0migration/311558-21kthreadcore20:49:493
339940,0migration/310805-21kthreadcore20:24:493
29715040,0irq/26-eth0-rx-0-21swapper/323:39:373
29715040,0irq/26-eth0-rx-0-21swapper/322:39:333
29715040,0irq/26-eth0-rx-0-21swapper/322:30:323
29715040,0irq/26-eth0-rx-0-21swapper/319:59:473
29715040,0irq/26-eth0-rx-0-21swapper/300:09:323
29715040,0irq/26-eth0-rx-0-21swapper/300:05:263
249940,0watchdog/20-21swapper/220:59:512
179940,0migration/18745-21aten_r1power_po20:49:431
179940,0migration/18179-21kthreadcore19:34:491
179940,0migration/16608-21kthreadcore22:24:491
179940,0migration/14688-21latency_hist19:09:331
179940,0migration/14053-21kthreadcore20:19:501
179940,0migration/13403-21fschecks_count21:09:451
179940,0migration/131961-21sh00:31:241
179940,0migration/130303-21kthreadcore19:49:511
179940,0migration/128758-21kthreadcore19:24:501
179940,0migration/127552-21latency_hist23:34:321
179940,0migration/126951-21kthreadcore00:09:491
179940,0migration/124793-21munin-node23:29:541
179940,0migration/124443-21munin-node21:39:471
179940,0migration/124232-21kthreadcore20:59:501
179940,0migration/11692-21latency_hist23:19:321
179940,0migration/116372-21kthreadcore19:39:501
179940,0migration/115490-21fschecks_time21:34:461
179940,0migration/115392-21aten_r1power_po20:54:431
179940,0migration/11345-21kthreadcore00:14:491
179940,0migration/113299-21sh21:32:151
179940,0migration/112633-21runrttasks22:45:171
169940,0watchdog/14882-21iostat23:39:461
169940,0watchdog/123756-21kthreadcore20:34:511
169940,0watchdog/123542-21ssh23:10:421
169940,0watchdog/121723-21cut00:24:501
169940,0watchdog/1179751ssh22:12:461
169940,0watchdog/115035-21ssh23:05:541
169940,0watchdog/10-21swapper/123:59:381
169940,0watchdog/10-21swapper/122:54:501
169940,0watchdog/10-21swapper/122:29:461
169940,0watchdog/10-21swapper/120:44:551
169940,0watchdog/10-21swapper/100:34:501
149940,0migration/028825-21kthreadcore20:14:500
149940,0migration/028411-21sa223:52:320
149940,0migration/02033-21timerandwakeup21:04:570
149940,0migration/016985-21kthreadcore20:29:490
149940,0migration/01550-21kthreadcore22:59:490
149940,0migration/015272-21kthreadcore21:14:490
45439933,0cyclictest3556-21lldpd19:32:522
45439932,0cyclictest7212-21sort20:44:502
45439932,0cyclictest23746-21sensors19:19:552
45439930,0cyclictest11429-21sh19:34:552
45429933,0cyclictest3556-21lldpd19:30:221
45429932,0cyclictest7583-21awk19:59:481
45429932,0cyclictest30176-21awk21:24:471
45429932,0cyclictest17932-21awk21:54:461
45429932,0cyclictest15282-21awk20:29:471
45429931,0cyclictest11823-21kthreadcore23:24:481
45429930,0cyclictest19702-21perl19:44:461
45419933,0cyclictest0-21swapper/023:34:410
45419932,0cyclictest4470-21ssh22:41:240
45419932,0cyclictest3514-21snmpd23:13:570
45419932,0cyclictest25755-21grep20:34:510
45419932,0cyclictest25670-21sensors20:09:520
45419932,0cyclictest19067-21munin-node19:44:460
45419930,0cyclictest5517-21basename19:34:330
45419930,0cyclictest20068-21iostat_ios23:29:480
29715030,0irq/26-eth0-rx-0-21swapper/320:59:453
29715030,0irq/26-eth0-rx-0-21swapper/320:20:283
169930,0watchdog/10-21swapper/123:14:501
169930,0watchdog/10-21swapper/122:14:541
169930,0watchdog/10-21swapper/122:04:351
169930,0watchdog/10-21swapper/119:54:551
169930,0watchdog/10-21swapper/100:19:341
149930,0migration/015880-21sh23:07:450
45419920,0cyclictest0-21swapper/019:54:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional