You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-13 - 10:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Sun Jul 13, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2246227170,0sleep00-21swapper/018:41:050
291350540,0irq/25-eth10-21swapper/318:43:563
2244924936,0sleep20-21swapper/218:40:552
2251823526,0sleep10-21swapper/118:41:421
249970,0watchdog/225066-21kthreadcore23:49:502
339960,0migration/39113-21sh22:38:253
339960,0migration/36312-21sh23:36:093
339960,0migration/325850-21sh22:07:083
339960,0migration/32248-21munin-node20:49:503
339960,0migration/317095-21sh22:43:323
28415060,0irq/27-eth0-tx-0-21swapper/223:56:012
28415060,0irq/27-eth0-tx-0-21swapper/223:46:582
28415060,0irq/27-eth0-tx-0-21swapper/223:15:112
28415060,0irq/27-eth0-tx-0-21swapper/223:09:112
28415060,0irq/27-eth0-tx-0-21swapper/222:52:252
28415060,0irq/27-eth0-tx-0-21swapper/222:43:422
28415060,0irq/27-eth0-tx-0-21swapper/222:24:032
28415060,0irq/27-eth0-tx-0-21swapper/222:13:592
28415060,0irq/27-eth0-tx-0-21swapper/221:55:312
28415060,0irq/27-eth0-tx-0-21swapper/221:42:342
28415060,0irq/27-eth0-tx-0-21swapper/221:31:322
28415060,0irq/27-eth0-tx-0-21swapper/221:24:402
28415060,0irq/27-eth0-tx-0-21swapper/221:09:552
28415060,0irq/27-eth0-tx-0-21swapper/221:01:272
28415060,0irq/27-eth0-tx-0-21swapper/220:48:012
28415060,0irq/27-eth0-tx-0-21swapper/020:24:440
28415060,0irq/27-eth0-tx-0-21swapper/019:54:300
28415060,0irq/27-eth0-tx-0-21swapper/019:44:540
28415060,0irq/27-eth0-tx-0-21swapper/018:51:340
28405060,0irq/26-eth0-rx-7650-21kthreadcore20:54:513
28405060,0irq/26-eth0-rx-7650-21kthreadcore20:54:513
28405060,0irq/26-eth0-rx-24111-21rm22:24:563
28405060,0irq/26-eth0-rx-0-21swapper/323:44:423
28405060,0irq/26-eth0-rx-0-21swapper/321:56:273
249960,0watchdog/220651-21chrt20:41:382
249960,0watchdog/217879-21kthreadcore21:19:502
249960,0watchdog/20-21swapper/222:17:302
149960,0migration/03447-21kthreadcore20:29:510
149960,0migration/02716-21sh21:31:390
149960,0migration/02201-21latency_hist22:14:310
339950,0migration/39447-21kthreadcore19:44:523
339950,0migration/38484-21gltestperf23:39:513
339950,0migration/37575-21kthreadcore21:14:523
339950,0migration/36548-21needreboot20:29:533
339950,0migration/36308-21kthreadcore22:54:503
339950,0migration/35038-21timerandwakeup19:39:573
339950,0migration/34062-21timerwakeupswit19:14:583
339950,0migration/34037-21aten_r1power_en18:54:463
339950,0migration/332076-21taskset20:01:513
339950,0migration/329593-21df_abs21:09:473
339950,0migration/326716-21kthreadcore23:49:513
339950,0migration/32452-21aten_r1power_cu22:14:423
339950,0migration/323332-21kthreadcore19:29:523
339950,0migration/323049-21taskset18:44:453
339950,0migration/322938-21munin-node19:04:543
339950,0migration/321758-21latency23:24:513
339950,0migration/321203-21kthreadcore19:54:503
339950,0migration/320182-21taskset20:19:353
339950,0migration/317750-21switchtime21:39:583
339950,0migration/317444-21cpuspeed_turbos23:04:473
339950,0migration/317273-21taskset22:00:093
339950,0migration/311984-21kthreadcore20:34:523
28415050,0irq/27-eth0-tx-23912-21ssh22:24:542
28415050,0irq/27-eth0-tx-18261-21munin-node19:29:480
28415050,0irq/27-eth0-tx-1480-21ssh21:50:202
28415050,0irq/27-eth0-tx-10596-21rt-features21:14:572
28415050,0irq/27-eth0-tx-0-21swapper/223:34:562
28415050,0irq/27-eth0-tx-0-21swapper/223:22:352
28415050,0irq/27-eth0-tx-0-21swapper/222:58:142
28415050,0irq/27-eth0-tx-0-21swapper/222:35:452
28415050,0irq/27-eth0-tx-0-21swapper/222:32:032
28415050,0irq/27-eth0-tx-0-21swapper/222:00:022
28415050,0irq/27-eth0-tx-0-21swapper/221:47:232
28415050,0irq/27-eth0-tx-0-21swapper/220:50:182
28415050,0irq/27-eth0-tx-0-21swapper/200:09:572
28415050,0irq/27-eth0-tx-0-21swapper/200:06:592
28415050,0irq/27-eth0-tx-0-21swapper/019:14:470
28405050,0irq/26-eth0-rx-6122-21ssh23:14:533
28405050,0irq/26-eth0-rx-13661-21kthreadcore22:59:523
28405050,0irq/26-eth0-rx-0-21swapper/323:54:473
28405050,0irq/26-eth0-rx-0-21swapper/323:20:503
28405050,0irq/26-eth0-rx-0-21swapper/323:10:443
28405050,0irq/26-eth0-rx-0-21swapper/322:48:193
28405050,0irq/26-eth0-rx-0-21swapper/322:20:053
28405050,0irq/26-eth0-rx-0-21swapper/322:10:183
28405050,0irq/26-eth0-rx-0-21swapper/321:34:503
28405050,0irq/26-eth0-rx-0-21swapper/321:25:213
28405050,0irq/26-eth0-rx-0-21swapper/321:23:563
28405050,0irq/26-eth0-rx-0-21swapper/321:04:433
28405050,0irq/26-eth0-rx-0-21swapper/320:59:503
28405050,0irq/26-eth0-rx-0-21swapper/300:05:183
28405050,0irq/26-eth0-rx-0-21swapper/300:00:473
259950,0migration/228122-21kthreadcore23:09:522
259950,0migration/219589-21fschecks_time22:04:492
259950,0migration/212695-21kthreadcore23:39:552
249950,0watchdog/20-21swapper/223:33:302
249950,0watchdog/20-21swapper/223:24:422
249950,0watchdog/20-21swapper/222:46:022
249950,0watchdog/20-21swapper/221:34:342
249950,0watchdog/20-21swapper/221:04:382
249950,0watchdog/20-21swapper/220:55:582
249950,0watchdog/20-21swapper/220:55:582
249950,0watchdog/20-21swapper/220:14:382
249950,0watchdog/20-21swapper/220:09:182
249950,0watchdog/20-21swapper/219:50:302
249950,0watchdog/20-21swapper/219:44:582
249950,0watchdog/20-21swapper/219:29:502
249950,0watchdog/20-21swapper/219:29:182
249950,0watchdog/20-21swapper/219:16:062
249950,0watchdog/20-21swapper/219:06:062
249950,0watchdog/20-21swapper/218:57:182
249950,0watchdog/20-21swapper/218:48:072
249950,0watchdog/20-21swapper/200:01:502
228359955,0cyclictest0-21swapper/219:34:352
228359950,0cyclictest0-21swapper/223:01:072
228339954,0cyclictest9499-21ls22:39:311
228339950,0cyclictest0-21swapper/122:05:191
149950,0migration/09718-21kthreadcore20:54:510
149950,0migration/09718-21kthreadcore20:54:510
149950,0migration/07630-21gltestperf20:09:460
149950,0migration/06906-21ntp_offset22:54:520
149950,0migration/06248-21taskset20:05:080
149950,0migration/05798-21kthreadcore23:14:510
149950,0migration/04569-21kthreadcore22:14:490
149950,0migration/03759-21kthreadcore23:54:530
149950,0migration/03623-21taskset20:51:270
149950,0migration/03173-21kthreadcore22:34:500
149950,0migration/031487-21taskset23:10:580
149950,0migration/031476-21taskset23:33:130
149950,0migration/029260-21kthreadcore21:49:520
149950,0migration/028876-21sh20:49:100
149950,0migration/027122-21irqrtprio19:59:490
149950,0migration/026730-21fschecks_count22:29:480
149950,0migration/025292-21sh22:27:240
149950,0migration/024212-21cpuspeed_turbos19:09:470
149950,0migration/022050-21kthreadcore21:24:530
149950,0migration/020661-21taskset20:41:460
149950,0migration/020340-21kthreadcore23:44:520
149950,0migration/018853-21taskset21:42:330
149950,0migration/018542-21kthreadcore20:14:520
149950,0migration/016716-21kthreadcore19:24:520
149950,0migration/016346-21sh23:24:380
149950,0migration/015576-21sh23:00:290
149950,0migration/015477-21kthreadcore20:59:500
149950,0migration/014217-21aten_r1power_cu00:04:450
149950,0migration/013756-21kthreadcore22:39:510
149950,0migration/012916-21kthreadcore21:59:560
339940,0migration/39583-21kthreadcore19:19:513
339940,0migration/332495-21latency_hist20:04:323
339940,0migration/328466-21kthreadcore20:24:493
339940,0migration/327677-21sh20:45:463
339940,0migration/323328-21kthreadcore00:09:513
339940,0migration/319721-21runrttasks20:16:533
339940,0migration/313227-21fschecks_time19:49:483
28415040,0irq/27-eth0-tx-0-21swapper/019:54:560
28405040,0irq/26-eth0-rx-25576-21fschecks_time19:34:483
28405040,0irq/26-eth0-rx-0-21swapper/323:29:473
28405040,0irq/26-eth0-rx-0-21swapper/322:49:483
28405040,0irq/26-eth0-rx-0-21swapper/322:29:463
28405040,0irq/26-eth0-rx-0-21swapper/321:53:413
28405040,0irq/26-eth0-rx-0-21swapper/321:44:473
28405040,0irq/26-eth0-rx-0-21swapper/321:29:423
28405040,0irq/26-eth0-rx-0-21swapper/320:39:523
28405040,0irq/26-eth0-rx-0-21swapper/319:29:283
28405040,0irq/26-eth0-rx-0-21swapper/319:09:543
259940,0migration/24110-21kthreadcore20:29:512
259940,0migration/225734-21kthreadcore19:09:512
249940,0watchdog/2396-21irqstats19:39:502
249940,0watchdog/230649-21if_eth1.218:49:512
249940,0watchdog/211792-21cut20:09:502
249940,0watchdog/20-21swapper/220:26:062
249940,0watchdog/20-21swapper/220:03:022
249940,0watchdog/20-21swapper/219:00:302
228359943,0cyclictest8715-21awk20:34:502
228359943,0cyclictest5852-21awk19:19:492
228359943,0cyclictest21062-21awk20:19:482
228359943,0cyclictest20104-21awk19:54:482
228339943,0cyclictest968-21sadc22:54:311
228339943,0cyclictest6800-21awk19:44:501
228339943,0cyclictest6445-21snmpd23:30:001
228339943,0cyclictest6445-21snmpd20:12:151
228339943,0cyclictest6445-21snmpd19:16:441
228339943,0cyclictest5763-21awk19:19:481
228339943,0cyclictest3425-21lldpd23:21:551
228339943,0cyclictest3425-21lldpd21:01:341
228339943,0cyclictest3425-21lldpd20:32:341
228339943,0cyclictest3110-21irqbalance20:45:061
228339943,0cyclictest29402-21awk21:29:481
228339943,0cyclictest27724-21awk20:24:481
228339943,0cyclictest18632-21awk22:44:471
228339943,0cyclictest17041-21sadc19:04:321
228339943,0cyclictest14975-21kworker/1:122:57:011
228339943,0cyclictest14975-21kworker/1:122:09:371
228339943,0cyclictest14975-21kworker/1:119:41:301
228339943,0cyclictest14975-21kworker/1:100:04:491
228339943,0cyclictest14381-21awk20:14:491
228339940,0cyclictest0-21swapper/123:59:181
228329943,0cyclictest7425-21awk23:59:470
228329943,0cyclictest6445-21snmpd18:55:440
228329943,0cyclictest3425-21lldpd19:22:040
228329943,0cyclictest30140-21awk21:09:500
228329943,0cyclictest22732-21awk00:09:490
228329943,0cyclictest13709-21awk21:19:480
228329940,0cyclictest17276-21munin-node19:04:500
228329940,0cyclictest10641-21munin-node18:59:540
179940,0migration/19714-21taskset22:39:321
179940,0migration/14905-21kthreadcore21:54:521
179940,0migration/14887-21aten_r1power_en21:14:461
179940,0migration/14107-21munin-node21:34:461
179940,0migration/13647-21kthreadcore23:34:501
179940,0migration/128949-21ntp_kernel_pll_23:49:531
179940,0migration/127097-21kthreadcore19:09:521
179940,0migration/126467-21fschecks_time19:59:471
179940,0migration/126185-21latency_hist22:29:311
179940,0migration/123690-21fschecks_time18:44:481
179940,0migration/122213-21kthreadcore22:24:531
179940,0migration/119897-21kthreadcore19:29:511
179940,0migration/118347-21sh21:40:561
179940,0migration/118026-21kthreadcore20:39:511
179940,0migration/116192-21kthreadcore19:24:521
179940,0migration/112968-21df_abs19:49:471
179940,0migration/111155-21munin-node22:19:451
169940,0watchdog/19443-21basename22:59:461
169940,0watchdog/13299-21munin-node22:14:461
169940,0watchdog/13160-21ssh21:10:341
169940,0watchdog/130103-21ls20:49:461
169940,0watchdog/125997-21grep21:24:541
169940,0watchdog/123960-21ssh23:29:221
169940,0watchdog/122738-21tr19:54:501
169940,0watchdog/121497-21kthreadcore20:19:501
169940,0watchdog/119741-21ssh21:44:381
169940,0watchdog/118092-21if_err_eth1.323:04:501
169940,0watchdog/117804-21ssh21:19:501
169940,0watchdog/111215-21grep23:59:501
169940,0watchdog/110307-21kthreadcore23:39:541
169940,0watchdog/10-21swapper/123:14:501
169940,0watchdog/10-21swapper/123:09:581
169940,0watchdog/10-21swapper/121:49:581
169940,0watchdog/10-21swapper/118:49:511
149940,0migration/028427-21kthreadcore19:34:520
149940,0migration/024147-21kthreadcore22:04:530
149940,0migration/020798-21kthreadcore22:44:490
149940,0migration/014679-21sh23:42:380
339930,0migration/37315-21fschecks_count20:09:463
28415030,0irq/27-eth0-tx-0-21swapper/020:34:480
28405030,0irq/26-eth0-rx-0-21swapper/318:59:443
28405030,0irq/26-eth0-rx-0-21swapper/318:49:463
228339933,0cyclictest3425-21lldpd00:09:411
228339933,0cyclictest14975-21kworker/1:123:44:331
228339933,0cyclictest14975-21kworker/1:121:59:371
228339933,0cyclictest14975-21kworker/1:121:04:331
228339933,0cyclictest14975-21kworker/1:120:54:351
228339933,0cyclictest14975-21kworker/1:120:54:351
228339933,0cyclictest14975-21kworker/1:120:34:481
228339933,0cyclictest14975-21kworker/1:120:04:481
228339933,0cyclictest14975-21kworker/1:119:34:461
228329933,0cyclictest4057-21awk21:54:490
228329932,0cyclictest4896-21sensors19:39:560
228329932,0cyclictest3425-21lldpd23:37:580
228329932,0cyclictest23902-21awk18:44:500
228329932,0cyclictest10645-21sensors21:14:570
228329930,0cyclictest9088-21perl23:19:490
228329930,0cyclictest17607-21diskstats23:04:480
228329930,0cyclictest0-21swapper/022:22:390
179930,0migration/14040-21aten_r1power_en18:54:461
169930,0watchdog/122549-21latency19:04:501
149930,0migration/028175-21sh21:07:340
149930,0migration/019763-21sh21:44:420
149930,0migration/011145-21sh21:38:040
228329921,0cyclictest20336-21aten_r1power_vo20:19:450
228329920,0cyclictest23356-21latency_hist23:49:310
228329920,0cyclictest0-21swapper/022:49:310
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional