You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 08:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Nov 22, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350350,0irq/25-eth10-21swapper/318:36:043
492923423,0sleep00-21swapper/018:37:310
481223324,0sleep20-21swapper/218:36:092
510122824,0sleep10-21swapper/118:39:321
51799976,0cyclictest26302-21munin-node19:19:571
51809960,0cyclictest0-21swapper/223:34:582
35160,0ktimersoftd/30-21swapper/320:46:343
339960,0migration/39453-21sh22:40:133
339960,0migration/38814-21munin-node22:19:573
339960,0migration/37574-21sh00:00:073
339960,0migration/36787-21kthreadcore20:39:553
339960,0migration/328741-21munin-node21:14:493
339960,0migration/327470-21irqrtprio19:19:543
339960,0migration/327054-21sh21:11:353
339960,0migration/326199-21sh22:52:103
339960,0migration/325712-21sh23:32:003
339960,0migration/317988-21sh23:07:073
28415060,0irq/27-eth0-tx-0-21swapper/223:51:542
28415060,0irq/27-eth0-tx-0-21swapper/223:31:302
28415060,0irq/27-eth0-tx-0-21swapper/223:27:042
28415060,0irq/27-eth0-tx-0-21swapper/223:27:042
28415060,0irq/27-eth0-tx-0-21swapper/223:17:452
28415060,0irq/27-eth0-tx-0-21swapper/223:10:302
28415060,0irq/27-eth0-tx-0-21swapper/223:06:002
28415060,0irq/27-eth0-tx-0-21swapper/223:02:172
28415060,0irq/27-eth0-tx-0-21swapper/222:57:022
28415060,0irq/27-eth0-tx-0-21swapper/222:32:552
28415060,0irq/27-eth0-tx-0-21swapper/222:15:052
28415060,0irq/27-eth0-tx-0-21swapper/221:55:092
28415060,0irq/27-eth0-tx-0-21swapper/221:45:252
28415060,0irq/27-eth0-tx-0-21swapper/221:12:122
28415060,0irq/27-eth0-tx-0-21swapper/221:06:252
28415060,0irq/27-eth0-tx-0-21swapper/221:01:212
28415060,0irq/27-eth0-tx-0-21swapper/220:59:172
28415060,0irq/27-eth0-tx-0-21swapper/220:50:442
28415060,0irq/27-eth0-tx-0-21swapper/220:49:212
28415060,0irq/27-eth0-tx-0-21swapper/220:41:552
28415060,0irq/27-eth0-tx-0-21swapper/200:03:532
28415060,0irq/27-eth0-tx-0-21swapper/020:05:420
28415060,0irq/27-eth0-tx-0-21swapper/018:59:010
28405060,0irq/26-eth0-rx-895-21sh23:35:593
28405060,0irq/26-eth0-rx-7631-21kernelversion19:04:523
28405060,0irq/26-eth0-rx-28395-21iostat_ios22:54:543
28405060,0irq/26-eth0-rx-11594-21ssh21:03:363
169960,0watchdog/19268-21ssh23:00:031
169960,0watchdog/16952-21grep18:39:521
169960,0watchdog/124028-21grep20:49:561
149960,0migration/09696-21switchtime21:39:590
8225050,0irq/24-0000:00:20400-21latency_hist19:39:372
51799955,0cyclictest3328-21diskmemload21:00:041
51799955,0cyclictest0-21swapper/121:40:451
51799954,0cyclictest7960-21grep20:14:561
51799954,0cyclictest20823-21awk21:49:501
51799950,0cyclictest0-21swapper/123:44:401
51799950,0cyclictest0-21swapper/122:41:081
51799950,0cyclictest0-21swapper/122:11:581
51799950,0cyclictest0-21swapper/120:42:341
51799950,0cyclictest0-21swapper/100:08:001
4150,0ktimersoftd/00-21swapper/020:56:120
4150,0ktimersoftd/00-21swapper/019:24:550
339950,0migration/327419-21munin-node20:05:023
339950,0migration/325847-21timerandwakeup21:49:583
339950,0migration/32226-21fschecks_time19:49:533
339950,0migration/32168-21runrttasks22:37:413
339950,0migration/319875-21munin-node22:29:513
339950,0migration/316652-21sendmail_mailst23:24:563
339950,0migration/316652-21sendmail_mailst23:24:563
339950,0migration/316577-21kthreadcore21:44:543
339950,0migration/314258-21kthreadcore22:04:553
339950,0migration/312194-21kthreadcore20:19:533
339950,0migration/31112-21fschecks_count19:24:493
339950,0migration/311109-21gltestperf23:44:523
28415050,0irq/27-eth0-tx-4088-21if_err_eth122:39:562
28415050,0irq/27-eth0-tx-27664-21cut23:54:532
28415050,0irq/27-eth0-tx-19668-21munin-node22:09:512
28415050,0irq/27-eth0-tx-14078-21kthreadcore22:44:532
28415050,0irq/27-eth0-tx-0-21swapper/223:47:552
28415050,0irq/27-eth0-tx-0-21swapper/223:21:362
28415050,0irq/27-eth0-tx-0-21swapper/222:50:302
28415050,0irq/27-eth0-tx-0-21swapper/222:21:442
28415050,0irq/27-eth0-tx-0-21swapper/222:05:122
28415050,0irq/27-eth0-tx-0-21swapper/222:01:382
28415050,0irq/27-eth0-tx-0-21swapper/221:50:452
28415050,0irq/27-eth0-tx-0-21swapper/221:40:102
28415050,0irq/27-eth0-tx-0-21swapper/221:27:292
28415050,0irq/27-eth0-tx-0-21swapper/221:21:422
28415050,0irq/27-eth0-tx-0-21swapper/221:18:242
28415050,0irq/27-eth0-tx-0-21swapper/019:40:230
28415050,0irq/27-eth0-tx-0-21swapper/019:29:510
28405050,0irq/26-eth0-rx-3501-21grep18:59:563
28405050,0irq/26-eth0-rx-31466-21ssh23:54:563
28405050,0irq/26-eth0-rx-19471-21ssh21:07:593
28405050,0irq/26-eth0-rx-10758-21seq22:02:493
28405050,0irq/26-eth0-rx-0-21swapper/323:23:213
28405050,0irq/26-eth0-rx-0-21swapper/323:14:523
28405050,0irq/26-eth0-rx-0-21swapper/323:12:043
28405050,0irq/26-eth0-rx-0-21swapper/322:47:283
28405050,0irq/26-eth0-rx-0-21swapper/322:17:053
28405050,0irq/26-eth0-rx-0-21swapper/321:58:453
28405050,0irq/26-eth0-rx-0-21swapper/321:40:043
28405050,0irq/26-eth0-rx-0-21swapper/321:25:233
28405050,0irq/26-eth0-rx-0-21swapper/321:23:513
28405050,0irq/26-eth0-rx-0-21swapper/320:53:223
259950,0migration/230408-21kthreadcore19:19:552
259950,0migration/228309-21hwlatdetect22:34:502
259950,0migration/22309-21latency_hist20:14:372
259950,0migration/220855-21munin-node21:29:522
259950,0migration/220829-21unixbench-2d19:59:592
259950,0migration/21862-21taskset21:35:012
259950,0migration/212849-21kthreadcore00:04:562
249950,0watchdog/22322-21ssh23:39:432
19150,0ktimersoftd/10-21swapper/100:04:321
179950,0migration/18559-21irqrtprio19:29:551
179950,0migration/13048-21fschecks_time23:39:501
179950,0migration/128073-21latency_hist20:54:371
179950,0migration/125106-21timerwakeupswit23:10:021
179950,0migration/123166-21kthreadcore22:49:541
179950,0migration/12277-21gltestperf19:49:531
179950,0migration/11762-21taskset22:16:011
169950,0watchdog/19735-21ssh22:20:311
169950,0watchdog/1821-21unixbench_singl19:45:001
169950,0watchdog/14808-21gltestperf21:19:521
169950,0watchdog/131002chrt20:37:121
169950,0watchdog/1268342sleep119:41:521
169950,0watchdog/125095-21sh19:14:561
169950,0watchdog/122796-21tr18:49:561
169950,0watchdog/121920-21expr20:04:521
169950,0watchdog/114552-21kthreadcore21:04:561
169950,0watchdog/10-21swapper/123:58:551
169950,0watchdog/10-21swapper/123:35:351
169950,0watchdog/10-21swapper/123:25:511
169950,0watchdog/10-21swapper/123:25:511
169950,0watchdog/10-21swapper/123:07:391
169950,0watchdog/10-21swapper/122:55:551
169950,0watchdog/10-21swapper/120:09:481
169950,0watchdog/10-21swapper/119:04:561
149950,0migration/09105-21hddtemp_smartct19:54:500
149950,0migration/06350-21kthreadcore23:59:550
149950,0migration/0519-21kthreadcore21:34:540
149950,0migration/04549-21munin-node22:19:550
149950,0migration/03637-21aten_r1power_cu20:39:480
149950,0migration/03262-21gltestperf20:14:520
149950,0migration/030899-21kthreadcore20:09:520
149950,0migration/030574-21kthreadcore21:14:530
149950,0migration/029284-21kthreadcore22:34:530
149950,0migration/029250-21kthreadcore19:19:550
149950,0migration/028850-21runrttasks19:44:530
149950,0migration/027590-21fschecks_count23:14:510
149950,0migration/02692-21latency_hist23:19:370
149950,0migration/026097-21munin-node21:09:570
149950,0migration/025471-21taskset22:50:040
149950,0migration/022329-21kthreadcore19:14:530
149950,0migration/022172-21kthreadcore20:49:550
149950,0migration/020339-21munin-node22:29:530
149950,0migration/019790-21latency_hist21:49:370
149950,0migration/01965-21taskset22:16:350
149950,0migration/019628-21munin-node19:34:590
149950,0migration/019414-21fschecks_count23:29:500
149950,0migration/019045-21latency_hist23:09:370
149950,0migration/017656-21ntp_kernel_pll_21:24:580
149950,0migration/017346-21kthreadcore19:59:540
149950,0migration/017271-21munin-node20:44:590
149950,0migration/016181-21taskset00:07:420
149950,0migration/015762-21missed_timers23:44:570
149950,0migration/015253-21kthreadcore21:04:560
149950,0migration/013253-21kthreadcore22:24:530
149950,0migration/010857-21sh21:01:420
8225040,0irq/24-0000:00:0-21swapper/220:34:402
51809943,0cyclictest6445-21snmpd20:25:332
51809943,0cyclictest6445-21snmpd18:47:472
51809943,0cyclictest1550-21awk19:24:512
51809943,0cyclictest12354-21awk22:24:522
51809943,0cyclictest12227-21grep19:04:592
51799943,0cyclictest9248-21awk19:54:511
51799943,0cyclictest7103-21kworker/1:220:44:511
51799943,0cyclictest6445-21snmpd20:01:481
51799942,0cyclictest12438-21kworker/1:121:55:431
51789944,0cyclictest0-21swapper/023:37:380
51789943,0cyclictest6445-21snmpd23:40:340
51789943,0cyclictest6445-21snmpd21:20:490
51789943,0cyclictest6445-21snmpd20:34:480
51789943,0cyclictest3926-21awk22:59:520
51789943,0cyclictest3425-21lldpd18:48:530
51789943,0cyclictest19044-21awk23:49:510
51789943,0cyclictest12153-21awk23:04:520
51789943,0cyclictest10094-21awk20:19:500
339940,0migration/38895-21kthreadcore18:39:523
339940,0migration/327999-21kthreadcore18:54:563
339940,0migration/325076-21kthreadcore20:29:543
339940,0migration/321869-21irqrtprio19:39:513
339940,0migration/321839-21kthreadcore21:29:553
339940,0migration/318171-21kthreadcore19:59:543
339940,0migration/316853-21kthreadcore19:34:553
339940,0migration/314215-21kthreadcore22:24:543
28415040,0irq/27-eth0-tx-0-21swapper/020:24:490
28415040,0irq/27-eth0-tx-0-21swapper/019:49:520
28415040,0irq/27-eth0-tx-0-21swapper/019:10:010
28415040,0irq/27-eth0-tx-0-21swapper/019:09:030
28415040,0irq/27-eth0-tx-0-21swapper/018:59:500
28405040,0irq/26-eth0-rx-25891-21ssh22:10:133
28405040,0irq/26-eth0-rx-0-21swapper/323:49:453
28405040,0irq/26-eth0-rx-0-21swapper/323:39:563
28405040,0irq/26-eth0-rx-0-21swapper/322:59:553
28405040,0irq/26-eth0-rx-0-21swapper/321:35:133
28405040,0irq/26-eth0-rx-0-21swapper/320:54:513
28405040,0irq/26-eth0-rx-0-21swapper/319:46:363
28405040,0irq/26-eth0-rx-0-21swapper/319:14:573
28405040,0irq/26-eth0-rx-0-21swapper/300:06:203
259940,0migration/223990-21kthreadcore20:29:542
259940,0migration/222295-21kthreadcore18:49:562
259940,0migration/221104-21kthreadcore19:14:532
259940,0migration/216037-21kthreadcore19:34:542
249940,0watchdog/232433-21fschecks_count18:59:522
249940,0watchdog/225750-21df_inode18:54:522
249940,0watchdog/213037-21hostname19:09:522
249940,0watchdog/210248-21ps20:19:522
249940,0watchdog/20-21swapper/220:14:562
249940,0watchdog/20-21swapper/219:29:562
249940,0watchdog/20-21swapper/218:39:522
20982240,0sleep00-21swapper/021:29:540
179940,0migration/1660-21irqcore18:59:541
179940,0migration/124403-21taskset23:50:291
179940,0migration/12439-21runrttasks21:36:411
179940,0migration/123600-21irqcore20:29:531
179940,0migration/121592-21kthreadcore21:29:551
179940,0migration/117968-21sh22:46:361
179940,0migration/115911-21runrttasks20:24:421
179940,0migration/114842-21kthreadcore21:24:561
179940,0migration/110221-21if_eth1.320:19:511
169940,0watchdog/19249-21ssh23:20:431
169940,0watchdog/16547-21runrttasks22:25:031
169940,0watchdog/10-21swapper/123:31:351
169940,0watchdog/10-21swapper/123:15:271
169940,0watchdog/10-21swapper/122:37:591
169940,0watchdog/10-21swapper/122:00:191
169940,0watchdog/10-21swapper/121:44:471
169940,0watchdog/10-21swapper/121:12:361
169940,0watchdog/10-21swapper/119:34:521
169940,0watchdog/10-21swapper/119:24:481
169940,0watchdog/10-21swapper/119:09:521
169940,0watchdog/10-21swapper/118:54:561
149940,0migration/032714-21sh23:57:130
149940,0migration/024271-21kthreadcore21:49:530
149940,0migration/022106-21kthreadcore22:09:550
8225030,0irq/24-0000:00:0-21swapper/219:50:012
51809932,0cyclictest9272-21awk19:54:512
51809930,0cyclictest32157-21cut19:44:542
51799933,0cyclictest7103-21kworker/1:218:44:551
51799933,0cyclictest12438-21kworker/1:122:29:511
51799933,0cyclictest12438-21kworker/1:122:04:451
51789933,0cyclictest0-21swapper/022:54:450
51789932,0cyclictest4816-21cut21:59:530
51789932,0cyclictest4148-21awk22:39:560
51789932,0cyclictest32227-21kthreadcore21:54:540
51789932,0cyclictest16266-21sed23:24:530
51789932,0cyclictest16266-21sed23:24:530
51789932,0cyclictest12590-21awk22:04:530
51789932,0cyclictest12179-21if_eth022:44:510
339930,0migration/317882-21kthreadcore20:24:543
28415030,0irq/27-eth0-tx-0-21swapper/020:29:490
28415030,0irq/27-eth0-tx-0-21swapper/018:49:500
28415030,0irq/27-eth0-tx-0-21swapper/018:39:470
28405030,0irq/26-eth0-rx-0-21swapper/320:09:473
28405030,0irq/26-eth0-rx-0-21swapper/319:57:103
28405030,0irq/26-eth0-rx-0-21swapper/319:12:223
28405030,0irq/26-eth0-rx-0-21swapper/318:49:503
249930,0watchdog/20-21swapper/220:04:522
169930,0watchdog/10-21swapper/121:14:441
329920,0watchdog/30-21swapper/320:34:593
28405020,0irq/26-eth0-rx-0-21swapper/320:14:573
28405020,0irq/26-eth0-rx-0-21swapper/319:29:503
28405020,0irq/26-eth0-rx-0-21swapper/318:44:453
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional