You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 12:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Mon Nov 17, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
351710,0ktimersoftd/31308-21perf18:39:383
776123425,0sleep20-21swapper/218:37:212
759823322,0sleep00-21swapper/018:35:270
777523127,0sleep10-21swapper/118:37:321
80419976,0cyclictest0-21swapper/023:40:090
28415070,0irq/27-eth0-tx-0-21swapper/223:12:582
80439960,0cyclictest0-21swapper/123:35:121
80419960,0cyclictest0-21swapper/020:20:050
339960,0migration/317550-21sh22:48:123
28415060,0irq/27-eth0-tx-6517-21kthreadcore22:20:072
28415060,0irq/27-eth0-tx-6269-21diskmemload21:30:082
28415060,0irq/27-eth0-tx-6250-21munin-node21:00:102
28415060,0irq/27-eth0-tx-3425-21lldpd22:26:482
28415060,0irq/27-eth0-tx-19211-21munin-node22:30:002
28415060,0irq/27-eth0-tx-0-21swapper/223:59:292
28415060,0irq/27-eth0-tx-0-21swapper/223:46:332
28415060,0irq/27-eth0-tx-0-21swapper/223:41:452
28415060,0irq/27-eth0-tx-0-21swapper/223:35:182
28415060,0irq/27-eth0-tx-0-21swapper/223:33:332
28415060,0irq/27-eth0-tx-0-21swapper/223:28:562
28415060,0irq/27-eth0-tx-0-21swapper/223:24:102
28415060,0irq/27-eth0-tx-0-21swapper/222:58:182
28415060,0irq/27-eth0-tx-0-21swapper/222:51:362
28415060,0irq/27-eth0-tx-0-21swapper/222:00:112
28415060,0irq/27-eth0-tx-0-21swapper/221:55:122
28415060,0irq/27-eth0-tx-0-21swapper/221:35:042
28415060,0irq/27-eth0-tx-0-21swapper/221:21:082
28415060,0irq/27-eth0-tx-0-21swapper/220:57:302
28415060,0irq/27-eth0-tx-0-21swapper/200:05:252
28405060,0irq/26-eth0-rx-7779-21kthreadcore21:40:073
28405060,0irq/26-eth0-rx-713-21entropy23:40:053
28405060,0irq/26-eth0-rx-24577-21wc22:30:073
249960,0watchdog/231496-21cat21:15:062
249960,0watchdog/227544-21users18:50:112
169960,0watchdog/111730-21ssh21:42:301
149960,0migration/06191-21sh00:01:200
149960,0migration/020139-21taskset21:49:110
8225050,0irq/24-0000:00:2241-21latency_hist22:39:472
80449954,0cyclictest615-21awk20:35:052
80449954,0cyclictest3110-21irqbalance20:16:222
80439950,0cyclictest6269-21diskmemload23:16:171
80439950,0cyclictest0-21swapper/123:42:361
80439950,0cyclictest0-21swapper/121:38:411
80419955,0cyclictest0-21swapper/022:12:580
80419955,0cyclictest0-21swapper/021:10:130
80419955,0cyclictest0-21swapper/021:10:130
80419955,0cyclictest0-21swapper/021:07:510
80419955,0cyclictest0-21swapper/020:58:430
339950,0migration/39513-21sh22:42:443
339950,0migration/34128-21aten_r1power_fr22:00:033
339950,0migration/33606-21kthreadcore20:35:073
339950,0migration/326683-21kthreadcore19:15:073
339950,0migration/326364-21munin-node20:30:053
339950,0migration/325475-21sh23:15:013
339950,0migration/323286-21turbostat19:39:483
339950,0migration/322765-21kthreadcore20:25:053
339950,0migration/322728-21munin-node21:30:073
339950,0migration/321581-21kthreadcore20:00:063
339950,0migration/321248-21kthreadcore22:50:063
339950,0migration/32083-21taskset22:39:173
339950,0migration/319815-21missed_timers20:45:103
339950,0migration/31168-21kthreadcore19:45:063
339950,0migration/311282-21munin-node21:20:093
339950,0migration/311032-21kthreadcore18:40:093
339950,0migration/310765-21latency20:15:063
339950,0migration/310608-21kthreadcore19:05:063
28415050,0irq/27-eth0-tx-32083-21munin-node00:00:052
28415050,0irq/27-eth0-tx-27808-21munin-node22:15:012
28415050,0irq/27-eth0-tx-18252-21ssh22:05:332
28415050,0irq/27-eth0-tx-0-21swapper/223:16:562
28415050,0irq/27-eth0-tx-0-21swapper/223:07:252
28415050,0irq/27-eth0-tx-0-21swapper/223:00:452
28415050,0irq/27-eth0-tx-0-21swapper/221:45:132
28415050,0irq/27-eth0-tx-0-21swapper/221:41:092
28415050,0irq/27-eth0-tx-0-21swapper/221:05:302
28415050,0irq/27-eth0-tx-0-21swapper/220:46:242
28415050,0irq/27-eth0-tx-0-21swapper/220:42:512
28405050,0irq/26-eth0-rx-9143-21sh23:04:003
28405050,0irq/26-eth0-rx-8381-21munin-node23:45:013
28405050,0irq/26-eth0-rx-5315-21ntpdc00:00:123
28405050,0irq/26-eth0-rx-3224-21ssh21:58:173
28405050,0irq/26-eth0-rx-29168-21sensors19:40:113
28405050,0irq/26-eth0-rx-24286-21cut23:54:593
28405050,0irq/26-eth0-rx-23341-21ssh23:52:563
28405050,0irq/26-eth0-rx-22979-21gltestperf21:10:033
28405050,0irq/26-eth0-rx-22979-21gltestperf21:10:033
28405050,0irq/26-eth0-rx-14839-21kthreadcore21:25:093
28405050,0irq/26-eth0-rx-14013-21kthreadcore22:25:073
28405050,0irq/26-eth0-rx-10698-21tune2fs19:30:043
28405050,0irq/26-eth0-rx-10464-21kthreadcore23:25:063
28405050,0irq/26-eth0-rx-0-21swapper/323:23:113
28405050,0irq/26-eth0-rx-0-21swapper/322:22:443
28405050,0irq/26-eth0-rx-0-21swapper/322:05:143
28405050,0irq/26-eth0-rx-0-21swapper/321:50:063
28405050,0irq/26-eth0-rx-0-21swapper/321:19:183
28405050,0irq/26-eth0-rx-0-21swapper/320:55:223
28405050,0irq/26-eth0-rx-0-21swapper/320:53:163
28405050,0irq/26-eth0-rx-0-21swapper/300:05:333
259950,0migration/227075-21sh21:51:162
259950,0migration/225014-21kthreadcore19:15:072
259950,0migration/224371-21kthreadcore21:10:072
259950,0migration/224371-21kthreadcore21:10:072
259950,0migration/223914-21taskset19:40:022
259950,0migration/220713-21idleruntime22:10:022
259950,0migration/216315-21switchtime22:45:132
249950,0watchdog/29150-21if_eth018:40:072
249950,0watchdog/219783-21ssh21:25:382
249950,0watchdog/211780-21kthreadcore19:05:072
249950,0watchdog/20-21swapper/220:50:072
179950,0migration/16339-21kthreadcore22:00:091
179950,0migration/14686-21munin-node22:20:061
179950,0migration/132087-21kthreadcore21:55:061
179950,0migration/125557-21sh22:30:471
179950,0migration/124108-21rt-features22:50:101
179950,0migration/118965-21kthreadcore20:45:091
179950,0migration/116409-21kthreadcore21:25:091
179950,0migration/110870-21taskset19:53:571
169950,0watchdog/11792-21telnet23:00:021
169950,0watchdog/10-21swapper/123:51:181
169950,0watchdog/10-21swapper/121:15:061
169950,0watchdog/10-21swapper/120:59:111
169950,0watchdog/10-21swapper/100:00:101
149950,0migration/09352-21sh23:04:410
149950,0migration/09073-21kthreadcore21:00:060
149950,0migration/0851-21sh22:35:290
149950,0migration/08170-21latency_hist18:39:480
149950,0migration/07466-21sh23:20:560
149950,0migration/07244-21kthreadcore22:40:080
149950,0migration/06132-21irqrtprio21:40:060
149950,0migration/05893-21cpuspeed_turbos21:20:010
149950,0migration/05239-21latency_hist20:14:480
149950,0migration/04651-21fschecks_time22:00:050
149950,0migration/032498-21kthreadcore18:55:060
149950,0migration/031694-21kthreadcore21:35:080
149950,0migration/031179-21taskset23:35:510
149950,0migration/029691-21irqrtprio21:55:050
149950,0migration/025574-21kthreadcore19:40:060
149950,0migration/023745-21irqrtprio19:15:060
149950,0migration/023085-21kthreadcore21:50:050
149950,0migration/022078-21kthreadcore20:00:060
149950,0migration/01938-21kthreadcore19:20:070
149950,0migration/018931-21aten_r1power_fr20:24:590
149950,0migration/017081-21munin-node22:25:120
149950,0migration/016725-21munin-node19:10:060
149950,0migration/015575-21timerandwakeup23:05:120
149950,0migration/015366-21kthreadcore21:25:090
149950,0migration/014635-21kthreadcore22:05:060
149950,0migration/012488-21timerwakeupswit20:40:140
149950,0migration/012004-21kthreadcore19:30:080
149950,0migration/010381-21sh22:22:480
80449943,0cyclictest6445-21snmpd19:35:462
80449943,0cyclictest3110-21irqbalance20:34:122
80449943,0cyclictest3110-21irqbalance20:25:322
80439944,0cyclictest31306-21kthreadcore18:55:051
80439944,0cyclictest0-21swapper/123:44:581
80439944,0cyclictest0-21swapper/122:11:001
80439943,0cyclictest6445-21snmpd22:26:181
80439943,0cyclictest6445-21snmpd19:04:461
80439943,0cyclictest640-21awk20:35:051
80439943,0cyclictest5887-21kworker/1:120:42:421
80439943,0cyclictest4459-21awk19:25:051
80439943,0cyclictest3110-21irqbalance22:15:321
80439943,0cyclictest3110-21irqbalance21:20:221
80439943,0cyclictest10654-21awk23:05:051
80419944,0cyclictest0-21swapper/022:53:100
80419944,0cyclictest0-21swapper/022:31:250
80419944,0cyclictest0-21swapper/019:09:460
80419943,0cyclictest664-21awk20:35:050
80419943,0cyclictest6445-21snmpd23:30:180
80419943,0cyclictest6445-21snmpd23:13:030
80419943,0cyclictest6445-21snmpd20:46:470
80419943,0cyclictest6346-21awk20:15:030
80419943,0cyclictest3391-21awk19:00:050
80419943,0cyclictest3110-21irqbalance22:15:220
80419943,0cyclictest27167-21awk22:55:040
80419943,0cyclictest26360-21awk23:15:080
80419943,0cyclictest23538-21awk20:50:080
80419943,0cyclictest17747-21awk19:35:060
80419943,0cyclictest15842-21awk18:45:040
339940,0migration/34703-21cpuspeed_turbos19:50:013
339940,0migration/323369-21diskmemload23:31:073
339940,0migration/321400-21latency_hist18:49:483
339940,0migration/317787-21cpuspeed_turbos23:10:003
28415040,0irq/27-eth0-tx-0-21swapper/223:49:492
28405040,0irq/26-eth0-rx-6492-21munin-node21:00:013
28405040,0irq/26-eth0-rx-27078-21port_sshpriv18:50:083
28405040,0irq/26-eth0-rx-26036-21ps23:35:063
28405040,0irq/26-eth0-rx-0-21swapper/323:05:123
28405040,0irq/26-eth0-rx-0-21swapper/322:56:023
28405040,0irq/26-eth0-rx-0-21swapper/322:14:543
28405040,0irq/26-eth0-rx-0-21swapper/322:10:033
28405040,0irq/26-eth0-rx-0-21swapper/321:45:023
28405040,0irq/26-eth0-rx-0-21swapper/321:35:093
28405040,0irq/26-eth0-rx-0-21swapper/321:05:063
28405040,0irq/26-eth0-rx-0-21swapper/320:40:043
28405040,0irq/26-eth0-rx-0-21swapper/320:20:003
28405040,0irq/26-eth0-rx-0-21swapper/319:28:223
28405040,0irq/26-eth0-rx-0-21swapper/319:22:563
259940,0migration/23892-21kthreadcore19:00:072
259940,0migration/227650-21fschecks_count22:35:072
259940,0migration/212247-21kthreadcore19:30:082
249940,0watchdog/29676-21cut19:50:072
249940,0watchdog/24333-21proc_pri20:10:112
249940,0watchdog/23461-21sensors19:45:112
249940,0watchdog/220663-21kthreadcore19:10:112
249940,0watchdog/216309-21tr20:20:072
249940,0watchdog/214936-21munin-node18:45:072
249940,0watchdog/212570-21kthreadcore19:55:072
249940,0watchdog/20-21swapper/220:05:032
249940,0watchdog/20-21swapper/219:20:072
179940,0migration/16041-21fschecks_time20:15:021
179940,0migration/13942-21taskset19:46:381
179940,0migration/13005-21kthreadcore20:10:071
179940,0migration/127792-21kthreadcore20:30:071
179940,0migration/126045-21kthreadcore18:50:041
179940,0migration/125025-21gltestperf23:55:021
179940,0migration/123310-21gltestperf19:15:031
179940,0migration/121935-21taskset19:10:231
179940,0migration/117672-21aten_r1power_po23:10:001
179940,0migration/114009-21kthreadcore22:05:061
179940,0migration/111099-21kthreadcore00:05:081
169940,0watchdog/16269-21diskmemload22:34:581
169940,0watchdog/16242-21ssh20:59:591
169940,0watchdog/132137-21grep19:20:071
169940,0watchdog/126852-21perl22:55:021
169940,0watchdog/126533-21kthreadcore19:40:071
169940,0watchdog/123347-21sed20:50:071
169940,0watchdog/116522-21taskset19:33:231
169940,0watchdog/10-21swapper/123:29:501
169940,0watchdog/10-21swapper/123:25:061
169940,0watchdog/10-21swapper/123:21:541
169940,0watchdog/10-21swapper/122:47:101
169940,0watchdog/10-21swapper/121:46:061
169940,0watchdog/10-21swapper/121:30:541
169940,0watchdog/10-21swapper/121:10:061
169940,0watchdog/10-21swapper/121:10:061
169940,0watchdog/10-21swapper/121:05:101
169940,0watchdog/10-21swapper/119:55:471
169940,0watchdog/10-21swapper/119:34:511
149940,0migration/09202-21sendmail_mailqu19:25:100
149940,0migration/08872-21fschecks_count23:45:030
149940,0migration/027226-21kthreadcore20:05:070
149940,0migration/026727-21kthreadcore23:55:050
149940,0migration/025387-21kthreadcore21:30:080
149940,0migration/020068-21kthreadcore23:50:100
80449932,0cyclictest4483-21awk19:25:052
80439933,0cyclictest5887-21kworker/1:122:39:481
80439933,0cyclictest5887-21kworker/1:121:49:541
80439933,0cyclictest5887-21kworker/1:120:24:521
80439933,0cyclictest5887-21kworker/1:120:19:501
80439933,0cyclictest5887-21kworker/1:119:59:521
80439933,0cyclictest5887-21kworker/1:119:04:501
80439933,0cyclictest5887-21kworker/1:118:44:521
80419933,0cyclictest0-21swapper/021:14:570
80419932,0cyclictest9965-21awk23:25:050
80419932,0cyclictest7899-21entropy00:05:040
80419932,0cyclictest3453-21sensors19:45:100
80419932,0cyclictest11200-21basename19:55:010
80419930,0cyclictest25482-21munin-node20:30:020
80419930,0cyclictest22199-21perl18:50:000
28405030,0irq/26-eth0-rx-0-21swapper/319:55:083
28405030,0irq/26-eth0-rx-0-21swapper/319:11:023
28405030,0irq/26-eth0-rx-0-21swapper/319:00:023
28405030,0irq/26-eth0-rx-0-21swapper/318:56:323
249930,0watchdog/20-21swapper/218:55:032
169930,0watchdog/10-21swapper/120:04:511
169930,0watchdog/10-21swapper/118:39:511
149930,0migration/010831-21runrttasks19:53:340
80449920,0cyclictest0-21swapper/219:59:482
80419920,0cyclictest0-21swapper/022:44:530
28405020,0irq/26-eth0-rx-0-21swapper/320:09:573
28405020,0irq/26-eth0-rx-0-21swapper/320:04:583
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional