You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-26 - 11:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Fri Dec 26, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304450710,0irq/25-eth10-21swapper/319:07:013
2341124332,0sleep10-21swapper/119:07:401
2317923624,0sleep00-21swapper/019:04:550
2344323324,0sleep20-21swapper/219:08:012
149970,0migration/024388-21sh21:46:120
339960,0migration/37990-21sh23:50:153
339960,0migration/332425-21aten_r1power_cu20:03:583
339960,0migration/330862-21pmu-power23:44:083
339960,0migration/327499-21runrttasks22:27:153
29725060,0irq/27-eth0-tx-23122-21cut23:59:042
29725060,0irq/27-eth0-tx-0-21swapper/222:27:252
29715060,0irq/26-eth0-rx-8172-21if_eth1.323:14:023
29715060,0irq/26-eth0-rx-24923-21munin-node22:04:143
235849966,0cyclictest21919-21diskmemload21:41:411
235849965,0cyclictest26074-21kthreadcore21:29:051
235839965,0cyclictest9815-21kthreadcore22:34:050
235839963,0cyclictest3960-21kworker/0:223:55:400
235839963,0cyclictest3960-21kworker/0:223:50:040
235839963,0cyclictest3960-21kworker/0:223:41:540
235839963,0cyclictest3960-21kworker/0:223:32:380
235839963,0cyclictest3960-21kworker/0:223:25:180
235839963,0cyclictest3960-21kworker/0:223:20:480
235839963,0cyclictest3960-21kworker/0:223:12:200
235839963,0cyclictest3960-21kworker/0:223:05:200
235839963,0cyclictest3960-21kworker/0:223:00:320
235839963,0cyclictest3960-21kworker/0:222:58:200
235839963,0cyclictest3960-21kworker/0:222:49:180
235839963,0cyclictest3960-21kworker/0:222:47:320
235839963,0cyclictest3960-21kworker/0:222:24:240
235839963,0cyclictest3960-21kworker/0:222:09:050
235839963,0cyclictest3960-21kworker/0:222:06:030
235839963,0cyclictest3960-21kworker/0:221:54:510
235839963,0cyclictest3960-21kworker/0:221:38:130
235839963,0cyclictest3960-21kworker/0:221:27:230
235839963,0cyclictest3960-21kworker/0:221:20:110
235839963,0cyclictest3960-21kworker/0:200:35:520
235839963,0cyclictest3960-21kworker/0:200:32:240
235839963,0cyclictest3960-21kworker/0:200:25:020
235839963,0cyclictest3960-21kworker/0:200:19:020
235839963,0cyclictest3960-21kworker/0:200:04:200
235839963,0cyclictest3960-21kworker/0:200:00:500
235839963,0cyclictest14961-21kworker/0:021:10:510
235839963,0cyclictest14961-21kworker/0:020:59:030
235839963,0cyclictest14961-21kworker/0:020:49:010
235839963,0cyclictest14961-21kworker/0:020:34:110
235839963,0cyclictest14961-21kworker/0:020:29:010
235839963,0cyclictest14961-21kworker/0:019:34:110
235839963,0cyclictest14961-21kworker/0:019:34:110
235839963,0cyclictest14961-21kworker/0:019:24:010
149960,0migration/016528-21missed_timers00:14:060
149960,0migration/016508-21kthreadcore20:14:080
149960,0migration/016502-21sh21:42:240
149960,0migration/011330-21latency_hist19:23:480
8215050,0irq/24-0000:00:11218-21munin-run19:23:482
339950,0migration/39059-21kthreadcore19:44:043
339950,0migration/35744-21irqrtprio22:34:043
339950,0migration/34009-21munin-node00:09:053
339950,0migration/330301-21sshd19:34:093
339950,0migration/330301-21sshd19:34:093
339950,0migration/330032-21kthreadcore20:49:053
339950,0migration/329887-21timerandwakeup23:24:113
339950,0migration/329368-21sh21:12:063
339950,0migration/327915-21munin-node20:24:043
339950,0migration/326299-21kthreadcore23:04:053
339950,0migration/324887-21timerwakeupswit19:54:093
339950,0migration/324430-21cpuspeed_turbos21:29:003
339950,0migration/313804-21kthreadcore21:59:053
339950,0migration/312932-21aten_r1power_cu22:38:573
339950,0migration/31279-21kthreadcore23:09:063
339950,0migration/312528-21munin-node22:19:033
339950,0migration/312237-21sh22:54:433
339950,0migration/312189-21kthreadcore21:39:053
339950,0migration/311508-21kthreadcore23:34:043
339950,0migration/310307-21kthreadcore20:09:063
29725050,0irq/27-eth0-tx-5702-21munin-node19:44:042
29725050,0irq/27-eth0-tx-31184-21ssh21:31:072
29725050,0irq/27-eth0-tx-1854-21swap22:29:102
29725050,0irq/27-eth0-tx-0-21swapper/223:56:042
29725050,0irq/27-eth0-tx-0-21swapper/223:25:472
29725050,0irq/27-eth0-tx-0-21swapper/223:23:462
29725050,0irq/27-eth0-tx-0-21swapper/223:18:152
29725050,0irq/27-eth0-tx-0-21swapper/222:04:422
29715050,0irq/26-eth0-rx-8080-21unixbench_singl20:54:133
29715050,0irq/26-eth0-rx-6830-21cut21:18:473
29715050,0irq/26-eth0-rx-6119-21cut21:54:053
29715050,0irq/26-eth0-rx-4960-21if_eth1.300:29:023
29715050,0irq/26-eth0-rx-4735-21grep23:29:073
29715050,0irq/26-eth0-rx-3514-21snmpd00:13:593
29715050,0irq/26-eth0-rx-30586-21munin-node22:49:043
29715050,0irq/26-eth0-rx-22220-21grep23:59:043
29715050,0irq/26-eth0-rx-18375-21if_eth1.321:44:033
29715050,0irq/26-eth0-rx-18218-21kthreadcore23:39:043
29715050,0irq/26-eth0-rx-17258-21kthreadcore22:59:063
29715050,0irq/26-eth0-rx-15503-21grep00:34:063
29715050,0irq/26-eth0-rx-0-21swapper/323:56:233
29715050,0irq/26-eth0-rx-0-21swapper/323:19:283
29715050,0irq/26-eth0-rx-0-21swapper/322:45:033
29715050,0irq/26-eth0-rx-0-21swapper/322:09:053
29715050,0irq/26-eth0-rx-0-21swapper/321:49:003
29715050,0irq/26-eth0-rx-0-21swapper/321:34:003
29715050,0irq/26-eth0-rx-0-21swapper/321:26:433
29715050,0irq/26-eth0-rx-0-21swapper/300:24:023
29715050,0irq/26-eth0-rx-0-21swapper/300:20:293
259950,0migration/26199-21sh21:17:592
259950,0migration/220553-21sh22:42:322
259950,0migration/21994-21sh22:11:452
235859955,0cyclictest0-21swapper/222:59:552
235859955,0cyclictest0-21swapper/222:46:502
235859955,0cyclictest0-21swapper/221:12:332
235859950,0cyclictest0-21swapper/221:57:382
235859950,0cyclictest0-21swapper/200:26:372
235849955,0cyclictest0-21swapper/123:14:241
235849955,0cyclictest0-21swapper/121:26:141
235849955,0cyclictest0-21swapper/119:19:101
235849950,0cyclictest0-21swapper/122:19:411
235849950,0cyclictest0-21swapper/100:37:291
235839955,0cyclictest0-21swapper/020:19:080
235839953,0cyclictest3960-21kworker/0:223:33:500
235839953,0cyclictest3960-21kworker/0:223:13:500
235839953,0cyclictest3960-21kworker/0:222:39:120
235839953,0cyclictest3960-21kworker/0:222:18:510
235839953,0cyclictest3960-21kworker/0:221:29:010
235839953,0cyclictest14961-21kworker/0:020:54:130
235839953,0cyclictest14961-21kworker/0:020:39:010
235839953,0cyclictest14961-21kworker/0:020:09:010
235839953,0cyclictest14961-21kworker/0:019:52:390
235839953,0cyclictest14961-21kworker/0:019:43:530
235839953,0cyclictest14961-21kworker/0:019:18:210
235839952,0cyclictest3960-21kworker/0:223:43:580
235839952,0cyclictest3960-21kworker/0:222:28:540
235839952,0cyclictest3960-21kworker/0:222:13:530
235839952,0cyclictest14961-21kworker/0:021:04:010
235839952,0cyclictest14961-21kworker/0:019:54:090
235839952,0cyclictest14961-21kworker/0:019:29:030
179950,0migration/1718-21munin-node20:04:011
179950,0migration/16112-21munin-node19:44:001
179950,0migration/1509-21kthreadcore19:39:061
179950,0migration/13963-21munin-node21:14:141
179950,0migration/131177-21taskset23:26:441
179950,0migration/12707-21sh22:30:221
179950,0migration/113844-21if_eth022:39:011
169950,0watchdog/132035-21kthreadcore00:24:051
149950,0migration/08430-21munin-node00:09:100
149950,0migration/03921-21munin-node21:14:130
149950,0migration/032549-21aten_r1power_po20:03:590
149950,0migration/028102-21kthreadcore21:49:040
149950,0migration/021922-21hwlatdetect20:44:050
149950,0migration/011768-21kthreadcore21:59:040
8215040,0irq/24-0000:00:8372-21cat23:33:472
8215040,0irq/24-0000:00:20909-21munin-run20:43:472
339940,0migration/3960-21kthreadcore19:14:063
339940,0migration/37305-21kthreadcore19:19:043
339940,0migration/35637-21kthreadcore22:14:053
339940,0migration/327645-21kthreadcore19:59:053
339940,0migration/32424-21kthreadcore20:29:063
339940,0migration/317018-21kthreadcore19:49:043
339940,0migration/310710-21kthreadcore20:59:053
29725040,0irq/27-eth0-tx-8024-21irqcore21:19:042
29725040,0irq/27-eth0-tx-5852-21munin-node22:54:022
29725040,0irq/27-eth0-tx-30445-21munin-node19:14:092
29725040,0irq/27-eth0-tx-21919-21diskmemload21:39:512
29725040,0irq/27-eth0-tx-21015-21ssh21:24:092
29725040,0irq/27-eth0-tx-16847-21ssh23:38:522
29725040,0irq/27-eth0-tx-0-21swapper/223:49:262
29725040,0irq/27-eth0-tx-0-21swapper/223:30:002
29725040,0irq/27-eth0-tx-0-21swapper/223:09:402
29725040,0irq/27-eth0-tx-0-21swapper/223:03:512
29725040,0irq/27-eth0-tx-0-21swapper/222:49:052
29725040,0irq/27-eth0-tx-0-21swapper/222:33:502
29725040,0irq/27-eth0-tx-0-21swapper/222:13:592
29725040,0irq/27-eth0-tx-0-21swapper/221:49:192
29725040,0irq/27-eth0-tx-0-21swapper/221:03:592
29725040,0irq/27-eth0-tx-0-21swapper/219:58:492
29725040,0irq/27-eth0-tx-0-21swapper/200:18:492
29715040,0irq/26-eth0-rx-0-21swapper/322:28:523
29715040,0irq/26-eth0-rx-0-21swapper/321:14:033
29715040,0irq/26-eth0-rx-0-21swapper/320:19:013
29715040,0irq/26-eth0-rx-0-21swapper/319:24:003
29715040,0irq/26-eth0-rx-0-21swapper/319:09:043
29715040,0irq/26-eth0-rx-0-21swapper/300:04:033
259940,0migration/28376-21pmu-power00:09:092
259940,0migration/2691-21munin-node21:34:002
259940,0migration/229421-21kthreadcore23:44:052
259940,0migration/226702-21latency_hist20:23:472
259940,0migration/222125-21kthreadcore20:19:042
259940,0migration/221238-21kthreadcore19:54:032
259940,0migration/217440-21kthreadcore00:34:072
249940,0watchdog/225470-21fschecks_time19:34:022
249940,0watchdog/225470-21fschecks_time19:34:022
249940,0watchdog/21950-21kthreadcore19:39:062
249940,0watchdog/219312-21unixbench_multi20:14:142
249940,0watchdog/218136-21ssh21:44:012
249940,0watchdog/20-21swapper/222:19:052
249940,0watchdog/20-21swapper/220:49:022
249940,0watchdog/20-21swapper/220:09:022
235859943,0cyclictest3556-21lldpd21:59:342
235859943,0cyclictest3556-21lldpd20:47:042
235859943,0cyclictest3556-21lldpd20:07:032
235859943,0cyclictest3556-21lldpd19:51:032
235859943,0cyclictest3556-21lldpd19:32:042
235859943,0cyclictest3556-21lldpd00:29:042
235859943,0cyclictest3556-21lldpd00:14:042
235859943,0cyclictest3556-21lldpd00:04:032
235859943,0cyclictest12305-21awk19:24:032
235859940,0cyclictest8793-21munin-node20:59:022
235849943,0cyclictest9361-21awk23:34:021
235849943,0cyclictest3556-21lldpd22:07:041
235849943,0cyclictest3514-21snmpd23:03:581
235849943,0cyclictest3514-21snmpd22:44:591
235849943,0cyclictest3514-21snmpd20:28:581
235849943,0cyclictest3514-21snmpd20:14:281
235849943,0cyclictest3241-21irqbalance21:19:161
235849943,0cyclictest15276-21awk20:39:021
235839942,0cyclictest14961-21kworker/0:020:23:490
235839942,0cyclictest14961-21kworker/0:019:39:110
235839942,0cyclictest14961-21kworker/0:019:09:010
179940,0migration/19995-21kthreadcore20:34:071
179940,0migration/1926-21munin-node23:29:051
179940,0migration/17833-21kthreadcore22:54:051
179940,0migration/15434-21kthreadcore00:29:031
179940,0migration/131330-21kthreadcore22:49:061
179940,0migration/127562-21latency21:09:081
179940,0migration/126768-21aten_r1power_cu20:23:591
179940,0migration/124917-21aten_r1power_vo19:34:001
179940,0migration/124917-21aten_r1power_vo19:34:001
179940,0migration/122485-21kthreadcore20:19:041
179940,0migration/115764-21kthreadcore22:59:051
179940,0migration/113671-21kernelversion19:49:031
169940,0watchdog/19772-21cut22:34:051
169940,0watchdog/16996-21kthreadcore21:54:051
169940,0watchdog/16913-21tr22:14:051
169940,0watchdog/12986-21df00:09:011
169940,0watchdog/128268-21fschecks_count20:49:021
169940,0watchdog/125334-21kthreadcore22:24:051
169940,0watchdog/125221-21ssh23:43:491
169940,0watchdog/12262-21ssh23:49:051
169940,0watchdog/118588-21ssh00:17:211
169940,0watchdog/11325-21ssh00:06:251
169940,0watchdog/10-21swapper/121:59:051
169940,0watchdog/10-21swapper/121:44:051
169940,0watchdog/10-21swapper/121:36:411
169940,0watchdog/10-21swapper/120:59:061
169940,0watchdog/10-21swapper/119:14:061
149940,0migration/026057-21df19:59:000
8215030,0irq/24-0000:00:0-21swapper/220:28:472
29715030,0irq/26-eth0-rx-0-21swapper/320:44:033
29715030,0irq/26-eth0-rx-0-21swapper/320:38:593
29715030,0irq/26-eth0-rx-0-21swapper/320:34:023
29715030,0irq/26-eth0-rx-0-21swapper/320:14:023
29715030,0irq/26-eth0-rx-0-21swapper/319:28:593
249930,0watchdog/20-21swapper/220:34:022
235859932,0cyclictest2935-21awk20:54:052
235859932,0cyclictest23524-21latency_hist19:08:482
235849933,0cyclictest0-21swapper/119:59:321
235849932,0cyclictest7721-21awk20:09:041
235849932,0cyclictest6097-21kthreadcore20:54:071
235849932,0cyclictest4088-21grep23:09:071
235849932,0cyclictest3514-21snmpd20:43:581
235849932,0cyclictest25768-21munin-node21:49:001
235849932,0cyclictest24702-21awk19:09:021
235849932,0cyclictest19804-21kthreadcore23:59:031
235849932,0cyclictest18836-21gltestperf19:29:011
235849930,0cyclictest12073-21perl19:24:011
235849930,0cyclictest0-21swapper/121:03:511
179930,0migration/121989-21runrttasks23:19:491
169930,0watchdog/121919-21diskmemload23:54:371
169930,0watchdog/10-21swapper/123:39:011
169930,0watchdog/10-21swapper/122:08:491
169930,0watchdog/10-21swapper/100:19:051
29715020,0irq/26-eth0-rx-0-21swapper/321:04:393
29715020,0irq/26-eth0-rx-0-21swapper/319:38:493
169920,0watchdog/10-21swapper/119:53:541
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional