You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 20:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Thu Nov 20, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350710,0irq/25-eth10-21swapper/306:37:433
253824432,0sleep20-21swapper/206:35:422
278523422,0sleep00-21swapper/006:38:320
2891122723,0sleep10-21swapper/106:34:491
29559976,0cyclictest23966-21tr09:24:580
35160,0ktimersoftd/30-21swapper/311:44:213
339960,0migration/32566-21runrttasks09:04:593
339960,0migration/325604-21sh11:32:213
339960,0migration/319667-21kthreadcore06:49:573
29579960,0cyclictest0-21swapper/208:59:572
29569960,0cyclictest0-21swapper/110:28:501
29559960,0cyclictest21993-21diskmemload10:04:560
28415060,0irq/27-eth0-tx-0-21swapper/211:52:442
28415060,0irq/27-eth0-tx-0-21swapper/211:29:342
28415060,0irq/27-eth0-tx-0-21swapper/210:22:522
28415060,0irq/27-eth0-tx-0-21swapper/007:24:560
28405060,0irq/26-eth0-rx-10496-21ssh10:39:273
169960,0watchdog/110595-21ssh10:16:551
169960,0watchdog/10-21swapper/107:26:151
169960,0watchdog/10-21swapper/107:15:351
149960,0migration/011574-21timerwakeupswit08:50:030
8225050,0irq/24-0000:00:30743-21munin-run07:24:402
8225050,0irq/24-0000:00:18436-21latency_hist07:39:402
339950,0migration/39794-21latency_hist10:59:403
339950,0migration/34249-21kthreadcore07:49:593
339950,0migration/327625-21kthreadcore09:24:593
339950,0migration/326827-21taskset11:34:513
339950,0migration/32575-21missed_timers09:49:593
339950,0migration/324890-21kthreadcore09:44:583
339950,0migration/323553-21munin-node08:09:553
339950,0migration/323285-21kthreadcore08:59:583
339950,0migration/322232-21taskset08:39:483
339950,0migration/321521-21taskset09:22:063
339950,0migration/320279-21kthreadcore07:14:593
339950,0migration/319601-21munin-node07:39:553
339950,0migration/319055-21gltestperf11:49:533
339950,0migration/317021-21kthreadcore09:59:593
339950,0migration/316334-21munin-node08:34:523
339950,0migration/314797-21munin-node09:39:593
339950,0migration/313568-21kthreadcore08:29:573
339950,0migration/312997-21munin-node10:19:583
339950,0migration/311493-21munin-node08:50:033
29579955,0cyclictest0-21swapper/210:24:562
29579950,0cyclictest0-21swapper/211:46:172
29579950,0cyclictest0-21swapper/210:45:062
29579950,0cyclictest0-21swapper/210:17:592
29579950,0cyclictest0-21swapper/209:53:332
29579950,0cyclictest0-21swapper/209:38:062
29579950,0cyclictest0-21swapper/209:05:092
29579950,0cyclictest0-21swapper/208:48:552
29569955,0cyclictest0-21swapper/111:01:191
29569955,0cyclictest0-21swapper/110:51:231
29569955,0cyclictest0-21swapper/110:35:471
29569955,0cyclictest0-21swapper/109:19:381
29569954,0cyclictest25572-21kworker/1:208:39:581
29569950,0cyclictest0-21swapper/107:54:581
29559955,0cyclictest21993-21diskmemload11:45:310
29559955,0cyclictest0-21swapper/011:12:060
29559955,0cyclictest0-21swapper/009:49:290
29559950,0cyclictest0-21swapper/011:37:190
29559950,0cyclictest0-21swapper/010:45:180
29559950,0cyclictest0-21swapper/010:35:410
29559950,0cyclictest0-21swapper/010:25:070
29559950,0cyclictest0-21swapper/009:10:320
28415050,0irq/27-eth0-tx-8906-21ssh10:57:112
28415050,0irq/27-eth0-tx-15583-21aten2.4-expect08:34:490
28415050,0irq/27-eth0-tx-13827-21munin-node08:05:010
28415050,0irq/27-eth0-tx-0-21swapper/212:05:522
28415050,0irq/27-eth0-tx-0-21swapper/211:40:042
28415050,0irq/27-eth0-tx-0-21swapper/211:20:542
28415050,0irq/27-eth0-tx-0-21swapper/210:59:592
28415050,0irq/27-eth0-tx-0-21swapper/210:52:372
28415050,0irq/27-eth0-tx-0-21swapper/210:38:242
28415050,0irq/27-eth0-tx-0-21swapper/210:11:392
28415050,0irq/27-eth0-tx-0-21swapper/210:05:522
28415050,0irq/27-eth0-tx-0-21swapper/210:02:032
28415050,0irq/27-eth0-tx-0-21swapper/209:49:112
28415050,0irq/27-eth0-tx-0-21swapper/209:41:032
28415050,0irq/27-eth0-tx-0-21swapper/209:21:232
28405050,0irq/26-eth0-rx-5349-21sed07:04:563
28405050,0irq/26-eth0-rx-30003-21cut11:15:013
28405050,0irq/26-eth0-rx-29845-21kthreadcore10:09:583
28405050,0irq/26-eth0-rx-0-21swapper/311:59:543
28405050,0irq/26-eth0-rx-0-21swapper/311:58:113
28405050,0irq/26-eth0-rx-0-21swapper/311:48:233
28405050,0irq/26-eth0-rx-0-21swapper/311:25:003
28405050,0irq/26-eth0-rx-0-21swapper/311:06:143
28405050,0irq/26-eth0-rx-0-21swapper/310:49:253
28405050,0irq/26-eth0-rx-0-21swapper/310:29:453
28405050,0irq/26-eth0-rx-0-21swapper/310:29:203
28405050,0irq/26-eth0-rx-0-21swapper/310:14:573
28405050,0irq/26-eth0-rx-0-21swapper/310:07:533
28405050,0irq/26-eth0-rx-0-21swapper/309:55:303
28405050,0irq/26-eth0-rx-0-21swapper/309:29:573
28405050,0irq/26-eth0-rx-0-21swapper/309:16:243
28405050,0irq/26-eth0-rx-0-21swapper/309:10:013
28405050,0irq/26-eth0-rx-0-21swapper/308:46:023
27150,0ktimersoftd/20-21swapper/209:11:372
259950,0migration/2913-21sh11:36:302
259950,0migration/2315-21hwlatdetect07:49:562
259950,0migration/229530-21kthreadcore11:15:012
259950,0migration/228277-21if_eth110:29:582
259950,0migration/224984-21missed_timers11:09:582
259950,0migration/213176-21kthreadcore10:39:562
19150,0ktimersoftd/10-21swapper/111:44:211
19150,0ktimersoftd/10-21swapper/110:08:211
179950,0migration/17559-21kthreadcore09:34:561
179950,0migration/14819-21fschecks_time07:04:541
179950,0migration/131572-21irqcore09:29:561
179950,0migration/128214-21munin-node10:29:581
179950,0migration/122938-21taskset06:53:551
179950,0migration/121534-21irqcore08:09:541
179950,0migration/118375-21taskset10:21:371
179950,0migration/111665-21idleruntime10:39:531
169950,0watchdog/14095-21ssh09:52:311
169950,0watchdog/115758-21cat11:44:591
169950,0watchdog/11557-21ssh11:38:231
169950,0watchdog/112157-21kthreadcore06:44:591
169950,0watchdog/111158-21ssh09:10:071
149950,0migration/09376-21sh10:58:370
149950,0migration/03989-21idleruntime-cro07:04:400
149950,0migration/0300-21kthreadcore09:29:570
149950,0migration/02801-21latency_hist11:19:400
149950,0migration/027578-21cpuspeed_turbos10:29:550
149950,0migration/021382-21aten_r1power_po08:59:520
149950,0migration/019794-21taskset09:17:340
149950,0migration/018830-21aten_r1power_cu11:29:510
149950,0migration/01692-21kthreadcore09:04:560
149950,0migration/015112-21kthreadcore11:00:010
149950,0migration/013687-21kthreadcore06:45:000
149950,0migration/013667-21kthreadcore08:29:570
339940,0migration/3425-21kthreadcore07:24:553
339940,0migration/327575-21kthreadcore06:54:583
339940,0migration/326910-21kthreadcore07:19:573
339940,0migration/311624-21taskset07:32:553
339940,0migration/311009-21aten2.4-expect10:39:503
29579944,0cyclictest0-21swapper/209:34:292
29579944,0cyclictest0-21swapper/208:54:452
29579943,0cyclictest8129-21awk07:59:562
29579943,0cyclictest6445-21snmpd08:24:042
29579943,0cyclictest5932-21awk09:54:552
29579943,0cyclictest5101-21awk07:04:552
29579943,0cyclictest2903-21awk11:59:542
29579943,0cyclictest27290-21awk11:54:532
29579943,0cyclictest19805-21awk11:29:562
29579943,0cyclictest18435-21sadc07:39:402
29579943,0cyclictest16329-21sadc06:49:402
29569944,0cyclictest14544-21sensors08:30:011
29569943,0cyclictest6445-21snmpd06:40:031
29569943,0cyclictest6261-21awk07:29:561
29569943,0cyclictest6015-21awk09:54:561
29569943,0cyclictest3425-21lldpd09:48:171
29569943,0cyclictest3425-21lldpd08:27:451
29569943,0cyclictest3425-21lldpd07:19:541
29569943,0cyclictest25572-21kworker/1:208:49:441
29569943,0cyclictest25572-21kworker/1:208:34:561
29569943,0cyclictest12851-21awk07:34:541
29569943,0cyclictest12724-21ps08:00:001
29559944,0cyclictest0-21swapper/011:04:510
29559943,0cyclictest6445-21snmpd07:29:490
29559943,0cyclictest4173-21awk06:39:570
29559943,0cyclictest3425-21lldpd09:23:390
29559943,0cyclictest3425-21lldpd08:44:250
29559943,0cyclictest3178-21awk11:39:550
29559943,0cyclictest31209-21ps08:44:580
29559943,0cyclictest3110-21irqbalance12:07:540
29559943,0cyclictest3110-21irqbalance11:59:440
29559943,0cyclictest3110-21irqbalance11:54:540
29559943,0cyclictest3110-21irqbalance11:50:040
29559943,0cyclictest3110-21irqbalance11:24:440
29559943,0cyclictest3110-21irqbalance11:14:440
29559943,0cyclictest3110-21irqbalance10:00:150
29559943,0cyclictest3110-21irqbalance09:57:550
29559943,0cyclictest3110-21irqbalance07:59:450
29559943,0cyclictest3110-21irqbalance07:55:450
29559943,0cyclictest3110-21irqbalance07:49:550
29559943,0cyclictest3110-21irqbalance07:45:150
29559943,0cyclictest3110-21irqbalance07:35:050
29559943,0cyclictest3110-21irqbalance07:14:550
29559943,0cyclictest3110-21irqbalance06:56:250
29559943,0cyclictest21385-21awk08:09:530
28415040,0irq/27-eth0-tx-6372-21iceccd11:04:442
28415040,0irq/27-eth0-tx-0-21swapper/008:19:530
28415040,0irq/27-eth0-tx-0-21swapper/007:39:520
28415040,0irq/27-eth0-tx-0-21swapper/007:04:510
28405040,0irq/26-eth0-rx-27814-21kernelversion10:49:583
28405040,0irq/26-eth0-rx-0-21swapper/312:05:313
28405040,0irq/26-eth0-rx-0-21swapper/311:19:423
28405040,0irq/26-eth0-rx-0-21swapper/311:12:413
28405040,0irq/26-eth0-rx-0-21swapper/309:34:423
28405040,0irq/26-eth0-rx-0-21swapper/308:54:523
259940,0migration/228368-21rt-features09:25:022
259940,0migration/223002-21gltestperf08:39:552
259940,0migration/217121-21fschecks_time06:49:532
249940,0watchdog/221993-21diskmemload09:17:152
249940,0watchdog/220220-21kthreadcore07:14:592
249940,0watchdog/20-21swapper/208:04:592
249940,0watchdog/20-21swapper/207:24:552
249940,0watchdog/20-21swapper/206:39:592
179940,0migration/132100-21aten_r1power_po07:49:541
179940,0migration/128294-21aten_r1power_cu10:09:521
179940,0migration/125145-21kthreadcore06:54:571
179940,0migration/122888-21cpuspeed_turbos09:24:541
179940,0migration/119668-21if_eth1.210:44:561
179940,0migration/118808-21munin-node08:54:591
179940,0migration/118555-21runrttasks11:07:241
179940,0migration/116363-21kthreadcore08:05:001
179940,0migration/114605-21sh09:39:571
179940,0migration/113761-21if_err_eth1.309:59:561
169940,0watchdog/18387-21ssh12:00:151
169940,0watchdog/131831-21kthreadcore08:44:591
169940,0watchdog/128600-21kthreadcore11:54:551
169940,0watchdog/126549-21cut08:59:591
169940,0watchdog/12282-21ssh11:18:431
169940,0watchdog/116055-21python12:05:031
169940,0watchdog/10-21swapper/111:53:431
169940,0watchdog/10-21swapper/111:34:071
169940,0watchdog/10-21swapper/111:25:111
169940,0watchdog/10-21swapper/111:20:231
169940,0watchdog/10-21swapper/111:12:551
169940,0watchdog/10-21swapper/110:54:591
169940,0watchdog/10-21swapper/109:24:111
169940,0watchdog/10-21swapper/107:40:031
149940,0migration/09872-21kthreadcore09:34:570
149940,0migration/08202-21aten_r1power_fr08:24:560
149940,0migration/032094-21kthreadcore08:14:570
149940,0migration/029412-21kthreadcore07:19:580
149940,0migration/027887-21kthreadcore10:49:580
149940,0migration/017617-21timerandwakeup10:20:050
149940,0migration/016614-21sensors_temp07:10:020
149940,0migration/015128-21kthreadcore10:39:560
149940,0migration/015023-21kthreadcore08:54:570
29579933,0cyclictest3425-21lldpd08:09:542
29579933,0cyclictest3425-21lldpd06:54:532
29579932,0cyclictest3425-21lldpd07:57:132
29579932,0cyclictest16616-21sensors07:10:022
29579932,0cyclictest16426-21awk08:34:532
29579932,0cyclictest11008-21sensors07:30:022
29579930,0cyclictest0-21swapper/208:32:122
29569933,0cyclictest25572-21kworker/1:208:19:421
29569933,0cyclictest25572-21kworker/1:208:15:221
29569933,0cyclictest25572-21kworker/1:206:59:501
29559932,0cyclictest6445-21snmpd09:53:000
29559932,0cyclictest29173-21awk10:09:560
29559930,0cyclictest14224-21df_inode09:39:560
28415030,0irq/27-eth0-tx-0-21swapper/208:49:502
28415030,0irq/27-eth0-tx-0-21swapper/006:52:270
28405030,0irq/26-eth0-rx-0-21swapper/311:00:123
28405030,0irq/26-eth0-rx-0-21swapper/308:25:533
28405030,0irq/26-eth0-rx-0-21swapper/308:20:183
28405030,0irq/26-eth0-rx-0-21swapper/308:14:513
28405030,0irq/26-eth0-rx-0-21swapper/308:05:023
28405030,0irq/26-eth0-rx-0-21swapper/308:02:093
28405030,0irq/26-eth0-rx-0-21swapper/307:55:133
28405030,0irq/26-eth0-rx-0-21swapper/307:45:043
28405030,0irq/26-eth0-rx-0-21swapper/307:34:593
28405030,0irq/26-eth0-rx-0-21swapper/307:00:033
28405030,0irq/26-eth0-rx-0-21swapper/306:46:373
28405030,0irq/26-eth0-rx-0-21swapper/306:39:483
259930,0migration/28385-21runrttasks08:26:512
259930,0migration/229204-21kthreadcore08:14:562
249930,0watchdog/20-21swapper/207:44:552
249930,0watchdog/20-21swapper/207:00:032
169930,0watchdog/126527-21ps07:44:591
169930,0watchdog/10-21swapper/109:04:431
169930,0watchdog/10-21swapper/107:09:551
29559921,0cyclictest0-21swapper/010:14:500
28405020,0irq/26-eth0-rx-0-21swapper/307:10:073
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional