You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 22:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Tue Nov 18, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350980,0irq/25-eth10-21swapper/306:35:543
519724735,0sleep00-21swapper/006:38:550
506624536,0sleep10-21swapper/106:37:241
508023425,0sleep20-21swapper/206:37:332
53389976,0cyclictest12601-21cpuspeed_turbos06:45:010
339960,0migration/37832-21sh10:05:463
339960,0migration/32829-21aten_r1power_po09:44:583
339960,0migration/325411-21missed_timers11:40:043
339960,0migration/313105-21sh10:51:173
28415060,0irq/27-eth0-tx-20850-21munin-node07:40:010
28415060,0irq/27-eth0-tx-19833-21munin-node07:14:590
28415060,0irq/27-eth0-tx-0-21swapper/212:04:272
28415060,0irq/27-eth0-tx-0-21swapper/211:46:292
28415060,0irq/27-eth0-tx-0-21swapper/211:35:202
28415060,0irq/27-eth0-tx-0-21swapper/211:34:382
28415060,0irq/27-eth0-tx-0-21swapper/210:52:532
28415060,0irq/27-eth0-tx-0-21swapper/210:34:442
28415060,0irq/27-eth0-tx-0-21swapper/209:45:432
28415060,0irq/27-eth0-tx-0-21swapper/209:06:122
28415060,0irq/27-eth0-tx-0-21swapper/209:02:042
28415060,0irq/27-eth0-tx-0-21swapper/008:20:070
28405060,0irq/26-eth0-rx-21619-21diskstats11:20:043
28405060,0irq/26-eth0-rx-16644-21wc09:05:113
259960,0migration/231609-21kthreadcore12:05:062
149960,0migration/03872-21sh11:28:540
149960,0migration/023876-21sh10:17:280
149960,0migration/013359-21sh10:52:080
149960,0migration/01295-21sh09:40:360
8225050,0irq/24-0000:00:18696-21munin-run06:49:452
8225050,0irq/24-0000:00:16104-21perf08:24:452
53419955,0cyclictest0-21swapper/208:42:402
53419954,0cyclictest6445-21snmpd07:05:462
53419950,0cyclictest0-21swapper/211:04:152
53419950,0cyclictest0-21swapper/210:15:412
53409953,0cyclictest7648-21kworker/1:109:00:581
53409950,0cyclictest0-21swapper/110:42:281
53389955,0cyclictest0-21swapper/012:05:130
53389955,0cyclictest0-21swapper/011:55:450
53389955,0cyclictest0-21swapper/011:42:020
53389955,0cyclictest0-21swapper/007:30:060
53389950,0cyclictest0-21swapper/011:51:110
53389950,0cyclictest0-21swapper/010:00:020
53389950,0cyclictest0-21swapper/009:27:240
35150,0ktimersoftd/30-21swapper/311:28:193
339950,0migration/3820-21kthreadcore11:05:033
339950,0migration/36840-21runrttasks10:26:423
339950,0migration/35177-21missed_timers07:00:033
339950,0migration/34882-21kthreadcore08:15:033
339950,0migration/34297-21missed_timers10:45:043
339950,0migration/33542-21latency_hist11:49:443
339950,0migration/332575-21taskset07:20:503
339950,0migration/32277-21kthreadcore07:25:053
339950,0migration/322377-21kthreadcore09:55:033
339950,0migration/317067-21kthreadcore09:30:043
339950,0migration/316287-21kthreadcore11:15:053
339950,0migration/314661-21taskset07:57:013
339950,0migration/314356-21aten_r1power_vo07:35:013
339950,0migration/313406-21kthreadcore09:50:023
339950,0migration/312044-21taskset06:44:443
339950,0migration/311931-21aten_r1power_cu08:44:573
339950,0migration/31112-21munin-node12:05:093
28415050,0irq/27-eth0-tx-31730-21ssh10:22:562
28415050,0irq/27-eth0-tx-18914-21ssh11:57:512
28415050,0irq/27-eth0-tx-13183-21irqrtprio08:45:032
28415050,0irq/27-eth0-tx-0-21swapper/211:51:112
28415050,0irq/27-eth0-tx-0-21swapper/211:25:482
28415050,0irq/27-eth0-tx-0-21swapper/211:22:552
28415050,0irq/27-eth0-tx-0-21swapper/211:16:342
28415050,0irq/27-eth0-tx-0-21swapper/211:09:512
28415050,0irq/27-eth0-tx-0-21swapper/211:05:082
28415050,0irq/27-eth0-tx-0-21swapper/210:58:012
28415050,0irq/27-eth0-tx-0-21swapper/210:45:182
28415050,0irq/27-eth0-tx-0-21swapper/210:36:142
28415050,0irq/27-eth0-tx-0-21swapper/210:28:412
28415050,0irq/27-eth0-tx-0-21swapper/210:05:492
28415050,0irq/27-eth0-tx-0-21swapper/209:51:502
28415050,0irq/27-eth0-tx-0-21swapper/209:36:012
28415050,0irq/27-eth0-tx-0-21swapper/209:30:422
28415050,0irq/27-eth0-tx-0-21swapper/209:27:112
28415050,0irq/27-eth0-tx-0-21swapper/209:20:592
28415050,0irq/27-eth0-tx-0-21swapper/209:10:412
28415050,0irq/27-eth0-tx-0-21swapper/208:55:432
28415050,0irq/27-eth0-tx-0-21swapper/208:50:232
28415050,0irq/27-eth0-tx-0-21swapper/008:30:010
28415050,0irq/27-eth0-tx-0-21swapper/006:59:560
28415050,0irq/27-eth0-tx-0-21swapper/006:50:050
28405050,0irq/26-eth0-rx-701-21rt-features08:55:073
28405050,0irq/26-eth0-rx-3380-21df_abs09:00:033
28405050,0irq/26-eth0-rx-27158-21turbostat09:39:443
28405050,0irq/26-eth0-rx-26638-21timerwakeupswit07:40:093
28405050,0irq/26-eth0-rx-25984-21kthreadcore10:40:023
28405050,0irq/26-eth0-rx-24008-21iostat_ios08:30:023
28405050,0irq/26-eth0-rx-19288-21grep10:35:063
28405050,0irq/26-eth0-rx-18192-21kthreadcore08:25:043
28405050,0irq/26-eth0-rx-16157-21kthreadcore11:55:033
28405050,0irq/26-eth0-rx-0-21swapper/312:00:413
28405050,0irq/26-eth0-rx-0-21swapper/311:50:023
28405050,0irq/26-eth0-rx-0-21swapper/311:36:573
28405050,0irq/26-eth0-rx-0-21swapper/311:13:533
28405050,0irq/26-eth0-rx-0-21swapper/310:14:593
28405050,0irq/26-eth0-rx-0-21swapper/309:27:113
28405050,0irq/26-eth0-rx-0-21swapper/309:18:503
249950,0watchdog/29296-21chrt08:19:032
249950,0watchdog/226652-21tune2fs09:59:592
249950,0watchdog/20-21swapper/208:05:112
19150,0ktimersoftd/10-21swapper/111:28:191
179950,0migration/123884-21kthreadcore11:00:001
179950,0migration/123085-21kthreadcore11:40:031
179950,0migration/122720-21sh10:38:051
179950,0migration/122641-21munin-node10:15:071
179950,0migration/11457-21taskset09:41:171
179950,0migration/113737-21kthreadcore06:45:051
169950,0watchdog/19890-21cat10:30:031
169950,0watchdog/16547-21runrttasks10:23:071
169950,0watchdog/15241-21ssh10:46:031
169950,0watchdog/14134-21df_inode11:49:591
169950,0watchdog/121219-21cut08:50:031
169950,0watchdog/1164282sleep110:13:311
169950,0watchdog/10-21swapper/111:06:191
169950,0watchdog/10-21swapper/109:45:111
149950,0migration/08753-21gltestperf10:30:000
149950,0migration/07250-21taskset07:29:200
149950,0migration/032728-21kthreadcore08:10:140
149950,0migration/032728-21kthreadcore08:10:140
149950,0migration/030950-21munin-node06:55:040
149950,0migration/030179-21sh10:42:290
149950,0migration/02884-21cpu09:44:590
149950,0migration/028066-21fschecks_time08:54:580
149950,0migration/027285-21idleruntime07:19:580
149950,0migration/025533-21kthreadcore11:00:010
149950,0migration/025222-21kthreadcore08:05:070
149950,0migration/024703-21kthreadcore11:20:090
149950,0migration/024684-21kthreadcore08:50:040
149950,0migration/019859-21taskset10:55:060
149950,0migration/019171-21taskset09:34:160
149950,0migration/016774-21kthreadcore08:45:040
149950,0migration/016340-21kthreadcore11:35:080
149950,0migration/015210-21kthreadcore11:15:040
149950,0migration/014188-21kthreadcore09:05:070
149950,0migration/014178-21latency_hist07:34:450
149950,0migration/012181-21runrttasks11:11:520
149950,0migration/011223-21kthreadcore07:05:030
149950,0migration/011040-21sh11:31:340
8225040,0irq/24-0000:00:0-21swapper/206:50:322
53419943,0cyclictest3425-21lldpd08:35:362
53419943,0cyclictest3425-21lldpd07:55:222
53419943,0cyclictest3425-21lldpd07:37:232
53419943,0cyclictest3425-21lldpd07:25:512
53419943,0cyclictest3425-21lldpd07:12:212
53419943,0cyclictest3425-21lldpd07:01:502
53419943,0cyclictest3425-21lldpd06:55:482
53409944,0cyclictest0-21swapper/109:08:021
53409943,0cyclictest7648-21kworker/1:109:35:501
53409943,0cyclictest6445-21snmpd09:20:161
53409943,0cyclictest6445-21snmpd08:41:011
53409943,0cyclictest6445-21snmpd08:27:011
53409943,0cyclictest6445-21snmpd08:16:161
53409943,0cyclictest26238-21sadc07:19:451
53409943,0cyclictest22162-21awk11:20:071
53409943,0cyclictest15039-21awk07:35:041
53409943,0cyclictest13046-21awk11:35:061
53409943,0cyclictest10547-21awk08:20:111
53389944,0cyclictest0-21swapper/010:46:040
53389943,0cyclictest6445-21snmpd10:14:160
53389943,0cyclictest25816-21sadc09:59:440
53389943,0cyclictest2411-21awk10:05:020
53389943,0cyclictest17212-21awk08:25:030
339940,0migration/3896-21kthreadcore08:10:143
339940,0migration/3896-21kthreadcore08:10:143
339940,0migration/328449-21munin-node09:40:023
339940,0migration/325001-21diskmemload09:10:133
339940,0migration/323094-21kthreadcore06:50:023
339940,0migration/322927-21kthreadcore08:50:043
339940,0migration/32100-21df_abs07:49:573
339940,0migration/314947-21kthreadcore06:45:063
339940,0migration/312546-21kthreadcore10:30:043
28415040,0irq/27-eth0-tx-6372-21iceccd10:09:532
28415040,0irq/27-eth0-tx-6372-21iceccd09:14:592
28415040,0irq/27-eth0-tx-1776-21aten2.4-expect07:49:560
28415040,0irq/27-eth0-tx-15170-21telnet07:59:560
28415040,0irq/27-eth0-tx-0-21swapper/211:40:412
28415040,0irq/27-eth0-tx-0-21swapper/008:40:030
28405040,0irq/26-eth0-rx-10368-21ssh08:41:023
28405040,0irq/26-eth0-rx-0-21swapper/311:30:043
28405040,0irq/26-eth0-rx-0-21swapper/311:00:273
28405040,0irq/26-eth0-rx-0-21swapper/310:56:003
28405040,0irq/26-eth0-rx-0-21swapper/310:20:033
28405040,0irq/26-eth0-rx-0-21swapper/310:10:423
28405040,0irq/26-eth0-rx-0-21swapper/310:00:013
28405040,0irq/26-eth0-rx-0-21swapper/309:20:213
259940,0migration/2573-21latency_hist07:24:452
259940,0migration/24281-21kthreadcore07:50:022
259940,0migration/228203-21kthreadcore10:40:022
259940,0migration/211718-21runrttasks06:40:582
249940,0watchdog/28844-21kthreadcore07:30:032
249940,0watchdog/221179-21df_abs07:39:592
249940,0watchdog/20-21swapper/209:40:032
249940,0watchdog/20-21swapper/208:00:032
179940,0migration/19562-21irqcore07:55:041
179940,0migration/18746-21kthreadcore11:10:051
179940,0migration/17262-21kthreadcore09:25:021
179940,0migration/1704-21runrttasks07:45:091
179940,0migration/16923-21kthreadcore07:50:031
179940,0migration/14544-21kthreadcore10:05:041
179940,0migration/129878-21kthreadcore08:55:021
179940,0migration/128556-21aten_r1power_cu08:09:571
179940,0migration/128556-21aten_r1power_cu08:09:571
179940,0migration/128233-21kthreadcore08:30:041
179940,0migration/127315-21kthreadcore06:55:021
179940,0migration/126682-21aten_r1power_cu09:14:571
179940,0migration/118831-21fschecks_count09:54:591
179940,0migration/116349-21switchtime09:50:071
179940,0migration/115616-21kthreadcore08:45:041
179940,0migration/112324-21fschecks_count11:54:591
169940,0watchdog/131279-21grep08:35:031
169940,0watchdog/127898-21ssh12:04:551
169940,0watchdog/126654-21cut09:59:591
169940,0watchdog/125041-21kthreadcore12:00:071
169940,0watchdog/122627-21smartctl08:05:031
169940,0watchdog/118583-21ssh09:32:191
169940,0watchdog/115463-21if_eth1.310:55:031
169940,0watchdog/115143-21munin-node07:59:591
169940,0watchdog/113285-21ssh10:51:551
169940,0watchdog/10-21swapper/111:45:031
169940,0watchdog/10-21swapper/111:30:031
169940,0watchdog/10-21swapper/111:15:111
169940,0watchdog/10-21swapper/110:25:111
169940,0watchdog/10-21swapper/109:10:031
169940,0watchdog/10-21swapper/107:40:031
169940,0watchdog/10-21swapper/107:25:351
169940,0watchdog/10-21swapper/107:04:591
169940,0watchdog/10-21swapper/106:42:231
149940,0migration/07969-21kthreadcore06:40:050
149940,0migration/032099-21kthreadcore08:35:030
149940,0migration/030937-21if_err_eth111:05:000
149940,0migration/023482-21kthreadcore09:35:040
149940,0migration/011099-21fschecks_count09:49:580
149940,0migration/010109-21diskmemload09:02:570
53419933,0cyclictest3425-21lldpd07:44:522
53419932,0cyclictest3425-21lldpd07:14:512
53419930,0cyclictest28568-21port_sshpriv08:30:072
53419930,0cyclictest2601-21latency_hist08:14:452
53419930,0cyclictest2601-21latency_hist08:14:452
53419930,0cyclictest16092-21latency_hist08:24:452
53409933,0cyclictest7648-21kworker/1:107:29:511
53409933,0cyclictest7648-21kworker/1:107:19:531
53409933,0cyclictest7648-21kworker/1:106:59:471
53409933,0cyclictest7648-21kworker/1:106:49:471
53389933,0cyclictest0-21swapper/010:36:560
53389933,0cyclictest0-21swapper/009:14:000
53389933,0cyclictest0-21swapper/008:18:110
53389932,0cyclictest3768-21diskmemload10:19:510
53389932,0cyclictest27609-21awk09:15:030
53389932,0cyclictest1561-21cat11:45:100
53389932,0cyclictest14163-21awk07:09:590
53389930,0cyclictest3566-21munin-run09:24:440
53389930,0cyclictest20614-21cut12:00:030
53389930,0cyclictest0-21swapper/010:25:030
28415030,0irq/27-eth0-tx-0-21swapper/209:54:452
28415030,0irq/27-eth0-tx-0-21swapper/007:55:020
28405030,0irq/26-eth0-rx-0-21swapper/308:34:583
28405030,0irq/26-eth0-rx-0-21swapper/308:20:153
28405030,0irq/26-eth0-rx-0-21swapper/308:05:033
28405030,0irq/26-eth0-rx-0-21swapper/308:03:143
28405030,0irq/26-eth0-rx-0-21swapper/307:44:583
28405030,0irq/26-eth0-rx-0-21swapper/307:29:583
28405030,0irq/26-eth0-rx-0-21swapper/307:09:483
28405030,0irq/26-eth0-rx-0-21swapper/307:04:453
28405030,0irq/26-eth0-rx-0-21swapper/306:55:163
169930,0watchdog/10-21swapper/107:09:471
53389920,0cyclictest0-21swapper/007:44:580
28405020,0irq/26-eth0-rx-0-21swapper/307:15:513
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional