You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-25 - 00:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Fri Oct 24, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350820,0irq/25-eth10-21swapper/306:35:593
1935123523,0sleep00-21swapper/006:36:080
1965123430,0sleep20-21swapper/206:39:342
1953823424,0sleep10-21swapper/106:38:171
28415070,0irq/27-eth0-tx-0-21swapper/007:01:010
249970,0watchdog/20-21swapper/208:58:592
198169976,0cyclictest0-21swapper/011:26:000
339960,0migration/329036-21sh11:57:023
339960,0migration/327687-21aten_r1power_cu07:10:493
339960,0migration/321166-21sh11:11:083
339960,0migration/317474-21munin-node07:00:563
339960,0migration/314366-21sh11:30:433
28415060,0irq/27-eth0-tx-28945-21ssh11:37:052
28415060,0irq/27-eth0-tx-0-21swapper/212:03:352
28415060,0irq/27-eth0-tx-0-21swapper/211:52:112
28415060,0irq/27-eth0-tx-0-21swapper/211:41:222
28415060,0irq/27-eth0-tx-0-21swapper/211:22:582
28415060,0irq/27-eth0-tx-0-21swapper/211:19:362
28415060,0irq/27-eth0-tx-0-21swapper/210:59:362
28415060,0irq/27-eth0-tx-0-21swapper/210:37:522
28415060,0irq/27-eth0-tx-0-21swapper/210:18:202
28415060,0irq/27-eth0-tx-0-21swapper/209:51:392
28415060,0irq/27-eth0-tx-0-21swapper/209:40:092
28415060,0irq/27-eth0-tx-0-21swapper/209:31:532
28415060,0irq/27-eth0-tx-0-21swapper/209:28:012
28415060,0irq/27-eth0-tx-0-21swapper/209:22:022
28415060,0irq/27-eth0-tx-0-21swapper/209:11:022
28415060,0irq/27-eth0-tx-0-21swapper/209:07:032
28415060,0irq/27-eth0-tx-0-21swapper/008:15:520
28415060,0irq/27-eth0-tx-0-21swapper/008:10:540
28415060,0irq/27-eth0-tx-0-21swapper/008:06:130
28415060,0irq/27-eth0-tx-0-21swapper/007:27:410
28405060,0irq/26-eth0-rx-32171-21df_abs11:00:563
28405060,0irq/26-eth0-rx-25725-21ssh10:15:333
28405060,0irq/26-eth0-rx-2308-21users07:36:043
28405060,0irq/26-eth0-rx-17579-21port_sshpriv09:01:003
28405060,0irq/26-eth0-rx-10729-21ssh08:57:553
249960,0watchdog/20-21swapper/210:50:582
249960,0watchdog/20-21swapper/208:54:112
249960,0watchdog/20-21swapper/207:55:432
198209960,0cyclictest0-21swapper/209:19:092
198169966,0cyclictest943-21sensors07:10:590
198169960,0cyclictest0-21swapper/011:23:000
198169960,0cyclictest0-21swapper/009:06:000
149960,0migration/029937-21sh11:18:480
149960,0migration/029313-21latency10:56:000
149960,0migration/024669-21kthreadcore07:05:590
149960,0migration/015677-21sh10:26:500
339950,0migration/38486-21kthreadcore09:15:593
339950,0migration/37490-21kthreadcore08:06:083
339950,0migration/35760-21latency10:40:593
339950,0migration/33297-21kthreadcore12:00:593
339950,0migration/331614-21kthreadcore09:31:003
339950,0migration/327486-21runrttasks06:45:523
339950,0migration/325929-21kthreadcore07:55:583
339950,0migration/324310-21latency_hist10:10:403
339950,0migration/321663-21kthreadcore07:50:593
339950,0migration/321169-21aten_r1power_po07:05:543
339950,0migration/320169-21aten_r1power_vo06:40:513
339950,0migration/319521-21kthreadcore10:30:553
339950,0migration/31857-21if_eth006:50:553
339950,0migration/318210-21aten_r1power_po09:45:513
339950,0migration/316571-21runrttasks10:50:543
339950,0migration/316452-21kthreadcore11:50:593
339950,0migration/314865-21kthreadcore07:45:583
339950,0migration/313689-21latency10:46:023
339950,0migration/312512-21diskmemload11:46:113
339950,0migration/312433-21munin-node09:20:583
339950,0migration/311450-21kthreadcore07:20:583
339950,0migration/310298-21kthreadcore11:05:593
28501250,0sleep328588-21port_sshpriv11:16:013
28415050,0irq/27-eth0-tx-6386-21ssh12:05:462
28415050,0irq/27-eth0-tx-6372-21iceccd06:46:480
28415050,0irq/27-eth0-tx-12380-21diskmemload10:20:512
28415050,0irq/27-eth0-tx-0-21swapper/211:45:562
28415050,0irq/27-eth0-tx-0-21swapper/211:33:352
28415050,0irq/27-eth0-tx-0-21swapper/211:25:582
28415050,0irq/27-eth0-tx-0-21swapper/211:14:202
28415050,0irq/27-eth0-tx-0-21swapper/211:06:102
28415050,0irq/27-eth0-tx-0-21swapper/210:46:212
28415050,0irq/27-eth0-tx-0-21swapper/210:44:082
28415050,0irq/27-eth0-tx-0-21swapper/210:12:232
28415050,0irq/27-eth0-tx-0-21swapper/210:03:342
28415050,0irq/27-eth0-tx-0-21swapper/209:00:582
28415050,0irq/27-eth0-tx-0-21swapper/208:45:452
28415050,0irq/27-eth0-tx-0-21swapper/008:41:060
28415050,0irq/27-eth0-tx-0-21swapper/007:40:560
28405050,0irq/26-eth0-rx-4815-21ssh11:42:243
28405050,0irq/26-eth0-rx-3125-21ssh08:54:033
28405050,0irq/26-eth0-rx-26951-21iostat_ios09:50:543
28405050,0irq/26-eth0-rx-24239-21cpuspeed_turbos10:55:543
28405050,0irq/26-eth0-rx-23922-21kthreadcore09:25:573
28405050,0irq/26-eth0-rx-21786-21kthreadcore08:15:593
28405050,0irq/26-eth0-rx-14722-21grep10:26:003
28405050,0irq/26-eth0-rx-12832-21ssh11:26:393
28405050,0irq/26-eth0-rx-12579-21vmstat12:06:043
28405050,0irq/26-eth0-rx-0-21swapper/311:21:353
28405050,0irq/26-eth0-rx-0-21swapper/310:37:073
28405050,0irq/26-eth0-rx-0-21swapper/309:05:503
259950,0migration/217502-21df_abs10:30:512
259950,0migration/215905-21kthreadcore08:40:572
249950,0watchdog/227080-21cut11:55:582
249950,0watchdog/20-21swapper/211:02:582
249950,0watchdog/20-21swapper/210:25:582
249950,0watchdog/20-21swapper/209:47:022
249950,0watchdog/20-21swapper/209:43:422
249950,0watchdog/20-21swapper/208:28:472
249950,0watchdog/20-21swapper/208:23:192
249950,0watchdog/20-21swapper/208:19:192
249950,0watchdog/20-21swapper/208:02:512
249950,0watchdog/20-21swapper/207:55:392
249950,0watchdog/20-21swapper/207:50:312
249950,0watchdog/20-21swapper/207:39:032
249950,0watchdog/20-21swapper/207:32:472
249950,0watchdog/20-21swapper/207:29:392
249950,0watchdog/20-21swapper/207:07:072
249950,0watchdog/20-21swapper/207:00:032
198179950,0cyclictest0-21swapper/111:32:131
198169955,0cyclictest0-21swapper/010:09:130
198169954,0cyclictest0-21swapper/008:57:190
198169950,0cyclictest0-21swapper/010:12:320
198169950,0cyclictest0-21swapper/009:38:380
179950,0migration/127219-21irqrtprio10:15:551
179950,0migration/117782-21taskset09:45:341
169950,0watchdog/18922-21cut08:55:591
169950,0watchdog/126592-21kthreadcore08:25:551
169950,0watchdog/11034-21perl10:40:541
149950,0migration/0826-21kthreadcore11:40:560
149950,0migration/06630-21sh10:41:100
149950,0migration/06091-21kthreadcore09:55:570
149950,0migration/0504-21kthreadcore08:00:560
149950,0migration/0340-21munin-node11:00:590
149950,0migration/031088-21kthreadcore10:15:560
149950,0migration/027988-21aten_r1power_fr09:10:530
149950,0migration/027004-21irqcore09:50:550
149950,0migration/026944-21kthreadcore07:55:580
149950,0migration/026653-21kthreadcore08:25:550
149950,0migration/024056-21timerwakeupswit09:46:030
149950,0migration/023045-21munin-node10:31:010
149950,0migration/021801-21pmu-power10:51:020
149950,0migration/020140-21sh09:25:550
149950,0migration/019147-21taskset09:04:120
149950,0migration/018315-21kthreadcore11:10:580
149950,0migration/017432-21munin-node07:50:580
149950,0migration/016277-21kthreadcore09:21:010
149950,0migration/014239-21timerandwakeup10:46:060
149950,0migration/013715-21taskset11:07:140
149950,0migration/013211-21kthreadcore10:01:020
149950,0migration/013188-21kthreadcore09:40:590
149950,0migration/012995-21kthreadcore07:45:570
339940,0migration/35090-21kthreadcore07:41:043
339940,0migration/323239-21hddtemp_smartct11:35:523
28415040,0irq/27-eth0-tx-0-21swapper/008:35:560
28405040,0irq/26-eth0-rx-1828-21ssh09:11:353
28405040,0irq/26-eth0-rx-0-21swapper/310:19:273
28405040,0irq/26-eth0-rx-0-21swapper/310:03:043
28405040,0irq/26-eth0-rx-0-21swapper/309:55:573
28405040,0irq/26-eth0-rx-0-21swapper/309:40:593
28405040,0irq/26-eth0-rx-0-21swapper/309:35:443
28405040,0irq/26-eth0-rx-0-21swapper/308:46:053
28405040,0irq/26-eth0-rx-0-21swapper/308:40:513
28405040,0irq/26-eth0-rx-0-21swapper/306:55:573
259940,0migration/229143-21kthreadcore06:45:542
249940,0watchdog/28035-21vmstat07:16:032
249940,0watchdog/20-21swapper/209:56:022
249940,0watchdog/20-21swapper/208:37:432
249940,0watchdog/20-21swapper/208:10:432
249940,0watchdog/20-21swapper/208:06:072
249940,0watchdog/20-21swapper/207:00:432
249940,0watchdog/20-21swapper/206:51:232
249940,0watchdog/20-21swapper/206:43:152
198209943,0cyclictest3425-21lldpd10:05:452
198209943,0cyclictest32413-21awk08:31:052
198209943,0cyclictest3110-21irqbalance07:40:442
198209943,0cyclictest3110-21irqbalance07:20:442
198209943,0cyclictest3110-21irqbalance07:10:442
198179943,0cyclictest6445-21snmpd11:40:521
198179943,0cyclictest6445-21snmpd08:45:521
198179943,0cyclictest3425-21lldpd11:02:401
198179943,0cyclictest3425-21lldpd07:20:561
198179943,0cyclictest31771-21awk11:20:581
198179943,0cyclictest2746-21sadc07:40:401
198179943,0cyclictest25946-21awk10:35:581
198179943,0cyclictest2178-21awk09:55:551
198179943,0cyclictest11678-21awk08:10:571
198179943,0cyclictest11653-21kworker/1:210:45:581
198179943,0cyclictest10307-21awk10:25:571
198179943,0cyclictest10039-21awk10:00:591
198179940,0cyclictest12380-21diskmemload11:11:441
198169944,0cyclictest0-21swapper/012:06:310
198169943,0cyclictest8616-21awk06:55:550
198169943,0cyclictest3425-21lldpd08:22:120
198169943,0cyclictest23458-21awk11:55:560
198169943,0cyclictest23385-21awk11:35:530
179940,0migration/17350-21irqcore11:45:551
179940,0migration/17242-21aten_r1power_en11:05:531
179940,0migration/14440-21kthreadcore09:35:591
179940,0migration/13053-21kthreadcore10:21:001
179940,0migration/128314-21timerandwakeup11:36:011
179940,0migration/128223-21kthreadcore09:50:551
179940,0migration/127885-21irqrtprio06:45:541
179940,0migration/127229-21cpuspeed_turbos09:30:551
179940,0migration/126081-21kthreadcore07:55:581
179940,0migration/124821-21sh10:12:161
179940,0migration/123052-21aten_r1power_fr11:15:511
179940,0migration/122986-21fschecks_count11:55:541
179940,0migration/121177-21kthreadcore09:45:571
179940,0migration/121130-21munin-node07:05:531
179940,0migration/119811-21kthreadcore07:25:591
179940,0migration/119219-21kthreadcore07:00:561
179940,0migration/115386-21munin-node07:46:021
179940,0migration/111955-21fschecks_count09:20:561
169940,0watchdog/1949-21cut08:31:071
169940,0watchdog/14237-21fschecks_time09:15:541
169940,0watchdog/131252-21cut08:50:591
169940,0watchdog/131068-21forks12:00:541
169940,0watchdog/125264-21ssh09:26:141
169940,0watchdog/125258-21irqrtprio10:55:581
169940,0watchdog/122783-21ssh10:52:381
169940,0watchdog/119186-21kthreadcore07:50:591
169940,0watchdog/118893-21ssh08:41:431
169940,0watchdog/117911-21fschecks_count08:15:551
169940,0watchdog/115647-21irqrtprio11:50:581
169940,0watchdog/111593-21kthreadcore12:05:581
169940,0watchdog/10-21swapper/107:16:031
149940,0migration/03307-21runrttasks08:54:400
149940,0migration/024800-21kthreadcore06:40:570
149940,0migration/01258-21kthreadcore08:31:070
339930,0migration/35501-21taskset08:33:393
339930,0migration/324754-21kthreadcore07:30:583
339930,0migration/317301-21kthreadcore07:25:583
28415030,0irq/27-eth0-tx-1074-21aten2.4-expect06:50:520
28405030,0irq/26-eth0-rx-0-21swapper/310:21:283
28405030,0irq/26-eth0-rx-0-21swapper/308:35:523
28405030,0irq/26-eth0-rx-0-21swapper/308:25:503
28405030,0irq/26-eth0-rx-0-21swapper/308:10:543
28405030,0irq/26-eth0-rx-0-21swapper/308:00:553
28405030,0irq/26-eth0-rx-0-21swapper/307:16:033
198179933,0cyclictest6767-21kworker/1:008:35:421
198179933,0cyclictest6767-21kworker/1:008:20:401
198179933,0cyclictest6767-21kworker/1:008:05:421
198179933,0cyclictest6767-21kworker/1:008:00:421
198179933,0cyclictest6767-21kworker/1:007:40:461
198179933,0cyclictest6767-21kworker/1:007:30:421
198179933,0cyclictest6767-21kworker/1:007:10:421
198179933,0cyclictest6767-21kworker/1:006:55:521
198179933,0cyclictest6767-21kworker/1:006:50:421
198179933,0cyclictest6767-21kworker/1:006:40:421
198179933,0cyclictest11653-21kworker/1:210:30:441
198179933,0cyclictest11653-21kworker/1:210:05:561
198179933,0cyclictest11653-21kworker/1:209:10:521
198179933,0cyclictest11653-21kworker/1:209:05:481
198179932,0cyclictest11653-21kworker/1:209:00:541
198169932,0cyclictest27848-21awk09:30:570
198169932,0cyclictest22582-21kthreadcore08:45:580
198169932,0cyclictest2153-21grep10:20:570
198169932,0cyclictest14150-21sadc11:50:400
198169931,0cyclictest28161-21tr10:36:010
198169931,0cyclictest0-21swapper/007:37:020
198169930,0cyclictest8687-21munin-node07:20:530
198169930,0cyclictest22026-21munin-node07:30:570
179930,0migration/17943-21kthreadcore11:25:591
149930,0migration/030638-21sh12:00:500
149930,0migration/021087-21sh11:32:320
28405020,0irq/26-eth0-rx-0-21swapper/308:20:423
198169921,0cyclictest3663-21aten_r1power_vo09:15:520
198169920,0cyclictest0-21swapper/011:50:480
198169920,0cyclictest0-21swapper/007:15:510
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional