You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Fri May 02, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350440,0irq/25-eth10-21swapper/318:46:333
2416723523,0sleep10-21swapper/118:46:111
2405723127,0sleep20-21swapper/218:44:522
2399522925,0sleep00-21swapper/018:44:100
339960,0migration/36968-21sh21:36:433
339960,0migration/35366-21sh21:52:493
339960,0migration/331224-21munin-node23:32:253
339960,0migration/330931-21sh21:11:003
339960,0migration/330081-21sh23:11:223
339960,0migration/32833-21munin-node00:12:353
28415060,0irq/27-eth0-tx-5219-21ssh22:13:012
28415060,0irq/27-eth0-tx-22433-21munin-node23:27:272
28415060,0irq/27-eth0-tx-13739-21ssh21:38:102
28415060,0irq/27-eth0-tx-0-21swapper/223:37:552
28415060,0irq/27-eth0-tx-0-21swapper/223:32:332
28415060,0irq/27-eth0-tx-0-21swapper/223:25:282
28415060,0irq/27-eth0-tx-0-21swapper/223:18:152
28415060,0irq/27-eth0-tx-0-21swapper/222:57:012
28415060,0irq/27-eth0-tx-0-21swapper/222:46:442
28415060,0irq/27-eth0-tx-0-21swapper/222:30:052
28415060,0irq/27-eth0-tx-0-21swapper/222:21:152
28415060,0irq/27-eth0-tx-0-21swapper/221:57:412
28415060,0irq/27-eth0-tx-0-21swapper/221:47:462
28415060,0irq/27-eth0-tx-0-21swapper/221:18:462
28415060,0irq/27-eth0-tx-0-21swapper/221:08:062
28415060,0irq/27-eth0-tx-0-21swapper/220:55:012
28415060,0irq/27-eth0-tx-0-21swapper/220:51:142
28415060,0irq/27-eth0-tx-0-21swapper/200:12:142
28415060,0irq/27-eth0-tx-0-21swapper/200:09:592
28415060,0irq/27-eth0-tx-0-21swapper/200:01:462
28415060,0irq/27-eth0-tx-0-21swapper/020:39:270
28415060,0irq/27-eth0-tx-0-21swapper/020:07:240
28405060,0irq/26-eth0-rx-3103-21cut20:32:283
28405060,0irq/26-eth0-rx-25214-21cut20:22:343
243269960,0cyclictest22576-21diskmemload21:55:581
243269960,0cyclictest0-21swapper/123:39:231
243269960,0cyclictest0-21swapper/121:13:141
243259966,0cyclictest5903-21kthreadcore19:42:330
249960,0watchdog/21148-21seq20:02:582
249960,0watchdog/20-21swapper/222:58:422
249960,0watchdog/20-21swapper/220:57:582
149960,0migration/06382-21taskset22:16:220
149960,0migration/05707-21sh20:55:080
149960,0migration/03365-21missed_timers23:52:300
149960,0migration/029977-21sh23:52:030
35150,0ktimersoftd/30-21swapper/300:05:283
339950,0migration/38669-21kthreadcore23:57:313
339950,0migration/3824-21aten_r1power_fr19:42:263
339950,0migration/37861-21if_eth1.320:57:333
339950,0migration/35240-21needreboot19:17:383
339950,0migration/35039-21sh20:53:193
339950,0migration/34869-21taskset18:52:423
339950,0migration/332028-21munin-node22:32:283
339950,0migration/330675-21taskset18:48:573
339950,0migration/32748-21munin-node20:07:283
339950,0migration/325481-21kthreadcore20:47:323
339950,0migration/325375-21kthreadcore23:27:323
339950,0migration/323204-21munin-node22:47:253
339950,0migration/32210-21kthreadcore23:52:303
339950,0migration/32172-21taskset20:28:123
339950,0migration/321480-21munin-node21:42:383
339950,0migration/321037-21missed_timers21:22:323
339950,0migration/319925-21kthreadcore22:22:303
339950,0migration/316419-21kthreadcore19:52:323
339950,0migration/315536-21taskset20:38:053
339950,0migration/315291-21kthreadcore19:27:313
339950,0migration/313942-21taskset21:18:143
28415050,0irq/27-eth0-tx-32508-21idleruntime21:12:292
28415050,0irq/27-eth0-tx-28495-21munin-node20:27:270
28415050,0irq/27-eth0-tx-22993-21sed23:07:272
28415050,0irq/27-eth0-tx-22779-21munin-node22:47:332
28415050,0irq/27-eth0-tx-14239-21munin-node23:42:272
28415050,0irq/27-eth0-tx-0-21swapper/223:50:052
28415050,0irq/27-eth0-tx-0-21swapper/223:13:082
28415050,0irq/27-eth0-tx-0-21swapper/223:02:392
28415050,0irq/27-eth0-tx-0-21swapper/222:33:022
28415050,0irq/27-eth0-tx-0-21swapper/222:23:502
28415050,0irq/27-eth0-tx-0-21swapper/222:07:212
28415050,0irq/27-eth0-tx-0-21swapper/222:03:212
28415050,0irq/27-eth0-tx-0-21swapper/221:53:162
28415050,0irq/27-eth0-tx-0-21swapper/221:43:082
28415050,0irq/27-eth0-tx-0-21swapper/221:32:392
28415050,0irq/27-eth0-tx-0-21swapper/221:28:422
28415050,0irq/27-eth0-tx-0-21swapper/221:04:352
28415050,0irq/27-eth0-tx-0-21swapper/200:03:182
28415050,0irq/27-eth0-tx-0-21swapper/019:52:280
28415050,0irq/27-eth0-tx-0-21swapper/019:47:350
28405050,0irq/26-eth0-rx-9091-21kthreadcore23:17:293
28405050,0irq/26-eth0-rx-6547-21runrttasks23:02:203
28405050,0irq/26-eth0-rx-5636-21ssh22:55:323
28405050,0irq/26-eth0-rx-4661-21sh23:13:143
28405050,0irq/26-eth0-rx-21492-21aten_r1power_cu00:07:233
28405050,0irq/26-eth0-rx-20848-21python20:17:403
28405050,0irq/26-eth0-rx-15033-21fschecks_time23:22:273
28405050,0irq/26-eth0-rx-0-21swapper/323:42:293
28405050,0irq/26-eth0-rx-0-21swapper/322:57:263
28405050,0irq/26-eth0-rx-0-21swapper/322:42:303
28405050,0irq/26-eth0-rx-0-21swapper/322:12:243
28405050,0irq/26-eth0-rx-0-21swapper/322:09:423
28405050,0irq/26-eth0-rx-0-21swapper/321:57:303
28405050,0irq/26-eth0-rx-0-21swapper/321:37:303
28405050,0irq/26-eth0-rx-0-21swapper/321:14:383
259950,0migration/27892-21kthreadcore19:22:332
259950,0migration/26791-21aten_r1power_en22:37:242
259950,0migration/222152-21sh21:23:462
259950,0migration/220713-21munin-node19:32:302
259950,0migration/219677-21kthreadcore20:17:352
243269955,0cyclictest0-21swapper/123:49:291
243269955,0cyclictest0-21swapper/122:37:541
243269955,0cyclictest0-21swapper/121:25:511
243269955,0cyclictest0-21swapper/121:19:331
243269955,0cyclictest0-21swapper/119:07:261
243269950,0cyclictest0-21swapper/123:11:381
243269950,0cyclictest0-21swapper/122:29:331
243269950,0cyclictest0-21swapper/121:43:501
243269950,0cyclictest0-21swapper/100:08:331
243259955,0cyclictest22576-21diskmemload22:34:330
243259955,0cyclictest0-21swapper/021:46:190
243259955,0cyclictest0-21swapper/021:27:420
243259955,0cyclictest0-21swapper/019:57:130
243259954,0cyclictest0-21swapper/020:59:480
243259950,0cyclictest0-21swapper/021:54:300
249950,0watchdog/20-21swapper/223:54:462
249950,0watchdog/20-21swapper/220:27:342
249950,0watchdog/20-21swapper/219:44:142
249950,0watchdog/20-21swapper/219:07:542
249950,0watchdog/20-21swapper/219:06:102
179950,0migration/19886-21kthreadcore19:47:321
179950,0migration/123498-21aten_r1power_fr21:47:221
179950,0migration/121844-21taskset23:05:521
149950,0migration/0989-21kthreadcore21:12:320
149950,0migration/09658-21kthreadcore23:17:290
149950,0migration/09437-21kthreadcore19:22:330
149950,0migration/07526-21if_eth1.222:57:290
149950,0migration/03475-21kthreadcore21:32:300
149950,0migration/032538-21kthreadcore23:12:290
149950,0migration/032231-21kthreadcore22:52:300
149950,0migration/026976-21kthreadcore23:27:330
149950,0migration/026959-21kthreadcore18:47:300
149950,0migration/023735-21sh20:47:300
149950,0migration/023099-21fschecks_count23:07:270
149950,0migration/021115-21kthreadcore19:07:290
149950,0migration/018651-21kthreadcore00:02:300
149950,0migration/018565-21kthreadcore20:42:300
149950,0migration/018511-21kthreadcore22:42:300
149950,0migration/016185-21munin-node21:02:310
149950,0migration/013155-21if_eth1.219:02:280
149950,0migration/011262-21kthreadcore21:57:300
339940,0migration/39013-21kthreadcore22:17:293
339940,0migration/36145-21runrttasks23:37:153
339940,0migration/331938-21taskset19:17:043
339940,0migration/326461-21latency_hist19:37:123
339940,0migration/324995-21kthreadcore22:27:313
339940,0migration/324516-21kthreadcore23:47:293
339940,0migration/322190-21taskset20:42:543
339940,0migration/312616-21taskset19:23:083
339940,0migration/310162-21kthreadcore19:47:323
28415040,0irq/27-eth0-tx-27692-21telnet20:02:260
28415040,0irq/27-eth0-tx-0-21swapper/020:22:280
28415040,0irq/27-eth0-tx-0-21swapper/019:37:260
28415040,0irq/27-eth0-tx-0-21swapper/018:52:240
28405040,0irq/26-eth0-rx-0-21swapper/322:37:343
28405040,0irq/26-eth0-rx-0-21swapper/322:02:133
28405040,0irq/26-eth0-rx-0-21swapper/321:47:253
28405040,0irq/26-eth0-rx-0-21swapper/321:27:273
28405040,0irq/26-eth0-rx-0-21swapper/321:02:253
28405040,0irq/26-eth0-rx-0-21swapper/318:58:533
259940,0migration/26398-21kthreadcore20:32:332
259940,0migration/21993-21cpuspeed_turbos20:07:252
259940,0migration/214788-21fschecks_time19:52:282
259940,0migration/211100-21kthreadcore20:37:332
243279944,0cyclictest30057-21kworker/2:118:52:252
243279944,0cyclictest28274-21kworker/2:120:42:292
243279944,0cyclictest18672-21kworker/2:020:12:252
243279944,0cyclictest18672-21kworker/2:019:57:232
243279944,0cyclictest18672-21kworker/2:019:47:212
243279944,0cyclictest18672-21kworker/2:019:37:272
243279944,0cyclictest18672-21kworker/2:019:17:252
243269944,0cyclictest14962-21kworker/1:219:44:531
243269944,0cyclictest0-21swapper/123:12:161
243269944,0cyclictest0-21swapper/120:52:161
243269943,0cyclictest3657-21awk20:32:301
243269943,0cyclictest3425-21lldpd19:57:571
243269943,0cyclictest3425-21lldpd19:17:261
243269943,0cyclictest32161-21awk22:12:271
243269943,0cyclictest31520-21ps23:32:271
243269943,0cyclictest3110-21irqbalance00:12:261
243269943,0cyclictest29362-21awk20:27:301
243269943,0cyclictest24538-21awk21:27:281
243269943,0cyclictest23809-21awk20:47:311
243269943,0cyclictest18649-21sadc19:07:111
243269943,0cyclictest14917-21sadc20:17:111
243259943,0cyclictest9450-21awk20:12:270
243259943,0cyclictest8477-21awk21:17:290
243259943,0cyclictest7064-21awk23:37:260
243259943,0cyclictest3425-21lldpd23:07:070
243259943,0cyclictest24397-21awk21:07:320
243259943,0cyclictest16178-21awk22:02:290
243259943,0cyclictest15425-21awk23:22:290
249940,0watchdog/222466-21cut20:22:302
249940,0watchdog/218738-21port_sshpriv19:27:342
249940,0watchdog/20-21swapper/219:12:142
249940,0watchdog/20-21swapper/218:57:222
249940,0watchdog/20-21swapper/218:47:502
179940,0migration/19321-21kthreadcore23:17:291
179940,0migration/19275-21kthreadcore23:57:311
179940,0migration/16246-21taskset21:34:421
179940,0migration/129219-21taskset23:29:231
179940,0migration/127701-21irqrtprio19:37:301
179940,0migration/127307-21kthreadcore18:47:311
179940,0migration/123184-21latency_hist22:07:121
179940,0migration/118496-21kthreadcore20:17:351
179940,0migration/118429-21kthreadcore23:42:301
179940,0migration/117212-21kthreadcore23:22:311
179940,0migration/115539-21kthreadcore19:52:321
179940,0migration/114008-21cpuspeed_turbos00:02:251
179940,0migration/112967-21rt-features21:57:341
179940,0migration/112009-21kthreadcore20:37:331
179940,0migration/111389-21kthreadcore19:22:341
169940,0watchdog/17974-21kthreadcore22:57:301
169940,0watchdog/131247-21diskstats22:52:261
169940,0watchdog/120813-21awk19:32:301
169940,0watchdog/120645-21uname21:02:341
169940,0watchdog/112334-21latency22:17:301
169940,0watchdog/10-21swapper/122:42:301
169940,0watchdog/10-21swapper/122:32:261
169940,0watchdog/10-21swapper/121:37:261
169940,0watchdog/10-21swapper/120:42:301
169940,0watchdog/10-21swapper/120:22:301
169940,0watchdog/10-21swapper/120:02:261
149940,0migration/029496-21cpuspeed_turbos00:12:250
149940,0migration/024492-21irqcore21:47:270
149940,0migration/017942-21kthreadcore21:22:310
149940,0migration/014445-21sh22:42:070
28415030,0irq/27-eth0-tx-0-21swapper/019:12:340
28405030,0irq/26-eth0-rx-0-21swapper/320:12:233
28405030,0irq/26-eth0-rx-0-21swapper/319:32:273
28405030,0irq/26-eth0-rx-0-21swapper/319:07:333
243269932,0cyclictest8792-21cut20:57:341
243269932,0cyclictest31195-21awk23:52:271
243269932,0cyclictest3110-21irqbalance22:02:161
243269932,0cyclictest26538-21awk19:12:261
243269930,0cyclictest2119-21sed20:07:251
243259932,0cyclictest8741-21grep20:32:390
243259932,0cyclictest7868-21awk22:17:270
243259932,0cyclictest6763-21awk23:57:290
243259932,0cyclictest6447-21awk18:57:280
243259932,0cyclictest25540-21kthreadcore22:47:290
243259932,0cyclictest24175-21kthreadcore19:32:320
243259932,0cyclictest20774-21ssh23:43:470
243259932,0cyclictest15870-21awk22:22:270
243259932,0cyclictest15427-21grep20:17:290
243259932,0cyclictest14186-21awk19:27:290
243259932,0cyclictest14031-21sh21:38:580
243259931,0cyclictest25593-21kthreadcore22:07:310
243259931,0cyclictest0-21swapper/019:18:450
243259930,0cyclictest31193-21perl23:32:250
179930,0migration/15525-21aten2.4-expect18:57:231
169930,0watchdog/10-21swapper/122:47:181
169930,0watchdog/10-21swapper/122:22:261
169930,0watchdog/10-21swapper/121:07:341
169930,0watchdog/10-21swapper/119:29:341
169930,0watchdog/10-21swapper/118:52:301
339920,0migration/327933-21diskstats20:02:273
28405020,0irq/26-eth0-rx-0-21swapper/319:57:263
28405020,0irq/26-eth0-rx-0-21swapper/319:02:143
243259921,0cyclictest21335-21latency_hist00:07:110
243259920,0cyclictest0-21swapper/022:27:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional