You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 12:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Tue Dec 02, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2913501040,0irq/25-eth10-21swapper/318:37:543
2011027675,0sleep00-21swapper/018:37:160
2022227372,0sleep20-21swapper/218:38:302
2026623927,0sleep10-21swapper/118:39:051
20352991313,0cyclictest23936-21sensors22:39:361
339960,0migration/317978-21munin-node23:39:313
339960,0migration/317784-21sh21:15:333
339960,0migration/317549-21sh22:38:233
339960,0migration/31262-21idleruntime-cro18:49:153
339960,0migration/310996-21sh21:14:063
28415060,0irq/27-eth0-tx-31437-21ssh23:45:142
28415060,0irq/27-eth0-tx-26720-21munin-node22:04:332
28415060,0irq/27-eth0-tx-25766-21ssh23:24:192
28415060,0irq/27-eth0-tx-0-21swapper/223:58:172
28415060,0irq/27-eth0-tx-0-21swapper/223:53:002
28415060,0irq/27-eth0-tx-0-21swapper/223:39:562
28415060,0irq/27-eth0-tx-0-21swapper/223:36:512
28415060,0irq/27-eth0-tx-0-21swapper/223:30:532
28415060,0irq/27-eth0-tx-0-21swapper/223:04:322
28415060,0irq/27-eth0-tx-0-21swapper/222:56:542
28415060,0irq/27-eth0-tx-0-21swapper/222:45:162
28415060,0irq/27-eth0-tx-0-21swapper/222:28:232
28415060,0irq/27-eth0-tx-0-21swapper/222:19:232
28415060,0irq/27-eth0-tx-0-21swapper/222:16:572
28415060,0irq/27-eth0-tx-0-21swapper/222:11:422
28415060,0irq/27-eth0-tx-0-21swapper/221:48:552
28415060,0irq/27-eth0-tx-0-21swapper/221:43:542
28415060,0irq/27-eth0-tx-0-21swapper/221:24:452
28415060,0irq/27-eth0-tx-0-21swapper/221:02:092
28415060,0irq/27-eth0-tx-0-21swapper/221:02:092
28415060,0irq/27-eth0-tx-0-21swapper/220:53:542
28415060,0irq/27-eth0-tx-0-21swapper/220:48:232
28415060,0irq/27-eth0-tx-0-21swapper/200:06:242
28405060,0irq/26-eth0-rx-24493-21sensors_temp21:59:383
28405060,0irq/26-eth0-rx-16811-21ssh23:39:013
4150,0ktimersoftd/00-21swapper/022:45:320
339950,0migration/36025-21latency23:49:313
339950,0migration/35976-21kthreadcore20:04:323
339950,0migration/3581-21munin-node21:04:353
339950,0migration/34069-21fschecks_count19:39:293
339950,0migration/331192-21runrttasks23:24:363
339950,0migration/330843-21taskset20:21:543
339950,0migration/328768-21kthreadcore19:54:353
339950,0migration/325670-21fschecks_time23:44:293
339950,0migration/324805-21kthreadcore19:29:353
339950,0migration/324511-21munin-node22:19:393
339950,0migration/323423-21kthreadcore22:39:333
339950,0migration/320049-21kthreadcore19:49:343
339950,0migration/318436-21fschecks_count20:14:303
339950,0migration/317221-21diskmemload21:34:423
339950,0migration/316595-21fschecks_time19:24:303
339950,0migration/316554-21munin-node22:14:413
339950,0migration/316363-21kthreadcore20:09:333
339950,0migration/313072-21kthreadcore21:54:333
28415050,0irq/27-eth0-tx-5749-21kthreadcore21:09:352
28415050,0irq/27-eth0-tx-23747-21awk22:59:382
28415050,0irq/27-eth0-tx-22661-21munin-node19:29:300
28415050,0irq/27-eth0-tx-17402-21iostat_ios23:59:302
28415050,0irq/27-eth0-tx-13432-21kthreadcore21:14:332
28415050,0irq/27-eth0-tx-0-21swapper/223:20:252
28415050,0irq/27-eth0-tx-0-21swapper/223:14:582
28415050,0irq/27-eth0-tx-0-21swapper/223:09:202
28415050,0irq/27-eth0-tx-0-21swapper/222:51:232
28415050,0irq/27-eth0-tx-0-21swapper/222:39:212
28415050,0irq/27-eth0-tx-0-21swapper/222:34:272
28415050,0irq/27-eth0-tx-0-21swapper/222:30:132
28415050,0irq/27-eth0-tx-0-21swapper/221:59:202
28415050,0irq/27-eth0-tx-0-21swapper/221:50:502
28415050,0irq/27-eth0-tx-0-21swapper/221:34:182
28415050,0irq/27-eth0-tx-0-21swapper/221:32:302
28415050,0irq/27-eth0-tx-0-21swapper/221:04:262
28415050,0irq/27-eth0-tx-0-21swapper/220:57:352
28415050,0irq/27-eth0-tx-0-21swapper/020:24:220
28415050,0irq/27-eth0-tx-0-21swapper/020:09:280
28415050,0irq/27-eth0-tx-0-21swapper/019:21:520
28415050,0irq/27-eth0-tx-0-21swapper/018:44:350
28405050,0irq/26-eth0-rx-8247-21uptime22:29:413
28405050,0irq/26-eth0-rx-31371-21sshd00:08:023
28405050,0irq/26-eth0-rx-20503-21kthreadcore23:19:323
28405050,0irq/26-eth0-rx-18351-21df_inode22:59:303
28405050,0irq/26-eth0-rx-15375-21missed_timers22:54:353
28405050,0irq/26-eth0-rx-0-21swapper/323:32:083
28405050,0irq/26-eth0-rx-0-21swapper/323:11:193
28405050,0irq/26-eth0-rx-0-21swapper/323:04:333
28405050,0irq/26-eth0-rx-0-21swapper/322:27:393
28405050,0irq/26-eth0-rx-0-21swapper/322:14:023
28405050,0irq/26-eth0-rx-0-21swapper/322:04:543
28405050,0irq/26-eth0-rx-0-21swapper/321:41:473
28405050,0irq/26-eth0-rx-0-21swapper/321:19:283
28405050,0irq/26-eth0-rx-0-21swapper/321:01:463
28405050,0irq/26-eth0-rx-0-21swapper/321:01:463
28405050,0irq/26-eth0-rx-0-21swapper/320:49:323
28405050,0irq/26-eth0-rx-0-21swapper/320:39:383
259950,0migration/232534-21kthreadcore19:09:342
259950,0migration/228352-21munin-node19:29:392
259950,0migration/226167-21sh20:42:412
259950,0migration/220497-21irqcore21:19:322
203539950,0cyclictest0-21swapper/220:10:252
203529955,0cyclictest0-21swapper/123:21:531
203529955,0cyclictest0-21swapper/121:48:431
203529955,0cyclictest0-21swapper/119:44:381
203529950,0cyclictest0-21swapper/122:13:191
203529950,0cyclictest0-21swapper/121:37:311
203529950,0cyclictest0-21swapper/120:54:001
203519955,0cyclictest0-21swapper/023:58:130
203519955,0cyclictest0-21swapper/023:32:410
203519955,0cyclictest0-21swapper/022:14:190
203519955,0cyclictest0-21swapper/020:04:250
203519954,0cyclictest20456-21ps20:59:320
203519954,0cyclictest20456-21ps20:59:320
203519950,0cyclictest0-21swapper/023:43:240
203519950,0cyclictest0-21swapper/022:25:120
203519950,0cyclictest0-21swapper/022:10:420
203519950,0cyclictest0-21swapper/021:21:590
203519950,0cyclictest0-21swapper/020:50:270
203519950,0cyclictest0-21swapper/020:45:340
179950,0migration/15574-21kthreadcore22:49:331
179950,0migration/132210-21users23:04:411
179950,0migration/131798-21kthreadcore19:34:371
179950,0migration/131187-21kthreadcore22:04:351
179950,0migration/12708-21kthreadcore23:49:301
179950,0migration/126843-21kthreadcore20:19:331
179950,0migration/121372-21idleruntime18:39:301
179950,0migration/121051-21kthreadcore19:49:351
179950,0migration/120370-21munin-node18:59:361
179950,0migration/118575-21fschecks_time22:59:301
179950,0migration/117810-21turbostat.cron22:39:091
179950,0migration/110939-21ksmtuned20:07:241
169950,0watchdog/124435-21cut21:19:341
169950,0watchdog/122672chrt20:48:381
169950,0watchdog/117968-21ssh22:18:421
169950,0watchdog/117443-21ssh23:19:011
169950,0watchdog/10-21swapper/123:09:331
169950,0watchdog/10-21swapper/122:45:461
169950,0watchdog/10-21swapper/119:14:381
149950,0migration/022925-21sh00:00:440
149950,0migration/017202-21taskset22:37:240
149950,0migration/014963-21kthreadcore20:34:330
149950,0migration/012288-21kthreadcore21:54:330
339940,0migration/38782-21cpuspeed_turbos23:54:283
339940,0migration/3740-21kthreadcore20:24:353
339940,0migration/331679-21kthreadcore20:44:333
339940,0migration/32930-21sh21:29:223
339940,0migration/328946-21kthreadcore21:24:343
339940,0migration/328226-21latency_hist19:09:153
339940,0migration/328052-21kthreadcore22:44:333
339940,0migration/325184-21kthreadcore18:39:333
339940,0migration/322863-21sh00:00:333
339940,0migration/320196-21kthreadcore18:59:353
339940,0migration/313378-21kthreadcore18:54:353
339940,0migration/311645-21kthreadcore19:44:333
28415040,0irq/27-eth0-tx-0-21swapper/020:14:280
28415040,0irq/27-eth0-tx-0-21swapper/019:59:170
28415040,0irq/27-eth0-tx-0-21swapper/019:34:390
28415040,0irq/27-eth0-tx-0-21swapper/019:09:310
28415040,0irq/27-eth0-tx-0-21swapper/018:54:340
28415040,0irq/27-eth0-tx-0-21swapper/018:41:280
28405040,0irq/26-eth0-rx-4160-21ksmtuned20:01:243
28405040,0irq/26-eth0-rx-0-21swapper/323:14:523
28405040,0irq/26-eth0-rx-0-21swapper/322:50:333
28405040,0irq/26-eth0-rx-0-21swapper/321:49:573
28405040,0irq/26-eth0-rx-0-21swapper/321:46:213
28405040,0irq/26-eth0-rx-0-21swapper/320:54:573
27140,0ktimersoftd/20-21swapper/219:44:242
259940,0migration/26834-21kthreadcore19:14:332
259940,0migration/224869-21cpuspeed_turbos20:19:292
259940,0migration/222725-21kthreadcore18:39:322
259940,0migration/215593-21kthreadcore21:54:342
259940,0migration/213364-21kthreadcore18:54:352
259940,0migration/211423-21kthreadcore19:19:332
249940,0watchdog/230006-21fschecks_time19:34:342
249940,0watchdog/218636-21kthreadcore19:24:342
249940,0watchdog/217282-21kthreadcore18:59:342
203539944,0cyclictest0-21swapper/219:59:322
203539943,0cyclictest3425-21lldpd20:29:232
203539943,0cyclictest3425-21lldpd20:26:232
203539943,0cyclictest22593-21awk19:04:312
203539943,0cyclictest17798-21awk19:49:322
203529944,0cyclictest18779-21diskmemload23:35:551
203529943,0cyclictest32206-21awk20:24:331
203529943,0cyclictest30987-21kworker/1:221:24:251
203529943,0cyclictest29148-21awk19:09:311
203529943,0cyclictest26100-21awk23:44:311
203529943,0cyclictest22641-21awk19:04:311
203529943,0cyclictest18071-21awk23:39:321
203529943,0cyclictest12252-21awk20:09:311
203529940,0cyclictest13305-21grep22:54:341
203519944,0cyclictest0-21swapper/022:56:250
203519944,0cyclictest0-21swapper/022:40:200
203519944,0cyclictest0-21swapper/021:27:490
203519944,0cyclictest0-21swapper/021:05:440
203519944,0cyclictest0-21swapper/000:05:190
203519943,0cyclictest6500-21awk20:29:320
203519943,0cyclictest6445-21snmpd23:26:190
203519943,0cyclictest6445-21snmpd23:16:490
203519943,0cyclictest6445-21snmpd23:00:340
203519943,0cyclictest6445-21snmpd19:27:480
203519943,0cyclictest3425-21lldpd21:31:550
203519943,0cyclictest3425-21lldpd21:09:240
203519943,0cyclictest32719-21sadc23:49:150
203519943,0cyclictest2438-21awk18:49:310
203519943,0cyclictest20012-21awk21:39:300
203519943,0cyclictest11192-21awk19:44:310
203519940,0cyclictest16911-21awk23:39:150
179940,0migration/17927-21taskset18:53:071
179940,0migration/17913-21kthreadcore19:39:341
179940,0migration/13699-21switchtime19:59:401
179940,0migration/130580-21kthreadcore21:04:321
179940,0migration/128717-21taskset19:29:541
179940,0migration/127552-21irqcore22:24:321
179940,0migration/127188-21latency_hist18:44:151
179940,0migration/12638-21kernelversion23:29:321
179940,0migration/124194-21sh22:19:371
179940,0migration/122660-21kthreadcore21:39:321
179940,0migration/119066-21aten_r1power_cu20:39:261
179940,0migration/116540-21missed_timers20:54:371
179940,0migration/115418-21kthreadcore21:54:341
179940,0migration/110653-21kthreadcore18:54:341
169940,0watchdog/125117-21python20:59:381
169940,0watchdog/125117-21python20:59:381
169940,0watchdog/114233-21kthreadcore21:14:341
169940,0watchdog/10-21swapper/123:56:251
169940,0watchdog/10-21swapper/123:24:291
169940,0watchdog/10-21swapper/122:31:141
169940,0watchdog/10-21swapper/121:59:221
169940,0watchdog/10-21swapper/121:50:381
169940,0watchdog/10-21swapper/121:09:181
169940,0watchdog/10-21swapper/120:34:341
169940,0watchdog/10-21swapper/120:29:341
169940,0watchdog/10-21swapper/119:24:381
169940,0watchdog/10-21swapper/100:04:291
149940,0migration/09865-21taskset21:52:340
149940,0migration/06299-21munin-node23:49:340
149940,0migration/03150-21kernelversion22:49:320
149940,0migration/029055-21kthreadcore21:44:320
149940,0migration/028193-21kthreadcore23:04:340
149940,0migration/024690-21sh22:19:570
149940,0migration/023670-21aten2.4-expect19:54:280
149940,0migration/022234-21latency19:49:350
149940,0migration/019302-21kthreadcore23:19:310
149940,0migration/017667-21taskset21:36:000
149940,0migration/013095-21kthreadcore20:54:350
28415030,0irq/27-eth0-tx-0-21swapper/020:39:270
28415030,0irq/27-eth0-tx-0-21swapper/020:19:380
28415030,0irq/27-eth0-tx-0-21swapper/018:59:340
28405030,0irq/26-eth0-rx-21758-21aten_r1power_po19:04:273
28405030,0irq/26-eth0-rx-0-21swapper/320:36:393
28405030,0irq/26-eth0-rx-0-21swapper/320:29:283
28405030,0irq/26-eth0-rx-0-21swapper/319:36:273
28405030,0irq/26-eth0-rx-0-21swapper/319:20:523
249930,0watchdog/20-21swapper/219:39:342
249930,0watchdog/20-21swapper/218:44:222
203539933,0cyclictest3425-21lldpd18:49:222
203539932,0cyclictest24554-21awk19:54:332
203539932,0cyclictest17996-21basename20:14:282
203539931,0cyclictest4578-21cat20:04:152
203539930,0cyclictest12727-21diskstats20:34:282
203529933,0cyclictest30987-21kworker/1:223:59:231
203529933,0cyclictest30987-21kworker/1:221:29:171
203529933,0cyclictest30987-21kworker/1:220:14:291
203529933,0cyclictest30987-21kworker/1:219:54:171
203519932,0cyclictest7896-21sensors22:29:390
203519932,0cyclictest6445-21snmpd21:14:340
203519932,0cyclictest532-21ssh22:05:060
203519932,0cyclictest3925-21grep19:39:290
203519932,0cyclictest3246-21tune2fs19:14:290
203519932,0cyclictest2362-21sort23:09:300
203519932,0cyclictest22499-21awk19:04:300
203519930,0cyclictest24506-21python21:59:390
169930,0watchdog/10-21swapper/119:19:181
28405020,0irq/26-eth0-rx-0-21swapper/319:14:383
28405020,0irq/26-eth0-rx-0-21swapper/318:49:303
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional