You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-06 - 09:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Thu Nov 06, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1216027473,0sleep10-21swapper/118:38:071
595324948,0sleep30-21swapper/318:35:173
1209723724,0sleep00-21swapper/018:37:220
1217123228,0sleep20-21swapper/218:38:162
35160,0ktimersoftd/30-21swapper/322:29:283
35160,0ktimersoftd/30-21swapper/321:56:423
339960,0migration/39065-21sh22:39:143
339960,0migration/329659-21latency23:50:333
339960,0migration/323990-21sh23:50:103
339960,0migration/321880-21missed_timers20:20:323
339960,0migration/317635-21sh21:02:023
28415060,0irq/27-eth0-tx-912-21ssh22:55:022
28415060,0irq/27-eth0-tx-31537-21ssh23:31:252
28415060,0irq/27-eth0-tx-0-21swapper/223:52:162
28415060,0irq/27-eth0-tx-0-21swapper/223:42:152
28415060,0irq/27-eth0-tx-0-21swapper/223:39:022
28415060,0irq/27-eth0-tx-0-21swapper/223:29:502
28415060,0irq/27-eth0-tx-0-21swapper/223:21:042
28415060,0irq/27-eth0-tx-0-21swapper/223:11:402
28415060,0irq/27-eth0-tx-0-21swapper/223:00:232
28415060,0irq/27-eth0-tx-0-21swapper/222:55:382
28415060,0irq/27-eth0-tx-0-21swapper/222:36:162
28415060,0irq/27-eth0-tx-0-21swapper/222:25:142
28415060,0irq/27-eth0-tx-0-21swapper/222:21:052
28415060,0irq/27-eth0-tx-0-21swapper/222:10:482
28415060,0irq/27-eth0-tx-0-21swapper/221:41:572
28415060,0irq/27-eth0-tx-0-21swapper/221:27:412
28415060,0irq/27-eth0-tx-0-21swapper/220:59:072
28415060,0irq/27-eth0-tx-0-21swapper/220:52:072
28415060,0irq/27-eth0-tx-0-21swapper/020:20:360
28415060,0irq/27-eth0-tx-0-21swapper/019:43:550
28405060,0irq/26-eth0-rx-24244-21ssh23:07:413
27160,0ktimersoftd/20-21swapper/221:56:422
249960,0watchdog/25669-21cut23:15:312
249960,0watchdog/20-21swapper/222:34:032
249960,0watchdog/20-21swapper/222:01:592
249960,0watchdog/20-21swapper/221:45:272
249960,0watchdog/20-21swapper/221:36:312
249960,0watchdog/20-21swapper/221:13:072
124089960,0cyclictest0-21swapper/121:45:271
124079960,0cyclictest0-21swapper/023:13:060
124079960,0cyclictest0-21swapper/021:46:040
124079960,0cyclictest0-21swapper/021:41:400
124079960,0cyclictest0-21swapper/019:45:280
8225050,0irq/24-0000:00:16405-21cat20:20:122
4150,0ktimersoftd/00-21swapper/021:04:180
339950,0migration/38890-21munin-node21:15:373
339950,0migration/35709-21timerandwakeup23:55:373
339950,0migration/33814-21irqrtprio21:35:293
339950,0migration/33313-21aten_r1power_vo20:10:273
339950,0migration/332079-21kthreadcore19:40:313
339950,0migration/330557-21kthreadcore23:10:293
339950,0migration/328970-21timerandwakeup20:25:353
339950,0migration/325795-21latency_hist18:50:123
339950,0migration/320082-21munin-node21:25:323
339950,0migration/319481-21kthreadcore19:30:323
339950,0migration/316158-21taskset22:20:423
339950,0migration/311701-21idleruntime20:40:263
339950,0migration/310393-21taskset20:37:453
28415050,0irq/27-eth0-tx-351-21ssh21:50:472
28415050,0irq/27-eth0-tx-32114-21kthreadcore21:30:302
28415050,0irq/27-eth0-tx-24191-21ssh23:07:322
28415050,0irq/27-eth0-tx-21273-21munin-node19:35:300
28415050,0irq/27-eth0-tx-15202-21munin-node00:05:262
28415050,0irq/27-eth0-tx-13600-21munin-node19:05:320
28415050,0irq/27-eth0-tx-10622-21diskmemload21:22:032
28415050,0irq/27-eth0-tx-0-21swapper/223:55:482
28415050,0irq/27-eth0-tx-0-21swapper/223:46:522
28415050,0irq/27-eth0-tx-0-21swapper/222:16:132
28415050,0irq/27-eth0-tx-0-21swapper/221:17:182
28415050,0irq/27-eth0-tx-0-21swapper/220:46:082
28415050,0irq/27-eth0-tx-0-21swapper/200:03:392
28415050,0irq/27-eth0-tx-0-21swapper/020:05:250
28415050,0irq/27-eth0-tx-0-21swapper/019:55:280
28415050,0irq/27-eth0-tx-0-21swapper/019:15:240
28415050,0irq/27-eth0-tx-0-21swapper/018:45:350
28405050,0irq/26-eth0-rx-8004-21ssh22:57:113
28405050,0irq/26-eth0-rx-735-21grep21:10:363
28405050,0irq/26-eth0-rx-32750-21smartctl20:50:343
28405050,0irq/26-eth0-rx-1272-21ssh21:32:353
28405050,0irq/26-eth0-rx-10380-21kthreadcore19:50:283
28405050,0irq/26-eth0-rx-0-21swapper/323:42:053
28405050,0irq/26-eth0-rx-0-21swapper/323:38:093
28405050,0irq/26-eth0-rx-0-21swapper/323:30:323
28405050,0irq/26-eth0-rx-0-21swapper/323:30:093
28405050,0irq/26-eth0-rx-0-21swapper/323:20:333
28405050,0irq/26-eth0-rx-0-21swapper/323:15:183
28405050,0irq/26-eth0-rx-0-21swapper/323:01:493
28405050,0irq/26-eth0-rx-0-21swapper/322:05:323
28405050,0irq/26-eth0-rx-0-21swapper/322:04:443
28405050,0irq/26-eth0-rx-0-21swapper/321:52:093
28405050,0irq/26-eth0-rx-0-21swapper/321:47:163
28405050,0irq/26-eth0-rx-0-21swapper/321:40:323
28405050,0irq/26-eth0-rx-0-21swapper/321:05:123
28405050,0irq/26-eth0-rx-0-21swapper/320:55:203
28405050,0irq/26-eth0-rx-0-21swapper/320:45:353
28405050,0irq/26-eth0-rx-0-21swapper/300:05:533
259950,0migration/25687-21ntp_kernel_err18:55:322
249950,0watchdog/20-21swapper/222:48:112
249950,0watchdog/20-21swapper/222:40:352
249950,0watchdog/20-21swapper/222:05:272
249950,0watchdog/20-21swapper/221:09:272
249950,0watchdog/20-21swapper/220:40:512
249950,0watchdog/20-21swapper/218:40:352
179950,0migration/1685-21turbostat.cron23:15:111
149950,0migration/06575-21kthreadcore20:10:320
149950,0migration/031318-21missed_timers18:50:300
149950,0migration/022033-21munin-node23:45:350
149950,0migration/021525-21munin-node19:10:260
149950,0migration/019497-21kthreadcore22:25:320
149950,0migration/011113-21kthreadcore23:20:320
149950,0migration/011007-21kthreadcore19:50:280
149950,0migration/010644-21sh21:00:070
124089953,0cyclictest3620-21kworker/1:023:23:481
124079955,0cyclictest0-21swapper/022:23:360
124079955,0cyclictest0-21swapper/021:23:320
124079955,0cyclictest0-21swapper/021:15:480
124079955,0cyclictest0-21swapper/021:10:460
124079954,0cyclictest26416-21grep23:30:290
124079950,0cyclictest0-21swapper/023:26:490
124079950,0cyclictest0-21swapper/021:52:510
124079950,0cyclictest0-21swapper/021:34:150
124079950,0cyclictest0-21swapper/021:26:130
124079950,0cyclictest0-21swapper/020:50:540
339940,0migration/33611-21kthreadcore18:55:303
339940,0migration/331564-21runrttasks22:30:333
339940,0migration/320370-21aten_r1power_po19:10:223
339940,0migration/319502-21kthreadcore19:55:323
339940,0migration/316276-21diskmemload22:41:513
339940,0migration/314731-21irqcore19:05:273
339940,0migration/314619-21kthreadcore21:20:293
339940,0migration/311562-21kthreadcore19:00:293
339940,0migration/310815-21kthreadcore19:25:293
28415040,0irq/27-eth0-tx-22275-21telnet20:00:240
28415040,0irq/27-eth0-tx-12115-21irqrtprio20:40:270
28415040,0irq/27-eth0-tx-11984-21iostat21:00:282
28415040,0irq/27-eth0-tx-0-21swapper/019:25:230
28415040,0irq/27-eth0-tx-0-21swapper/018:43:520
28405040,0irq/26-eth0-rx-0-21swapper/322:51:493
28405040,0irq/26-eth0-rx-0-21swapper/322:46:393
28405040,0irq/26-eth0-rx-0-21swapper/322:16:433
28405040,0irq/26-eth0-rx-0-21swapper/322:10:273
28405040,0irq/26-eth0-rx-0-21swapper/320:15:293
28405040,0irq/26-eth0-rx-0-21swapper/320:05:243
28405040,0irq/26-eth0-rx-0-21swapper/319:35:303
28405040,0irq/26-eth0-rx-0-21swapper/318:50:253
28405040,0irq/26-eth0-rx-0-21swapper/300:01:353
259940,0migration/29715-21kthreadcore19:00:292
259940,0migration/231740-21kthreadcore20:05:302
259940,0migration/227883-21kthreadcore18:50:282
259940,0migration/227354-21munin-node19:15:262
259940,0migration/224252-21kthreadcore19:10:272
249940,0watchdog/227919-21python20:00:352
249940,0watchdog/21995-21kthreadcore20:30:312
249940,0watchdog/20-21swapper/220:35:312
249940,0watchdog/20-21swapper/220:20:272
249940,0watchdog/20-21swapper/220:10:552
249940,0watchdog/20-21swapper/219:55:272
249940,0watchdog/20-21swapper/219:41:312
249940,0watchdog/20-21swapper/219:25:232
249940,0watchdog/20-21swapper/219:20:392
249940,0watchdog/20-21swapper/219:05:512
19641243,0sleep13110-21irqbalance19:05:571
179940,0migration/18708-21kthreadcore19:00:281
179940,0migration/16158-21kthreadcore20:55:301
179940,0migration/15628-21kthreadcore23:15:311
179940,0migration/128937-21kthreadcore21:10:291
179940,0migration/128370-21kthreadcore23:30:291
179940,0migration/127607-21missed_timers20:00:321
179940,0migration/124578-21kthreadcore19:35:301
179940,0migration/122664-21runrttasks20:21:321
179940,0migration/120695-21kthreadcore21:25:321
179940,0migration/120694-21kthreadcore23:25:331
169940,0watchdog/17816-21ssh22:56:391
169940,0watchdog/13421-21fschecks_time21:15:271
169940,0watchdog/131549-21cut19:40:311
169940,0watchdog/115249-21needreboot20:15:311
169940,0watchdog/111751-21if_err_eth121:20:271
169940,0watchdog/10-21swapper/123:40:311
169940,0watchdog/10-21swapper/123:05:191
169940,0watchdog/10-21swapper/121:35:551
149940,0migration/03643-21kthreadcore23:55:310
149940,0migration/028261-21kthreadcore22:10:300
149940,0migration/025186-21kthreadcore20:25:290
149940,0migration/022157-21kthreadcore23:05:320
149940,0migration/02184-21ksmtuned21:55:210
149940,0migration/01490-21latency_hist22:35:120
124109943,0cyclictest6445-21snmpd19:51:582
124109943,0cyclictest6445-21snmpd18:49:582
124109943,0cyclictest3425-21lldpd19:47:062
124109943,0cyclictest3425-21lldpd19:36:062
124089944,0cyclictest0-21swapper/122:00:351
124089943,0cyclictest8810-21awk19:25:271
124089943,0cyclictest8310-21cut20:10:321
124089943,0cyclictest3620-21kworker/1:023:57:321
124089943,0cyclictest3620-21kworker/1:023:46:321
124089943,0cyclictest3620-21kworker/1:023:36:221
124089943,0cyclictest3620-21kworker/1:023:00:241
124089943,0cyclictest3620-21kworker/1:022:44:301
124089943,0cyclictest3620-21kworker/1:022:25:421
124089943,0cyclictest3620-21kworker/1:022:10:141
124089943,0cyclictest3620-21kworker/1:022:09:281
124089943,0cyclictest3620-21kworker/1:020:48:421
124089943,0cyclictest3620-21kworker/1:000:00:341
124089943,0cyclictest3425-21lldpd18:41:361
124089943,0cyclictest3110-21irqbalance21:56:561
124089943,0cyclictest2869-21awk22:15:321
124089943,0cyclictest27865-21awk20:50:271
124089943,0cyclictest27446-21awk21:50:241
124089943,0cyclictest26782-21grep22:30:271
124089943,0cyclictest26584-21awk22:50:251
124089943,0cyclictest2498-21awk22:35:291
124089943,0cyclictest18689-21awk22:45:281
124089943,0cyclictest16084-21awk00:05:281
124089943,0cyclictest15497-21awk19:30:291
124089943,0cyclictest11833-21awk21:00:271
124089943,0cyclictest11622-21awk21:40:301
124089943,0cyclictest10933-21awk22:20:271
124089943,0cyclictest1062-21awk18:55:281
124089940,0cyclictest0-21swapper/121:31:441
124079944,0cyclictest0-21swapper/023:15:470
124079944,0cyclictest0-21swapper/023:02:230
124079944,0cyclictest0-21swapper/022:54:390
124079944,0cyclictest0-21swapper/022:47:150
124079944,0cyclictest0-21swapper/022:33:220
124079944,0cyclictest0-21swapper/022:06:440
124079943,0cyclictest8450-21awk00:00:280
124079943,0cyclictest6445-21snmpd23:40:290
124079943,0cyclictest6445-21snmpd23:35:140
124079943,0cyclictest5071-21awk20:35:300
124079943,0cyclictest3425-21lldpd00:06:180
124079943,0cyclictest2173-21ps19:20:310
124079943,0cyclictest20030-21awk21:05:280
124079943,0cyclictest15584-21awk19:30:290
124079943,0cyclictest1116-21awk18:55:280
124079943,0cyclictest11132-21awk22:00:260
124079943,0cyclictest10348-21awk21:40:120
28415030,0irq/27-eth0-tx-0-21swapper/020:19:070
28415030,0irq/27-eth0-tx-0-21swapper/019:00:240
28405030,0irq/26-eth0-rx-0-21swapper/320:30:253
28405030,0irq/26-eth0-rx-0-21swapper/319:15:243
28405030,0irq/26-eth0-rx-0-21swapper/318:44:313
179930,0migration/19051-21aten_r1power_po19:50:231
169930,0watchdog/10-21swapper/119:45:231
169930,0watchdog/10-21swapper/118:50:231
124109932,0cyclictest24090-21awk20:25:272
124109932,0cyclictest15394-21hddtemp_smartct19:30:282
124089933,0cyclictest3620-21kworker/1:023:50:121
124089933,0cyclictest3620-21kworker/1:021:05:241
124089933,0cyclictest3620-21kworker/1:020:40:261
124089933,0cyclictest3620-21kworker/1:020:35:141
124089933,0cyclictest3620-21kworker/1:020:25:221
124089933,0cyclictest3620-21kworker/1:020:05:141
124089933,0cyclictest3620-21kworker/1:019:15:201
124089933,0cyclictest3620-21kworker/1:019:10:201
124089933,0cyclictest3620-21kworker/1:018:45:141
124089932,0cyclictest30695-21awk20:30:281
124089932,0cyclictest2018-21awk19:20:301
124089932,0cyclictest16449-21awk19:55:291
124079932,0cyclictest6445-21snmpd22:59:360
124079932,0cyclictest6445-21snmpd22:41:290
124079932,0cyclictest6445-21snmpd20:34:280
124079932,0cyclictest3971-21tr22:15:340
124079932,0cyclictest19794-21awk20:45:290
124079930,0cyclictest24521-21cpu23:50:270
28405020,0irq/26-eth0-rx-0-21swapper/320:01:313
28405020,0irq/26-eth0-rx-0-21swapper/319:45:143
28405020,0irq/26-eth0-rx-0-21swapper/319:20:183
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional