You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 02:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Thu Nov 20, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2590426766,0sleep10-21swapper/118:37:331
2581823828,0sleep30-21swapper/318:36:313
2602223422,0sleep00-21swapper/018:38:560
2596623127,0sleep20-21swapper/218:38:182
35160,0ktimersoftd/30-21swapper/320:09:593
339960,0migration/34995-21sh23:48:283
339960,0migration/323153-21sh22:19:003
28415060,0irq/27-eth0-tx-4679-21ssh23:05:052
28415060,0irq/27-eth0-tx-31642-21aten_r1power_po19:20:020
28415060,0irq/27-eth0-tx-15856-21ssh21:33:012
28415060,0irq/27-eth0-tx-0-21swapper/223:34:092
28415060,0irq/27-eth0-tx-0-21swapper/222:54:182
28415060,0irq/27-eth0-tx-0-21swapper/222:45:342
28415060,0irq/27-eth0-tx-0-21swapper/222:43:332
28415060,0irq/27-eth0-tx-0-21swapper/222:15:212
28415060,0irq/27-eth0-tx-0-21swapper/222:08:162
28415060,0irq/27-eth0-tx-0-21swapper/221:50:532
28415060,0irq/27-eth0-tx-0-21swapper/221:46:312
28415060,0irq/27-eth0-tx-0-21swapper/221:20:382
28415060,0irq/27-eth0-tx-0-21swapper/221:12:542
28415060,0irq/27-eth0-tx-0-21swapper/221:07:502
28415060,0irq/27-eth0-tx-0-21swapper/220:46:552
28415060,0irq/27-eth0-tx-0-21swapper/200:09:142
28415060,0irq/27-eth0-tx-0-21swapper/200:00:372
28405060,0irq/26-eth0-rx-32434-21ssh20:43:413
28405060,0irq/26-eth0-rx-28362-21tr22:00:003
28405060,0irq/26-eth0-rx-12681-21cut21:09:583
28405060,0irq/26-eth0-rx-0-21swapper/322:49:153
261609966,0cyclictest24853-21diskmemload21:08:111
261609960,0cyclictest0-21swapper/121:43:021
261599960,0cyclictest0-21swapper/023:33:010
249960,0watchdog/221787-21ssh22:57:202
249960,0watchdog/219701-21ssh23:55:202
149960,0migration/0395-21latency_hist20:44:410
149960,0migration/024801-21sh21:19:380
149960,0migration/024585-21kthreadcore23:20:010
149960,0migration/022034-21latency21:34:580
339950,0migration/38180-21latency_hist21:49:413
339950,0migration/3776-21aten_r1power_vo21:24:533
339950,0migration/37509-21kthreadcore20:24:563
339950,0migration/35617-21kthreadcore20:00:013
339950,0migration/35078-21runrttasks23:27:113
339950,0migration/331534-21sh21:00:333
339950,0migration/331089-21timerwakeupswit21:20:053
339950,0migration/330138-21taskset20:15:423
339950,0migration/329714-21taskset19:06:293
339950,0migration/327860-21taskset18:52:203
339950,0migration/325953-21munin-node21:39:553
339950,0migration/324505-21kthreadcore19:50:013
339950,0migration/319149-21kthreadcore23:35:003
339950,0migration/314843-21kthreadcore20:04:593
339950,0migration/314531-21taskset23:14:493
339950,0migration/310625-21kthreadcore22:49:573
339950,0migration/310297-21kthreadcore22:29:593
28415050,0irq/27-eth0-tx-25167-21kthreadcore23:00:002
28415050,0irq/27-eth0-tx-24061-21munin-node20:15:040
28415050,0irq/27-eth0-tx-17189-21fschecks_count21:34:542
28415050,0irq/27-eth0-tx-0-21swapper/223:52:182
28415050,0irq/27-eth0-tx-0-21swapper/223:45:132
28415050,0irq/27-eth0-tx-0-21swapper/223:43:042
28415050,0irq/27-eth0-tx-0-21swapper/223:35:062
28415050,0irq/27-eth0-tx-0-21swapper/223:24:272
28415050,0irq/27-eth0-tx-0-21swapper/223:19:342
28415050,0irq/27-eth0-tx-0-21swapper/222:34:512
28415050,0irq/27-eth0-tx-0-21swapper/222:27:402
28415050,0irq/27-eth0-tx-0-21swapper/222:21:062
28415050,0irq/27-eth0-tx-0-21swapper/222:03:352
28415050,0irq/27-eth0-tx-0-21swapper/221:41:062
28415050,0irq/27-eth0-tx-0-21swapper/221:28:462
28415050,0irq/27-eth0-tx-0-21swapper/221:15:322
28415050,0irq/27-eth0-tx-0-21swapper/220:57:542
28415050,0irq/27-eth0-tx-0-21swapper/220:51:202
28415050,0irq/27-eth0-tx-0-21swapper/019:52:540
28405050,0irq/26-eth0-rx-6054-21sensors_temp21:45:043
28405050,0irq/26-eth0-rx-25547-21cut22:39:583
28405050,0irq/26-eth0-rx-22564-21aten_r1power_en23:19:553
28405050,0irq/26-eth0-rx-22165-21kthreadcore21:14:563
28405050,0irq/26-eth0-rx-21190-21kthreadcore21:34:583
28405050,0irq/26-eth0-rx-18852-21kthreadcore22:55:013
28405050,0irq/26-eth0-rx-13308-21open_inodes22:10:043
28405050,0irq/26-eth0-rx-0-21swapper/323:54:523
28405050,0irq/26-eth0-rx-0-21swapper/323:50:453
28405050,0irq/26-eth0-rx-0-21swapper/323:40:493
28405050,0irq/26-eth0-rx-0-21swapper/323:33:413
28405050,0irq/26-eth0-rx-0-21swapper/323:09:563
28405050,0irq/26-eth0-rx-0-21swapper/323:00:013
28405050,0irq/26-eth0-rx-0-21swapper/322:36:413
28405050,0irq/26-eth0-rx-0-21swapper/321:54:573
28405050,0irq/26-eth0-rx-0-21swapper/321:30:013
28405050,0irq/26-eth0-rx-0-21swapper/320:55:023
28405050,0irq/26-eth0-rx-0-21swapper/320:49:583
28405050,0irq/26-eth0-rx-0-21swapper/320:48:213
28405050,0irq/26-eth0-rx-0-21swapper/320:29:573
28405050,0irq/26-eth0-rx-0-21swapper/300:05:003
261619955,0cyclictest0-21swapper/221:57:292
261619955,0cyclictest0-21swapper/220:35:022
261619950,0cyclictest0-21swapper/222:31:422
261619950,0cyclictest0-21swapper/222:11:502
261619950,0cyclictest0-21swapper/221:01:222
261609955,0cyclictest24853-21diskmemload21:28:501
261609955,0cyclictest24853-21diskmemload20:59:401
261609955,0cyclictest0-21swapper/122:21:021
261609955,0cyclictest0-21swapper/121:17:011
261609950,0cyclictest0-21swapper/122:11:151
261599955,0cyclictest0-21swapper/021:47:320
261599950,0cyclictest0-21swapper/023:52:240
261599950,0cyclictest0-21swapper/023:39:100
261599950,0cyclictest0-21swapper/023:14:530
261599950,0cyclictest0-21swapper/021:05:080
259950,0migration/228238-21latency_hist18:54:422
259950,0migration/210497-21turbostat20:04:412
19150,0ktimersoftd/10-21swapper/100:04:271
169950,0watchdog/17292-21cut20:24:561
169950,0watchdog/127722-21cat20:40:001
169950,0watchdog/12733-21netstat19:55:001
169950,0watchdog/124853-21diskmemload23:02:081
169950,0watchdog/124035-21ssh21:59:441
169950,0watchdog/10-21swapper/123:45:441
169950,0watchdog/10-21swapper/123:15:561
169950,0watchdog/10-21swapper/119:45:001
149950,0migration/08463-21kthreadcore22:49:560
149950,0migration/07353-21sh21:26:270
149950,0migration/05290-21taskset23:06:540
149950,0migration/05062-21sh22:45:190
149950,0migration/04033-21kthreadcore20:44:580
149950,0migration/03681-21switchtime23:45:050
149950,0migration/031185-21fschecks_count20:19:570
149950,0migration/029595-21kthreadcore20:59:580
149950,0migration/029235-21kthreadcore22:00:000
149950,0migration/025161-21aten_r1power_en21:19:530
149950,0migration/024481-21kthreadcore23:39:580
149950,0migration/023479-21runrttasks22:39:530
149950,0migration/022505-21taskset19:46:020
149950,0migration/020490-21taskset23:57:510
149950,0migration/017219-21irqcore21:54:560
149950,0migration/015857-21sh21:33:010
149950,0migration/015203-21kthreadcore20:29:590
149950,0migration/013386-21unixbench_singl22:30:050
8225040,0irq/24-0000:00:0-21swapper/219:14:412
339940,0migration/36551-21needreboot21:05:003
339940,0migration/330683-21latency_hist19:14:413
339940,0migration/32693-21kthreadcore22:25:013
28415040,0irq/27-eth0-tx-0-21swapper/223:10:012
28415040,0irq/27-eth0-tx-0-21swapper/020:34:530
28415040,0irq/27-eth0-tx-0-21swapper/020:09:540
28415040,0irq/27-eth0-tx-0-21swapper/019:31:590
28405040,0irq/26-eth0-rx-0-21swapper/323:59:513
28405040,0irq/26-eth0-rx-0-21swapper/323:05:403
28405040,0irq/26-eth0-rx-0-21swapper/322:19:533
28405040,0irq/26-eth0-rx-0-21swapper/322:04:543
28405040,0irq/26-eth0-rx-0-21swapper/319:39:553
261619943,0cyclictest5921-21awk20:24:542
261619943,0cyclictest3425-21lldpd20:14:552
261619943,0cyclictest3425-21lldpd20:05:552
261619943,0cyclictest31505-21awk23:24:572
261619943,0cyclictest24050-21awk19:50:002
261619943,0cyclictest22830-21sadc19:49:412
261619943,0cyclictest1913-21awk19:30:142
261619943,0cyclictest12728-21awk20:29:572
261619940,0cyclictest0-21swapper/219:23:002
261609944,0cyclictest0-21swapper/123:57:021
261609944,0cyclictest0-21swapper/122:35:031
261609944,0cyclictest0-21swapper/122:25:171
261609944,0cyclictest0-21swapper/122:15:491
261609943,0cyclictest9174-21awk21:49:561
261609943,0cyclictest6445-21snmpd23:49:471
261609943,0cyclictest6445-21snmpd20:09:471
261609943,0cyclictest6445-21snmpd18:51:021
261609943,0cyclictest3425-21lldpd21:22:271
261609943,0cyclictest3425-21lldpd20:37:551
261609943,0cyclictest32137-21awk22:44:571
261609943,0cyclictest31684-21awk20:19:591
261609943,0cyclictest27428-21sadc18:49:421
261609943,0cyclictest24992-21awk20:14:561
261609943,0cyclictest1571-21awk20:44:561
261609943,0cyclictest10566-21awk19:39:581
261599944,0cyclictest0-21swapper/023:26:330
261599944,0cyclictest0-21swapper/021:51:100
261599943,0cyclictest8975-21grep19:35:050
261599943,0cyclictest6445-21snmpd23:13:020
261599943,0cyclictest4892-21awk19:59:590
261599943,0cyclictest473-21awk22:04:560
261599943,0cyclictest4333-21sadc00:09:410
261599943,0cyclictest22583-21sadc22:59:410
261599943,0cyclictest11593-21awk20:04:570
259940,0migration/29678-21latency_hist19:39:412
259940,0migration/230743-21sh20:40:032
259940,0migration/22614-21kthreadcore20:20:012
249940,0watchdog/20-21swapper/219:55:002
179940,0migration/13993-21sendmail_mailtr23:25:031
179940,0migration/125734-21kthreadcore23:20:011
179940,0migration/124096-21kthreadcore23:39:581
169940,0watchdog/130902-21grep00:05:001
169940,0watchdog/123305-21aten2.4-expect22:39:521
169940,0watchdog/120454-21open_files22:55:041
169940,0watchdog/116138-21ssh21:12:321
169940,0watchdog/115608-21ssh21:32:161
169940,0watchdog/10-21swapper/123:35:001
169940,0watchdog/10-21swapper/123:11:281
169940,0watchdog/10-21swapper/122:49:521
169940,0watchdog/10-21swapper/122:33:361
169940,0watchdog/10-21swapper/121:54:441
169940,0watchdog/10-21swapper/121:39:041
169940,0watchdog/10-21swapper/120:49:561
169940,0watchdog/10-21swapper/120:00:001
169940,0watchdog/10-21swapper/119:38:441
149940,0migration/08871-21irqcore22:10:000
149940,0migration/028866-21timerwakeupswit23:00:060
149940,0migration/025815-21kthreadcore22:19:570
339930,0migration/331879-21taskset19:21:343
28415030,0irq/27-eth0-tx-0-21swapper/019:39:550
28415030,0irq/27-eth0-tx-0-21swapper/019:24:480
28405030,0irq/26-eth0-rx-0-21swapper/320:34:473
28405030,0irq/26-eth0-rx-0-21swapper/320:21:313
28405030,0irq/26-eth0-rx-0-21swapper/319:54:533
28405030,0irq/26-eth0-rx-0-21swapper/319:44:493
28405030,0irq/26-eth0-rx-0-21swapper/319:31:093
28405030,0irq/26-eth0-rx-0-21swapper/319:27:043
28405030,0irq/26-eth0-rx-0-21swapper/319:14:443
28405030,0irq/26-eth0-rx-0-21swapper/319:01:273
28405030,0irq/26-eth0-rx-0-21swapper/318:44:343
261619933,0cyclictest3425-21lldpd19:08:542
261619933,0cyclictest3425-21lldpd18:45:542
261619933,0cyclictest3425-21lldpd18:40:242
261619932,0cyclictest4012-21awk19:34:572
261619932,0cyclictest3425-21lldpd18:50:242
261619932,0cyclictest32508-21aten2.4-expect19:24:482
261619930,0cyclictest18673-21irqstats20:09:592
261609933,0cyclictest0-21swapper/119:19:431
261609932,0cyclictest31289-21df_inode23:04:531
261609932,0cyclictest1671-21tune2fs19:29:581
261609932,0cyclictest12632-21awk20:29:571
261609932,0cyclictest12508-21cat20:04:581
261599933,0cyclictest0-21swapper/020:29:020
261599933,0cyclictest0-21swapper/018:53:290
261599932,0cyclictest6445-21snmpd19:18:300
261599932,0cyclictest6445-21snmpd19:07:320
261599932,0cyclictest6445-21snmpd18:48:240
261599932,0cyclictest3425-21lldpd22:14:580
261599932,0cyclictest30594-21awk19:54:560
261599932,0cyclictest28715-21sadc18:59:410
261599932,0cyclictest24974-21grep21:39:510
261599932,0cyclictest22237-21awk00:00:000
261599932,0cyclictest21532-21sh22:35:200
261599932,0cyclictest16702-21sh20:54:090
261599930,0cyclictest11861-21tr21:09:570
249930,0watchdog/20-21swapper/219:10:282
249930,0watchdog/20-21swapper/219:01:402
169930,0watchdog/130274-21taskset19:10:281
169930,0watchdog/10-21swapper/123:29:521
169930,0watchdog/10-21swapper/122:04:561
169930,0watchdog/10-21swapper/121:44:561
169930,0watchdog/10-21swapper/120:59:481
169930,0watchdog/10-21swapper/119:49:561
169930,0watchdog/10-21swapper/119:25:081
169930,0watchdog/10-21swapper/119:18:561
169930,0watchdog/10-21swapper/119:04:441
169930,0watchdog/10-21swapper/119:00:121
169930,0watchdog/10-21swapper/118:55:121
169930,0watchdog/10-21swapper/118:40:361
28405020,0irq/26-eth0-rx-0-21swapper/319:34:463
28405020,0irq/26-eth0-rx-0-21swapper/318:55:183
28405020,0irq/26-eth0-rx-0-21swapper/318:44:453
261599921,0cyclictest3411-21lldpd18:39:550
261599921,0cyclictest31564-21cut22:24:410
261599921,0cyclictest29997-21latency_hist19:09:410
261599921,0cyclictest28712-21latency_hist18:59:410
261599920,0cyclictest0-21swapper/020:54:450
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional