You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-20 - 18:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Tue May 20, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2576925041,0sleep20-21swapper/206:44:482
291350440,0irq/25-eth10-21swapper/306:46:073
2570822925,0sleep10-21swapper/106:44:071
2561222817,0sleep00-21swapper/006:42:580
28415070,0irq/27-eth0-tx-0-21swapper/211:46:502
28415070,0irq/27-eth0-tx-0-21swapper/208:58:192
249970,0watchdog/226121-21ssh10:43:062
339960,0migration/35835-21sh09:28:133
339960,0migration/329781-21sh09:45:253
339960,0migration/323917-21needreboot12:01:533
339960,0migration/320756-21cpuspeed_turbos07:06:453
28415060,0irq/27-eth0-tx-28846-21ssh09:18:452
28415060,0irq/27-eth0-tx-25028-21munin-node08:31:490
28415060,0irq/27-eth0-tx-0-21swapper/211:57:172
28415060,0irq/27-eth0-tx-0-21swapper/211:54:392
28415060,0irq/27-eth0-tx-0-21swapper/211:42:522
28415060,0irq/27-eth0-tx-0-21swapper/211:36:522
28415060,0irq/27-eth0-tx-0-21swapper/211:26:402
28415060,0irq/27-eth0-tx-0-21swapper/211:22:092
28415060,0irq/27-eth0-tx-0-21swapper/211:18:302
28415060,0irq/27-eth0-tx-0-21swapper/211:15:252
28415060,0irq/27-eth0-tx-0-21swapper/211:06:262
28415060,0irq/27-eth0-tx-0-21swapper/210:01:212
28415060,0irq/27-eth0-tx-0-21swapper/209:08:332
28415060,0irq/27-eth0-tx-0-21swapper/208:54:102
28415060,0irq/27-eth0-tx-0-21swapper/008:01:450
28405060,0irq/26-eth0-rx-25100-21rm10:41:513
28405060,0irq/26-eth0-rx-17243-21port_sshpriv10:36:523
28405060,0irq/26-eth0-rx-11459-21iostat_ios11:56:483
28405060,0irq/26-eth0-rx-0-21swapper/310:28:153
249960,0watchdog/23605-21kthreadcore09:01:502
249960,0watchdog/22952-21df11:31:452
249960,0watchdog/20-21swapper/210:57:582
249960,0watchdog/20-21swapper/209:41:502
249960,0watchdog/20-21swapper/209:32:582
249960,0watchdog/20-21swapper/207:56:382
149960,0migration/03461-21munin-node07:41:490
149960,0migration/019538-21irqrtprio08:26:480
149960,0migration/016556-21munin-node08:16:520
4150,0ktimersoftd/00-21swapper/010:22:220
4150,0ktimersoftd/00-21swapper/009:39:510
339950,0migration/39436-21kthreadcore09:31:443
339950,0migration/38337-21kthreadcore08:46:493
339950,0migration/37593-21taskset07:17:313
339950,0migration/35336-21kthreadcore10:31:523
339950,0migration/329994-21idleruntime08:01:483
339950,0migration/329377-21sh08:58:343
339950,0migration/329320-21kthreadcore07:11:493
339950,0migration/32804-21kthreadcore08:41:533
339950,0migration/326343-21sh11:25:273
339950,0migration/32598-21diskmemload11:11:103
339950,0migration/322491-21kthreadcore08:26:503
339950,0migration/321693-21irqrtprio10:21:483
339950,0migration/319096-21latency07:01:503
339950,0migration/318933-21kthreadcore07:51:513
339950,0migration/315892-21munin-node09:36:483
339950,0migration/312393-21kthreadcore06:56:503
339950,0migration/311901-21ntp_offset09:06:553
339950,0migration/310856-21aten_r1power_po11:36:443
28415050,0irq/27-eth0-tx-7173-21munin-node06:56:510
28415050,0irq/27-eth0-tx-26011-21pluginstate10:21:522
28415050,0irq/27-eth0-tx-0-21swapper/212:11:442
28415050,0irq/27-eth0-tx-0-21swapper/210:51:332
28415050,0irq/27-eth0-tx-0-21swapper/210:46:442
28415050,0irq/27-eth0-tx-0-21swapper/210:16:392
28415050,0irq/27-eth0-tx-0-21swapper/210:12:502
28415050,0irq/27-eth0-tx-0-21swapper/210:01:392
28415050,0irq/27-eth0-tx-0-21swapper/209:51:472
28415050,0irq/27-eth0-tx-0-21swapper/209:28:152
28415050,0irq/27-eth0-tx-0-21swapper/209:25:102
28415050,0irq/27-eth0-tx-0-21swapper/209:11:582
28415050,0irq/27-eth0-tx-0-21swapper/008:06:470
28415050,0irq/27-eth0-tx-0-21swapper/007:01:540
28405050,0irq/26-eth0-rx-5003-21cut11:31:503
28405050,0irq/26-eth0-rx-31565-21irqrtprio09:46:503
28405050,0irq/26-eth0-rx-30168-21kthreadcore11:26:493
28405050,0irq/26-eth0-rx-29970-21aten_r1power_en09:21:453
28405050,0irq/26-eth0-rx-2735-21tune2fs12:11:453
28405050,0irq/26-eth0-rx-21068-21taskset07:28:403
28405050,0irq/26-eth0-rx-19709-21ssh10:20:103
28405050,0irq/26-eth0-rx-14680-21sed08:51:313
28405050,0irq/26-eth0-rx-10919-21ssh10:56:203
28405050,0irq/26-eth0-rx-0-21swapper/311:01:503
28405050,0irq/26-eth0-rx-0-21swapper/310:58:063
28405050,0irq/26-eth0-rx-0-21swapper/310:13:153
28405050,0irq/26-eth0-rx-0-21swapper/310:05:433
28405050,0irq/26-eth0-rx-0-21swapper/309:16:523
27150,0ktimersoftd/20-21swapper/208:21:352
259919954,0cyclictest3322-21kworker/2:210:26:512
259919954,0cyclictest3322-21kworker/2:210:06:492
259909955,0cyclictest6520-21diskmemload12:06:441
259909955,0cyclictest0-21swapper/109:57:461
259909954,0cyclictest11494-21kworker/1:011:32:391
259909954,0cyclictest11494-21kworker/1:010:31:531
259909954,0cyclictest11494-21kworker/1:010:07:491
259909954,0cyclictest11494-21kworker/1:009:10:431
259909954,0cyclictest11494-21kworker/1:007:21:511
259909950,0cyclictest0-21swapper/110:50:541
259909950,0cyclictest0-21swapper/109:39:201
259950,0migration/2917-21munin-node08:41:512
259950,0migration/27072-21latency_hist06:56:312
259950,0migration/228152-21latency_hist07:36:312
259950,0migration/214006-21sh08:50:042
259950,0migration/212252-21kthreadcore08:16:482
259950,0migration/210013-21idleruntime07:46:472
249950,0watchdog/218476-21aten_r1power_en12:01:452
249950,0watchdog/20-21swapper/212:10:452
249950,0watchdog/20-21swapper/211:06:502
249950,0watchdog/20-21swapper/210:32:542
249950,0watchdog/20-21swapper/209:46:382
249950,0watchdog/20-21swapper/209:38:102
249950,0watchdog/20-21swapper/208:26:342
249950,0watchdog/20-21swapper/208:06:582
249950,0watchdog/20-21swapper/207:41:582
2081250,0sleep00-21swapper/011:51:270
179950,0migration/125819-21kthreadcore09:41:471
169950,0watchdog/122606-21cut07:06:501
149950,0migration/09792-21fschecks_time07:46:460
149950,0migration/09774-21taskset11:55:280
149950,0migration/09681-21kthreadcore08:46:490
149950,0migration/08231-21aten_r1power_en07:21:450
149950,0migration/07354-21cpuspeed_turbos09:31:400
149950,0migration/06988-21kthreadcore10:31:530
149950,0migration/06103-21kthreadcore11:11:490
149950,0migration/05754-21taskset09:27:590
149950,0migration/05329-21kthreadcore10:51:490
149950,0migration/04230-21taskset09:23:470
149950,0migration/03987-21sh10:09:580
149950,0migration/03221-21kthreadcore06:51:480
149950,0migration/031786-21kthreadcore09:46:500
149950,0migration/031252-21fschecks_count09:01:450
149950,0migration/029258-21kthreadcore06:46:490
149950,0migration/028120-21munin-node08:56:510
149950,0migration/027678-21sh10:02:350
149950,0migration/027523-21irqcore12:06:470
149950,0migration/026571-21kthreadcore07:31:520
149950,0migration/026477-21munin-node07:06:560
149950,0migration/025462-21kthreadcore07:56:490
149950,0migration/022860-21kthreadcore10:41:500
149950,0migration/022443-21kthreadcore12:01:520
149950,0migration/018989-21kthreadcore09:11:510
149950,0migration/018084-21sh11:40:260
149950,0migration/01637-21taskset11:29:330
149950,0migration/015894-21irqcore08:51:490
149950,0migration/014084-21kthreadcore11:16:490
149950,0migration/013706-21kthreadcore10:56:490
8225040,0irq/24-0000:00:14802-21cat07:26:312
339940,0migration/327805-21taskset07:34:323
339940,0migration/319368-21fschecks_time11:41:463
339940,0migration/315164-21kthreadcore08:16:493
28415040,0irq/27-eth0-tx-0-21swapper/007:20:180
28405040,0irq/26-eth0-rx-0-21swapper/312:06:493
28405040,0irq/26-eth0-rx-0-21swapper/311:51:533
28405040,0irq/26-eth0-rx-0-21swapper/311:51:073
28405040,0irq/26-eth0-rx-0-21swapper/311:17:053
28405040,0irq/26-eth0-rx-0-21swapper/311:11:493
28405040,0irq/26-eth0-rx-0-21swapper/310:47:173
28405040,0irq/26-eth0-rx-0-21swapper/310:09:043
28405040,0irq/26-eth0-rx-0-21swapper/309:58:303
28405040,0irq/26-eth0-rx-0-21swapper/309:52:323
28405040,0irq/26-eth0-rx-0-21swapper/309:14:223
28405040,0irq/26-eth0-rx-0-21swapper/309:01:493
259919944,0cyclictest3322-21kworker/2:210:36:372
259919944,0cyclictest3322-21kworker/2:208:38:052
259919944,0cyclictest3322-21kworker/2:208:01:312
259919944,0cyclictest3322-21kworker/2:207:21:312
259919944,0cyclictest3322-21kworker/2:207:16:312
259919944,0cyclictest3322-21kworker/2:206:46:312
259919943,0cyclictest3425-21lldpd08:11:592
259919943,0cyclictest16706-21awk07:51:482
259909944,0cyclictest5313-21ssh09:26:331
259909944,0cyclictest11494-21kworker/1:011:56:371
259909944,0cyclictest11494-21kworker/1:011:41:491
259909944,0cyclictest11494-21kworker/1:011:17:171
259909944,0cyclictest11494-21kworker/1:011:11:351
259909944,0cyclictest11494-21kworker/1:011:06:431
259909944,0cyclictest11494-21kworker/1:009:46:331
259909944,0cyclictest11494-21kworker/1:009:31:331
259909944,0cyclictest11494-21kworker/1:009:16:491
259909944,0cyclictest11494-21kworker/1:008:26:331
259909944,0cyclictest11494-21kworker/1:008:21:451
259909944,0cyclictest11494-21kworker/1:008:16:411
259909944,0cyclictest11494-21kworker/1:008:06:431
259909944,0cyclictest11494-21kworker/1:007:51:311
259909944,0cyclictest11494-21kworker/1:007:47:091
259909944,0cyclictest11494-21kworker/1:007:41:471
259909944,0cyclictest11494-21kworker/1:007:37:031
259909944,0cyclictest11494-21kworker/1:007:31:351
259909944,0cyclictest11494-21kworker/1:007:01:331
259909944,0cyclictest11494-21kworker/1:006:56:451
259909944,0cyclictest11494-21kworker/1:006:51:371
259909944,0cyclictest0-21swapper/110:51:321
259909943,0cyclictest3425-21lldpd08:46:371
259909943,0cyclictest3319-21awk11:51:461
259909943,0cyclictest3110-21irqbalance09:52:051
259909943,0cyclictest3110-21irqbalance08:41:461
259909943,0cyclictest3110-21irqbalance08:36:461
259909943,0cyclictest3110-21irqbalance08:11:561
259909943,0cyclictest3110-21irqbalance08:01:461
259909943,0cyclictest30833-21awk09:21:491
259909943,0cyclictest2937-21awk12:11:461
259909943,0cyclictest28038-21awk07:11:471
259909943,0cyclictest25570-21grep08:31:461
259909943,0cyclictest22240-21awk10:01:471
259909943,0cyclictest13616-21awk10:16:461
259909943,0cyclictest11739-21awk11:36:471
259909943,0cyclictest11494-21kworker/1:011:26:471
259909943,0cyclictest11494-21kworker/1:010:11:531
259899943,0cyclictest993-21awk08:41:510
259899943,0cyclictest9890-21sadc12:16:310
259899943,0cyclictest6445-21snmpd11:21:350
259899943,0cyclictest5762-21awk10:11:490
259899943,0cyclictest3583-21awk11:31:480
259899943,0cyclictest23140-21awk09:16:500
259940,0migration/225254-21kthreadcore07:31:522
259940,0migration/220647-21aten2.4-expect07:06:432
259940,0migration/218012-21kthreadcore07:01:492
249940,0watchdog/213251-21awk06:57:062
249940,0watchdog/20-21swapper/208:31:462
249940,0watchdog/20-21swapper/207:11:342
179940,0migration/1479-21kthreadcore09:01:491
179940,0migration/126747-21fschecks_time06:46:461
179940,0migration/124551-21munin-node11:21:521
179940,0migration/124288-21kthreadcore10:41:501
179940,0migration/123676-21munin-node08:56:481
179940,0migration/122877-21kthreadcore11:01:491
179940,0migration/120830-21sh09:13:121
179940,0migration/119399-21kthreadcore07:27:011
179940,0migration/117568-21taskset10:57:141
179940,0migration/115441-21fschecks_time08:51:471
179940,0migration/11481-21aten_r1power_fr07:16:431
169940,0watchdog/123091-21perl07:56:461
169940,0watchdog/10-21swapper/111:46:491
169940,0watchdog/10-21swapper/110:36:461
169940,0watchdog/10-21swapper/110:26:461
169940,0watchdog/10-21swapper/110:21:501
149940,0migration/032157-21kthreadcore07:36:520
149940,0migration/030858-21kthreadcore11:06:490
149940,0migration/020754-21kthreadcore07:51:510
149940,0migration/017307-21kthreadcore09:56:500
149940,0migration/017292-21runrttasks11:59:060
149940,0migration/015591-21kthreadcore10:36:490
35130,0ktimersoftd/30-21swapper/308:39:143
339930,0migration/39335-21aten_r1power_vo07:46:443
28405030,0irq/26-eth0-rx-0-21swapper/308:35:293
28405030,0irq/26-eth0-rx-0-21swapper/308:06:453
28405030,0irq/26-eth0-rx-0-21swapper/307:36:593
28405030,0irq/26-eth0-rx-0-21swapper/307:21:323
28405030,0irq/26-eth0-rx-0-21swapper/306:53:223
28405030,0irq/26-eth0-rx-0-21swapper/306:46:433
259909933,0cyclictest11494-21kworker/1:012:01:371
259899933,0cyclictest20114-21awk11:01:460
259899932,0cyclictest28444-21awk10:46:480
259899932,0cyclictest28053-21awk07:11:470
259899932,0cyclictest15781-21awk07:26:580
259899932,0cyclictest12128-21sensors09:06:570
259899931,0cyclictest0-21swapper/008:12:290
259899930,0cyclictest0-21swapper/009:41:450
149930,0migration/02565-21sh10:28:440
149930,0migration/012014-21sh09:52:080
4120,0ktimersoftd/031688-21munin-node08:36:430
35120,0ktimersoftd/38545-21users07:41:583
28405020,0irq/26-eth0-rx-0-21swapper/308:22:383
28405020,0irq/26-eth0-rx-0-21swapper/308:12:093
28405020,0irq/26-eth0-rx-0-21swapper/308:00:033
259899921,0cyclictest18627-21latency_hist11:41:310
259899920,0cyclictest12866-21cpuspeed_turbos10:16:430
259899920,0cyclictest0-21swapper/008:21:310
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional