You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 07:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Fri May 09, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350440,0irq/25-eth10-21swapper/318:46:563
3206223329,0sleep20-21swapper/218:46:552
3192023322,0sleep00-21swapper/018:45:170
3189823228,0sleep10-21swapper/118:45:021
321329970,0cyclictest0-21swapper/023:27:080
339960,0migration/37964-21sh23:12:593
339960,0migration/34025-21runrttasks21:50:113
339960,0migration/324456-21sh23:02:353
339960,0migration/321774-21sh21:21:263
339960,0migration/31999-21munin-node19:12:143
339960,0migration/317947-21sh23:01:513
339960,0migration/31202-21runrttasks23:11:443
321399965,0cyclictest2954-21sed19:37:122
321339965,0cyclictest21415-21ps23:02:171
321339960,0cyclictest30211-21diskmemload23:32:251
321339960,0cyclictest0-21swapper/122:37:141
28415060,0irq/27-eth0-tx-9091-21ssh23:16:352
28415060,0irq/27-eth0-tx-32532chrt21:47:492
28415060,0irq/27-eth0-tx-0-21swapper/223:51:482
28415060,0irq/27-eth0-tx-0-21swapper/223:27:552
28415060,0irq/27-eth0-tx-0-21swapper/223:22:562
28415060,0irq/27-eth0-tx-0-21swapper/223:17:082
28415060,0irq/27-eth0-tx-0-21swapper/222:53:462
28415060,0irq/27-eth0-tx-0-21swapper/222:31:462
28415060,0irq/27-eth0-tx-0-21swapper/222:24:312
28415060,0irq/27-eth0-tx-0-21swapper/222:13:532
28415060,0irq/27-eth0-tx-0-21swapper/221:38:222
28415060,0irq/27-eth0-tx-0-21swapper/221:35:252
28415060,0irq/27-eth0-tx-0-21swapper/221:18:432
28415060,0irq/27-eth0-tx-0-21swapper/221:06:162
28415060,0irq/27-eth0-tx-0-21swapper/220:52:192
28415060,0irq/27-eth0-tx-0-21swapper/200:12:492
28405060,0irq/26-eth0-rx-1812-21tr21:07:153
28405060,0irq/26-eth0-rx-17696-21perl20:37:113
249960,0watchdog/213443-21kthreadcore19:42:152
249960,0watchdog/20-21swapper/221:22:192
249960,0watchdog/20-21swapper/220:49:072
149960,0migration/014388-21sh23:59:570
149960,0migration/013802-21fschecks_count18:57:070
339950,0migration/39598-21aten_r1power_po23:17:093
339950,0migration/37767-21kthreadcore21:32:173
339950,0migration/36498-21rt-features23:32:183
339950,0migration/36366-21taskset18:51:323
339950,0migration/331830-21latency19:07:153
339950,0migration/330391-21cpuspeed_turbos21:27:123
339950,0migration/330070-21fschecks_time20:22:123
339950,0migration/329224-21kthreadcore22:27:143
339950,0migration/326904-21missed_timers21:42:173
339950,0migration/326459-21kthreadcore23:47:153
339950,0migration/32514-21taskset20:01:453
339950,0migration/324537-21kthreadcore19:52:133
339950,0migration/323320-21idleruntime21:02:173
339950,0migration/322226-21aten_r1power_cu21:22:083
339950,0migration/319880-21aten_r1power_fr22:22:093
339950,0migration/313585-21munin-node21:57:093
339950,0migration/313580-21munin-node19:17:193
339950,0migration/31255-21taskset19:33:473
339950,0migration/31173-21taskset22:49:023
321399950,0cyclictest0-21swapper/222:02:302
321339955,0cyclictest0-21swapper/122:13:161
321339955,0cyclictest0-21swapper/121:55:271
321339950,0cyclictest0-21swapper/123:15:321
321339950,0cyclictest0-21swapper/122:59:371
321329955,0cyclictest0-21swapper/020:58:470
321329955,0cyclictest0-21swapper/019:37:070
321329955,0cyclictest0-21swapper/000:06:140
28415050,0irq/27-eth0-tx-9486-21munin-node20:07:110
28415050,0irq/27-eth0-tx-6372-21iceccd20:12:280
28415050,0irq/27-eth0-tx-32290-21ssh23:07:372
28415050,0irq/27-eth0-tx-28485-21munin-node19:57:070
28415050,0irq/27-eth0-tx-23783-21munin-node20:42:100
28415050,0irq/27-eth0-tx-22885-21munin-node00:07:112
28415050,0irq/27-eth0-tx-11073-21munin-node22:37:082
28415050,0irq/27-eth0-tx-0-21swapper/223:37:052
28415050,0irq/27-eth0-tx-0-21swapper/223:32:142
28415050,0irq/27-eth0-tx-0-21swapper/223:02:372
28415050,0irq/27-eth0-tx-0-21swapper/222:47:202
28415050,0irq/27-eth0-tx-0-21swapper/222:43:052
28415050,0irq/27-eth0-tx-0-21swapper/222:32:232
28415050,0irq/27-eth0-tx-0-21swapper/222:18:122
28415050,0irq/27-eth0-tx-0-21swapper/222:08:142
28415050,0irq/27-eth0-tx-0-21swapper/220:57:182
28405050,0irq/26-eth0-rx-3144-21ssh22:32:043
28405050,0irq/26-eth0-rx-22375-21ssh00:06:073
28405050,0irq/26-eth0-rx-16290-21kthreadcore19:22:143
28405050,0irq/26-eth0-rx-12964-21ps22:17:133
28405050,0irq/26-eth0-rx-0-21swapper/323:42:543
28405050,0irq/26-eth0-rx-0-21swapper/322:42:153
28405050,0irq/26-eth0-rx-0-21swapper/322:37:193
28405050,0irq/26-eth0-rx-0-21swapper/322:12:123
28405050,0irq/26-eth0-rx-0-21swapper/321:54:143
28405050,0irq/26-eth0-rx-0-21swapper/320:57:153
28405050,0irq/26-eth0-rx-0-21swapper/320:47:093
259950,0migration/25304-21munin-node18:47:142
249950,0watchdog/240212sleep221:27:312
249950,0watchdog/20-21swapper/223:57:312
249950,0watchdog/20-21swapper/223:55:072
249950,0watchdog/20-21swapper/223:46:112
249950,0watchdog/20-21swapper/222:58:232
249950,0watchdog/20-21swapper/222:01:032
249950,0watchdog/20-21swapper/221:53:432
249950,0watchdog/20-21swapper/221:45:072
249950,0watchdog/20-21swapper/221:16:272
249950,0watchdog/20-21swapper/221:08:272
249950,0watchdog/20-21swapper/220:20:072
249950,0watchdog/20-21swapper/220:02:112
249950,0watchdog/20-21swapper/219:28:112
249950,0watchdog/20-21swapper/200:02:312
19150,0ktimersoftd/10-21swapper/123:41:121
179950,0migration/130596-21munin-node23:27:131
169950,0watchdog/11445-21cut20:47:151
149950,0migration/09459-21munin-node20:27:160
149950,0migration/08256-21sh23:13:540
149950,0migration/0710-21sh23:10:230
149950,0migration/06867-21kthreadcore22:12:150
149950,0migration/032485-21kthreadcore21:27:170
149950,0migration/031459-21if_eth1.320:47:130
149950,0migration/029407-21munin-node22:07:110
149950,0migration/028725-21unixbench-2d20:17:180
149950,0migration/025551-21smart_sda22:22:190
149950,0migration/023588-21irqcore21:02:180
149950,0migration/022868-21kthreadcore22:02:140
149950,0migration/020447-21latency_hist22:01:560
149950,0migration/020031-21kthreadcore23:22:150
149950,0migration/019515-21sh22:21:550
149950,0migration/016223-21kthreadcore22:37:140
149950,0migration/015805-21kthreadcore21:37:150
149950,0migration/015056-21idleruntime19:22:120
149950,0migration/013882-21kthreadcore23:17:160
149950,0migration/013040-21sh19:17:150
149950,0migration/012946-21taskset18:55:300
149950,0migration/01161-21irqrtprio18:47:120
149950,0migration/011322-21sh21:53:140
8225040,0irq/24-0000:00:0-21swapper/219:02:112
339940,0migration/39586-21kthreadcore23:57:133
339940,0migration/36892-21kthreadcore20:27:123
339940,0migration/34874-21kthreadcore22:52:123
339940,0migration/326233-21irqcore23:27:113
339940,0migration/321355-21kthreadcore23:22:153
339940,0migration/31678-21kthreadcore00:12:133
339940,0migration/316140-21if_eth1.219:47:133
339940,0migration/312200-21kthreadcore19:42:153
321339944,0cyclictest28602-21kworker/1:221:43:481
321339944,0cyclictest0-21swapper/119:17:101
321339943,0cyclictest7591-21awk18:52:121
321339943,0cyclictest6445-21snmpd21:32:351
321339943,0cyclictest6445-21snmpd20:26:351
321339943,0cyclictest6445-21snmpd20:02:351
321339943,0cyclictest6445-21snmpd19:37:351
321339943,0cyclictest6445-21snmpd19:35:351
321339943,0cyclictest6445-21snmpd19:03:501
321339943,0cyclictest6445-21snmpd18:48:051
321339943,0cyclictest4708-21awk20:27:101
321339943,0cyclictest374-21awk23:52:151
321339943,0cyclictest3425-21lldpd22:07:391
321339943,0cyclictest3425-21lldpd20:13:131
321339943,0cyclictest3425-21lldpd20:07:431
321339943,0cyclictest3425-21lldpd19:42:131
321339943,0cyclictest28602-21kworker/1:221:48:041
321339943,0cyclictest23219-21awk21:22:121
321339943,0cyclictest22837-21awk19:52:111
321339943,0cyclictest21743-21awk19:27:101
321339943,0cyclictest19979-21awk22:42:131
321339943,0cyclictest1690-21awk19:12:121
321339943,0cyclictest14449-21awk21:37:131
321329943,0cyclictest6445-21snmpd21:10:200
321329943,0cyclictest6445-21snmpd19:06:350
321329943,0cyclictest3425-21lldpd19:51:430
321329943,0cyclictest3307-21awk22:52:100
321329943,0cyclictest30184-21awk21:47:110
321329943,0cyclictest28581-21awk22:27:120
321329943,0cyclictest27455-21awk19:07:110
321329943,0cyclictest23978-21sadc23:46:560
321329943,0cyclictest22344-21awk21:42:130
321329943,0cyclictest1705-21awk19:12:120
321329943,0cyclictest11418-21awk20:32:140
321329943,0cyclictest11373-21awk22:57:170
321329940,0cyclictest29493-21munin-node20:22:150
28405040,0irq/26-eth0-rx-7763-21kthreadcore21:12:133
28405040,0irq/26-eth0-rx-24219-21kthreadcore19:27:123
28405040,0irq/26-eth0-rx-0-21swapper/323:37:013
28405040,0irq/26-eth0-rx-0-21swapper/322:02:063
28405040,0irq/26-eth0-rx-0-21swapper/321:37:123
28405040,0irq/26-eth0-rx-0-21swapper/320:52:513
28405040,0irq/26-eth0-rx-0-21swapper/300:09:353
259940,0migration/227554-21munin-node19:32:072
259940,0migration/21761-21kthreadcore20:22:162
249940,0watchdog/20-21swapper/220:42:032
249940,0watchdog/20-21swapper/220:37:072
249940,0watchdog/20-21swapper/220:26:592
249940,0watchdog/20-21swapper/220:12:072
249940,0watchdog/20-21swapper/220:01:392
249940,0watchdog/20-21swapper/219:52:192
249940,0watchdog/20-21swapper/219:46:592
249940,0watchdog/20-21swapper/219:21:592
249940,0watchdog/20-21swapper/219:19:272
249940,0watchdog/20-21swapper/219:12:472
249940,0watchdog/20-21swapper/219:06:592
249940,0watchdog/20-21swapper/218:57:112
249940,0watchdog/20-21swapper/218:51:592
179940,0migration/127664-21kthreadcore23:47:151
179940,0migration/111887-21taskset22:17:011
169940,0watchdog/17386-21kthreadcore22:32:151
169940,0watchdog/131617-21kthreadcore19:57:111
169940,0watchdog/127028-21ssh22:03:031
169940,0watchdog/124180-21diskstats20:42:111
169940,0watchdog/123825-21interrupts20:17:111
169940,0watchdog/119518-21kthreadcore23:22:151
169940,0watchdog/118948-21cut19:47:151
169940,0watchdog/115898-21perl00:02:111
169940,0watchdog/112895-21telnet21:57:071
169940,0watchdog/110730-21kthreadcore20:52:151
169940,0watchdog/10-21swapper/122:27:191
169940,0watchdog/10-21swapper/122:22:311
169940,0watchdog/10-21swapper/121:19:151
169940,0watchdog/10-21swapper/119:22:151
149940,0migration/031738-21kthreadcore19:32:130
149940,0migration/023832-21kthreadcore19:52:130
149940,0migration/017152-21kthreadcore21:17:140
149940,0migration/012600-21kthreadcore19:42:150
321339933,0cyclictest28602-21kworker/1:223:57:041
321339933,0cyclictest28602-21kworker/1:223:42:041
321339933,0cyclictest28602-21kworker/1:220:57:441
321339932,0cyclictest6445-21snmpd18:57:051
321339932,0cyclictest3368-21ntp_kernel_pll_21:27:191
321339932,0cyclictest28602-21kworker/1:221:06:581
321339932,0cyclictest27798-21awk22:47:121
321339932,0cyclictest27573-21awk19:07:121
321339932,0cyclictest14368-21grep23:17:161
321339930,0cyclictest0-21swapper/122:52:061
321329933,0cyclictest20003-21awk22:42:130
321329932,0cyclictest9668-21cut21:12:140
321329932,0cyclictest9250-21sadc20:06:560
321329932,0cyclictest4101-21awk22:32:120
321329932,0cyclictest3425-21lldpd22:50:480
321329932,0cyclictest23849-21awk00:07:130
321329932,0cyclictest21798-21kthreadcore23:02:170
321329932,0cyclictest11385-21kthreadcore23:37:160
321329930,0cyclictest7505-21taskset23:34:100
28415030,0irq/27-eth0-tx-0-21swapper/020:37:120
28405030,0irq/26-eth0-rx-0-21swapper/323:52:273
28405030,0irq/26-eth0-rx-0-21swapper/322:07:173
28405030,0irq/26-eth0-rx-0-21swapper/320:32:103
28405030,0irq/26-eth0-rx-0-21swapper/320:12:113
28405030,0irq/26-eth0-rx-0-21swapper/320:07:093
28405030,0irq/26-eth0-rx-0-21swapper/320:05:453
28405030,0irq/26-eth0-rx-0-21swapper/319:37:083
28405030,0irq/26-eth0-rx-0-21swapper/319:04:463
28405030,0irq/26-eth0-rx-0-21swapper/318:57:473
28405030,0irq/26-eth0-rx-0-21swapper/318:54:463
249930,0watchdog/20-21swapper/220:32:112
249930,0watchdog/20-21swapper/220:06:592
179930,0migration/129370-21kthreadcore23:07:171
169930,0watchdog/10-21swapper/121:12:431
169930,0watchdog/10-21swapper/121:02:191
169930,0watchdog/10-21swapper/120:37:151
169930,0watchdog/10-21swapper/120:32:191
169930,0watchdog/10-21swapper/100:12:231
169930,0watchdog/10-21swapper/100:07:511
321329921,0cyclictest30216-21latency_hist00:11:560
321329921,0cyclictest20915-21tail19:27:060
321329921,0cyclictest0-21swapper/021:31:570
321329921,0cyclictest0-21swapper/020:52:070
321329920,0cyclictest31909-21latency_hist23:51:560
321329920,0cyclictest22096-21latency_hist21:21:570
321329920,0cyclictest0-21swapper/023:46:570
329920,0watchdog/30-21swapper/320:17:113
28405020,0irq/26-eth0-rx-0-21swapper/320:42:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional