You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 11:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Wed Sep 17, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350450,0irq/25-eth10-21swapper/318:37:263
1310623725,0sleep00-21swapper/018:38:010
1327223526,0sleep20-21swapper/218:39:562
1322623430,0sleep10-21swapper/118:39:261
35160,0ktimersoftd/30-21swapper/323:52:203
339960,0migration/37201-21sh00:06:493
339960,0migration/3675-21sh23:40:193
339960,0migration/326383-21munin-node22:12:253
339960,0migration/326320-21sh22:33:043
339960,0migration/318799-21sh22:50:523
339960,0migration/31212-21munin-node20:52:243
28415060,0irq/27-eth0-tx-0-21swapper/223:53:112
28415060,0irq/27-eth0-tx-0-21swapper/223:50:472
28415060,0irq/27-eth0-tx-0-21swapper/223:37:112
28415060,0irq/27-eth0-tx-0-21swapper/223:17:312
28415060,0irq/27-eth0-tx-0-21swapper/223:13:462
28415060,0irq/27-eth0-tx-0-21swapper/223:02:422
28415060,0irq/27-eth0-tx-0-21swapper/222:57:452
28415060,0irq/27-eth0-tx-0-21swapper/222:53:592
28415060,0irq/27-eth0-tx-0-21swapper/222:50:282
28415060,0irq/27-eth0-tx-0-21swapper/222:45:302
28415060,0irq/27-eth0-tx-0-21swapper/222:38:352
28415060,0irq/27-eth0-tx-0-21swapper/222:27:232
28415060,0irq/27-eth0-tx-0-21swapper/222:04:142
28415060,0irq/27-eth0-tx-0-21swapper/221:53:042
28415060,0irq/27-eth0-tx-0-21swapper/221:49:502
28415060,0irq/27-eth0-tx-0-21swapper/221:43:102
28415060,0irq/27-eth0-tx-0-21swapper/221:27:572
28415060,0irq/27-eth0-tx-0-21swapper/220:59:212
28415060,0irq/27-eth0-tx-0-21swapper/220:47:152
28415060,0irq/27-eth0-tx-0-21swapper/220:43:312
28405060,0irq/26-eth0-rx-20710-21hddtemp_smartct21:27:173
28405060,0irq/26-eth0-rx-10584-21ssh22:23:213
28405060,0irq/26-eth0-rx-0-21swapper/320:50:353
249960,0watchdog/20-21swapper/223:09:422
249960,0watchdog/20-21swapper/222:34:262
249960,0watchdog/20-21swapper/219:30:302
149960,0migration/07903-21latency_hist19:02:040
149960,0migration/026452-21sh22:33:290
149960,0migration/022463-21timerandwakeup19:57:280
149960,0migration/021008-21gltestperf21:07:190
149960,0migration/019007-21sh22:51:290
149960,0migration/01787-21sh22:37:530
149960,0migration/016526-21munin-node23:07:240
149960,0migration/01618-21timerwakeupswit20:52:270
135259966,0cyclictest30608-21kworker/1:022:27:371
135259966,0cyclictest11727-21diskmemload21:45:241
8225050,0irq/24-0000:00:0-21swapper/221:37:342
339950,0migration/38599-21munin-node23:02:253
339950,0migration/36917-21kthreadcore19:22:213
339950,0migration/36556-21irqrtprio20:37:203
339950,0migration/322306-21munin-node21:52:193
339950,0migration/320033-21taskset21:49:053
339950,0migration/318119-21timerwakeupswit21:02:283
339950,0migration/318119-21timerwakeupswit21:02:283
339950,0migration/315319-21kthreadcore22:27:233
339950,0migration/31483-21unixbench_multi21:32:283
339950,0migration/314741-21munin-node18:42:213
339950,0migration/31243-21kthreadcore20:32:233
339950,0migration/310879-21latency_hist20:17:043
339950,0migration/310767-21kthreadcore23:47:233
339950,0migration/310420-21aten_r1power_fr23:27:143
339950,0migration/31022-21taskset18:56:273
28415050,0irq/27-eth0-tx-6372-21iceccd19:18:490
28415050,0irq/27-eth0-tx-32281-21munin-node00:02:272
28415050,0irq/27-eth0-tx-29552-21awk22:17:232
28415050,0irq/27-eth0-tx-28439-21perl21:32:182
28415050,0irq/27-eth0-tx-22362-21munin-node19:37:280
28415050,0irq/27-eth0-tx-11972-21munin-node20:42:250
28415050,0irq/27-eth0-tx-10741-21ssh22:23:522
28415050,0irq/27-eth0-tx-0-21swapper/223:57:142
28415050,0irq/27-eth0-tx-0-21swapper/223:32:202
28415050,0irq/27-eth0-tx-0-21swapper/223:27:132
28415050,0irq/27-eth0-tx-0-21swapper/223:22:542
28415050,0irq/27-eth0-tx-0-21swapper/222:15:282
28415050,0irq/27-eth0-tx-0-21swapper/222:09:172
28415050,0irq/27-eth0-tx-0-21swapper/221:25:362
28415050,0irq/27-eth0-tx-0-21swapper/221:18:082
28415050,0irq/27-eth0-tx-0-21swapper/221:11:062
28415050,0irq/27-eth0-tx-0-21swapper/018:47:270
28405050,0irq/26-eth0-rx-903-21tr19:42:213
28405050,0irq/26-eth0-rx-31424-21kthreadcore22:17:253
28405050,0irq/26-eth0-rx-25251-21taskset23:13:003
28405050,0irq/26-eth0-rx-22329-21munin-node19:12:203
28405050,0irq/26-eth0-rx-2203-21ssh21:12:463
28405050,0irq/26-eth0-rx-21252-21grep23:32:213
28405050,0irq/26-eth0-rx-13894-21kthreadcore19:27:253
28405050,0irq/26-eth0-rx-0-21swapper/323:25:133
28405050,0irq/26-eth0-rx-0-21swapper/323:09:423
28405050,0irq/26-eth0-rx-0-21swapper/322:42:223
28405050,0irq/26-eth0-rx-0-21swapper/322:39:423
28405050,0irq/26-eth0-rx-0-21swapper/322:12:033
28405050,0irq/26-eth0-rx-0-21swapper/322:03:343
28405050,0irq/26-eth0-rx-0-21swapper/322:01:193
28405050,0irq/26-eth0-rx-0-21swapper/321:41:413
28405050,0irq/26-eth0-rx-0-21swapper/321:18:203
28405050,0irq/26-eth0-rx-0-21swapper/321:09:253
28405050,0irq/26-eth0-rx-0-21swapper/320:57:283
28405050,0irq/26-eth0-rx-0-21swapper/300:07:183
259950,0migration/229433-21kthreadcore20:52:212
259950,0migration/210813-21latency_hist20:17:042
249950,0watchdog/20-21swapper/223:44:142
249950,0watchdog/20-21swapper/221:57:462
249950,0watchdog/20-21swapper/221:12:222
249950,0watchdog/20-21swapper/221:06:182
249950,0watchdog/20-21swapper/221:06:182
249950,0watchdog/20-21swapper/220:25:222
249950,0watchdog/20-21swapper/219:49:222
249950,0watchdog/20-21swapper/219:12:222
249950,0watchdog/20-21swapper/219:08:142
249950,0watchdog/20-21swapper/218:42:222
19150,0ktimersoftd/10-21swapper/123:07:421
179950,0migration/13363-21kthreadcore20:32:241
179950,0migration/13348-21kthreadcore00:02:261
179950,0migration/125671-21sh23:14:201
179950,0migration/124262-21taskset23:56:271
169950,0watchdog/117009-21grep21:47:221
149950,0migration/04060-21aten_r1power_vo20:57:200
149950,0migration/03359-21irqcore19:22:190
149950,0migration/032683-21kthreadcore19:42:210
149950,0migration/029749-21kthreadcore22:57:210
149950,0migration/029687-21irqcore22:17:240
149950,0migration/02861-21mdstat20:07:200
149950,0migration/026998-21munin-node19:12:250
149950,0migration/025559-21kthreadcore21:52:210
149950,0migration/025382-21irqcore20:27:210
149950,0migration/024595-21latency_hist23:57:030
149950,0migration/024152-21kthreadcore21:27:210
149950,0migration/022487-21latency23:52:240
149950,0migration/021143-21kthreadcore23:12:240
149950,0migration/020575-21kthreadcore19:32:220
149950,0migration/01461-21sh23:20:170
149950,0migration/014288-21fschecks_time18:42:180
149950,0migration/013820-21kthreadcore22:27:220
149950,0migration/013617-21gltestperf22:07:190
149950,0migration/01337-21kthreadcore21:57:240
149950,0migration/01240-21kthreadcore00:02:250
149950,0migration/011964-21runrttasks21:22:140
149950,0migration/011041-21sh21:20:090
149950,0migration/010979-21sh22:24:390
149950,0migration/010618-21irqrtprio23:47:220
135339954,0cyclictest8428-21awk00:07:212
135259955,0cyclictest3148-21kworker/1:123:42:251
135259955,0cyclictest30608-21kworker/1:021:59:271
135259955,0cyclictest0-21swapper/123:40:011
135259955,0cyclictest0-21swapper/122:47:021
135259955,0cyclictest0-21swapper/122:39:191
135259955,0cyclictest0-21swapper/122:18:491
135259955,0cyclictest0-21swapper/121:29:281
135259954,0cyclictest30608-21kworker/1:019:18:351
135259954,0cyclictest0-21swapper/121:33:531
135259950,0cyclictest0-21swapper/123:59:241
135259950,0cyclictest0-21swapper/123:29:181
135259950,0cyclictest0-21swapper/123:02:291
135259950,0cyclictest0-21swapper/122:13:461
135259950,0cyclictest0-21swapper/120:21:501
135249954,0cyclictest14758-21sadc00:12:030
135249950,0cyclictest0-21swapper/021:41:390
339940,0migration/34800-21kthreadcore19:47:233
339940,0migration/316130-21kthreadcore20:42:223
28415040,0irq/27-eth0-tx-0-21swapper/020:36:570
28415040,0irq/27-eth0-tx-0-21swapper/020:12:240
28405040,0irq/26-eth0-rx-20340-21python19:07:293
28405040,0irq/26-eth0-rx-0-21swapper/323:57:043
28405040,0irq/26-eth0-rx-0-21swapper/323:42:233
28405040,0irq/26-eth0-rx-0-21swapper/323:19:333
28405040,0irq/26-eth0-rx-0-21swapper/322:57:223
28405040,0irq/26-eth0-rx-0-21swapper/322:53:453
28405040,0irq/26-eth0-rx-0-21swapper/321:42:183
28405040,0irq/26-eth0-rx-0-21swapper/321:23:103
28405040,0irq/26-eth0-rx-0-21swapper/319:52:203
259940,0migration/25831-21kthreadcore18:57:192
259940,0migration/224391-21irqcore20:02:192
259940,0migration/21499-21unixbench_multi19:17:262
259940,0migration/21288-21kthreadcore20:32:232
259940,0migration/211395-21kthreadcore19:02:252
249940,0watchdog/227375-21kthreadcore20:27:222
249940,0watchdog/20-21swapper/219:52:182
249940,0watchdog/20-21swapper/219:32:182
249940,0watchdog/20-21swapper/219:22:462
179940,0migration/18215-21kthreadcore22:02:201
179940,0migration/17183-21kthreadcore21:17:221
179940,0migration/128416-21aten2.4-expect21:12:161
179940,0migration/127944-21if_err_eth1.218:52:191
179940,0migration/127645-21cpuspeed_turbos22:57:161
179940,0migration/127195-21df_abs23:17:141
179940,0migration/114358-21kthreadcore21:02:221
179940,0migration/114358-21kthreadcore21:02:221
169940,0watchdog/122363-21kthreadcore20:47:221
169940,0watchdog/120267-21smartctl22:52:181
169940,0watchdog/118331-21runrttasks21:23:061
169940,0watchdog/117400-21cat22:47:261
169940,0watchdog/10-21swapper/122:22:181
149940,0migration/06775-21kthreadcore23:22:230
149940,0migration/029541-21kthreadcore21:32:210
149940,0migration/028719-21kthreadcore23:37:230
149940,0migration/026546-21sh22:55:180
149940,0migration/019310-21kthreadcore19:07:250
135339943,0cyclictest6445-21snmpd18:51:042
135339943,0cyclictest30870-21awk20:07:172
135339943,0cyclictest29950-21awk19:42:192
135339943,0cyclictest23271-21awk19:37:222
135339943,0cyclictest17657-21awk19:57:212
135339943,0cyclictest11939-21awk20:17:192
135259944,0cyclictest3148-21kworker/1:123:32:211
135259944,0cyclictest30608-21kworker/1:022:32:211
135259944,0cyclictest30608-21kworker/1:022:07:351
135259944,0cyclictest30608-21kworker/1:021:56:511
135259944,0cyclictest30608-21kworker/1:021:38:071
135259944,0cyclictest30608-21kworker/1:020:57:251
135259944,0cyclictest30608-21kworker/1:020:37:211
135259944,0cyclictest30608-21kworker/1:020:22:111
135259944,0cyclictest30608-21kworker/1:020:12:111
135259944,0cyclictest30608-21kworker/1:019:47:371
135259944,0cyclictest30608-21kworker/1:019:42:511
135259944,0cyclictest30608-21kworker/1:019:37:171
135259944,0cyclictest30608-21kworker/1:019:32:111
135259944,0cyclictest30608-21kworker/1:019:22:091
135259944,0cyclictest30608-21kworker/1:019:02:091
135259944,0cyclictest30608-21kworker/1:018:57:071
135259944,0cyclictest30608-21kworker/1:018:47:091
135259944,0cyclictest30608-21kworker/1:018:42:131
135259943,0cyclictest6445-21snmpd19:07:051
135259943,0cyclictest3425-21lldpd21:07:221
135259943,0cyclictest3148-21kworker/1:123:47:171
135259943,0cyclictest3148-21kworker/1:123:23:211
135259943,0cyclictest3110-21irqbalance20:55:081
135259943,0cyclictest3110-21irqbalance20:27:381
135259943,0cyclictest3110-21irqbalance20:02:081
135259943,0cyclictest3110-21irqbalance19:57:181
135259943,0cyclictest3110-21irqbalance19:52:281
135259943,0cyclictest3110-21irqbalance19:27:081
135259943,0cyclictest3110-21irqbalance19:12:181
135259943,0cyclictest3110-21irqbalance00:08:181
135259943,0cyclictest30825-21awk20:07:171
135259943,0cyclictest12866-21awk20:42:191
135249944,0cyclictest0-21swapper/023:05:500
135249943,0cyclictest8904-21awk19:02:220
135249943,0cyclictest6445-21snmpd21:14:050
135249943,0cyclictest6445-21snmpd19:27:050
135249943,0cyclictest6278-21awk21:42:160
135249943,0cyclictest2482-21awk23:42:190
135249943,0cyclictest13066-21awk21:02:200
135249943,0cyclictest13066-21awk21:02:200
135249943,0cyclictest11236-21awk23:27:180
339930,0migration/317656-21munin-node19:57:213
28415030,0irq/27-eth0-tx-0-21swapper/020:17:150
28405030,0irq/26-eth0-rx-0-21swapper/320:30:333
28405030,0irq/26-eth0-rx-0-21swapper/320:25:483
28405030,0irq/26-eth0-rx-0-21swapper/320:13:003
28405030,0irq/26-eth0-rx-0-21swapper/320:07:193
28405030,0irq/26-eth0-rx-0-21swapper/320:02:143
28405030,0irq/26-eth0-rx-0-21swapper/319:39:213
28405030,0irq/26-eth0-rx-0-21swapper/319:35:163
28405030,0irq/26-eth0-rx-0-21swapper/319:19:513
28405030,0irq/26-eth0-rx-0-21swapper/318:57:133
28405030,0irq/26-eth0-rx-0-21swapper/318:47:183
135339930,0cyclictest5940-21perl20:37:172
135339930,0cyclictest27469-21diskstats18:52:172
135249932,0cyclictest26085-21ssh20:47:300
135249932,0cyclictest25111-21grep20:02:200
135249932,0cyclictest19119-21sensors21:47:260
135249932,0cyclictest18530-21awk20:22:190
135249932,0cyclictest10883-21awk19:52:180
135249931,0cyclictest22563-21kthreadcore22:12:210
135249930,0cyclictest4197-21perl22:42:200
135249930,0cyclictest4023-21perl19:47:190
135249930,0cyclictest27733-21perl18:52:180
135249930,0cyclictest0-21swapper/020:37:180
11185230,0sleep011727-21diskmemload22:02:560
329920,0watchdog/30-21swapper/319:02:303
135249921,0cyclictest0-21swapper/023:32:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional