You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 07:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Tue Sep 16, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350560,0irq/25-eth10-21swapper/318:41:453
847023329,0sleep00-21swapper/018:39:060
870423127,0sleep10-21swapper/118:41:471
831122723,0sleep20-21swapper/218:37:322
28415070,0irq/27-eth0-tx-23394-21kthreadcore22:12:242
28415070,0irq/27-eth0-tx-0-21swapper/223:34:582
88009960,0cyclictest0-21swapper/223:39:322
88009960,0cyclictest0-21swapper/223:39:322
87999965,0cyclictest20817-21sed23:12:231
87999960,0cyclictest6860-21diskmemload21:23:131
8225060,0irq/24-0000:00:11242-21cat23:27:062
339960,0migration/35485-21sh20:59:143
339960,0migration/35099-21sh22:01:363
339960,0migration/33620-21sh22:18:493
339960,0migration/329613-21sh21:56:353
339960,0migration/325798-21missed_timers22:52:263
339960,0migration/311798-21sh23:27:193
28415060,0irq/27-eth0-tx-21883-21ssh21:09:262
28415060,0irq/27-eth0-tx-16019-21cut23:07:242
28415060,0irq/27-eth0-tx-13574-21aten2.4-expect22:07:192
28415060,0irq/27-eth0-tx-0-21swapper/223:58:422
28415060,0irq/27-eth0-tx-0-21swapper/223:54:062
28415060,0irq/27-eth0-tx-0-21swapper/223:49:212
28415060,0irq/27-eth0-tx-0-21swapper/223:06:372
28415060,0irq/27-eth0-tx-0-21swapper/222:58:552
28415060,0irq/27-eth0-tx-0-21swapper/222:51:552
28415060,0irq/27-eth0-tx-0-21swapper/222:39:552
28415060,0irq/27-eth0-tx-0-21swapper/222:26:332
28415060,0irq/27-eth0-tx-0-21swapper/222:17:482
28415060,0irq/27-eth0-tx-0-21swapper/221:50:532
28415060,0irq/27-eth0-tx-0-21swapper/221:42:562
28415060,0irq/27-eth0-tx-0-21swapper/221:34:242
28415060,0irq/27-eth0-tx-0-21swapper/221:17:162
28415060,0irq/27-eth0-tx-0-21swapper/220:57:152
28415060,0irq/27-eth0-tx-0-21swapper/020:42:280
28415060,0irq/27-eth0-tx-0-21swapper/020:17:230
28405060,0irq/26-eth0-rx-8168-21kthreadcore22:22:243
28405060,0irq/26-eth0-rx-26788-21perl20:32:213
249960,0watchdog/20-21swapper/223:14:482
249960,0watchdog/20-21swapper/220:47:282
87999955,0cyclictest0-21swapper/123:03:581
87999955,0cyclictest0-21swapper/121:20:141
87999954,0cyclictest12144-21munin-run22:47:061
87999950,0cyclictest0-21swapper/123:34:141
87999950,0cyclictest0-21swapper/122:13:021
87999950,0cyclictest0-21swapper/121:59:571
87999950,0cyclictest0-21swapper/100:03:521
87989955,0cyclictest0-21swapper/021:44:050
87989955,0cyclictest0-21swapper/021:21:040
87989955,0cyclictest0-21swapper/019:52:180
87989950,0cyclictest0-21swapper/023:18:220
87989950,0cyclictest0-21swapper/022:45:530
87989950,0cyclictest0-21swapper/021:58:070
87989950,0cyclictest0-21swapper/021:24:030
87989950,0cyclictest0-21swapper/021:05:460
87989950,0cyclictest0-21swapper/020:47:290
87989950,0cyclictest0-21swapper/020:22:060
35150,0ktimersoftd/30-21swapper/323:50:093
35150,0ktimersoftd/30-21swapper/320:45:323
339950,0migration/39777-21fschecks_time00:07:223
339950,0migration/39589-21kthreadcore20:17:243
339950,0migration/38824-21sh00:06:463
339950,0migration/38121-21kthreadcore23:02:213
339950,0migration/35779-21cpuspeed_turbos22:02:183
339950,0migration/35219-21munin-node20:12:293
339950,0migration/330821-21cpuspeed_turbos21:37:173
339950,0migration/330495-21kthreadcore23:17:223
339950,0migration/330112-21sh21:36:353
339950,0migration/329554-21irqrtprio22:57:243
339950,0migration/327406-21aten_r1power_vo23:37:203
339950,0migration/327406-21aten_r1power_vo23:37:203
339950,0migration/325064-21latency_hist20:07:063
339950,0migration/324779-21sh23:53:023
339950,0migration/324711-21kthreadcore22:32:253
339950,0migration/324693-21kthreadcore20:52:233
339950,0migration/322658-21sh22:12:243
339950,0migration/322644-21taskset20:51:513
339950,0migration/32136-21kthreadcore19:47:243
339950,0migration/321155-21kthreadcore20:02:203
339950,0migration/320906-21sh22:11:243
339950,0migration/317466-21kthreadcore21:07:223
339950,0migration/317331-21fschecks_time19:12:203
339950,0migration/316918-21kthreadcore23:07:253
339950,0migration/31461-21irqcore20:37:213
339950,0migration/314440-21sh21:06:243
339950,0migration/314205-21aten_r1power_fr21:47:193
339950,0migration/311029-21kthreadcore21:22:243
28415050,0irq/27-eth0-tx-7433-21fschecks_time21:22:202
28415050,0irq/27-eth0-tx-28186-21fschecks_count23:17:182
28415050,0irq/27-eth0-tx-28096-21kthreadcore20:52:242
28415050,0irq/27-eth0-tx-25121-21munin-node20:07:210
28415050,0irq/27-eth0-tx-23253-21munin-node19:17:200
28415050,0irq/27-eth0-tx-0-21swapper/222:55:142
28415050,0irq/27-eth0-tx-0-21swapper/222:36:002
28415050,0irq/27-eth0-tx-0-21swapper/221:54:592
28415050,0irq/27-eth0-tx-0-21swapper/221:12:182
28415050,0irq/27-eth0-tx-0-21swapper/200:09:012
28415050,0irq/27-eth0-tx-0-21swapper/200:02:182
28405050,0irq/26-eth0-rx-8793-21pmu-power19:02:263
28405050,0irq/26-eth0-rx-3242-21fschecks_count23:42:223
28405050,0irq/26-eth0-rx-2152-21processes22:37:293
28405050,0irq/26-eth0-rx-21445-21sshd21:29:213
28405050,0irq/26-eth0-rx-11204-21df_abs19:32:193
28405050,0irq/26-eth0-rx-0-21swapper/323:59:133
28405050,0irq/26-eth0-rx-0-21swapper/323:22:523
28405050,0irq/26-eth0-rx-0-21swapper/322:42:383
28405050,0irq/26-eth0-rx-0-21swapper/321:46:093
28405050,0irq/26-eth0-rx-0-21swapper/321:18:333
259950,0migration/225860-21gltestperf20:07:212
249950,0watchdog/24689-21ssh21:38:362
249950,0watchdog/20-21swapper/223:43:282
249950,0watchdog/20-21swapper/223:31:202
249950,0watchdog/20-21swapper/222:46:122
249950,0watchdog/20-21swapper/222:27:242
249950,0watchdog/20-21swapper/222:02:162
249950,0watchdog/20-21swapper/221:59:362
249950,0watchdog/20-21swapper/221:27:122
249950,0watchdog/20-21swapper/221:02:122
249950,0watchdog/20-21swapper/220:42:242
249950,0watchdog/20-21swapper/220:32:242
249950,0watchdog/20-21swapper/220:30:322
249950,0watchdog/20-21swapper/220:25:282
249950,0watchdog/20-21swapper/220:15:242
249950,0watchdog/20-21swapper/220:02:162
249950,0watchdog/20-21swapper/219:58:322
249950,0watchdog/20-21swapper/219:52:082
249950,0watchdog/20-21swapper/219:45:362
249950,0watchdog/20-21swapper/219:37:202
249950,0watchdog/20-21swapper/219:28:522
249950,0watchdog/20-21swapper/219:18:322
249950,0watchdog/20-21swapper/219:02:202
19150,0ktimersoftd/10-21swapper/120:54:061
179950,0migration/16719-21sh22:02:221
179950,0migration/12849-21switchtime22:17:291
179950,0migration/123430-21cpuspeed_turbos21:12:191
179950,0migration/118102-21sh23:07:551
149950,0migration/0851-21kthreadcore19:47:230
149950,0migration/07611-21kthreadcore22:22:240
149950,0migration/05430-21latency_hist22:02:060
149950,0migration/031971-21kthreadcore22:57:250
149950,0migration/023347-21kthreadcore20:02:210
149950,0migration/015069-21sh21:07:170
149950,0migration/014015-21irqcore22:27:220
88009943,0cyclictest9916-21awk18:42:212
88009943,0cyclictest3110-21irqbalance20:37:302
88009943,0cyclictest3110-21irqbalance20:17:202
88009943,0cyclictest30868-21awk19:22:212
88009943,0cyclictest2985-21sadc19:02:052
88009943,0cyclictest17472-21awk19:12:212
88009943,0cyclictest16569-21awk18:47:222
87999944,0cyclictest0-21swapper/121:45:551
87999944,0cyclictest0-21swapper/120:43:581
87999943,0cyclictest3110-21irqbalance23:22:301
87999943,0cyclictest3110-21irqbalance23:18:101
87999943,0cyclictest3110-21irqbalance22:07:201
87999943,0cyclictest3110-21irqbalance21:09:401
87999943,0cyclictest3110-21irqbalance21:00:001
87999943,0cyclictest3110-21irqbalance19:52:201
87999943,0cyclictest21309-21awk22:52:221
87999943,0cyclictest17592-21awk19:12:221
87999943,0cyclictest16545-21awk18:47:211
87999943,0cyclictest10062-21awk00:07:231
87989944,0cyclictest6860-21diskmemload21:52:260
87989944,0cyclictest0-21swapper/021:38:240
87989944,0cyclictest0-21swapper/021:33:410
87989943,0cyclictest3110-21irqbalance23:57:100
87989943,0cyclictest3110-21irqbalance23:51:000
87989943,0cyclictest3110-21irqbalance00:02:200
87989943,0cyclictest26962-21awk20:32:220
87989943,0cyclictest24034-21awk21:12:220
87989943,0cyclictest22352-21awk22:12:220
87989943,0cyclictest20657-21awk23:12:220
87989943,0cyclictest18501-21awk19:37:240
87989943,0cyclictest12317-21awk23:27:210
87989943,0cyclictest11708-21awk19:32:210
87989943,0cyclictest10808-21awk19:07:210
87989943,0cyclictest10636-21sadc19:32:050
87989940,0cyclictest0-21swapper/019:57:240
339940,0migration/320932-21kthreadcore20:27:223
339940,0migration/313219-21kthreadcore19:57:233
28415040,0irq/27-eth0-tx-0-21swapper/019:43:220
28415040,0irq/27-eth0-tx-0-21swapper/019:02:130
28405040,0irq/26-eth0-rx-3869-21sshd19:26:223
28405040,0irq/26-eth0-rx-14834-21timerwakeupswit18:42:283
28405040,0irq/26-eth0-rx-0-21swapper/323:33:273
28405040,0irq/26-eth0-rx-0-21swapper/323:12:193
28405040,0irq/26-eth0-rx-0-21swapper/322:47:073
28405040,0irq/26-eth0-rx-0-21swapper/322:27:093
28405040,0irq/26-eth0-rx-0-21swapper/321:12:233
28405040,0irq/26-eth0-rx-0-21swapper/319:27:253
259940,0migration/2367-21kthreadcore19:47:232
249940,0watchdog/211605-21sed19:32:202
249940,0watchdog/20-21swapper/219:07:162
249940,0watchdog/20-21swapper/218:52:082
179940,0migration/17876-21kthreadcore19:02:241
179940,0migration/130663-21aten_r1power_po21:37:171
179940,0migration/128279-21sh23:37:251
179940,0migration/128279-21sh23:37:251
179940,0migration/128164-21kthreadcore19:42:261
179940,0migration/12686-21taskset23:42:101
179940,0migration/125582-21kthreadcore21:32:261
179940,0migration/115887-21kthreadcore22:27:231
179940,0migration/114501-21kthreadcore19:57:231
169940,0watchdog/128702-21df_inode22:57:201
169940,0watchdog/127400-21kthreadcore18:52:241
169940,0watchdog/127399-21kthreadcore20:32:241
169940,0watchdog/124118-21sensors20:02:241
169940,0watchdog/121221-21chrt21:28:401
169940,0watchdog/120852-21kthreadcore23:52:241
169940,0watchdog/120257-21kthreadcore20:47:281
169940,0watchdog/113089-21sh21:02:441
169940,0watchdog/111607-21gltestperf19:32:201
169940,0watchdog/10-21swapper/123:47:241
169940,0watchdog/10-21swapper/122:32:241
169940,0watchdog/10-21swapper/122:22:241
169940,0watchdog/10-21swapper/121:52:161
169940,0watchdog/10-21swapper/121:47:281
149940,0migration/09798-21hwlatdetect18:42:200
149940,0migration/09743-21taskset23:45:120
149940,0migration/06851-21sh20:42:060
149940,0migration/05136-21switchtime20:12:290
149940,0migration/031253-21cpu20:57:180
149940,0migration/031231-21kthreadcore19:22:220
149940,0migration/027364-21kthreadcore18:52:240
149940,0migration/022896-21kthreadcore22:52:240
149940,0migration/021087-21kthreadcore23:32:240
149940,0migration/020160-21kthreadcore23:52:240
149940,0migration/020032-21fschecks_time20:27:190
149940,0migration/019413-21sh22:50:120
149940,0migration/017910-21kernelversion19:12:230
149940,0migration/01749-21sh23:39:540
149940,0migration/01749-21sh23:39:540
149940,0migration/011932-21kthreadcore00:07:260
149940,0migration/010589-21taskset23:25:310
149940,0migration/01031-21kthreadcore22:37:260
87999933,0cyclictest3425-21lldpd22:48:351
87999933,0cyclictest3425-21lldpd19:47:541
87999933,0cyclictest3425-21lldpd19:29:241
87999932,0cyclictest6445-21snmpd20:19:241
87999932,0cyclictest6445-21snmpd19:07:141
87999932,0cyclictest6445-21snmpd18:42:091
87999932,0cyclictest3110-21irqbalance20:12:101
87999932,0cyclictest27107-21awk23:57:211
87999932,0cyclictest20155-21tail20:27:191
87999932,0cyclictest18476-21awk19:37:241
87999932,0cyclictest12244-21awk23:27:201
87989932,0cyclictest6445-21snmpd21:27:430
87989932,0cyclictest4705-21awk23:02:180
87989932,0cyclictest3425-21lldpd21:48:330
87989932,0cyclictest23331-21grep20:52:180
87989932,0cyclictest12795-21awk23:07:220
87989930,0cyclictest21344-21df_inode22:32:200
4130,0ktimersoftd/00-21swapper/018:48:080
339930,0migration/329271-21taskset19:17:403
28415030,0irq/27-eth0-tx-0-21swapper/018:57:180
28405030,0irq/26-eth0-rx-0-21swapper/320:22:323
28405030,0irq/26-eth0-rx-0-21swapper/319:52:163
28405030,0irq/26-eth0-rx-0-21swapper/319:42:203
28405030,0irq/26-eth0-rx-0-21swapper/319:41:003
28405030,0irq/26-eth0-rx-0-21swapper/319:07:173
28405030,0irq/26-eth0-rx-0-21swapper/318:57:293
28405030,0irq/26-eth0-rx-0-21swapper/318:52:403
28405030,0irq/26-eth0-rx-0-21swapper/318:47:173
19748230,0sleep00-21swapper/022:07:530
179930,0migration/125848-21kthreadcore19:17:231
169930,0watchdog/16547-21runrttasks22:37:201
169930,0watchdog/132033-21munin-node18:57:241
169930,0watchdog/10-21swapper/120:07:241
87999921,0cyclictest32704-21cut20:37:061
87999920,0cyclictest0-21swapper/120:22:081
87999920,0cyclictest0-21swapper/119:22:161
87989920,0cyclictest29377-21sh22:17:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional