You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-23 - 03:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Wed Sep 23, 2026 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
899827977,0sleep20-21swapper/218:40:302
907127776,0sleep10-21swapper/118:41:221
304450740,0irq/25-eth10-21swapper/318:38:213
895423724,0sleep00-21swapper/018:39:560
339970,0migration/328941-21kthreadcore23:58:243
92969966,0cyclictest7367-21diskmemload23:31:261
92969965,0cyclictest4797-21sadc22:43:071
92959965,0cyclictest27196-21dump-pmu-power23:58:070
35160,0ktimersoftd/30-21swapper/322:53:243
339960,0migration/34859-21sh21:38:373
339960,0migration/330020-21sh21:57:153
339960,0migration/320325-21sh22:30:233
339960,0migration/312664-21runrttasks22:04:213
29715060,0irq/26-eth0-rx-19742-21open_inodes22:08:273
29715060,0irq/26-eth0-rx-0-21swapper/323:19:463
19160,0ktimersoftd/10-21swapper/121:28:181
149960,0migration/02985-21sh22:59:260
149960,0migration/029591-21sh21:34:340
149960,0migration/012886-21sh22:26:530
92969955,0cyclictest7367-21diskmemload22:01:341
92969950,0cyclictest0-21swapper/122:30:251
92969950,0cyclictest0-21swapper/121:13:101
92959955,0cyclictest0-21swapper/023:52:030
92959955,0cyclictest0-21swapper/023:52:030
92959955,0cyclictest0-21swapper/023:26:310
92959955,0cyclictest0-21swapper/023:04:360
92959955,0cyclictest0-21swapper/022:55:180
92959955,0cyclictest0-21swapper/022:50:060
92959955,0cyclictest0-21swapper/022:13:120
92959950,0cyclictest0-21swapper/023:44:390
92959950,0cyclictest0-21swapper/021:47:210
92959950,0cyclictest0-21swapper/019:22:370
92959950,0cyclictest0-21swapper/000:12:230
339950,0migration/39889-21kthreadcore23:03:233
339950,0migration/38260-21kthreadcore22:23:243
339950,0migration/36068-21fschecks_count19:53:213
339950,0migration/35317-21munin-node19:28:223
339950,0migration/34128-21taskset23:02:453
339950,0migration/332589-21munin-node20:33:323
339950,0migration/332298-21aten_r1power_cu20:13:183
339950,0migration/33121-21taskset18:59:103
339950,0migration/329827-21latency_hist22:18:073
339950,0migration/326545-21runrttasks23:34:353
339950,0migration/325531-21kthreadcore20:53:263
339950,0migration/324632-21munin-node20:03:303
339950,0migration/323900-21kthreadcore20:28:243
339950,0migration/321891-21kthreadcore19:38:253
339950,0migration/321587-21munin-node18:48:273
339950,0migration/316845-21kthreadcore22:48:243
339950,0migration/314728-21kthreadcore23:48:253
339950,0migration/314728-21kthreadcore23:48:253
339950,0migration/313964-21sh21:04:073
29725050,0irq/27-eth0-tx-31471-21kthreadcore23:18:252
29725050,0irq/27-eth0-tx-25615-21munin-node20:08:230
29725050,0irq/27-eth0-tx-2469-21ssh00:02:252
29725050,0irq/27-eth0-tx-22079-21munin-node22:13:282
29725050,0irq/27-eth0-tx-20965-21switchtime21:28:292
29725050,0irq/27-eth0-tx-0-21swapper/223:53:252
29725050,0irq/27-eth0-tx-0-21swapper/223:40:222
29725050,0irq/27-eth0-tx-0-21swapper/221:48:262
29725050,0irq/27-eth0-tx-0-21swapper/221:24:042
29725050,0irq/27-eth0-tx-0-21swapper/221:06:322
29725050,0irq/27-eth0-tx-0-21swapper/220:48:202
29725050,0irq/27-eth0-tx-0-21swapper/020:02:450
29725050,0irq/27-eth0-tx-0-21swapper/019:48:260
29715050,0irq/26-eth0-rx-5943-21cut00:03:233
29715050,0irq/26-eth0-rx-5655-21kthreadcore23:23:243
29715050,0irq/26-eth0-rx-5049-21kthreadcore19:03:243
29715050,0irq/26-eth0-rx-23956-21tr23:53:243
29715050,0irq/26-eth0-rx-17723-21tune2fs19:13:213
29715050,0irq/26-eth0-rx-16523-21if_err_eth1.321:08:223
29715050,0irq/26-eth0-rx-0-21swapper/323:43:263
29715050,0irq/26-eth0-rx-0-21swapper/323:38:423
29715050,0irq/26-eth0-rx-0-21swapper/323:32:443
29715050,0irq/26-eth0-rx-0-21swapper/323:13:273
29715050,0irq/26-eth0-rx-0-21swapper/323:12:043
29715050,0irq/26-eth0-rx-0-21swapper/322:46:453
29715050,0irq/26-eth0-rx-0-21swapper/322:39:443
29715050,0irq/26-eth0-rx-0-21swapper/322:33:103
29715050,0irq/26-eth0-rx-0-21swapper/322:00:393
29715050,0irq/26-eth0-rx-0-21swapper/321:43:303
29715050,0irq/26-eth0-rx-0-21swapper/321:28:193
29715050,0irq/26-eth0-rx-0-21swapper/321:23:233
29715050,0irq/26-eth0-rx-0-21swapper/320:58:213
29715050,0irq/26-eth0-rx-0-21swapper/319:23:263
29715050,0irq/26-eth0-rx-0-21swapper/300:08:183
259950,0migration/230272-21sendmail_mailqu19:43:282
259950,0migration/229258-21sh21:54:562
259950,0migration/226156-21munin-node23:33:312
249950,0watchdog/29113-21kthreadcore22:03:252
19150,0ktimersoftd/10-21swapper/100:10:421
19150,0ktimersoftd/10-21swapper/100:03:171
179950,0migration/15482-21munin-node23:03:201
179950,0migration/125096-21kthreadcore18:53:241
179950,0migration/12367-21missed_timers18:58:261
179950,0migration/118088-21irqcore19:13:241
179950,0migration/11806-21sendmail_mailqu23:18:291
179950,0migration/11556-21sh23:59:401
179950,0migration/112590-21taskset22:47:371
169950,0watchdog/121983-21ssh20:48:541
149950,0migration/06527-21latency_hist20:18:070
149950,0migration/032255-21kthreadcore22:38:250
149950,0migration/031882-21aten_r1power_cu20:58:190
149950,0migration/031482-21kthreadcore18:58:250
149950,0migration/030944-21if_eth022:18:230
149950,0migration/027039-21kthreadcore19:43:240
149950,0migration/025680-21kthreadcore22:33:250
149950,0migration/021872-21kthreadcore20:03:250
149950,0migration/021854-21sh21:30:170
149950,0migration/021254-21irqrtprio23:33:240
149950,0migration/020855-21kthreadcore20:48:270
149950,0migration/018319-21sendmail_mailst23:08:280
149950,0migration/017906-21kthreadcore21:48:240
149950,0migration/016188-21kthreadcore23:28:240
149950,0migration/013705-21fschecks_count20:23:220
149950,0migration/010201-21kthreadcore20:43:290
92979943,0cyclictest3556-21lldpd22:53:182
92979943,0cyclictest3556-21lldpd20:18:462
92979943,0cyclictest3556-21lldpd20:14:152
92979943,0cyclictest3556-21lldpd20:02:162
92979943,0cyclictest3556-21lldpd19:18:142
92979943,0cyclictest3556-21lldpd19:14:152
92979943,0cyclictest3556-21lldpd19:07:452
92979943,0cyclictest3556-21lldpd18:53:152
92979943,0cyclictest3556-21lldpd18:48:142
92979943,0cyclictest3514-21snmpd22:23:112
92979943,0cyclictest3514-21snmpd22:09:412
92979943,0cyclictest3514-21snmpd20:38:112
92979943,0cyclictest18987-21awk19:38:232
92979943,0cyclictest16534-21awk21:08:222
92979943,0cyclictest12548-21awk23:48:232
92979943,0cyclictest12548-21awk23:48:232
92979943,0cyclictest11856-21awk00:08:212
92979942,0cyclictest27848-21grep21:13:232
92969944,0cyclictest0-21swapper/121:43:081
92969943,0cyclictest8543-21awk20:43:271
92969943,0cyclictest8475-21awk21:03:211
92969943,0cyclictest7214-21awk22:03:241
92969943,0cyclictest5068-21awk23:23:231
92969943,0cyclictest3556-21lldpd20:55:151
92969943,0cyclictest3514-21snmpd20:38:261
92969943,0cyclictest3514-21snmpd19:53:111
92969943,0cyclictest3241-21irqbalance23:43:151
92969943,0cyclictest27446-21awk20:33:221
92969943,0cyclictest24648-21awk19:18:221
92969943,0cyclictest23598-21awk21:53:281
92969943,0cyclictest20899-21awk20:28:221
92969943,0cyclictest19898-21awk20:03:231
92969943,0cyclictest16582-21awk21:08:221
92959944,0cyclictest0-21swapper/022:29:170
92959944,0cyclictest0-21swapper/021:04:470
92959944,0cyclictest0-21swapper/019:40:350
92959943,0cyclictest3514-21snmpd21:41:260
92959943,0cyclictest3241-21irqbalance23:38:350
92959943,0cyclictest3241-21irqbalance21:22:360
92959943,0cyclictest3241-21irqbalance20:13:260
92959943,0cyclictest2864-21kworker/0:018:48:250
92959943,0cyclictest28354-21awk23:58:220
92959943,0cyclictest23768-21awk18:53:230
92959943,0cyclictest11323-21awk19:08:240
339940,0migration/33625-21kthreadcore20:38:273
339940,0migration/330135-21diskmemload21:36:123
339940,0migration/329987-21taskset22:18:113
339940,0migration/317397-21kthreadcore20:23:263
29725040,0irq/27-eth0-tx-6665-21kthreadcore22:43:282
29725040,0irq/27-eth0-tx-31846-21ssh20:58:142
29725040,0irq/27-eth0-tx-30022-21munin-node22:18:202
29725040,0irq/27-eth0-tx-0-21swapper/223:29:172
29725040,0irq/27-eth0-tx-0-21swapper/223:23:132
29725040,0irq/27-eth0-tx-0-21swapper/223:08:182
29725040,0irq/27-eth0-tx-0-21swapper/222:48:162
29725040,0irq/27-eth0-tx-0-21swapper/222:38:202
29725040,0irq/27-eth0-tx-0-21swapper/222:28:272
29725040,0irq/27-eth0-tx-0-21swapper/220:53:312
29725040,0irq/27-eth0-tx-0-21swapper/220:45:202
29725040,0irq/27-eth0-tx-0-21swapper/200:03:352
29725040,0irq/27-eth0-tx-0-21swapper/020:33:240
29715040,0irq/26-eth0-rx-25022-21kthreadcore19:18:233
29715040,0irq/26-eth0-rx-0-21swapper/321:48:243
29715040,0irq/26-eth0-rx-0-21swapper/321:18:373
29715040,0irq/26-eth0-rx-0-21swapper/321:13:433
29715040,0irq/26-eth0-rx-0-21swapper/320:52:263
29715040,0irq/26-eth0-rx-0-21swapper/320:43:183
29715040,0irq/26-eth0-rx-0-21swapper/319:48:193
29715040,0irq/26-eth0-rx-0-21swapper/319:08:233
259940,0migration/28994-21kthreadcore23:03:222
259940,0migration/26253-21fschecks_time19:53:212
259940,0migration/26083-21kthreadcore23:43:262
259940,0migration/23502-21kthreadcore19:48:252
259940,0migration/231056-21aten_r1power_cu21:38:182
259940,0migration/230486-21aten_r1power_vo19:23:202
259940,0migration/228100-21kthreadcore20:08:232
259940,0migration/228003-21kthreadcore20:33:242
259940,0migration/225169-21kthreadcore23:13:232
259940,0migration/222794-21kthreadcore20:03:252
259940,0migration/222303-21hddtemp_smartct22:33:212
259940,0migration/215764-21kthreadcore19:33:252
249940,0watchdog/28943-21cat21:43:292
249940,0watchdog/231493-21ssh21:18:132
249940,0watchdog/22758-21switchtime18:58:302
249940,0watchdog/216242-21kthreadcore20:23:262
249940,0watchdog/21316-21kthreadcore21:58:252
249940,0watchdog/212649-21kthreadcore19:08:262
249940,0watchdog/20-21swapper/222:58:252
249940,0watchdog/20-21swapper/221:33:252
179940,0migration/18476-21kthreadcore22:23:241
179940,0migration/132405-21kthreadcore21:38:241
179940,0migration/132025-21fschecks_count21:18:221
179940,0migration/128038-21timerwakeupswit22:13:281
179940,0migration/124693-21aten_r1power_en19:43:181
179940,0migration/121525-21fschecks_time22:53:201
179940,0migration/120567-21timerandwakeup21:48:291
179940,0migration/119478-21runrttasks20:25:311
179940,0migration/119079-21unixbench_multi22:48:301
179940,0migration/116775-21ntp_kernel_pll_19:33:271
179940,0migration/116268-21munin-node18:48:191
179940,0migration/116043-21kthreadcore22:08:231
179940,0migration/112686-21irqcore23:48:231
179940,0migration/112686-21irqcore23:48:231
179940,0migration/112324-21aten_r1power_cu19:58:181
179940,0migration/111960-21kthreadcore21:23:251
179940,0migration/110005-21fschecks_count18:43:201
169940,0watchdog/130586-21grep22:18:211
169940,0watchdog/12821-21cut20:58:261
169940,0watchdog/12807-21ssh23:41:411
169940,0watchdog/125347-21cat23:53:291
169940,0watchdog/122768-21kthreadcore23:33:251
169940,0watchdog/11890-21tr19:23:261
169940,0watchdog/112651-21kthreadcore19:08:261
169940,0watchdog/11228-21tr22:58:251
149940,0migration/022582-21runrttasks21:53:110
149940,0migration/019450-21kthreadcore22:08:250
149940,0migration/017694-21kthreadcore21:08:240
149940,0migration/013470-21sh21:23:580
149940,0migration/010429-21switchtime19:28:300
92979932,0cyclictest5550-21awk19:28:232
92979932,0cyclictest3556-21lldpd20:28:162
92979932,0cyclictest3556-21lldpd18:47:442
92969932,0cyclictest7355-21gltestperf20:18:241
92969932,0cyclictest4629-21awk19:03:221
92969932,0cyclictest26626-21irqcore20:08:221
92969932,0cyclictest22471-21awk22:33:231
92969932,0cyclictest18860-21awk19:38:221
92969931,0cyclictest6466-21latency_hist20:18:071
92969930,0cyclictest4481-21sed19:48:281
92959933,0cyclictest2864-21kworker/0:023:13:440
92959933,0cyclictest2864-21kworker/0:022:43:240
92959933,0cyclictest2864-21kworker/0:021:58:210
92959933,0cyclictest2864-21kworker/0:021:13:190
92959933,0cyclictest2864-21kworker/0:020:53:170
92959933,0cyclictest2864-21kworker/0:020:38:130
92959933,0cyclictest2864-21kworker/0:019:53:130
92959933,0cyclictest2864-21kworker/0:019:23:130
92959933,0cyclictest2864-21kworker/0:019:13:110
92959933,0cyclictest2864-21kworker/0:019:03:150
92959933,0cyclictest2864-21kworker/0:018:43:150
92959933,0cyclictest2864-21kworker/0:000:03:200
92959932,0cyclictest3514-21snmpd22:03:100
92959932,0cyclictest21436-21kthreadcore20:28:230
92959932,0cyclictest1832-21sensors23:18:290
92959930,0cyclictest0-21swapper/019:33:270
29715030,0irq/26-eth0-rx-0-21swapper/319:43:243
29715030,0irq/26-eth0-rx-0-21swapper/319:33:263
29715030,0irq/26-eth0-rx-0-21swapper/318:53:193
29715030,0irq/26-eth0-rx-0-21swapper/318:47:373
169930,0watchdog/120505-21munin-node23:13:211
169930,0watchdog/10-21swapper/123:08:211
169930,0watchdog/10-21swapper/121:33:251
169930,0watchdog/10-21swapper/119:30:461
339920,0migration/313318-21iostat_ios19:58:233
29715020,0irq/26-eth0-rx-0-21swapper/320:18:253
29715020,0irq/26-eth0-rx-0-21swapper/320:08:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional