You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-01 - 09:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Thu Jan 01, 2026 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304450620,0irq/25-eth10-21swapper/319:04:043
878223723,0sleep00-21swapper/019:03:420
900223524,0sleep10-21swapper/119:06:151
887123526,0sleep20-21swapper/219:04:442
92459966,0cyclictest0-21swapper/223:50:442
92459960,0cyclictest0-21swapper/222:58:512
92459960,0cyclictest0-21swapper/221:34:022
92429966,0cyclictest3514-21snmpd21:57:110
4160,0ktimersoftd/06559-21runrttasks23:20:160
339960,0migration/39248-21sh21:27:473
339960,0migration/33873-21switchtime00:18:363
339960,0migration/332753-21sh21:38:353
339960,0migration/332709-21sh21:22:303
339960,0migration/327536-21sh21:57:163
339960,0migration/326611-21munin-node22:13:323
339960,0migration/323512-21sh23:11:333
339960,0migration/320359-21sh22:29:293
339960,0migration/318403-21sh21:33:243
29715060,0irq/26-eth0-rx-23553-21df00:33:253
29715060,0irq/26-eth0-rx-21928-21ssh22:50:373
29715060,0irq/26-eth0-rx-12407-21sh00:06:013
27160,0ktimersoftd/20-21swapper/221:59:112
149960,0migration/09215-21kthreadcore20:43:310
149960,0migration/029700-21sh22:36:540
149960,0migration/021414-21sh00:29:380
149960,0migration/014114-21idleruntime20:48:280
149960,0migration/011308-21munin-node22:23:340
92459955,0cyclictest0-21swapper/222:58:042
92459955,0cyclictest0-21swapper/221:12:262
92459950,0cyclictest0-21swapper/221:46:582
92439955,0cyclictest0-21swapper/123:03:091
92439954,0cyclictest14267-21awk20:48:291
92439950,0cyclictest0-21swapper/122:25:101
92429955,0cyclictest0-21swapper/022:48:540
92429955,0cyclictest0-21swapper/021:37:000
92429955,0cyclictest0-21swapper/021:15:530
92429950,0cyclictest0-21swapper/023:28:090
92429950,0cyclictest0-21swapper/021:52:180
8215050,0irq/24-0000:00:22925-21latency_hist00:33:132
8215050,0irq/24-0000:00:11067-21munin-run19:58:142
339950,0migration/39165-21kthreadcore23:03:363
339950,0migration/36364-21kthreadcore23:58:333
339950,0migration/34235-21kthreadcore23:18:313
339950,0migration/33847-21taskset22:02:523
339950,0migration/326330-21taskset20:57:063
339950,0migration/325765-21munin-node20:33:273
339950,0migration/324419-21gltestperf19:43:273
339950,0migration/323267-21latency20:03:323
339950,0migration/321200-21kthreadcore20:28:313
339950,0migration/320878-21kthreadcore19:13:303
339950,0migration/318111-21kthreadcore00:28:313
339950,0migration/315900-21sh23:43:353
339950,0migration/315490-21kthreadcore19:33:343
339950,0migration/312990-21cpuspeed_turbos22:08:273
339950,0migration/311971-21kthreadcore19:08:313
29725050,0irq/27-eth0-tx-9454-21munin-node19:08:322
29725050,0irq/27-eth0-tx-6559-21runrttasks00:08:232
29725050,0irq/27-eth0-tx-3877-21ssh22:20:072
29725050,0irq/27-eth0-tx-31436-21ssh23:15:302
29725050,0irq/27-eth0-tx-23403-21ps22:33:292
29725050,0irq/27-eth0-tx-18008-21munin-node20:03:372
29725050,0irq/27-eth0-tx-0-21swapper/222:46:412
29725050,0irq/27-eth0-tx-0-21swapper/221:54:002
29725050,0irq/27-eth0-tx-0-21swapper/221:15:442
29715050,0irq/26-eth0-rx-7054-21kthreadcore21:43:313
29715050,0irq/26-eth0-rx-3514-21snmpd20:13:303
29715050,0irq/26-eth0-rx-29095-21smartctl22:53:353
29715050,0irq/26-eth0-rx-26788-21cut23:13:323
29715050,0irq/26-eth0-rx-17716-21ssh21:48:373
29715050,0irq/26-eth0-rx-16491-21ssh21:29:443
29715050,0irq/26-eth0-rx-10827-21kthreadcore21:08:303
29715050,0irq/26-eth0-rx-0-21swapper/323:38:203
29715050,0irq/26-eth0-rx-0-21swapper/323:34:083
29715050,0irq/26-eth0-rx-0-21swapper/323:32:363
29715050,0irq/26-eth0-rx-0-21swapper/322:58:273
29715050,0irq/26-eth0-rx-0-21swapper/322:43:303
29715050,0irq/26-eth0-rx-0-21swapper/322:33:353
29715050,0irq/26-eth0-rx-0-21swapper/322:24:353
29715050,0irq/26-eth0-rx-0-21swapper/322:04:073
29715050,0irq/26-eth0-rx-0-21swapper/320:23:273
29715050,0irq/26-eth0-rx-0-21swapper/300:23:253
29715050,0irq/26-eth0-rx-0-21swapper/300:13:483
29715050,0irq/26-eth0-rx-0-21swapper/300:13:483
27150,0ktimersoftd/20-21swapper/222:30:482
259950,0migration/23598-21latency_hist19:28:142
259950,0migration/227672-21munin-node20:58:302
259950,0migration/227179-21kthreadcore20:08:342
259950,0migration/218806-21kthreadcore23:28:312
259950,0migration/21832-21kthreadcore23:18:302
259950,0migration/211872-21kthreadcore20:43:312
249950,0watchdog/225739-21ps21:18:322
249950,0watchdog/217149-21sleep23:45:472
249950,0watchdog/210449-21perl21:28:282
179950,0migration/18191-21sh23:39:221
179950,0migration/17414-21munin-node22:43:281
179950,0migration/13376-21latency00:18:311
179950,0migration/131595-21kthreadcore22:18:301
179950,0migration/126184-21taskset21:54:001
179950,0migration/119725-21sh22:10:401
179950,0migration/118937-21munin-node20:03:291
179950,0migration/117627-21mdstat20:23:331
179950,0migration/116087-21kthreadcore22:28:311
179950,0migration/114503-21turbostat.cron00:28:081
179950,0migration/114451-21needreboot23:23:321
179950,0migration/110804-21taskset19:55:231
169950,0watchdog/13352-21fschecks_count23:58:271
149950,0migration/09199-21kthreadcore23:03:360
149950,0migration/08680-21if_eth1.221:08:280
149950,0migration/04921-21unixbench_singl20:13:370
149950,0migration/03535-21df_abs21:43:260
149950,0migration/032289-21taskset21:21:360
149950,0migration/029277-21latency_hist19:23:140
149950,0migration/028842-21fschecks_time21:58:270
149950,0migration/028714-21kthreadcore20:58:310
149950,0migration/026706-21kthreadcore23:13:320
149950,0migration/025957-21cpuspeed_turbos23:33:240
149950,0migration/025051-40yum-updatesd23:49:420
149950,0migration/020794-21kthreadcore23:28:320
149950,0migration/017565-21kthreadcore22:28:310
149950,0migration/017469-21kthreadcore19:13:290
149950,0migration/01742-21munin-node21:23:250
149950,0migration/014099-21kthreadcore20:23:320
149950,0migration/012500-21unixbench-2d00:23:370
149950,0migration/011272-21kthreadcore21:28:310
92459943,0cyclictest5359-21grep20:38:332
92459943,0cyclictest3556-21lldpd19:36:272
92459943,0cyclictest3556-21lldpd19:23:572
92459943,0cyclictest3556-21lldpd19:15:272
92439944,0cyclictest0-21swapper/121:38:201
92439943,0cyclictest481-21cut23:18:241
92439943,0cyclictest3556-21lldpd21:48:571
92439943,0cyclictest3556-21lldpd21:07:571
92439943,0cyclictest3556-21lldpd20:56:271
92439943,0cyclictest3514-21snmpd23:31:441
92439943,0cyclictest3514-21snmpd23:04:141
92439943,0cyclictest3514-21snmpd22:51:291
92439943,0cyclictest3514-21snmpd22:07:441
92439943,0cyclictest3514-21snmpd21:14:291
92439943,0cyclictest3514-21snmpd20:38:291
92439943,0cyclictest3514-21snmpd00:04:591
92439943,0cyclictest3241-21irqbalance21:59:521
92439943,0cyclictest24208-21awk00:33:301
92439943,0cyclictest22882-21awk00:13:281
92439943,0cyclictest22882-21awk00:13:281
92439943,0cyclictest15762-21awk00:28:281
92429944,0cyclictest0-21swapper/021:03:250
92429943,0cyclictest3893-21awk23:58:310
92429943,0cyclictest3556-21lldpd19:33:270
92429943,0cyclictest3514-21snmpd22:10:290
92429943,0cyclictest3514-21snmpd20:09:000
92429943,0cyclictest3241-21irqbalance22:45:320
92429943,0cyclictest3241-21irqbalance22:18:220
92429943,0cyclictest3241-21irqbalance22:15:120
92429943,0cyclictest3241-21irqbalance21:38:220
92429943,0cyclictest3241-21irqbalance19:08:330
92429943,0cyclictest3241-21irqbalance00:34:510
92429943,0cyclictest3241-21irqbalance00:16:510
92429943,0cyclictest3241-21irqbalance00:16:510
92429943,0cyclictest3241-21irqbalance00:10:210
92429943,0cyclictest26643-21awk20:33:310
92429943,0cyclictest12132-21awk19:58:300
339940,0migration/329273-21kthreadcore20:58:313
339940,0migration/32133-21kthreadcore20:38:313
339940,0migration/31722-21hwlatdetect21:03:283
339940,0migration/314240-21kthreadcore19:58:333
339940,0migration/310859-21kthreadcore20:43:313
29725040,0irq/27-eth0-tx-9723-21gltestperf23:23:272
29725040,0irq/27-eth0-tx-7776-21kthreadcore22:03:302
29725040,0irq/27-eth0-tx-6607-21ssh00:23:252
29725040,0irq/27-eth0-tx-2550-21grep22:38:312
29725040,0irq/27-eth0-tx-12559-21latency_hist22:08:142
29725040,0irq/27-eth0-tx-0-21swapper/223:54:212
29725040,0irq/27-eth0-tx-0-21swapper/223:08:232
29725040,0irq/27-eth0-tx-0-21swapper/222:48:222
29725040,0irq/27-eth0-tx-0-21swapper/222:23:292
29725040,0irq/27-eth0-tx-0-21swapper/221:38:272
29725040,0irq/27-eth0-tx-0-21swapper/221:23:242
29725040,0irq/27-eth0-tx-0-21swapper/220:13:302
29725040,0irq/27-eth0-tx-0-21swapper/219:58:262
29725040,0irq/27-eth0-tx-0-21swapper/219:51:392
29725040,0irq/27-eth0-tx-0-21swapper/200:19:032
29725040,0irq/27-eth0-tx-0-21swapper/200:03:232
29715040,0irq/26-eth0-rx-0-21swapper/323:53:493
29715040,0irq/26-eth0-rx-0-21swapper/323:49:503
29715040,0irq/26-eth0-rx-0-21swapper/323:23:213
29715040,0irq/26-eth0-rx-0-21swapper/322:38:293
29715040,0irq/26-eth0-rx-0-21swapper/322:19:033
29715040,0irq/26-eth0-rx-0-21swapper/321:13:203
29715040,0irq/26-eth0-rx-0-21swapper/320:48:323
29715040,0irq/26-eth0-rx-0-21swapper/319:48:253
259940,0migration/24922-21kthreadcore21:03:322
259940,0migration/228517-21kthreadcore20:33:332
259940,0migration/226143-21kthreadcore19:43:302
259940,0migration/225409-21kthreadcore22:13:312
259940,0migration/222628-21fschecks_time00:13:272
259940,0migration/222628-21fschecks_time00:13:272
259940,0migration/219950-21kthreadcore00:28:312
259940,0migration/21614-21munin-node23:38:242
259940,0migration/214364-21irqcore20:48:302
259940,0migration/212584-21cpuspeed_turbos20:23:272
249940,0watchdog/23353-21fschecks_count23:58:272
249940,0watchdog/225852-21cut19:18:322
249940,0watchdog/20-21swapper/223:33:232
249940,0watchdog/20-21swapper/223:03:312
249940,0watchdog/20-21swapper/221:48:402
179940,0migration/19005-21taskset21:27:141
179940,0migration/18547-21idleruntime21:08:281
179940,0migration/17228-21fschecks_time20:43:271
179940,0migration/132465-21kthreadcore22:38:311
179940,0migration/130295-21latency_hist19:48:141
179940,0migration/128434-21kthreadcore20:08:341
179940,0migration/127010-21munin-node23:33:291
179940,0migration/126115-21fschecks_count20:33:291
179940,0migration/124435-21kthreadcore19:18:321
179940,0migration/122115-21taskset19:13:491
179940,0migration/121894-21kthreadcore21:33:311
179940,0migration/117974-21kthreadcore23:08:461
179940,0migration/112180-21kthreadcore19:08:311
179940,0migration/111142-21gltestperf19:33:301
169940,0watchdog/13945-21ssh21:43:281
169940,0watchdog/131160-21kthreadcore19:23:321
169940,0watchdog/130335-21sh22:55:431
169940,0watchdog/125743-21sed21:18:321
169940,0watchdog/12441-21ssh23:58:031
169940,0watchdog/115606-21mailstats21:28:361
169940,0watchdog/111686-21df_inode19:58:281
169940,0watchdog/10-21swapper/123:48:311
169940,0watchdog/10-21swapper/123:43:271
169940,0watchdog/10-21swapper/123:15:231
169940,0watchdog/10-21swapper/120:58:321
169940,0watchdog/10-21swapper/100:08:271
149940,0migration/0688-21kthreadcore19:48:290
149940,0migration/04178-21sh00:18:370
149940,0migration/031064-21fschecks_count22:38:270
149940,0migration/01985-21df_abs23:38:250
149940,0migration/01737-21runrttasks23:56:230
149940,0migration/011913-21runrttasks20:21:070
149940,0migration/011232-21latency_hist00:03:130
92459933,0cyclictest3556-21lldpd20:19:272
92459932,0cyclictest0-21swapper/220:29:392
92459930,0cyclictest0-21swapper/220:53:142
92439933,0cyclictest3556-21lldpd20:28:271
92439932,0cyclictest4761-21cut19:28:301
92439932,0cyclictest4624-21sensors20:13:361
92439932,0cyclictest3514-21snmpd19:46:151
92439932,0cyclictest3241-21irqbalance20:18:321
92439932,0cyclictest23077-21awk22:33:271
92429932,0cyclictest9916-21rm22:03:330
92429932,0cyclictest5648-21sensors20:38:360
92429932,0cyclictest3514-21snmpd19:45:450
92429932,0cyclictest3241-21irqbalance19:28:320
92429932,0cyclictest23838-21sh23:12:190
92429932,0cyclictest23658-21sensors20:03:350
92429932,0cyclictest10999-21awk23:43:280
92429931,0cyclictest0-21swapper/020:53:310
92429930,0cyclictest19387-21diskstats20:28:270
29715030,0irq/26-eth0-rx-0-21swapper/320:18:343
29715030,0irq/26-eth0-rx-0-21swapper/320:08:323
29715030,0irq/26-eth0-rx-0-21swapper/319:53:263
29715030,0irq/26-eth0-rx-0-21swapper/319:43:093
29715030,0irq/26-eth0-rx-0-21swapper/319:28:253
29715030,0irq/26-eth0-rx-0-21swapper/319:18:263
29715030,0irq/26-eth0-rx-0-21swapper/300:09:073
249930,0watchdog/20-21swapper/219:38:362
179930,0migration/119156-21kthreadcore19:38:361
169930,0watchdog/10-21swapper/122:13:241
149930,0migration/023267-21sh22:53:180
92429921,0cyclictest3241-21irqbalance19:53:220
92429921,0cyclictest0-21swapper/019:38:290
92429921,0cyclictest0-21swapper/019:18:140
92429920,0cyclictest0-21swapper/022:58:190
329920,0watchdog/30-21swapper/319:23:333
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional