You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 12:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Wed Mar 25, 2026 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304450590,0irq/25-eth10-21swapper/318:55:383
2734623626,0sleep20-21swapper/218:58:002
2751423524,0sleep00-21swapper/018:59:580
2658023424,0sleep10-21swapper/118:55:471
276279983,0cyclictest19279-21awk20:30:390
276279980,0cyclictest22193-21munin-node19:20:380
276279980,0cyclictest11798-21munin-node20:25:380
276279973,0cyclictest3556-21lldpd19:01:130
276279972,0cyclictest729-21awk23:05:370
276279972,0cyclictest31760-21awk20:15:400
276279972,0cyclictest31583-21dump-pmu-power23:25:240
276279972,0cyclictest2993-21awk19:05:400
276279972,0cyclictest29592-21turbostat19:50:250
276279972,0cyclictest28042-21grep22:20:420
276279972,0cyclictest2653-21awk21:05:390
276279972,0cyclictest25681-21sadc21:40:250
276279972,0cyclictest2443-21awk21:45:400
276279972,0cyclictest16095-21awk23:55:400
276279971,0cyclictest32446-21awk23:45:410
276279971,0cyclictest21967-21sadc19:20:250
276279971,0cyclictest18606-21awk21:55:420
276279970,0cyclictest930-21ssh21:44:200
276279970,0cyclictest8601-21netstat19:30:440
276279970,0cyclictest5064-21munin-node20:20:430
276279970,0cyclictest24381-21irqstats00:00:420
276279970,0cyclictest2156-21munin-node22:05:400
276279970,0cyclictest19739-21cut00:15:430
276279970,0cyclictest16499-21munin-node19:40:470
276279970,0cyclictest16238-21ssh22:13:190
276279970,0cyclictest16034-21ssh22:55:210
276279970,0cyclictest13216-21awk23:30:460
276279970,0cyclictest1229-21ssh21:24:040
276279970,0cyclictest10134-21tune2fs21:50:380
276279970,0cyclictest10134-21tune2fs21:50:380
276279970,0cyclictest0-21swapper/020:10:460
339960,0migration/330707-21sh23:22:513
339960,0migration/324078-21sh22:17:573
339960,0migration/317726-21sh21:14:563
339960,0migration/315550-21munin-node23:35:363
29715060,0irq/26-eth0-rx-25882-21ssh21:20:103
29715060,0irq/26-eth0-rx-17309-21if_eth1.322:55:403
29715060,0irq/26-eth0-rx-1450-21ssh22:45:403
276309966,0cyclictest0-21swapper/220:35:422
276279961,0cyclictest31857-21aten_r1power_po20:40:360
276279961,0cyclictest25714-21grep20:35:380
276279960,0cyclictest9962-21aten_r1power_po19:35:360
276279960,0cyclictest9955-21cat21:30:250
276279960,0cyclictest9745-21latency_hist21:10:250
276279960,0cyclictest9216-21tr19:10:370
276279960,0cyclictest8786-21sh22:30:330
276279960,0cyclictest7914-21sh23:10:300
276279960,0cyclictest6702-21rm00:10:280
276279960,0cyclictest654-21sh22:25:250
276279960,0cyclictest639-21awk22:45:370
276279960,0cyclictest6338-21cpuspeed_turbos20:45:350
276279960,0cyclictest4059-21cut19:55:250
276279960,0cyclictest31842-21sh23:25:330
276279960,0cyclictest31021-21cat00:05:240
276279960,0cyclictest30636-21rm00:25:330
276279960,0cyclictest29742-21date19:50:250
276279960,0cyclictest28949-21munin-node19:25:380
276279960,0cyclictest26244-21cpu21:00:370
276279960,0cyclictest25548-21rm22:00:260
276279960,0cyclictest24669-21sh22:40:350
276279960,0cyclictest24288-21sh23:00:320
276279960,0cyclictest23483-21basename23:40:350
276279960,0cyclictest22623-21cat00:20:240
276279960,0cyclictest19503-21aten_r1power_fr20:55:380
276279960,0cyclictest1936-21sh21:25:290
276279960,0cyclictest18817-21uniq21:15:380
276279960,0cyclictest17642-21entropy22:15:390
276279960,0cyclictest17587-21aten_r1power_vo20:05:340
276279960,0cyclictest16752-21sh22:35:340
276279960,0cyclictest16286-21ssh22:55:280
276279960,0cyclictest15863-21sh23:15:260
276279960,0cyclictest15382-21sh23:35:250
276279960,0cyclictest12734-21munin-node20:50:370
276279960,0cyclictest11018-21cpuspeed_turbos20:00:400
276279960,0cyclictest0-21swapper/023:50:240
339950,0migration/38661-21kthreadcore20:45:403
339950,0migration/34287-21munin-node00:05:453
339950,0migration/33790-21taskset19:54:143
339950,0migration/332249-21kthreadcore20:15:413
339950,0migration/331322-21kthreadcore19:25:433
339950,0migration/329191-21sendmail_mailst23:40:453
339950,0migration/327001-21kthreadcore20:10:423
339950,0migration/325770-21kthreadcore19:45:413
339950,0migration/325494-21idleruntime22:40:403
339950,0migration/32509-21fschecks_count21:25:383
339950,0migration/321636-21kthreadcore20:05:403
339950,0migration/320706-21kthreadcore21:55:433
339950,0migration/320528-21kthreadcore21:35:423
339950,0migration/32010-21munin-node22:05:393
339950,0migration/319823-21munin-node20:55:403
339950,0migration/317436-21fschecks_time22:35:383
339950,0migration/315397-21latency21:30:433
339950,0migration/315166-21munin-node23:55:353
339950,0migration/312113-21cpuspeed_turbos20:25:383
339950,0migration/311879-21kthreadcore21:50:423
339950,0migration/311879-21kthreadcore21:50:423
339950,0migration/311524-21taskset20:25:033
29725050,0irq/27-eth0-tx-9203-21df_inode22:30:382
29725050,0irq/27-eth0-tx-8682-21df22:50:362
29725050,0irq/27-eth0-tx-7214-21munin-node21:05:422
29725050,0irq/27-eth0-tx-13028-21sh23:30:452
29715050,0irq/26-eth0-rx-6559-21runrttasks21:41:053
29715050,0irq/26-eth0-rx-31099-21yum20:35:493
29715050,0irq/26-eth0-rx-2928-21grep00:25:433
29715050,0irq/26-eth0-rx-21001-21ssh00:16:093
29715050,0irq/26-eth0-rx-11089-21kthreadcore23:10:423
29715050,0irq/26-eth0-rx-0-21swapper/323:45:283
29715050,0irq/26-eth0-rx-0-21swapper/323:25:373
29715050,0irq/26-eth0-rx-0-21swapper/323:16:153
29715050,0irq/26-eth0-rx-0-21swapper/323:07:403
29715050,0irq/26-eth0-rx-0-21swapper/322:50:413
29715050,0irq/26-eth0-rx-0-21swapper/322:34:143
29715050,0irq/26-eth0-rx-0-21swapper/322:24:113
29715050,0irq/26-eth0-rx-0-21swapper/322:12:313
29715050,0irq/26-eth0-rx-0-21swapper/322:03:413
29715050,0irq/26-eth0-rx-0-21swapper/321:45:473
29715050,0irq/26-eth0-rx-0-21swapper/321:20:433
276309955,0cyclictest1293-21ssh21:45:232
276309955,0cyclictest0-21swapper/223:28:492
276309955,0cyclictest0-21swapper/222:08:152
276309950,0cyclictest0-21swapper/222:26:422
276309950,0cyclictest0-21swapper/221:37:112
276309950,0cyclictest0-21swapper/221:14:102
276299955,0cyclictest0-21swapper/121:13:191
276299955,0cyclictest0-21swapper/119:05:461
276299950,0cyclictest0-21swapper/122:44:261
276299950,0cyclictest0-21swapper/122:28:591
27150,0ktimersoftd/20-21swapper/221:52:572
27150,0ktimersoftd/20-21swapper/221:52:572
259950,0migration/29840-21latency_hist21:30:252
259950,0migration/26670-21latency_hist00:10:242
259950,0migration/2537-21munin-node23:05:362
259950,0migration/227466-21kthreadcore23:20:422
259950,0migration/225902-21kthreadcore23:00:402
259950,0migration/222208-21switchtime19:40:472
259950,0migration/218437-21aten_r1power_cu20:30:352
249950,0watchdog/219919-21kthreadcore21:55:432
249950,0watchdog/217638-21kthreadcore20:50:432
179950,0migration/123716-21mdstat20:30:431
179950,0migration/119249-21kthreadcore22:35:411
179950,0migration/116776-21irqrtprio19:15:411
179950,0migration/113532-21munin-node23:30:481
339940,0migration/3650-21diskmemload22:25:253
339940,0migration/33646-21kthreadcore20:40:433
339940,0migration/327909-21kthreadcore00:20:413
339940,0migration/325030-21kthreadcore00:00:433
339940,0migration/317040-21kthreadcore20:50:433
339940,0migration/316892-21df_inode19:40:383
339940,0migration/314506-21kthreadcore19:35:423
29725040,0irq/27-eth0-tx-31088-21ssh00:05:302
29725040,0irq/27-eth0-tx-26624-21interrupts22:00:422
29725040,0irq/27-eth0-tx-25205-21munin-node22:20:372
29725040,0irq/27-eth0-tx-15119-21munin-node23:55:362
29725040,0irq/27-eth0-tx-0-21swapper/223:50:262
29725040,0irq/27-eth0-tx-0-21swapper/223:45:382
29725040,0irq/27-eth0-tx-0-21swapper/223:35:542
29725040,0irq/27-eth0-tx-0-21swapper/223:10:512
29725040,0irq/27-eth0-tx-0-21swapper/222:45:412
29725040,0irq/27-eth0-tx-0-21swapper/222:35:352
29725040,0irq/27-eth0-tx-0-21swapper/221:45:462
29725040,0irq/27-eth0-tx-0-21swapper/200:25:352
29725040,0irq/27-eth0-tx-0-21swapper/200:15:252
29715040,0irq/26-eth0-rx-10879-21kthreadcore23:30:423
29715040,0irq/26-eth0-rx-0-21swapper/323:50:523
29715040,0irq/26-eth0-rx-0-21swapper/323:00:543
29715040,0irq/26-eth0-rx-0-21swapper/321:05:263
29715040,0irq/26-eth0-rx-0-21swapper/321:00:443
29715040,0irq/26-eth0-rx-0-21swapper/300:11:153
276309943,0cyclictest5035-21awk19:55:402
276309943,0cyclictest3556-21lldpd19:13:122
276309943,0cyclictest3556-21lldpd19:00:432
276309943,0cyclictest23582-21awk00:20:382
276309943,0cyclictest20333-21awk20:55:422
276309943,0cyclictest16871-21awk23:15:412
276309943,0cyclictest10239-21awk22:10:392
276309941,0cyclictest0-21swapper/222:15:432
276299944,0cyclictest0-21swapper/123:12:591
276299944,0cyclictest0-21swapper/121:28:551
276299944,0cyclictest0-21swapper/120:45:431
276299943,0cyclictest9300-21awk22:50:391
276299943,0cyclictest8040-21awk23:50:381
276299943,0cyclictest771-21awk23:05:381
276299943,0cyclictest5917-21awk20:20:411
276299943,0cyclictest3514-21snmpd00:25:471
276299943,0cyclictest32620-21awk20:40:401
276299943,0cyclictest3241-21irqbalance23:41:131
276299943,0cyclictest3241-21irqbalance22:47:531
276299943,0cyclictest3241-21irqbalance21:40:331
276299943,0cyclictest3241-21irqbalance21:36:131
276299943,0cyclictest3241-21irqbalance21:05:331
276299943,0cyclictest3241-21irqbalance21:01:231
276299943,0cyclictest31618-21awk20:15:391
276299943,0cyclictest26032-21awk20:35:401
276299943,0cyclictest24839-21awk23:20:401
276299943,0cyclictest24128-21awk19:45:391
276299943,0cyclictest2157-21awk22:05:401
276299943,0cyclictest16820-21awk23:15:401
276299943,0cyclictest11705-21awk20:00:431
276299943,0cyclictest10649-21awk21:50:401
276299943,0cyclictest10649-21awk21:50:401
276299943,0cyclictest10212-21awk22:10:391
259940,0migration/24749-21missed_timers20:40:432
259940,0migration/232120-21switchtime21:20:482
259940,0migration/230042-21cpuspeed_turbos19:50:382
259940,0migration/227476-21kthreadcore00:00:442
259940,0migration/227144-21irqcore21:00:412
259940,0migration/225346-21kthreadcore23:40:402
259940,0migration/225037-21df_abs22:40:382
259940,0migration/223478-21cpuspeed_turbos19:45:362
259940,0migration/219898-21kthreadcore22:55:422
259940,0migration/217899-21fschecks_count20:05:362
249940,0watchdog/215364-21cut21:30:432
249940,0watchdog/20-21swapper/221:15:392
249940,0watchdog/20-21swapper/220:15:432
19140,0ktimersoftd/110564-21sadc20:00:251
179940,0migration/19713-21idleruntime19:10:391
179940,0migration/16876-21kthreadcore19:30:431
179940,0migration/14280-21kthreadcore21:45:421
179940,0migration/132292-21kthreadcore19:00:431
179940,0migration/131903-21aten_r1power_en23:25:371
179940,0migration/131131-21taskset00:05:361
179940,0migration/129299-21kthreadcore23:00:421
179940,0migration/128989-21taskset00:21:071
179940,0migration/124320-21cpu20:10:371
179940,0migration/123909-21kthreadcore19:20:421
179940,0migration/123335-21cpuspeed_turbos00:00:371
179940,0migration/114598-21latency_hist00:15:241
169940,0watchdog/19571-21cut22:30:391
169940,0watchdog/1867-21ssh21:23:111
169940,0watchdog/15348-21ssh23:46:151
169940,0watchdog/127451-21kthreadcore22:00:431
169940,0watchdog/120883-21ssh23:55:471
169940,0watchdog/120114-21kthreadcore22:15:431
169940,0watchdog/113955-21ssh00:13:581
169940,0watchdog/10-21swapper/123:36:111
169940,0watchdog/10-21swapper/122:55:431
169940,0watchdog/10-21swapper/121:32:231
169940,0watchdog/10-21swapper/121:15:391
8215030,0irq/24-0000:00:0-21swapper/220:00:422
29715030,0irq/26-eth0-rx-0-21swapper/320:04:283
29715030,0irq/26-eth0-rx-0-21swapper/319:59:483
29715030,0irq/26-eth0-rx-0-21swapper/319:20:373
29715030,0irq/26-eth0-rx-0-21swapper/319:15:433
29715030,0irq/26-eth0-rx-0-21swapper/319:04:273
276309932,0cyclictest7171-21cut19:30:432
276309932,0cyclictest6980-21awk20:45:382
276309932,0cyclictest3121-21awk19:05:412
276309932,0cyclictest29689-21awk19:25:412
276309932,0cyclictest27851-21sensors19:20:462
276309932,0cyclictest12650-21awk20:25:402
276309932,0cyclictest10717-21awk19:35:392
276309930,0cyclictest16212-21perl19:15:392
276299933,0cyclictest17336-21awk19:40:391
276299932,0cyclictest3241-21irqbalance20:55:431
276299932,0cyclictest12777-21awk20:25:411
259930,0migration/26344-21irqrtprio20:20:432
249930,0watchdog/20-21swapper/220:10:272
179930,0migration/131044-21kthreadcore19:25:431
179930,0migration/127168-21kthreadcore22:20:421
169930,0watchdog/10-21swapper/121:55:391
169930,0watchdog/10-21swapper/120:50:431
169930,0watchdog/10-21swapper/120:05:391
169930,0watchdog/10-21swapper/119:50:441
169930,0watchdog/10-21swapper/119:36:521
35120,0ktimersoftd/324365-21munin-node20:30:473
339920,0migration/32523-21diskstats19:05:383
29715020,0irq/26-eth0-rx-0-21swapper/319:30:493
29715020,0irq/26-eth0-rx-0-21swapper/319:10:433
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional