You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 19:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Sun Nov 30, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
731525243,0sleep20-21swapper/206:38:102
737724131,0sleep30-21swapper/306:38:553
728223725,0sleep10-21swapper/106:37:491
703123524,0sleep00-21swapper/006:34:520
249970,0watchdog/20-21swapper/211:59:372
74889960,0cyclictest0-21swapper/209:32:502
74879960,0cyclictest0-21swapper/111:17:181
74869960,0cyclictest0-21swapper/009:21:410
4160,0ktimersoftd/011455-21sh12:06:170
339960,0migration/35325-21sh11:01:113
339960,0migration/32934-21sh11:59:443
339960,0migration/325828-21munin-node08:29:333
339960,0migration/320468-21sh11:52:353
339960,0migration/311137-21sh09:43:163
28415060,0irq/27-eth0-tx-28407-21munin-node09:34:412
28415060,0irq/27-eth0-tx-0-21swapper/212:07:262
28415060,0irq/27-eth0-tx-0-21swapper/211:57:232
28415060,0irq/27-eth0-tx-0-21swapper/211:49:362
28415060,0irq/27-eth0-tx-0-21swapper/211:31:522
28415060,0irq/27-eth0-tx-0-21swapper/211:24:582
28415060,0irq/27-eth0-tx-0-21swapper/211:22:442
28415060,0irq/27-eth0-tx-0-21swapper/211:19:172
28415060,0irq/27-eth0-tx-0-21swapper/211:08:422
28415060,0irq/27-eth0-tx-0-21swapper/210:39:032
28415060,0irq/27-eth0-tx-0-21swapper/210:19:352
28415060,0irq/27-eth0-tx-0-21swapper/210:11:032
28415060,0irq/27-eth0-tx-0-21swapper/209:39:342
28415060,0irq/27-eth0-tx-0-21swapper/209:26:522
28415060,0irq/27-eth0-tx-0-21swapper/209:10:082
28415060,0irq/27-eth0-tx-0-21swapper/008:40:310
28415060,0irq/27-eth0-tx-0-21swapper/007:29:320
28405060,0irq/26-eth0-rx-935-21irqcore10:39:443
28405060,0irq/26-eth0-rx-7348-21memory06:59:373
28405060,0irq/26-eth0-rx-12246-21cut10:24:363
259960,0migration/25799-21fschecks_count08:59:322
249960,0watchdog/223919-21cut11:34:372
169960,0watchdog/16883-21diskstats11:04:331
169960,0watchdog/10-21swapper/110:15:291
169960,0watchdog/10-21swapper/109:54:371
149960,0migration/012397-21kthreadcore10:24:360
74889950,0cyclictest0-21swapper/210:44:532
74879955,0cyclictest0-21swapper/111:35:071
74879955,0cyclictest0-21swapper/111:31:291
74879955,0cyclictest0-21swapper/110:04:091
74879954,0cyclictest12829-21ps08:19:351
74879950,0cyclictest0-21swapper/109:35:551
74869955,0cyclictest0-21swapper/011:09:120
74869955,0cyclictest0-21swapper/010:20:100
74869955,0cyclictest0-21swapper/009:39:490
74869950,0cyclictest0-21swapper/011:19:460
74869950,0cyclictest0-21swapper/010:50:270
74869950,0cyclictest0-21swapper/010:31:220
339950,0migration/38642-21switchtime09:59:433
339950,0migration/38626-21kthreadcore11:44:373
339950,0migration/3525-21missed_timers06:54:383
339950,0migration/35076-21cpuspeed_turbos12:04:323
339950,0migration/34887-21proc_pri08:34:393
339950,0migration/34587-21kthreadcore07:24:363
339950,0migration/34275-21runrttasks11:20:333
339950,0migration/332069-21latency_hist10:39:183
339950,0migration/330719-21taskset10:59:073
339950,0migration/330402-21hddtemp_smartct08:54:333
339950,0migration/329553-21latency_hist11:39:183
339950,0migration/326610-21fschecks_time10:14:343
339950,0migration/324423-21df_abs08:04:323
339950,0migration/32419-21needreboot09:14:373
339950,0migration/324109-21taskset10:34:133
339950,0migration/323018-21sh11:14:333
339950,0migration/321144-21sh11:12:013
339950,0migration/320342-21hwlatdetect09:49:353
339950,0migration/319204-21kthreadcore07:34:373
339950,0migration/315103-21taskset07:09:053
339950,0migration/314977-21fschecks_time06:44:343
339950,0migration/314075-21kthreadcore08:19:353
28415050,0irq/27-eth0-tx-6372-21iceccd07:34:430
28415050,0irq/27-eth0-tx-3481-21ssh09:15:492
28415050,0irq/27-eth0-tx-15366-21munin-node07:09:410
28415050,0irq/27-eth0-tx-0-21swapper/211:13:592
28415050,0irq/27-eth0-tx-0-21swapper/210:59:562
28415050,0irq/27-eth0-tx-0-21swapper/210:50:352
28415050,0irq/27-eth0-tx-0-21swapper/210:26:052
28415050,0irq/27-eth0-tx-0-21swapper/210:04:382
28415050,0irq/27-eth0-tx-0-21swapper/210:00:072
28415050,0irq/27-eth0-tx-0-21swapper/209:44:542
28415050,0irq/27-eth0-tx-0-21swapper/209:20:112
28415050,0irq/27-eth0-tx-0-21swapper/208:56:162
28415050,0irq/27-eth0-tx-0-21swapper/208:50:582
28415050,0irq/27-eth0-tx-0-21swapper/208:47:032
28415050,0irq/27-eth0-tx-0-21swapper/208:43:242
28415050,0irq/27-eth0-tx-0-21swapper/008:29:300
28415050,0irq/27-eth0-tx-0-21swapper/008:04:400
28415050,0irq/27-eth0-tx-0-21swapper/008:01:340
28415050,0irq/27-eth0-tx-0-21swapper/007:53:250
28405050,0irq/26-eth0-rx-8699-21kthreadcore08:59:353
28405050,0irq/26-eth0-rx-284422sleep307:39:383
28405050,0irq/26-eth0-rx-284422sleep307:39:383
28405050,0irq/26-eth0-rx-24169-21kthreadcore09:09:353
28405050,0irq/26-eth0-rx-21555-21cpuspeed_turbos11:54:323
28405050,0irq/26-eth0-rx-17864-21yum09:44:443
28405050,0irq/26-eth0-rx-14183-21df08:44:363
28405050,0irq/26-eth0-rx-0-21swapper/311:31:583
28405050,0irq/26-eth0-rx-0-21swapper/310:53:593
28405050,0irq/26-eth0-rx-0-21swapper/310:47:133
28405050,0irq/26-eth0-rx-0-21swapper/310:21:023
28405050,0irq/26-eth0-rx-0-21swapper/310:08:573
28405050,0irq/26-eth0-rx-0-21swapper/309:35:073
28405050,0irq/26-eth0-rx-0-21swapper/309:32:513
28405050,0irq/26-eth0-rx-0-21swapper/309:19:323
28405050,0irq/26-eth0-rx-0-21swapper/308:40:483
259950,0migration/221839-21kthreadcore10:29:372
259950,0migration/221479-21kthreadcore09:49:382
259950,0migration/21862-21latency_hist06:59:192
249950,0watchdog/227975-21fschecks_time09:54:332
249950,0watchdog/216979-21perl07:34:332
249950,0watchdog/20-21swapper/210:55:172
179950,0migration/19574-21proc_pri09:39:401
179950,0migration/15937-21fschecks_time11:44:331
179950,0migration/132016-21kthreadcore07:19:491
179950,0migration/130103-21sh11:39:321
179950,0migration/127666-21sh09:09:551
179950,0migration/122219-21munin-node08:49:301
179950,0migration/113013-21taskset08:42:271
179950,0migration/111524-21latency_hist08:19:181
169950,0watchdog/17278-21kthreadcore11:24:371
169950,0watchdog/130741-21sed07:44:371
169950,0watchdog/120876-21proc_pri07:09:411
169950,0watchdog/118268-21grep08:44:411
169950,0watchdog/10-21swapper/111:13:531
169950,0watchdog/10-21swapper/110:19:331
169950,0watchdog/10-21swapper/109:29:411
169950,0watchdog/10-21swapper/109:23:171
169950,0watchdog/10-21swapper/108:24:371
149950,0migration/09464-21kthreadcore06:39:350
149950,0migration/0890-21kthreadcore11:39:370
149950,0migration/07982-21kthreadcore07:24:370
149950,0migration/06000-21kthreadcore06:59:360
149950,0migration/04562-21sh10:59:410
149950,0migration/03559-21hwlatdetect09:59:340
149950,0migration/032751-21irqcore08:34:350
149950,0migration/028710-21taskset07:19:320
149950,0migration/027595-21sh11:55:180
149950,0migration/027181-21latency_hist09:54:180
149950,0migration/025944-21kthreadcore10:54:360
149950,0migration/02499-21kthreadcore08:54:370
149950,0migration/024888-21kthreadcore11:14:360
149950,0migration/023229-21kthreadcore06:49:410
149950,0migration/022250-21kthreadcore09:29:370
149950,0migration/022145-21aten_r1power_vo07:14:310
149950,0migration/02140-21latency11:59:380
149950,0migration/021309-21sh11:33:350
149950,0migration/021072-21sh08:47:540
149950,0migration/01705-21kthreadcore07:44:390
149950,0migration/016289-21munin-node07:54:400
149950,0migration/01543-21sh09:56:130
149950,0migration/013227-21sh11:28:190
149950,0migration/012478-21sh11:47:320
149950,0migration/011383-21kthreadcore10:44:380
149950,0migration/010993-21kthreadcore07:04:370
74889944,0cyclictest0-21swapper/207:49:342
74889943,0cyclictest8604-21awk06:39:342
74889943,0cyclictest5927-21awk08:14:342
74889943,0cyclictest25842-21awk08:29:332
74889942,0cyclictest0-21swapper/211:39:382
74879944,0cyclictest17825-21kworker/1:110:26:521
74879944,0cyclictest0-21swapper/110:59:291
74879943,0cyclictest6445-21snmpd10:39:191
74879943,0cyclictest3425-21lldpd07:55:251
74879943,0cyclictest3425-21lldpd07:05:391
74879943,0cyclictest3425-21lldpd06:39:341
74879943,0cyclictest22208-21awk11:54:341
74879943,0cyclictest18289-21awk07:59:341
74879943,0cyclictest17697-21awk10:29:341
74869944,0cyclictest0-21swapper/009:35:520
74869943,0cyclictest6445-21snmpd11:34:340
74869943,0cyclictest6445-21snmpd11:12:190
74869943,0cyclictest6445-21snmpd08:24:180
74869943,0cyclictest3425-21lldpd09:14:190
74869943,0cyclictest22842-21awk08:49:330
74869943,0cyclictest12672-21awk09:44:340
339940,0migration/37741-21aten_r1power_en06:39:303
339940,0migration/35273-21irqstats07:49:353
339940,0migration/325468-21kthreadcore06:49:413
339940,0migration/319487-21irqcore08:24:363
339940,0migration/316873-21kthreadcore07:09:373
339940,0migration/314914-21kthreadcore09:24:383
28415040,0irq/27-eth0-tx-3077-21cut10:39:452
28415040,0irq/27-eth0-tx-0-21swapper/211:45:152
28415040,0irq/27-eth0-tx-0-21swapper/209:04:382
28405040,0irq/26-eth0-rx-0-21swapper/311:42:223
28405040,0irq/26-eth0-rx-0-21swapper/311:24:233
28405040,0irq/26-eth0-rx-0-21swapper/311:07:283
28405040,0irq/26-eth0-rx-0-21swapper/310:09:293
28405040,0irq/26-eth0-rx-0-21swapper/309:54:373
28405040,0irq/26-eth0-rx-0-21swapper/309:04:413
28405040,0irq/26-eth0-rx-0-21swapper/308:53:243
28405040,0irq/26-eth0-rx-0-21swapper/307:20:593
259940,0migration/231843-21kthreadcore07:19:492
259940,0migration/226888-21kthreadcore08:04:362
259940,0migration/224242-21irqrtprio07:39:362
259940,0migration/224242-21irqrtprio07:39:362
259940,0migration/221493-21kthreadcore07:59:362
259940,0migration/21831-21kthreadcore08:34:362
259940,0migration/21335-21kthreadcore08:09:372
259940,0migration/212550-21if_err_eth1.308:19:332
249940,0watchdog/211070-21tr07:04:372
249940,0watchdog/20-21swapper/210:14:332
249940,0watchdog/20-21swapper/207:44:372
249940,0watchdog/20-21swapper/206:54:372
249940,0watchdog/20-21swapper/206:49:372
179940,0migration/17383-21kthreadcore08:59:351
179940,0migration/14789-21sh12:04:281
179940,0migration/13027-21aten2.4-expect07:24:301
179940,0migration/125460-21fschecks_count08:29:321
179940,0migration/120140-21runrttasks11:51:361
179940,0migration/117388-21kthreadcore09:24:391
179940,0migration/117088-21kthreadcore06:44:371
179940,0migration/116095-21taskset07:33:341
179940,0migration/110944-21kthreadcore10:44:381
169940,0watchdog/14359-21ssh11:20:491
169940,0watchdog/132592-21tr06:54:371
169940,0watchdog/130055-21ssh10:56:571
169940,0watchdog/12426-21port_sshpriv11:59:411
169940,0watchdog/122763-21smartctl07:14:331
169940,0watchdog/117871-21ssh09:44:451
169940,0watchdog/110575-21ssh10:04:211
169940,0watchdog/10-21swapper/110:38:331
169940,0watchdog/10-21swapper/110:11:291
169940,0watchdog/10-21swapper/109:49:451
169940,0watchdog/10-21swapper/109:14:491
169940,0watchdog/10-21swapper/108:54:331
169940,0watchdog/10-21swapper/107:39:331
169940,0watchdog/10-21swapper/107:39:331
169940,0watchdog/10-21swapper/107:34:371
169940,0watchdog/10-21swapper/106:49:371
149940,0migration/04899-21latency_hist08:14:190
149940,0migration/025919-21sh10:14:270
149940,0migration/020962-21taskset09:08:530
149940,0migration/018452-21kthreadcore06:44:380
149940,0migration/017690-21sh10:08:000
149940,0migration/017030-21missed_timers08:19:370
8225030,0irq/24-0000:00:0-21swapper/206:44:192
74889932,0cyclictest3918-21awk07:24:342
74889932,0cyclictest19201-21awk08:24:342
74889932,0cyclictest10102-21df_inode07:29:322
74879933,0cyclictest17825-21kworker/1:108:09:201
74879933,0cyclictest17825-21kworker/1:107:49:221
74879933,0cyclictest17825-21kworker/1:106:59:301
74879933,0cyclictest14231-21awk09:04:331
74879932,0cyclictest31918-21egrep08:34:311
74879930,0cyclictest24668-21perl08:04:331
74869932,0cyclictest6445-21snmpd11:52:160
74869932,0cyclictest6245-21awk08:59:330
74869932,0cyclictest32237-21basename10:39:300
74869932,0cyclictest31492-21ssh10:37:580
74869932,0cyclictest28534-21awk06:54:340
74869932,0cyclictest19367-21awk10:09:340
74869932,0cyclictest19104-21ssh09:25:380
74869930,0cyclictest24694-21grep09:09:350
74869930,0cyclictest0-21swapper/008:09:310
339930,0migration/317023-21runrttasks07:58:343
28415030,0irq/27-eth0-tx-0-21swapper/007:43:320
28415030,0irq/27-eth0-tx-0-21swapper/007:43:320
28405030,0irq/26-eth0-rx-0-21swapper/308:14:303
28405030,0irq/26-eth0-rx-0-21swapper/308:09:393
28405030,0irq/26-eth0-rx-0-21swapper/307:44:333
259930,0migration/221052-21switchtime07:09:432
249930,0watchdog/20-21swapper/207:54:372
169930,0watchdog/10-21swapper/110:49:211
74889921,0cyclictest22082-21aten2.4-expect07:14:302
329920,0watchdog/327080-21kthreadcore07:14:373
28405020,0irq/26-eth0-rx-0-21swapper/307:59:313
28405020,0irq/26-eth0-rx-0-21swapper/307:29:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional