You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 20:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Mon Mar 16, 2026 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304450350,0irq/25-eth10-21swapper/306:56:183
2424723425,0sleep20-21swapper/207:00:532
2406023424,0sleep10-21swapper/106:58:401
1790222622,0sleep00-21swapper/006:56:180
339960,0migration/37357-21kthreadcore12:11:293
339960,0migration/34427-21munin-node09:06:353
339960,0migration/320239-21needreboot09:36:313
339960,0migration/320048-21munin-node07:21:283
339960,0migration/314937-21munin-node10:56:293
339960,0migration/311822-21sh10:51:473
29725060,0irq/27-eth0-tx-9378-21munin-node08:51:330
29725060,0irq/27-eth0-tx-32197-21munin-node07:31:310
29725060,0irq/27-eth0-tx-28407-21munin-node08:41:260
29725060,0irq/27-eth0-tx-22747-21munin-node09:01:290
29715060,0irq/26-eth0-rx-15731-21ps09:16:283
243459960,0cyclictest0-21swapper/108:56:311
243449960,0cyclictest0-21swapper/009:30:230
149960,0migration/05132-21latency_hist11:11:130
149960,0migration/028787-21sh10:02:040
149960,0migration/022038-21sh09:19:380
149960,0migration/020569-21ntp_offset08:31:330
149960,0migration/013782-21sh09:55:030
149960,0migration/011355-21sh11:32:010
8215050,0irq/24-0000:00:7281-21latency_hist08:01:132
8215050,0irq/24-0000:00:20523-21cat12:01:132
35150,0ktimersoftd/30-21swapper/308:56:353
339950,0migration/37629-21irqrtprio09:51:283
339950,0migration/35698-21fschecks_time11:31:273
339950,0migration/3425-21kthreadcore09:26:293
339950,0migration/330963-21irqcore10:46:283
339950,0migration/330804-21fschecks_time10:26:253
339950,0migration/327377-21unixbench_multi11:21:353
339950,0migration/32704-21kthreadcore09:46:303
339950,0migration/326909-21kthreadcore09:21:303
339950,0migration/326798-21munin-node12:01:363
339950,0migration/326677-21kthreadcore10:01:313
339950,0migration/32607-21proc_pri11:26:333
339950,0migration/325217-21kthreadcore09:01:293
339950,0migration/322289-21latency_hist10:01:133
339950,0migration/321849-21latency_hist10:41:133
339950,0migration/320636-21cpuspeed_turbos12:21:243
339950,0migration/318368-21taskset07:18:043
339950,0migration/31506-21pmu-power08:41:333
339950,0migration/314185-21taskset10:16:193
339950,0migration/312813-21runrttasks10:33:453
339950,0migration/310026-21fschecks_time08:51:263
29725050,0irq/27-eth0-tx-28707-21which09:21:342
29725050,0irq/27-eth0-tx-27766-21users11:01:372
29725050,0irq/27-eth0-tx-14343-21forks10:36:252
29725050,0irq/27-eth0-tx-13871-21ssh09:14:462
29725050,0irq/27-eth0-tx-13632-21munin-node10:56:272
29725050,0irq/27-eth0-tx-0-21swapper/211:48:242
29725050,0irq/27-eth0-tx-0-21swapper/211:30:192
29725050,0irq/27-eth0-tx-0-21swapper/209:05:382
29725050,0irq/27-eth0-tx-0-21swapper/008:16:250
29715050,0irq/26-eth0-rx-6071-21hwlatdetect11:11:263
29715050,0irq/26-eth0-rx-23494-21kthreadcore11:41:293
29715050,0irq/26-eth0-rx-18935-21ssh11:56:493
29715050,0irq/26-eth0-rx-0-21swapper/312:16:223
29715050,0irq/26-eth0-rx-0-21swapper/312:07:573
29715050,0irq/26-eth0-rx-0-21swapper/311:36:313
29715050,0irq/26-eth0-rx-0-21swapper/311:16:543
29715050,0irq/26-eth0-rx-0-21swapper/311:06:273
29715050,0irq/26-eth0-rx-0-21swapper/310:10:293
29715050,0irq/26-eth0-rx-0-21swapper/309:44:163
29715050,0irq/26-eth0-rx-0-21swapper/309:31:193
29715050,0irq/26-eth0-rx-0-21swapper/309:11:363
29715050,0irq/26-eth0-rx-0-21swapper/307:51:273
29715050,0irq/26-eth0-rx-0-21swapper/307:36:263
259950,0migration/24926-21fschecks_time12:11:252
259950,0migration/230451-21idleruntime11:06:252
259950,0migration/222390-21kthreadcore07:46:302
243469955,0cyclictest0-21swapper/212:26:242
243469955,0cyclictest0-21swapper/210:22:102
243469955,0cyclictest0-21swapper/209:54:442
243469950,0cyclictest0-21swapper/211:40:032
243469950,0cyclictest0-21swapper/210:28:192
243469950,0cyclictest0-21swapper/210:16:482
243469950,0cyclictest0-21swapper/209:19:062
243459955,0cyclictest22504-21diskmemload11:10:511
243459955,0cyclictest0-21swapper/111:30:431
243459955,0cyclictest0-21swapper/110:49:531
243459955,0cyclictest0-21swapper/109:19:421
243459954,0cyclictest7894-21kthreadcore09:11:301
243459950,0cyclictest0-21swapper/112:29:031
243459950,0cyclictest0-21swapper/112:02:151
243459950,0cyclictest0-21swapper/110:03:411
243449955,0cyclictest0-21swapper/012:17:490
243449955,0cyclictest0-21swapper/011:55:450
243449955,0cyclictest0-21swapper/011:18:320
243449955,0cyclictest0-21swapper/011:13:580
243449950,0cyclictest0-21swapper/010:14:040
249950,0watchdog/2703-21grep12:06:302
249950,0watchdog/227904-21ssh12:04:462
249950,0watchdog/20-21swapper/209:43:312
19150,0ktimersoftd/10-21swapper/111:11:571
179950,0migration/14952-21taskset09:47:541
179950,0migration/14230-21munin-node10:06:361
179950,0migration/131248-21switchtime07:26:361
179950,0migration/129531-21taskset09:03:371
179950,0migration/128869-21latency_hist11:46:131
179950,0migration/119935-21sh11:59:461
179950,0migration/111346-21rt-features10:51:331
169950,0watchdog/114239-21cut08:06:231
149950,0migration/0787-21kthreadcore10:26:280
149950,0migration/05561-21taskset10:51:200
149950,0migration/05411-21latency_hist07:11:130
149950,0migration/05304-21sh12:11:270
149950,0migration/05086-21kthreadcore08:46:290
149950,0migration/031734-21kthreadcore11:26:290
149950,0migration/031618-21taskset07:27:220
149950,0migration/030635-21latency_hist09:06:130
149950,0migration/02908-21kthreadcore10:06:300
149950,0migration/026748-21munin-node07:51:250
149950,0migration/026700-21kthreadcore09:21:300
149950,0migration/026460-21kthreadcore07:01:300
149950,0migration/022893-21irqcore08:36:290
149950,0migration/022581-21df_abs10:21:250
149950,0migration/02217-21sh12:26:530
149950,0migration/02206-21pmu-power11:46:320
149950,0migration/021468-21gltestperf12:01:260
149950,0migration/02037-21cpuspeed_turbos08:21:240
149950,0migration/017839-21kthreadcore11:56:290
149950,0migration/017385-21kthreadcore11:36:310
149950,0migration/016651-21kthreadcore10:56:290
149950,0migration/016185-21kthreadcore09:56:290
149950,0migration/016145-21kthreadcore08:06:290
149950,0migration/015240-21kthreadcore07:41:290
149950,0migration/014927-21if_eth1.310:36:270
149950,0migration/014417-21sh09:35:570
8215040,0irq/24-0000:00:0-21swapper/208:36:132
339940,0migration/38976-21kthreadcore08:01:313
339940,0migration/34335-21ntp_offset07:06:323
339940,0migration/331253-21kthreadcore08:16:313
339940,0migration/33087-21kthreadcore08:21:283
339940,0migration/324887-21kthreadcore08:11:313
339940,0migration/320521-21fschecks_time07:46:263
339940,0migration/316558-21kthreadcore08:31:303
29725040,0irq/27-eth0-tx-30448-21ssh10:06:182
29725040,0irq/27-eth0-tx-0-21swapper/212:16:232
29725040,0irq/27-eth0-tx-0-21swapper/211:51:242
29725040,0irq/27-eth0-tx-0-21swapper/211:21:282
29725040,0irq/27-eth0-tx-0-21swapper/210:51:392
29725040,0irq/27-eth0-tx-0-21swapper/210:46:252
29725040,0irq/27-eth0-tx-0-21swapper/210:41:252
29725040,0irq/27-eth0-tx-0-21swapper/209:56:202
29725040,0irq/27-eth0-tx-0-21swapper/209:36:432
29725040,0irq/27-eth0-tx-0-21swapper/209:26:212
29725040,0irq/27-eth0-tx-0-21swapper/209:06:362
29725040,0irq/27-eth0-tx-0-21swapper/008:56:300
29725040,0irq/27-eth0-tx-0-21swapper/008:26:230
29725040,0irq/27-eth0-tx-0-21swapper/007:11:240
29715040,0irq/26-eth0-rx-0-21swapper/312:26:253
29715040,0irq/26-eth0-rx-0-21swapper/311:51:213
29715040,0irq/26-eth0-rx-0-21swapper/311:46:263
29715040,0irq/26-eth0-rx-0-21swapper/311:02:063
29715040,0irq/26-eth0-rx-0-21swapper/310:36:303
29715040,0irq/26-eth0-rx-0-21swapper/310:22:173
29715040,0irq/26-eth0-rx-0-21swapper/310:13:423
29715040,0irq/26-eth0-rx-0-21swapper/308:36:243
29715040,0irq/26-eth0-rx-0-21swapper/307:41:353
29715040,0irq/26-eth0-rx-0-21swapper/307:28:433
259940,0migration/27832-21kernelversion07:36:302
259940,0migration/25021-21kthreadcore08:46:292
259940,0migration/229836-21kthreadcore08:41:302
259940,0migration/228235-21gltestperf08:16:272
259940,0migration/227073-21kthreadcore07:26:302
259940,0migration/22651-21kthreadcore07:06:302
259940,0migration/224358-21open_files07:21:322
259940,0migration/219543-21kthreadcore08:56:302
259940,0migration/216740-21kthreadcore07:16:312
259940,0migration/21650-21kthreadcore07:31:312
259940,0migration/215549-21kthreadcore08:06:292
259940,0migration/210908-21kthreadcore08:51:292
259940,0migration/210573-21kthreadcore11:11:302
243469944,0cyclictest0-21swapper/211:42:442
243469943,0cyclictest6749-21awk10:31:282
243469943,0cyclictest3514-21snmpd08:13:542
243469943,0cyclictest31638-21awk09:46:272
243469943,0cyclictest21197-21awk12:21:262
243459943,0cyclictest32166-21awk07:06:281
243459943,0cyclictest29882-21awk11:46:261
243459943,0cyclictest2583-21awk08:21:261
243459943,0cyclictest23633-21awk09:41:271
243459943,0cyclictest14028-21awk07:41:271
243449944,0cyclictest0-21swapper/010:46:320
243449944,0cyclictest0-21swapper/009:43:190
243449944,0cyclictest0-21swapper/007:56:160
243449943,0cyclictest3556-21lldpd12:06:580
243449943,0cyclictest3556-21lldpd11:43:280
243449943,0cyclictest3556-21lldpd10:19:270
243449943,0cyclictest3514-21snmpd09:12:390
243449943,0cyclictest22239-21awk11:21:270
249940,0watchdog/27313-21cat11:31:312
249940,0watchdog/26559-21runrttasks07:54:152
249940,0watchdog/226969-21kthreadcore10:01:312
249940,0watchdog/20-21swapper/211:16:312
249940,0watchdog/20-21swapper/210:11:272
249940,0watchdog/20-21swapper/209:31:232
179940,0migration/19369-21kthreadcore07:11:301
179940,0migration/17781-21munin-node09:31:291
179940,0migration/15954-21kthreadcore07:56:301
179940,0migration/132235-21kthreadcore10:26:281
179940,0migration/131994-21if_err_eth1.209:26:271
179940,0migration/128012-21runrttasks08:39:291
179940,0migration/126019-21kthreadcore09:21:301
179940,0migration/123283-21kthreadcore07:46:301
179940,0migration/118501-21taskset12:16:431
179940,0migration/117968-21switchtime07:16:361
179940,0migration/117191-21kthreadcore10:56:301
179940,0migration/113795-21sh10:15:491
179940,0migration/111841-21mdstat09:51:311
169940,0watchdog/17307-21smartctl07:36:281
169940,0watchdog/132706-21fschecks_count07:31:281
169940,0watchdog/129827-21ssh10:25:511
169940,0watchdog/126795-21ssh11:21:311
169940,0watchdog/1265912sleep112:21:501
169940,0watchdog/114464-21ssh09:56:231
169940,0watchdog/114127-21expr11:16:271
169940,0watchdog/113771-21perl11:36:271
169940,0watchdog/10-21swapper/110:41:591
169940,0watchdog/10-21swapper/109:36:231
169940,0watchdog/10-21swapper/108:01:311
149940,0migration/08566-21kthreadcore07:36:300
149940,0migration/028490-21sh11:03:400
149940,0migration/025709-21timerwakeupswit07:46:360
149940,0migration/020998-21kthreadcore07:21:290
149940,0migration/011258-21kthreadcore08:01:320
29725030,0irq/27-eth0-tx-0-21swapper/008:11:250
29715030,0irq/26-eth0-rx-0-21swapper/308:46:233
29715030,0irq/26-eth0-rx-0-21swapper/308:31:093
29715030,0irq/26-eth0-rx-0-21swapper/308:06:553
29715030,0irq/26-eth0-rx-0-21swapper/307:59:553
29715030,0irq/26-eth0-rx-0-21swapper/307:32:393
29715030,0irq/26-eth0-rx-0-21swapper/307:11:253
29715030,0irq/26-eth0-rx-0-21swapper/307:01:253
259930,0migration/29713-21irqrtprio08:26:282
243469933,0cyclictest3556-21lldpd07:56:262
243459932,0cyclictest9804-21missed_timers12:11:311
243459932,0cyclictest9468-21awk08:26:271
243459932,0cyclictest8376-21sensors08:46:331
243459932,0cyclictest6559-21runrttasks10:17:121
243459932,0cyclictest31696-21kthreadcore12:06:301
243459932,0cyclictest27484-21awk07:51:291
243459932,0cyclictest218522cat07:21:291
243459932,0cyclictest10127-21grep08:51:261
243459930,0cyclictest23874-21grep11:01:291
243449933,0cyclictest6952-21kworker/0:210:31:140
243449933,0cyclictest27043-21kworker/0:209:46:140
243449933,0cyclictest27043-21kworker/0:209:36:240
243449933,0cyclictest27043-21kworker/0:207:16:190
243449933,0cyclictest14718-21kworker/0:112:21:140
243449933,0cyclictest14718-21kworker/0:110:41:140
249930,0watchdog/20-21swapper/208:21:232
249930,0watchdog/20-21swapper/207:41:242
179930,0migration/132050-21kthreadcore09:06:281
179930,0migration/130736-21runrttasks07:03:341
169930,0watchdog/10-21swapper/111:51:151
169930,0watchdog/10-21swapper/111:31:311
169930,0watchdog/10-21swapper/110:36:271
169930,0watchdog/10-21swapper/110:31:511
169930,0watchdog/10-21swapper/108:31:311
169930,0watchdog/10-21swapper/108:11:391
243469920,0cyclictest0-21swapper/207:11:242
249920,0watchdog/20-21swapper/208:36:152
249920,0watchdog/20-21swapper/207:01:202
169920,0watchdog/10-21swapper/108:41:151
169920,0watchdog/10-21swapper/108:16:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional