You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 10:36
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8.osadl.org (updated Fri Nov 14, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25064622280,4sleep0151ktimers/023:29:410
213123199271,22cyclictest2586858-21turbostat00:20:003
2131231992522,1cyclictest0-21swapper/300:30:013
2131231992219,1cyclictest0-21swapper/323:25:013
2131231992118,1cyclictest0-21swapper/322:45:003
2131231992118,1cyclictest0-21swapper/321:55:143
2131231992118,1cyclictest0-21swapper/321:10:003
2131231992017,1cyclictest0-21swapper/319:20:003
213123199201,14cyclictest2320021-21taskset21:30:263
213123199201,14cyclictest2232972-21taskset20:30:143
23888072190,9sleep22390828-21grep22:15:172
2131231991916,1cyclictest0-21swapper/320:05:013
2131231991915,2cyclictest2226532-21turbostat20:25:003
213123199191,13cyclictest2442346-21chrt22:49:573
213123199191,13cyclictest2418693-21taskset22:35:143
213123199191,13cyclictest2330842-21taskset21:40:153
213123199187,7cyclictest2170602-21turbostat19:40:003
213123199182,12cyclictest2137325-21sleep19:14:593
2131231991815,1cyclictest0-21swapper/322:50:253
2131231991815,1cyclictest0-21swapper/319:45:003
2131231991815,1cyclictest0-21swapper/300:05:223
2131231991715,0cyclictest0-21swapper/323:50:033
2131231991714,1cyclictest0-21swapper/322:05:003
2131231991714,1cyclictest0-21swapper/300:10:183
213123199171,13cyclictest2609647-21chrt00:32:193
213123199171,13cyclictest2562028-21taskset00:03:333
213123199171,13cyclictest2376879-21taskset22:06:453
213123199171,12cyclictest2538533-21taskset23:49:553
213123199171,12cyclictest2411099-21taskset22:30:173
213123199171,12cyclictest2411099-21taskset22:30:163
213123199171,12cyclictest2210010-21taskset20:10:233
213123199171,12cyclictest2151755-21turbostat19:24:593
213123199170,13cyclictest2484135-21taskset23:15:163
213123199170,12cyclictest2251506-21taskset20:45:013
237089321611,3sleep12370934-21fschecks_count22:05:141
22194352160,8sleep22214400-21/usr/sbin/munin20:15:242
22194352160,8sleep22214400-21/usr/sbin/munin20:15:232
213123199165,8cyclictest2463980-21taskset23:00:183
2131231991614,1cyclictest0-21swapper/322:20:153
2131231991613,1cyclictest0-21swapper/323:05:143
2131231991613,1cyclictest0-21swapper/321:10:173
2131231991613,1cyclictest0-21swapper/300:35:473
2131231991613,1cyclictest0-21swapper/300:21:223
2131231991613,1cyclictest0-21swapper/300:00:013
213123199161,12cyclictest2529350-21chrt23:42:063
213123199161,12cyclictest2521710-21taskset23:37:563
213123199161,12cyclictest2456930-21chrt22:55:583
213123199161,12cyclictest2352514-21taskset21:50:413
213123199161,12cyclictest2304586-21chrt21:20:363
213123199161,12cyclictest2290774-21taskset21:15:063
213123199161,12cyclictest2214386-21taskset20:15:043
213123199161,12cyclictest2214386-21taskset20:15:033
213123199161,12cyclictest2207917-21taskset20:09:343
213123199161,12cyclictest2157641-21taskset19:26:233
213123199161,11cyclictest2268751-21chrt20:55:183
213123199160,8cyclictest2276020-21taskset21:05:013
213123199160,12cyclictest2232376-21chrt20:26:103
24188092150,12sleep02418875-21/usr/sbin/munin22:35:150
23845592151,4sleep22384567-21ssh22:11:012
2131231991512,1cyclictest0-21swapper/320:40:013
2131231991512,1cyclictest0-21swapper/319:30:103
213123199151,12cyclictest2505975-21taskset23:28:313
213123199151,11cyclictest2508329-21taskset23:30:203
213123199151,11cyclictest2392352-21taskset22:15:263
213123199151,11cyclictest2384396-21chrt22:10:283
213123199151,11cyclictest2328316-21taskset21:35:233
213123199151,11cyclictest2312500-21taskset21:25:323
213123199151,11cyclictest2257725-21taskset20:50:153
213123199151,11cyclictest2182845-21taskset19:45:313
213123199150,13cyclictest2409917-21chrt22:29:083
213123199150,12cyclictest2480826-21chrt23:10:443
213123199150,12cyclictest2338707-21taskset21:45:053
21886832141,10sleep02188903-21sed19:50:260
21760982141,9sleep12170864-21/usr/sbin/munin19:40:251
213123199141,11cyclictest2195348-21taskset19:57:213
2131231991411,1cyclictest0-21swapper/319:50:163
213123199140,11cyclictest2250943-21taskset20:40:563
147598130,8rtkit-daemon720-21snmpd00:20:240
25965322121,8sleep22598360-21tr00:25:212
25704072120,9sleep02570415-21ssh00:09:320
25553042121,8sleep02555403-21irqcore00:00:150
24683792120,9sleep22470292-21tr23:05:182
22528452120,3sleep22253815-21cut20:45:212
21642732120,9sleep22164274-21sshd19:33:442
21307262120,2sleep30-21swapper/319:05:483
21265412120,8sleep02128417-21cut19:05:180
147598120,8rtkit-daemon2515398-21cat23:35:170
147598120,3rtkit-daemon720-21snmpd00:35:141
26111492112,5sleep02611188-21apt-get00:35:130
25845102111,7sleep22584775-21unixbench_multi00:15:242
25300112110,8sleep12530018-21sh23:43:361
24100562110,8sleep02410064-21ssh22:29:260
23627602110,8sleep22362923-21sh22:00:142
23519482110,8sleep02352045-21grep21:50:240
23130572110,3sleep12313065-21ssh21:26:561
147598110,7rtkit-daemon474-21systemd-logind21:36:430
147598110,7rtkit-daemon474-21systemd-logind20:06:422
147598110,7rtkit-daemon313-21systemd-udevd22:01:430
147598110,6rtkit-daemon2344414-21sh21:45:250
147598110,1rtkit-daemon0-21swapper/000:26:260
26098572100,8sleep00-21swapper/000:32:460
25773772100,1sleep00-21swapper/000:12:140
25622272100,2sleep10-21swapper/100:04:081
24259072100,7sleep22425913-21ssh22:39:072
23378232100,7sleep02337831-21ssh21:43:370
21372982100,7sleep12137312-21idleruntime-cro19:14:591
147598100,5rtkit-daemon2276033-21cat21:05:012
147598100,2rtkit-daemon214-21plymouthd22:19:381
147598100,2rtkit-daemon214-21plymouthd22:09:050
2532298290,2sleep20-21swapper/223:45:192
2483362291,5sleep22484131-21irqrtprio23:15:162
2466028293,4sleep12450726-21kworker/u16:0+nfsiod23:03:301
2426662291,4sleep12426694-21ssh22:40:131
2403026290,6sleep22346035-21kworker/u16:4+rpciod22:25:152
2400450291,5sleep22400456-21ssh22:20:392
2386279290,3sleep12386287-21ssh22:14:531
2363142290,6sleep12363747-21irqrtprio22:00:181
2354684291,4sleep02354708-21apt-get21:55:110
2298996294,3sleep10-21swapper/121:20:141
2276756290,6sleep22277588-21ps21:05:162
2262822291,5sleep12262934-21python320:50:261
2220909290,2sleep10-21swapper/120:20:151
2209222290,6sleep22210008-21cut20:10:232
2204059290,6sleep12204976-21kthreadcore20:05:201
2201083290,6sleep02201342-21unixbench_singl20:00:240
2169890290,6sleep12170029-21timerandwakeup19:35:231
2147305290,6sleep12149259-21kthreadcore19:20:201
14759890,6rtkit-daemon22110-21packagekitd23:34:320
14759890,5rtkit-daemon476-21wpa_supplicant22:47:441
14759890,5rtkit-daemon214-21plymouthd23:40:390
14759890,4rtkit-daemon2433374-21ssh22:42:422
14759890,4rtkit-daemon2398468-21grep22:20:181
14759890,2rtkit-daemon214-21plymouthd22:26:491
2603167280,5sleep22603194-21http00:30:152
2570979281,4sleep22571016-21apt-get00:10:142
2545559280,5sleep12545567-21ssh23:52:381
2466673280,5sleep02466900-21latency_hist23:05:010
2448984281,4sleep12448986-21ssh22:51:081
2434783281,4sleep02434945-21diskstats22:45:170
2416039281,4sleep22416270-21unixbench_singl22:30:252
2416039281,4sleep22416270-21unixbench_singl22:30:242
2375910280,6sleep22376022-21switchtime22:05:242
2315170281,5sleep02315688-21/usr/sbin/munin21:30:190
2295792283,2sleep02295908-21ntpq21:15:210
2158422280,4sleep02157938-21apt-get19:30:020
14759880,4rtkit-daemon729-21lldpd20:40:302
14759880,4rtkit-daemon313-21systemd-udevd21:13:432
14759880,4rtkit-daemon2475012-21apt-get23:10:132
14759880,4rtkit-daemon22110-21packagekitd23:53:320
14759880,4rtkit-daemon214-21plymouthd23:55:180
14759880,4rtkit-daemon214-21plymouthd20:21:472
14759880,4rtkit-daemon214-21plymouthd19:57:042
14759880,4rtkit-daemon1-21systemd21:19:072
14759880,3rtkit-daemon515-21gmain23:45:330
14759880,3rtkit-daemon2306788-21apt-get21:25:120
14759880,3rtkit-daemon2306336-21ssh21:24:410
14759880,3rtkit-daemon214-21plymouthd20:00:192
14759880,3rtkit-daemon214-21plymouthd19:10:012
2602153270,4sleep12602160-21sh00:28:331
2586103271,3sleep12586111-21ssh00:18:191
2514151270,4sleep12514159-21ssh23:33:551
2513619270,2sleep20-21swapper/223:32:402
2490950270,5sleep12346035-21kworker/u16:4+rpciod23:20:051
2457372270,4sleep02457380-21ssh22:57:030
2410673271,3sleep02410833-21diskstats22:30:150
2410673271,3sleep02410833-21diskstats22:30:140
2393003270,5sleep00-21swapper/022:16:560
2385158270,1sleep00-21swapper/022:12:240
2351747270,4sleep22351889-21sed21:50:212
2338763271,3sleep22338795-21apt-get21:45:132
2337278270,1sleep20-21swapper/221:42:232
2322007270,5sleep22282191-21diskmemload21:34:022
2288549270,5sleep12288557-21ssh21:10:321
2282178271,4sleep12282198-21munin-run21:10:001
2270067270,4sleep12270100-21apt-get21:00:151
2265394270,4sleep12267374-21tr20:55:171
2250980270,1sleep00-21swapper/020:41:270
2240452273,2sleep2431ktimers/220:35:232
2226175270,1sleep00-21swapper/020:20:430
2157199270,5sleep20-21swapper/219:25:232
14759870,4rtkit-daemon214-21plymouthd19:37:002
14759870,4rtkit-daemon2137557-21munin-node00:16:350
14759870,3rtkit-daemon730-21ntpd19:16:112
14759870,3rtkit-daemon720-21snmpd20:55:062
14759870,3rtkit-daemon720-21snmpd19:10:072
14759870,3rtkit-daemon2496338-21python323:20:220
14759870,3rtkit-daemon2263670-21munin-run20:55:012
14759870,3rtkit-daemon2183452-21seq19:50:132
14759870,3rtkit-daemon214-21plymouthd22:59:402
14759870,3rtkit-daemon214-21plymouthd22:51:112
14759870,3rtkit-daemon214-21plymouthd22:31:091
14759870,3rtkit-daemon214-21plymouthd22:31:081
14759870,3rtkit-daemon214-21plymouthd19:20:272
14759870,3rtkit-daemon2137557-21munin-node20:31:012
14759870,3rtkit-daemon1-21systemd23:18:420
14759870,3rtkit-daemon1-21systemd23:00:112
14759870,3rtkit-daemon1-21systemd22:45:222
14759870,3rtkit-daemon1-21systemd20:28:442
14759870,2rtkit-daemon214-21plymouthd19:46:272
2617663260,1sleep20-21swapper/200:37:192
2577544260,1sleep10-21swapper/100:12:341
2568154260,4sleep12568235-21wc00:05:251
2556362260,3sleep22558145-21grep00:00:172
2546987261,3sleep12547147-21diskstats23:55:161
2539744260,3sleep22540544-21kthreadcore23:50:192
2530942260,3sleep12530979-21apt-get23:45:141
2506308260,4sleep12506316-21ssh23:29:221
2498980260,1sleep20-21swapper/223:25:142
2473239260,4sleep12425361-21kworker/u16:3+rpciod23:06:461
2472958260,1sleep00-21swapper/023:06:010
2457907261,3sleep10-21swapper/122:58:161
2394711260,2sleep00-21swapper/022:20:130
2361393260,4sleep12282191-21diskmemload21:57:461
2331137261,2sleep10-21swapper/121:40:181
2329848260,1sleep20-21swapper/221:38:392
2329073260,3sleep12329081-21ssh21:36:541
2297155260,1sleep10-21swapper/121:17:021
2282999260,3sleep02283131-21hddtemp_smartct21:10:150
2281421260,4sleep02281535-21ls21:05:230
2268892260,4sleep02268947-21ntp_states20:55:200
2263391260,1sleep0214-21plymouthd20:52:050
2232677260,3sleep02232696-21munin-run20:30:000
2220137260,1sleep10-21swapper/120:17:521
2220137260,1sleep10-21swapper/120:17:511
2220097260,1sleep00-21swapper/020:17:230
2220097260,1sleep00-21swapper/020:17:220
2207679260,1sleep00-21swapper/020:06:320
2201696260,1sleep10-21swapper/120:04:091
2189811261,2sleep00-21swapper/019:55:140
2183112260,1sleep00-21swapper/019:48:560
2176874260,3sleep12176785-21apt-get19:45:011
2176770261,2sleep02176873-21cmp19:45:010
2157608261,3sleep0720-21snmpd19:25:560
2139614260,1sleep00-21swapper/019:15:070
2130973260,1sleep10-21swapper/119:08:591
14759860,3rtkit-daemon515-21gmain19:41:202
14759860,2rtkit-daemon214-21plymouthd22:35:441
14759860,2rtkit-daemon214-21plymouthd21:20:162
2609119250,3sleep12609125-21ssh00:31:061
2593331250,3sleep20-21swapper/200:21:322
2587246251,2sleep10-21swapper/100:20:161
2552919250,3sleep20-21swapper/223:56:052
2516481250,2sleep10-21swapper/123:35:201
2497415250,3sleep20-21swapper/223:22:102
2488864252,1sleep10-21swapper/123:15:511
2482210251,2sleep02282191-21diskmemload23:14:030
2443333250,3sleep02443850-21cut22:50:190
2433532250,3sleep00-21swapper/022:43:050
2360768250,3sleep22360776-21ssh21:56:192
2352812250,3sleep10-21swapper/121:51:201
2346414250,3sleep12346464-21munin-run21:50:011
2322187250,3sleep10-21swapper/121:34:251
2313658250,3sleep20-21swapper/221:28:182
2257227250,1sleep10-21swapper/120:47:201
2251523250,2sleep00-21swapper/020:45:150
2246658250,2sleep10-21swapper/120:40:201
2244849250,3sleep00-21swapper/020:37:250
2244835250,1sleep10-21swapper/120:37:141
2238046250,3sleep02238146-21systemctl20:30:250
2226771250,2sleep10-21swapper/120:25:121
2213875250,3sleep00-21swapper/020:11:420
2208250250,3sleep12208406-21diskstats20:10:181
2189094250,1sleep10-21swapper/119:51:311
2166964250,2sleep02169026-21cut19:35:180
2159884250,2sleep10-21swapper/119:30:181
2151499250,3sleep00-21swapper/019:22:000
2145436250,3sleep1214-21plymouthd19:18:331
2137169250,3sleep00-21swapper/019:13:220
2566176241,2sleep2730-21ntpd00:05:222
2529502240,2sleep22529504-21ssh23:42:252
2522157240,2sleep22522165-21ssh23:39:012
2480952241,1sleep10-21swapper/123:11:071
2275207241,1sleep00-21swapper/021:00:250
2153714240,1sleep10-21swapper/119:25:181
2232714230,1sleep10-21swapper/120:30:021
2189618230,1sleep10-21swapper/119:55:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional