You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-04 - 06:51
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack5slot1.osadl.org (updated Tue Nov 04, 2025 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1565439930,0EcMasterDemoSyn1566429tAtEmLog_023:25:295
2835639920,0EcMasterDemoSyn2836629tAtEmLog_023:55:565
218839910,0EcMasterDemoSyn219829tAtEmLog_020:48:175
1326939910,0EcMasterDemoSyn1327929tAtEmLog_020:02:395
292322830,0sleep40-21swapper/422:50:524
2551639830,0EcMasterDemoSyn2552629tAtEmLog_022:04:205
1930139830,0EcMasterDemoSyn1931129tAtEmLog_019:06:525
356439820,0EcMasterDemoSyn357429tAtEmLog_021:49:085
3255639820,0EcMasterDemoSyn3256629tAtEmLog_022:09:255
2641239820,0EcMasterDemoSyn2642229tAtEmLog_019:22:055
2444339820,0EcMasterDemoSyn2445329tAtEmLog_023:10:165
2349739820,0EcMasterDemoSyn2350729tAtEmLog_023:30:335
2273039820,0EcMasterDemoSyn2274029tAtEmLog_020:22:565
1440539820,0EcMasterDemoSyn1441529tAtEmLog_021:33:555
1164039820,0EcMasterDemoSyn1165029tAtEmLog_021:08:345
1019239820,0EcMasterDemoSyn1020229tAtEmLog_000:06:035
823339810,0EcMasterDemoSyn824329tAtEmLog_019:52:305
454639810,0EcMasterDemoSyn455629tAtEmLog_020:53:215
3113139810,0EcMasterDemoSyn3114129tAtEmLog_019:32:135
2891439810,0EcMasterDemoSyn2892429tAtEmLog_021:44:035
2744539810,0EcMasterDemoSyn2745529tAtEmLog_020:33:045
2508739810,0EcMasterDemoSyn2509729tAtEmLog_020:28:005
2158039810,0EcMasterDemoSyn2159029tAtEmLog_021:38:595
1654939810,0EcMasterDemoSyn1655929tAtEmLog_000:36:305
1102139810,0EcMasterDemoSyn1103129tAtEmLog_022:39:505
1053739310,0EcMasterDemoSyn0-21swapper/521:54:125
19632992422,2cyclictest496-21kworker/4:3+events23:25:284
1962899200,10cyclictest16612-21sh21:12:520
352799190,16tDemoTimingTask5776-21chrt19:46:145
2168429190,10tAtEmLog_021689-21chrt19:11:575
19629991916,2cyclictest7980-21kworker/1:0+events23:15:191
1962899195,10cyclictest805-21runrttasks20:16:310
1963199181,13cyclictest16-1pr/legacy22:14:283
19629991815,2cyclictest7980-21kworker/1:0+events21:13:371
1962999180,10cyclictest20578-21ssh23:07:311
1962899186,9cyclictest5783-21sh22:13:080
1962899180,9cyclictest805-21runrttasks00:04:070
1962899180,14cyclictest19602-21sh21:36:590
352799170,14tDemoTimingTask3648-21chrt19:43:385
2350899170,7tDemoTimingTask29206-21chrt23:35:135
19632991714,2cyclictest496-21kworker/4:3+events00:11:064
1963299170,9cyclictest22712-21taskset20:22:504
1963199171,12cyclictest16-1pr/legacy21:49:073
1963199171,10cyclictest16-1pr/legacy21:28:503
1962899175,9cyclictest805-21runrttasks23:23:180
1962899170,9cyclictest805-21runrttasks20:49:000
1962899170,9cyclictest3372-21sh22:34:370
1962899170,9cyclictest25481-21sh22:04:190
1962899170,9cyclictest20181-21sh23:07:110
1962899170,14cyclictest31041-21sh22:07:550
1563799170,10tDemoTimingTask15887-21chrt20:10:005
19632991614,2cyclictest496-21kworker/4:3+events00:26:204
1963199164,9cyclictest28405-21cat19:25:213
1963199164,8cyclictest6170-21open_files21:50:193
19631991612,3cyclictest829-21runintdemo5min00:26:203
1963199161,11cyclictest30074-21taskset21:21:343
1963199161,11cyclictest16-1pr/legacy22:09:233
1963099164,8cyclictest11962-21aten2_r5power_p21:55:132
1963099163,8cyclictest5410-21taskset21:27:132
1963099162,10cyclictest11192-21ssh00:07:002
1963099161,11cyclictest18777-21ssh21:36:122
1963099160,9cyclictest1197-21ssh22:10:062
1962899165,2cyclictest0-21swapper/021:45:270
1962899164,9cyclictest16704-21sh22:43:180
1962899160,9cyclictest32127-21sh00:24:240
1962899160,8cyclictest17811-21runrttasks20:10:350
1962899160,6cyclictest17961-21ssh21:59:200
1799499160,9tDemoTimingTask18482-21taskset20:15:015
1963299151,10cyclictest29991-21latency_hist20:40:014
1963299150,9cyclictest1816-21ssh21:47:244
1963199159,5cyclictest22939-21tr22:25:173
1963199157,6cyclictest25489-21grep22:04:193
1963199154,6cyclictest23937-21tune2fs21:40:163
19631991511,3cyclictest24239-21needreboot20:25:203
19631991511,3cyclictest15522-21ssh21:35:003
1963199150,9cyclictest4053-21runrttasks19:45:083
1963199150,9cyclictest22910-21latency_hist21:40:003
1963199150,9cyclictest17389-21ssh22:20:443
1963099159,2cyclictest14061-21awk21:10:262
1963099155,6cyclictest10850-21rm23:22:542
1963099154,6cyclictest10853-21ssh00:32:072
1963099153,8cyclictest6900-21ssh21:50:322
1963099153,7cyclictest25716-21cat23:55:012
1963099152,9cyclictest29095-21ssh23:13:422
1963099152,9cyclictest20095-21fschecks_time22:00:152
1963099152,9cyclictest17708-21taskset00:37:422
1963099152,9cyclictest1303-21munin-run21:25:002
1963099152,9cyclictest11194-21ssh23:44:582
1963099150,9cyclictest21315-21ssh23:30:042
1963099150,9cyclictest19918-21ssh23:29:202
1962999154,6cyclictest477-21cut00:25:001
1962999154,6cyclictest28375-21switchtime22:50:241
1962999151,9cyclictest2551-21wc00:25:261
1962999151,10cyclictest9708-21ssh22:38:311
1962999151,10cyclictest829-21runintdemo5min21:44:031
1962999150,9cyclictest1850-21grep20:45:261
1962999150,7cyclictest2770-21ntpq19:40:221
1962899157,6cyclictest23868-21sed20:25:170
1962899151,10cyclictest7280-21sh21:50:520
1962899151,10cyclictest15583-21sh23:25:260
1962899151,10cyclictest13133-21runrttasks20:00:570
1962899150,9cyclictest19268-21sh00:39:170
1962899150,6cyclictest10161-21ssh23:43:580
1962899150,11cyclictest6748-21chrt20:56:270
1962899150,11cyclictest20914-21cat22:45:250
3235939140,1EcMasterDemoSyn671rcuc/521:23:465
1963299145,8cyclictest4425-21/usr/sbin/munin19:45:164
1963299145,8cyclictest24941-21/usr/sbin/munin23:10:194
1963299143,8cyclictest2464-21open_inodes22:10:204
1963299143,8cyclictest15679-21fschecks_count22:20:164
1963299141,9cyclictest8565-21ssh23:42:304
1963299141,9cyclictest8343-21/usr/sbin/munin22:15:174
1963299141,9cyclictest829-21runintdemo5min21:08:374
1963299141,9cyclictest7507-21/usr/sbin/munin19:50:204
1963299141,9cyclictest3932-21ssh22:55:284
1963299141,9cyclictest3271950mandb00:00:004
1963299141,9cyclictest30649-21awk22:30:234
1963299141,9cyclictest3027-21switchtime19:40:274
1963299141,9cyclictest19101-21ssh21:36:304
1963299141,9cyclictest13965-21sh22:40:474
1963299140,8cyclictest25505-21sh00:18:544
1963299140,11cyclictest22750-21perl19:15:164
1963299140,10cyclictest995-21sh23:37:354
1963299140,10cyclictest805-21runrttasks23:48:434
1963299140,10cyclictest805-21runrttasks22:04:124
1963299140,10cyclictest7900-21taskset22:36:554
1963299140,10cyclictest3847-21ssh21:25:374
1963299140,10cyclictest31044-21/usr/sbin/munin23:15:144
1963299140,10cyclictest28992-21runrttasks19:29:384
1963299140,10cyclictest28629-21ssh22:29:454
1963299140,10cyclictest27039-21df_inode00:20:164
1963299140,10cyclictest26881-21sh23:33:324
1963299140,10cyclictest23303-21ssh22:47:204
1963299140,10cyclictest23121-21ssh23:52:374
1963299140,10cyclictest21518-21sh21:15:324
1963299140,10cyclictest18590-21sh21:59:564
1963299140,10cyclictest16349-21/usr/sbin/munin23:05:174
1963299140,10cyclictest13872-21diskmemload00:35:094
1963299140,10cyclictest13066-21sh23:02:174
1963299140,10cyclictest12664-21ssh21:32:174
1963299140,10cyclictest11324-21ssh00:32:374
1963199141,9cyclictest4299-21/usr/sbin/munin20:50:283
1963199141,9cyclictest13057-21sh00:08:593
1963199141,9cyclictest12672-21sh23:01:573
1963199141,10cyclictest19429-21cpuspeed_turbos22:45:143
19631991410,3cyclictest12962-21needreboot21:55:183
1963199140,9cyclictest805-21runrttasks23:45:013
1963199140,11cyclictest22587-21ssh23:09:243
1963199140,10cyclictest9558-21ssh23:21:453
1963199140,10cyclictest9271-21ssh23:43:093
1963199140,10cyclictest805-21runrttasks22:17:053
1963199140,10cyclictest5221-21ssh00:02:543
1963199140,10cyclictest5062-21cut20:55:133
1963199140,10cyclictest429-21cut00:25:003
1963199140,10cyclictest31740-21expr23:15:173
1963199140,10cyclictest30315-21/usr/sbin/munin20:40:173
1963199140,10cyclictest30315-21/usr/sbin/munin20:40:173
1963199140,10cyclictest30113-21ssh23:57:373
1963199140,10cyclictest29178-21ssh22:50:493
1963199140,10cyclictest24827-21ssh23:54:183
1963199140,10cyclictest24016-21ssh21:17:493
1963199140,10cyclictest22856-21sh00:15:483
1963199140,10cyclictest22250-21/usr/sbin/munin19:15:203
1963199140,10cyclictest22196-21hddtemp_smartct23:30:163
1963199140,10cyclictest20885-21/usr/sbin/munin20:20:203
1963199140,10cyclictest20461-21chrt20:18:483
1963199140,10cyclictest19169-21sh23:28:383
1963199140,10cyclictest16944-21ssh21:13:113
1963199140,10cyclictest16819-21chrt20:10:163
1963199140,10cyclictest14460-21hddtemp_smartct20:05:153
1963199140,10cyclictest13592-21ssh22:40:263
1963199140,10cyclictest10820-21latency_hist20:00:003
1963099149,2cyclictest171rcu_preempt23:46:452
1963099146,7cyclictest19626-21cyclictest20:42:152
1963099146,7cyclictest19626-21cyclictest20:42:152
1963099144,8cyclictest31615-21sh22:53:002
1963099144,8cyclictest25459-21ssh22:49:152
1963099143,8cyclictest19813-21cpuspeed21:15:172
1963099143,8cyclictest10027-21smartctl23:00:182
1963099143,7cyclictest781-21postgres22:30:342
1963099143,7cyclictest1670-21grep20:45:252
1963099143,6cyclictest27959-21ntpq22:05:212
1963099143,6cyclictest11577-21ssh22:40:022
1963099142,8cyclictest5081-21ssh23:19:322
1963099142,8cyclictest2907-21ssh00:25:372
1963099142,8cyclictest28001-21rm00:20:222
1963099141,9cyclictest4716-21latency_hist20:55:012
1963099141,9cyclictest23711-21/usr/sbin/munin21:40:152
1963099141,9cyclictest2004-21sh22:55:112
1963099141,9cyclictest16978-21perl23:05:162
1963099140,10cyclictest9566-21fschecks_time21:30:192
1963099140,10cyclictest738-21snmpd20:55:512
1963099140,10cyclictest6717-21expr19:50:142
1963099140,10cyclictest4187-21sh22:35:032
1963099140,10cyclictest31686-21/usr/sbin/munin19:35:182
1963099140,10cyclictest30564-21cut21:45:152
1963099140,10cyclictest29317-21sh23:56:492
1963099140,10cyclictest24624-21/usr/sbin/munin19:20:242
1963099140,10cyclictest23235-21sshd20:25:132
1963099140,10cyclictest22678-21ssh22:25:162
1963099140,10cyclictest21666-21sed00:15:182
1963099140,10cyclictest17722-21sh00:11:462
1963099140,10cyclictest12028-21gltestperf20:00:162
1963099140,10cyclictest1173-21expr00:00:152
1963099140,10cyclictest11199-21gpg-connect-age20:00:002
1963099140,10cyclictest10977-21ssh22:16:152
1963099140,10cyclictest10006-21expr21:05:142
1962999141,9cyclictest4456-21runrttasks20:52:191
1962999141,9cyclictest3642-21rm22:11:001
1962999141,9cyclictest24624-21/usr/sbin/munin19:20:151
1962999141,9cyclictest13772-21latency_hist20:05:001
1962999141,9cyclictest12094-21runrttasks22:17:271
1962999141,10cyclictest4411-21ssh00:02:051
1962999141,10cyclictest30393-21sh21:45:091
1962999141,10cyclictest24065-21runrttasks19:17:041
1962999141,10cyclictest22139-21ssh22:25:021
19629991410,3cyclictest28945-21needreboot20:35:191
19629991410,3cyclictest28945-21needreboot20:35:191
1962999140,9cyclictest8767-21/usr/sbin/munin19:55:141
1962999140,9cyclictest12686-21ssh00:34:081
1962999140,10cyclictest9743-21/usr/sbin/munin21:05:171
1962999140,10cyclictest9707-21sh23:00:171
1962999140,10cyclictest938-21runrttasks19:35:321
1962999140,10cyclictest805-21runrttasks22:06:081
1962999140,10cyclictest6752-21sh22:58:051
1962999140,10cyclictest6133-21/usr/sbin/munin23:20:241
1962999140,10cyclictest4874-21sh21:50:071
1962999140,10cyclictest3835-21ssh21:25:371
1962999140,10cyclictest334-21systemd-journal19:15:011
1962999140,10cyclictest30931-21sh21:22:241
1962999140,10cyclictest29491-21ssh23:56:591
1962999140,10cyclictest29335-21/usr/sbin/munin19:30:171
1962999140,10cyclictest2866-21ssh23:39:101
1962999140,10cyclictest26975-21/usr/sbin/munin19:25:221
1962999140,10cyclictest26691-21ssh23:11:271
1962999140,10cyclictest24861-21ssh22:03:451
1962999140,10cyclictest23549-21ssh23:30:351
1962999140,10cyclictest21530-21ssh22:45:491
1962999140,10cyclictest20879-21chrt20:20:101
1962999140,10cyclictest19331-21df23:50:151
1962999140,10cyclictest19277-21sh22:22:321
1962999140,10cyclictest18645-21ssh21:36:051
1962999140,10cyclictest15940-21processes00:10:221
1962999140,10cyclictest14763-21sh21:56:301
1962999140,10cyclictest13906-21ssh21:33:301
1962999140,10cyclictest13898-21/usr/sbin/munin00:35:241
1962999140,10cyclictest10653-21ssh00:06:291
1962899146,6cyclictest334-21systemd-journal00:10:140
1962899145,3cyclictest805-21runrttasks19:31:310
1962899144,3cyclictest805-21runrttasks23:15:510
1962899143,8cyclictest752-21ntpd21:00:180
1962899140,7cyclictest7694-21sh00:05:080
1962899140,7cyclictest2519-21fschecks_count22:55:170
1962899140,7cyclictest23955-21tr23:10:130
1962899140,7cyclictest23690-21/usr/sbin/munin21:40:140
1962899140,7cyclictest12420-21sh21:31:580
1962899140,5cyclictest146-1kworker/0:1H+kblockd00:30:150
1962899140,12cyclictest27080-21sh21:20:130
1962899140,12cyclictest25094-21sh23:31:560
1962899140,12cyclictest25034-21ssh22:26:160
1962899140,12cyclictest18758-21ssh23:50:010
1962899140,12cyclictest18113-21sh22:21:230
1962899140,10cyclictest9088-21sh23:00:050
1962899140,10cyclictest805-21runrttasks19:35:160
1962899140,10cyclictest7499-21sh21:29:090
1962899140,10cyclictest7201-21sh22:36:180
1962899140,10cyclictest647-21sh00:25:020
1962899140,10cyclictest4126-21cat20:50:270
1962899140,10cyclictest29008-21sh23:35:090
1962899140,10cyclictest28262-21sensors_volt22:50:200
1962899140,10cyclictest27349-21forks19:25:120
1962899140,10cyclictest27275-21proc_pri23:55:220
1962899140,10cyclictest22613-21runrttasks20:21:280
1962899140,10cyclictest21136-21mailstats19:10:230
1962899140,10cyclictest20784-21ntp_offset21:15:230
1962899140,10cyclictest20711-21ssh00:15:090
1962899140,10cyclictest16783-21sh23:48:260
1962899140,10cyclictest11897-21sh22:17:110
824329130,11tAtEmLog_08762-21taskset19:55:105
2855729130,3tAtEmLog_01742-21chrt22:33:075
1963299131,9cyclictest9610-21ssh00:05:284
1963299131,9cyclictest4449-21sshd20:52:134
1963299130,9cyclictest805-21runrttasks20:35:094
1963299130,9cyclictest805-21runrttasks20:07:464
1963299130,9cyclictest7510-21cpu21:00:124
1963299130,9cyclictest6845-21taskset21:50:294
1963299130,9cyclictest5156-21cpu20:55:144
1963299130,9cyclictest367-21grep20:45:144
1963299130,9cyclictest32020-21chrt20:41:104
1963299130,9cyclictest32020-21chrt20:41:104
1963299130,9cyclictest31135-21sed19:37:174
1963299130,9cyclictest29946-21grep19:30:174
1963299130,9cyclictest29144-21ssh21:20:414
1963299130,9cyclictest26879-21latency_hist19:25:014
1963299130,9cyclictest26536-21ssh22:05:004
1963299130,9cyclictest23801-21fschecks_time20:25:174
1963299130,9cyclictest23150-21ssh21:40:044
1963299130,9cyclictest2149-21ps00:00:224
1963299130,9cyclictest20506-21taskset20:19:254
1963299130,9cyclictest16142-21/usr/sbin/munin20:10:134
1963299130,9cyclictest12095-21ssh21:10:014
1963299130,9cyclictest11807-21entropy20:00:154
1963299130,9cyclictest10634-21runrttasks19:57:404
1963299130,8cyclictest25585-21/usr/sbin/munin20:30:124
1963299130,7cyclictest6218-21ssh23:20:134
1963199139,3cyclictest21248-21gdbus23:35:063
1963199132,8cyclictest8260-21load21:00:163
1963199130,9cyclictest9743-21/usr/sbin/munin21:05:143
1963199130,9cyclictest7544-21/usr/sbin/munin00:30:113
1963199130,9cyclictest4351-21ssh22:35:123
1963199130,9cyclictest31711-21/usr/sbin/munin19:35:143
1963199130,9cyclictest30237-21cat20:40:013
1963199130,9cyclictest30237-21cat20:40:013
1963199130,9cyclictest29478-21/usr/sbin/munin22:30:233
1963199130,9cyclictest25750-21processes19:20:203
1963199130,9cyclictest25585-21/usr/sbin/munin20:30:183
1963199130,9cyclictest25101-21ssh23:10:203
1963199130,9cyclictest21138-21mailstats19:10:233
1963199130,9cyclictest2065-21/usr/sbin/munin22:55:243
1963199130,9cyclictest2043-21taskset20:46:283
1963199130,9cyclictest1712-21/usr/sbin/munin19:40:203
1963199130,9cyclictest14704-21/usr/sbin/munin00:10:223
1963199130,9cyclictest13776-21taskset00:35:033
1963199130,9cyclictest11429-21taskset20:00:033
1963199130,10cyclictest6412-21/usr/sbin/munin19:50:253
1963099133,8cyclictest15073-21ssh22:20:072
1963099131,8cyclictest29689-21df_inode19:30:152
1963099131,8cyclictest20928-21basename20:20:152
1963099131,8cyclictest14257-21fschecks_time20:05:142
1963099130,9cyclictest805-21runrttasks19:45:072
1963099130,9cyclictest7981-21grep21:00:142
1963099130,9cyclictest762-21sshd19:15:132
1963099130,9cyclictest28923-21sh23:35:042
1963099130,9cyclictest27939-21/usr/sbin/munin20:35:202
1963099130,9cyclictest27939-21/usr/sbin/munin20:35:202
1963099130,9cyclictest27069-21aten2_r5power_v19:25:112
1963099130,9cyclictest25590-21apt-get20:30:122
1963099130,9cyclictest19809-21latency_hist19:10:012
1963099130,9cyclictest1712-21/usr/sbin/munin19:40:182
1963099130,9cyclictest16133-21runrttasks20:10:042
1963099130,10cyclictest20945-21ntp_kernel_err19:10:222
1963099130,10cyclictest18497-21/usr/sbin/munin20:15:192
1962999130,9cyclictest6766-21runrttasks19:50:141
1962999130,9cyclictest4284-21sh23:40:101
1962999130,9cyclictest32074-21runrttasks20:41:541
1962999130,9cyclictest32074-21runrttasks20:41:541
1962999130,9cyclictest31823-21sh22:30:571
1962999130,9cyclictest25585-21/usr/sbin/munin20:30:191
1962999130,9cyclictest16142-21/usr/sbin/munin20:10:191
1962999130,9cyclictest14180-21sh22:41:011
1962999130,9cyclictest13820-21ssh23:45:381
1962999130,9cyclictest13787-21runrttasks20:05:071
1962999130,9cyclictest13687-21/usr/sbin/munin23:25:181
1962999130,10cyclictest7608-21cpuspeed_turbos21:00:121
1962999130,10cyclictest5311-21cpuspeed_turbos20:55:141
1962999130,10cyclictest4927-21/usr/sbin/munin19:45:191
1962999130,10cyclictest25118-21runrttasks20:28:021
1962999130,10cyclictest20629-21ssh00:15:031
1962999130,10cyclictest20464-21runrttasks20:18:511
1962999130,10cyclictest19510-21ssh21:15:121
1962899133,8cyclictest25786-21/usr/sbin/munin20:30:140
1962899131,9cyclictest805-21runrttasks20:06:040
1962899130,7cyclictest3400-21chrt19:40:570
1962899130,11cyclictest10099-21/usr/sbin/munin21:05:140
1962899130,10cyclictest9525-21/usr/sbin/munin19:55:190
1962899130,10cyclictest805-21runrttasks19:50:370
1962899130,10cyclictest4069-21/usr/sbin/munin19:45:140
1962899130,10cyclictest32002-21chrt20:40:540
1962899130,10cyclictest32002-21chrt20:40:540
1962899130,10cyclictest28888-21cat20:35:190
1962899130,10cyclictest28888-21cat20:35:190
1962899130,10cyclictest25296-21sh19:20:160
1962899130,10cyclictest23866-21sed19:15:260
717039120,0EcMasterDemoSyn0-21swapper/522:14:295
713599120,10tDemoTimingTask8367-21chrt21:29:565
2916239120,0EcMasterDemoSyn0-21swapper/500:21:175
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional