You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 03:34
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack5slot1.osadl.org (updated Thu Dec 04, 2025 00:44:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2367039940,0EcMasterDemoSyn0-21swapper/320:57:053
1094139900,0EcMasterDemoSyn1095129tAtEmLog_022:23:175
944939830,0EcMasterDemoSyn945929tAtEmLog_020:26:405
828239830,0EcMasterDemoSyn829229tAtEmLog_019:15:405
906439820,0EcMasterDemoSyn907429tAtEmLog_022:43:345
708939820,0EcMasterDemoSyn709929tAtEmLog_020:21:355
2751139820,0EcMasterDemoSyn2752129tAtEmLog_023:39:215
2635239820,0EcMasterDemoSyn2636229tAtEmLog_021:02:095
2615939820,0EcMasterDemoSyn2616929tAtEmLog_022:33:265
2131439820,0EcMasterDemoSyn2132429tAtEmLog_020:52:015
1422139820,0EcMasterDemoSyn1423129tAtEmLog_020:36:485
98339810,0EcMasterDemoSyn99329tAtEmLog_021:12:185
2053239810,0EcMasterDemoSyn2054229tAtEmLog_019:41:015
1999239810,0EcMasterDemoSyn2000229tAtEmLog_000:19:555
588539800,0EcMasterDemoSyn589529tAtEmLog_019:10:355
3207239580,0EcMasterDemoSyn3208229tAtEmLog_000:09:455
468239570,0EcMasterDemoSyn469229tAtEmLog_023:29:115
21778292928,0tAtEmLog_00-21swapper/523:19:035
394699185,9cyclictest781-21postgres19:21:372
393799185,9cyclictest805-21runrttasks19:21:280
393799185,9cyclictest1790-21sh23:43:430
393799180,14cyclictest24239-21postgres22:53:460
393799180,14cyclictest20533-21sh22:08:170
393799180,10cyclictest31321-21sh00:03:590
393799180,10cyclictest13381-21sh22:03:350
471329170,10tAtEmLog_04746-21chrt20:16:345
394299179,6cyclictest9466-21grep23:05:181
394299175,6cyclictest18007-21cut23:55:011
394299175,6cyclictest18007-21cut23:55:001
393799176,8cyclictest24557-21/usr/sbin/munin21:00:120
393799174,9cyclictest20629-21sh23:34:530
393799170,9cyclictest5208-21sh00:08:180
393799170,9cyclictest14958-21sh21:43:090
3004599170,7tDemoTimingTask30048-21chrt20:01:175
395199163,9cyclictest738-21snmpd00:10:053
394699166,8cyclictest14636-21/usr/sbin/munin00:15:182
394699164,9cyclictest680-21fschecks_time21:55:142
394299169,5cyclictest20870-21tr00:20:141
393799163,9cyclictest22460-21netstat23:35:200
393799160,9cyclictest805-21runrttasks19:27:520
393799160,9cyclictest18443-21diskmemload22:28:160
319729160,6tAtEmLog_04261-21taskset22:19:095
3004599160,6tDemoTimingTask30649-21chrt20:05:075
2766899160,10tDemoTimingTask27699-21taskset19:56:175
2177999160,9tDemoTimingTask15384-21sleep23:14:555
907599150,12tDemoTimingTask12686-21taskset22:45:205
395599154,8cyclictest754-21lldpd21:56:314
395599153,8cyclictest17228-21aten2_r5power_e20:45:124
395599152,9cyclictest20844-21ssh22:30:044
395599152,9cyclictest18623-21runintdemo5min22:28:244
395599151,10cyclictest23157-21sh22:52:454
395599150,9cyclictest13523-21sh23:50:594
395599150,9cyclictest13523-21sh23:50:594
395599150,10cyclictest10129-21ssh23:49:384
395199157,6cyclictest4224-21sort23:45:143
395199157,6cyclictest17501-21grep19:35:203
395199154,8cyclictest738-21snmpd20:51:223
395199154,8cyclictest11214-21cpu23:50:133
395199154,8cyclictest11214-21cpu23:50:133
395199151,10cyclictest805-21runrttasks23:03:213
395199151,10cyclictest24734-21ssh22:54:143
394699154,6cyclictest21079-21hostname22:30:132
3946991510,4cyclictest17066-21sh21:45:002
394699150,11cyclictest3593-21sed20:15:152
394299154,8cyclictest6641-21cut00:14:501
394299153,8cyclictest11202-21proc_pri20:30:191
394299152,9cyclictest23057-21/usr/sbin/munin22:10:221
394299151,10cyclictest8047-21ssh21:17:021
394299151,10cyclictest4683-21runrttasks22:19:321
394299151,10cyclictest23837-21diskmemload23:58:521
394299151,10cyclictest20600-21ssh23:12:561
3942991510,4cyclictest25210-21cut21:00:161
393799158,5cyclictest9065-21cut00:35:220
393799156,7cyclictest32128-21latency_hist21:55:000
393799154,8cyclictest17832-21sh23:10:290
393799153,9cyclictest738-21snmpd22:22:470
393799151,10cyclictest700-21sh22:38:020
393799151,10cyclictest16659-21runrttasks21:22:510
393799150,9cyclictest8612-21sh23:25:580
393799150,13cyclictest20228-21ssh21:25:160
393799150,12cyclictest28969-21diskmemload22:40:080
393799150,11cyclictest7203-21sh21:16:150
2209099156,7tDemoTimingTask23522-21taskset21:49:035
1444699150,9tDemoTimingTask16978-21taskset21:44:565
128699150,9tDemoTimingTask2959-21taskset22:39:555
395599143,8cyclictest781-21postgres21:27:424
395599143,6cyclictest5277-21latency_hist22:20:014
395599141,9cyclictest28408-21diskmemload22:13:304
395599141,9cyclictest26377-21cpuspeed_turbos22:55:144
395599141,9cyclictest18367-21ssh23:55:114
395599141,6cyclictest2698750cat00:25:014
3955991410,3cyclictest3165-21cpuspeed_turbos20:15:124
395599140,8cyclictest28969-21diskmemload23:36:114
395599140,10cyclictest9854-21rm00:35:364
395599140,10cyclictest8684-21chrt23:26:024
395599140,10cyclictest8625-21ssh21:17:314
395599140,10cyclictest8580-21sh22:21:174
395599140,10cyclictest481-21/usr/sbin/munin00:05:174
395599140,10cyclictest28419-21sh23:18:094
395599140,10cyclictest24870-21latency_hist21:50:014
395599140,10cyclictest18048-21sh23:10:404
395599140,10cyclictest1550-21ssh22:38:424
395599140,10cyclictest15490-21/usr/sbin/munin22:05:224
395599140,10cyclictest1499-21ssh23:21:194
395199145,7cyclictest24611-21/usr/sbin/munin22:10:263
395199141,9cyclictest805-21runrttasks20:09:423
395199141,9cyclictest762-21sshd23:27:403
395199141,9cyclictest6310-21diskmemload21:37:223
395199141,9cyclictest4850-21diskmemload21:57:533
395199141,9cyclictest28360-21/usr/sbin/munin22:35:213
395199141,9cyclictest26937-21ssh22:34:093
395199141,9cyclictest21228-21sh23:35:093
395199141,9cyclictest19502-21sh21:45:293
395199141,9cyclictest16150-21latency_hist19:35:003
395199141,9cyclictest15939-21aten2_r5power_c23:10:123
395199141,10cyclictest805-21runrttasks19:44:313
395199141,10cyclictest6097-21irqstats20:20:203
395199140,9cyclictest27203-21/usr/sbin/munin00:25:183
395199140,9cyclictest23143-21runrttasks19:49:003
395199140,7cyclictest32729-21cat23:20:253
395199140,10cyclictest9776-21apache222:05:073
395199140,10cyclictest9576-21unixbench_multi22:00:243
395199140,10cyclictest805-21runrttasks21:04:303
395199140,10cyclictest5560-21sh22:40:293
395199140,10cyclictest28458-21sh21:51:443
395199140,10cyclictest2798-21latency_hist20:15:003
395199140,10cyclictest27896-21runrttasks19:58:593
395199140,10cyclictest26901-21runrttasks21:05:023
395199140,10cyclictest25825-21ssh22:55:003
395199140,10cyclictest25704-21/usr/sbin/munin00:00:193
395199140,10cyclictest20421-21wc21:25:183
395199140,10cyclictest18167-21cut20:45:173
395199140,10cyclictest16509-21runrttasks23:31:153
395199140,10cyclictest15302-21ssh21:21:393
395199140,10cyclictest1485-21ssh21:12:433
395199140,10cyclictest1468-21/usr/sbin/munin00:30:253
395199140,10cyclictest11661-21df22:45:133
394699143,8cyclictest7936-21fschecks_time00:10:172
394699142,8cyclictest28567-21sensors00:25:222
394699141,9cyclictest805-21runrttasks19:59:012
394699141,9cyclictest31183-21ssh22:36:172
394699141,9cyclictest28602-21ssh23:40:012
394699141,9cyclictest27604-21sh21:50:592
394699141,9cyclictest17739-21sh22:05:472
394699141,10cyclictest32640-21runrttasks20:09:162
394699140,9cyclictest18746-21ssh22:50:042
394699140,10cyclictest9663-21sh21:18:262
394699140,10cyclictest8764-21memory22:00:172
394699140,10cyclictest805-21runrttasks22:55:132
394699140,10cyclictest805-21runrttasks20:04:272
394699140,10cyclictest805-21runrttasks19:25:022
394699140,10cyclictest7996-21sh21:38:502
394699140,10cyclictest738-21snmpd19:43:112
394699140,10cyclictest6597-21/usr/sbin/munin19:15:212
394699140,10cyclictest6423-21runrttasks23:46:132
394699140,10cyclictest5744-21apt-get22:20:122
394699140,10cyclictest5340-21/usr/sbin/munin20:20:162
394699140,10cyclictest3838-21sh21:14:572
394699140,10cyclictest2963-21ssh23:00:392
394699140,10cyclictest29174-21sh21:30:362
394699140,10cyclictest28494-21unixbench_multi21:05:242
394699140,10cyclictest28216-21sh00:01:022
394699140,10cyclictest25585-21ssh22:11:022
394699140,10cyclictest2535-21sed00:30:192
394699140,10cyclictest25349-21chrt19:52:022
394699140,10cyclictest24546-21chrt21:00:032
394699140,10cyclictest23349-21ssh23:15:052
394699140,10cyclictest22669-21sh23:57:442
394699140,10cyclictest22116-21awk00:20:242
394699140,10cyclictest19771-21sh23:12:132
394699140,10cyclictest1960-21sh00:05:262
394699140,10cyclictest19481-21ssh21:25:112
394699140,10cyclictest1643-21ssh22:16:512
394699140,10cyclictest13670-21/usr/sbin/munin23:30:162
394699140,10cyclictest13094-21ssh23:07:562
394699140,10cyclictest13050-21sh23:50:332
394699140,10cyclictest13050-21sh23:50:332
394699140,10cyclictest11808-21sh21:20:012
394699140,10cyclictest11307-21aten2_r5power_c22:45:112
394699140,10cyclictest11052-21ssh23:28:082
394699140,10cyclictest10136-21sh00:36:032
394299143,8cyclictest738-21snmpd20:15:391
394299142,9cyclictest26601-21smartctl19:55:161
394299141,9cyclictest31168-21runrttasks00:27:381
394299141,9cyclictest29968-21ssh22:57:271
394299141,9cyclictest29158-21sh21:30:351
394299141,9cyclictest20570-21ssh21:46:281
394299141,10cyclictest28858-21runrttasks21:08:421
394299140,10cyclictest9360-21sshd20:25:381
394299140,10cyclictest8571-21ssh22:21:161
394299140,10cyclictest8459-21runrttasks19:17:311
394299140,10cyclictest805-21runrttasks20:42:391
394299140,10cyclictest805-21runrttasks20:12:491
394299140,10cyclictest805-21runrttasks00:15:191
394299140,10cyclictest7189-21ssh23:04:261
394299140,10cyclictest5264-21ssh21:36:251
394299140,10cyclictest4636-21rm23:24:061
394299140,10cyclictest4534-21ssh00:31:421
394299140,10cyclictest4414-21ssh00:07:261
394299140,10cyclictest3964-21df22:40:121
394299140,10cyclictest3541-21ssh23:45:031
394299140,10cyclictest32606-21/usr/sbin/munin21:55:231
394299140,10cyclictest30555-21rm22:35:421
394299140,10cyclictest30030-21runintdemo5min20:01:181
394299140,10cyclictest29496-21basename21:10:121
394299140,10cyclictest28824-21/usr/sbin/munin23:40:201
394299140,10cyclictest26824-21ssh23:38:431
394299140,10cyclictest25704-21/usr/sbin/munin00:00:141
394299140,10cyclictest25062-21ssh21:50:101
394299140,10cyclictest23941-21run-parts21:00:001
394299140,10cyclictest23482-21/usr/sbin/munin23:15:181
394299140,10cyclictest23361-21ssh22:52:581
394299140,10cyclictest22212-21ssh21:26:111
394299140,10cyclictest21973-21missed_timers22:30:181
394299140,10cyclictest19359-21ssh23:33:481
394299140,10cyclictest16988-21fschecks_time19:35:161
394299140,10cyclictest16197-21sh21:22:271
394299140,10cyclictest15842-21runrttasks19:31:001
394299140,10cyclictest15704-21ssh22:25:501
394299140,10cyclictest15246-21ssh22:05:021
394299140,10cyclictest14482-21seq20:39:511
394299140,10cyclictest13498-21munin-plugin-st19:30:001
394299140,10cyclictest12922-21runrttasks21:41:251
394299140,10cyclictest11272-21/usr/sbin/munin22:45:181
394299140,10cyclictest10223-21ssh00:36:091
394299140,10cyclictest10106-21sh22:00:461
393799145,7cyclictest890-21sh00:29:570
393799144,6cyclictest6739-21cpu19:15:130
393799143,8cyclictest14496-21cut19:30:140
393799143,5cyclictest32278-21timerandwakeup22:15:270
393799141,9cyclictest28548-21diskmemload22:56:030
393799141,9cyclictest2566-21taskset23:22:150
393799140,9cyclictest6547-21cat19:15:000
393799140,9cyclictest4291-21sh21:35:330
393799140,8cyclictest31910-21diskmemload21:32:590
393799140,8cyclictest15579-21python300:15:260
393799140,7cyclictest7895-21tune2fs00:10:170
393799140,7cyclictest5022-21tr23:45:190
393799140,7cyclictest21727-21sh22:30:160
393799140,7cyclictest2155-21sed20:10:240
393799140,7cyclictest19263-21/usr/sbin/munin23:55:180
393799140,12cyclictest32663-21aten2_r5power_e21:55:110
393799140,12cyclictest17602-21apt-get21:45:140
393799140,10cyclictest6695-21find20:20:260
393799140,10cyclictest4826-21sh00:32:010
393799140,10cyclictest3714-21sh23:01:190
393799140,10cyclictest29470-21sh21:10:110
393799140,10cyclictest29139-21sh23:18:490
393799140,10cyclictest22548-21sh00:20:330
393799140,10cyclictest20370-21unixbench_multi19:40:230
393799140,10cyclictest13386-21sh23:50:520
393799140,10cyclictest13386-21sh23:50:520
393799140,10cyclictest11724-21sh23:06:360
393799140,10cyclictest11081-21sh22:45:000
786699130,7tDemoTimingTask11279-21taskset00:37:415
654699130,11tDemoTimingTask6819-21chrt23:04:055
395599132,8cyclictest7859-21sort20:25:114
395599132,8cyclictest6591-21sensors20:20:244
395599132,8cyclictest12606-21mailstats22:45:194
395599131,9cyclictest805-21runrttasks19:45:294
395599131,7cyclictest3832-21xargs21:35:254
395599130,9cyclictest9875-21ssh21:40:114
395599130,9cyclictest9845-21sh00:11:064
395599130,9cyclictest8975-21/usr/sbin/munin19:20:134
395599130,9cyclictest805-21runrttasks21:11:144
395599130,9cyclictest7746-21ssh22:00:084
395599130,9cyclictest6597-21/usr/sbin/munin19:15:174
395599130,9cyclictest6499-21latency_hist19:15:004
395599130,9cyclictest640-21tr20:10:134
395599130,9cyclictest5362-21sed22:40:224
395599130,9cyclictest30659-21/usr/sbin/munin20:05:264
395599130,9cyclictest30198-21chrt20:02:594
395599130,9cyclictest28609-21sh23:40:024
395599130,9cyclictest27236-21cpuspeed_turbos21:05:144
395599130,9cyclictest27148-21sh00:25:094
395599130,9cyclictest26998-21sh21:50:274
395599130,9cyclictest25231-21runrttasks21:00:164
395599130,9cyclictest25117-40mandb00:00:024
395599130,9cyclictest23575-21/usr/sbin/munin19:50:134
395599130,9cyclictest21846-21cat20:55:004
395599130,9cyclictest1998-21fschecks_time00:30:154
395599130,9cyclictest19513-21idleruntime19:40:164
395599130,9cyclictest17089-21grep19:35:154
395599130,9cyclictest15078-21wc00:15:204
395599130,9cyclictest14928-21diskmemload23:30:194
395599130,9cyclictest14347-21runrttasks20:38:004
395599130,9cyclictest14096-21/usr/sbin/munin19:30:204
395599130,9cyclictest1234-21df23:00:144
395599130,9cyclictest12048-21ssh23:06:554
395599130,9cyclictest12027-21/usr/sbin/munin21:20:174
395599130,9cyclictest11068-21grep20:30:184
395599130,10cyclictest31519-21ssh21:32:394
395599130,10cyclictest27106-21wc19:55:214
395599130,10cyclictest15347-21fschecks_time20:40:154
395599130,10cyclictest11443-21wget19:25:114
395199135,6cyclictest31257-21tune2fs22:15:183
395199131,7cyclictest10862-21/usr/sbin/munin20:30:173
395199130,9cyclictest9663-21runrttasks21:40:023
395199130,9cyclictest9659-21unixbench_multi00:35:283
395199130,9cyclictest8975-21/usr/sbin/munin19:20:193
395199130,9cyclictest805-21runrttasks20:38:083
395199130,9cyclictest805-21runrttasks19:50:133
395199130,9cyclictest7170-21runrttasks19:15:163
395199130,9cyclictest6133-21taskset22:20:143
395199130,9cyclictest5459-21python319:10:203
395199130,9cyclictest4288-21ssh21:15:053
395199130,9cyclictest4171-21sed19:09:593
395199130,9cyclictest31983-21ssh23:41:413
395199130,9cyclictest27079-21/usr/sbin/munin21:30:133
395199130,9cyclictest20466-21sh00:20:023
395199130,9cyclictest16902-21latency_hist20:45:003
395199130,9cyclictest14631-21tune2fs00:15:183
395199130,9cyclictest11966-21fschecks_time19:25:133
395199130,9cyclictest10793-21ssh23:05:443
395199130,10cyclictest939-21sh00:05:173
395199130,10cyclictest8378-21sort20:25:143
395199130,10cyclictest29617-21grep20:00:233
395199130,10cyclictest25394-21ssh23:15:283
395199130,10cyclictest20248-21sh23:55:263
395199130,10cyclictest13384-21sh22:25:083
394699135,6cyclictest4712-21fschecks_time19:10:152
394699132,8cyclictest752-21ntpd20:40:202
394699131,9cyclictest21884-21/usr/sbin/munin20:55:172
394699130,9cyclictest9179-21sed20:25:212
394699130,9cyclictest805-21runrttasks19:49:212
394699130,9cyclictest805-21runrttasks19:35:042
394699130,9cyclictest547-21/usr/sbin/munin20:10:182
394699130,9cyclictest31023-21/usr/sbin/munin23:20:222
394699130,9cyclictest21317-21/usr/sbin/munin23:35:172
394699130,9cyclictest17395-21ssh21:45:012
394699130,9cyclictest17178-21/usr/sbin/munin20:45:112
394699130,9cyclictest14096-21/usr/sbin/munin19:30:142
394699130,9cyclictest12666-21cpuspeed_turbos20:35:142
394699130,9cyclictest11356-21python320:30:222
394699130,10cyclictest4292-21tail22:40:142
394699130,10cyclictest19531-21/usr/sbin/munin20:50:202
394699130,10cyclictest13581-21wget22:25:142
394299131,9cyclictest281-21jbd2/md0-820:50:221
394299131,9cyclictest16-1pr/legacy20:21:341
394299130,9cyclictest805-21runrttasks19:22:061
394299130,9cyclictest6143-21/usr/sbin/munin23:25:231
394299130,9cyclictest32002-21cat20:05:261
394299130,9cyclictest25356-21runrttasks19:52:081
394299130,9cyclictest18823-21/usr/sbin/munin19:40:161
394299130,9cyclictest17178-21/usr/sbin/munin20:45:191
394299130,10cyclictest4689-21fschecks_time19:10:151
394299130,10cyclictest22985-21runrttasks19:46:511
393799131,9cyclictest17202-21/usr/sbin/munin19:35:180
393799130,9cyclictest9542-21taskset20:27:260
393799130,9cyclictest27703-21runrttasks19:56:190
393799130,9cyclictest22883-21sh22:10:050
393799130,9cyclictest11778-21chrt20:30:440
393799130,7cyclictest146-1kworker/0:1H+kblockd20:55:280
393799130,7cyclictest146-1kworker/0:1H+kblockd20:50:110
393799130,7cyclictest146-1kworker/0:1H+kblockd20:35:000
393799130,7cyclictest146-1kworker/0:1H+kblockd20:01:050
393799130,11cyclictest26900-21taskset21:05:020
393799130,11cyclictest16542-21sshd20:41:140
393799130,10cyclictest4312-21threads20:15:250
393799130,10cyclictest31700-21ntpq20:05:210
393799130,10cyclictest24851-21switchtime19:50:240
393799130,10cyclictest22944-21runrttasks19:46:180
393799130,10cyclictest18878-21runrttasks20:46:020
351999130,10tDemoTimingTask3525-21chrt19:05:335
2752299130,11tDemoTimingTask1021-21chrt23:43:005
2000399130,11tDemoTimingTask24345-21chrt00:22:325
1661399130,11tDemoTimingTask16711-21chrt20:42:515
1095299130,11tDemoTimingTask16231-21chrt22:26:175
490539120,0EcMasterDemoSyn0-21swapper/521:57:565
395599120,8cyclictest805-21runrttasks20:55:024
395199120,9cyclictest3432-21grep20:15:143
394699120,9cyclictest4016-21latency_hist19:09:592
394299120,9cyclictest3893-21latency_hist19:09:591
2797399120,10tDemoTimingTask28315-21taskset22:13:255
2416499120,10tDemoTimingTask31051-21taskset22:58:285
1991739120,0EcMasterDemoSyn0-21swapper/523:34:175
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional