You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-23 - 10:49
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Tue Dec 23, 2025 00:44:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
305869911262,42cyclictest24078-21cstates21:40:122
305809911230,47cyclictest0-21swapper/121:30:171
305809910737,34cyclictest27884-21systemd-run22:39:571
305809910340,36cyclictest21885-21sh22:29:591
30580991023,59cyclictest26358-21apt-get23:30:161
305809910057,37cyclictest24781-21idleruntime-cro22:34:561
30580999935,43cyclictest0-21swapper/121:15:221
30580999854,38cyclictest15651-21sh21:26:071
30580999711,43cyclictest0-21swapper/121:07:251
3058699963,38cyclictest4073-21/usr/sbin/munin21:05:212
3058099968,49cyclictest0-21swapper/120:21:321
3058099959,52cyclictest0-21swapper/122:51:161
3058699943,37cyclictest11579-21taskset23:05:272
30580999411,54cyclictest14898-21seq22:18:001
30586999370,15cyclictest30560-21cyclictest22:04:562
3058099922,75cyclictest0-21swapper/121:00:081
3058099919,51cyclictest0-21swapper/122:41:511
3058099912,77cyclictest0-21swapper/119:50:081
3058699903,36cyclictest25792-21latency_hist23:29:592
3058699903,36cyclictest25792-21latency_hist23:29:592
3058699894,37cyclictest14863-21/usr/sbin/munin20:00:302
3058699894,37cyclictest14863-21/usr/sbin/munin20:00:302
3058699892,26cyclictest714350ssh21:12:272
30580998947,35cyclictest23749-21/usr/sbin/munin00:19:561
30580998919,33cyclictest0-21swapper/119:31:081
30580998911,49cyclictest3143-21seq00:39:241
3058699883,38cyclictest29967-21latency_hist00:29:592
296392887,53sleep229660-21awk19:05:382
30586998747,34cyclictest23201-21cpu20:25:112
3058099877,49cyclictest0-21swapper/122:14:361
30580998762,17cyclictest27244-21fschecks_count21:45:191
3058099872,48cyclictest0-21swapper/121:40:291
3058699863,39cyclictest21644-21grep21:35:302
3058699863,39cyclictest21644-21grep21:35:302
3058699863,36cyclictest32485-21cpuspeed23:40:142
3058699863,36cyclictest11699-40mandb23:59:592
30569998665,15cyclictest7563-21latency_hist22:59:590
3058699854,51cyclictest8424-21apt-get19:40:082
3058699854,35cyclictest7894-21/usr/sbin/munin23:00:302
3058699853,39cyclictest3352-21sh00:39:552
3058699853,36cyclictest30067-21/usr/sbin/munin00:30:312
3058699853,32cyclictest346-21runrttasks19:25:382
30586998526,51cyclictest0-21swapper/223:15:102
30580998558,19cyclictest1623-21latency_hist22:49:591
30580998557,17cyclictest16905-21latency_hist23:14:591
30580998557,17cyclictest16905-21latency_hist23:14:591
30580998535,41cyclictest0-21swapper/120:09:591
30580998535,41cyclictest0-21swapper/120:09:591
30569998563,16cyclictest2759-21latency_hist23:44:580
30569998556,22cyclictest0-21swapper/022:01:060
3058699844,39cyclictest1273-21ntp_offset21:55:252
3058699844,39cyclictest1273-21ntp_offset21:55:252
3058699843,34cyclictest8373-21/usr/sbin/munin21:15:252
3058699843,23cyclictest15364-21/usr/sbin/munin00:05:192
3058699842,32cyclictest346-21runrttasks19:34:262
30580998440,37cyclictest0-21swapper/100:34:551
3058099843,70cyclictest21561-21proc_pri21:35:291
3058099843,70cyclictest21561-21proc_pri21:35:291
3019228417,36sleep00-21swapper/019:07:070
3058699833,37cyclictest1905-21grep20:59:592
3058699832,39cyclictest27268-21fschecks_count21:45:192
3058699832,38cyclictest10923-21sed22:10:332
30586998319,55cyclictest30218-21expr21:50:162
30580998343,33cyclictest3726-21/usr/sbin/munin22:00:251
30580998342,33cyclictest0-21swapper/100:05:331
3058099832,67cyclictest0-21swapper/119:25:091
30569998335,32cyclictest181rcu_preempt00:30:110
3058699823,36cyclictest21024-21/usr/sbin/munin00:15:362
3058699823,35cyclictest19687-21/usr/sbin/munin20:15:272
3058699823,35cyclictest19687-21/usr/sbin/munin20:15:272
3058699823,35cyclictest15038-21sh21:25:312
3058699823,29cyclictest31055-21sh22:44:592
3058699823,26cyclictest27307-21tail20:35:322
3058699823,25cyclictest32367-21grep19:15:022
3058699822,38cyclictest6998-21fschecks_count19:35:202
3058699822,38cyclictest6998-21fschecks_count19:35:202
30586998214,30cyclictest4892-21cpu22:55:142
30586998212,29cyclictest1-21systemd22:09:572
30580998247,28cyclictest0-21swapper/123:40:261
30580998238,39cyclictest0-21swapper/100:10:331
3058099823,69cyclictest0-21swapper/121:10:121
30580998230,38cyclictest352-21kworker/u19:0+events_unbound21:55:001
30580998230,38cyclictest352-21kworker/u19:0+events_unbound21:55:001
3058099822,72cyclictest0-21swapper/120:55:141
3058699814,33cyclictest30008-21open_files23:35:282
30586998139,35cyclictest0-21swapper/223:55:002
3058699813,38cyclictest11858-21fschecks_count19:50:192
3058699813,37cyclictest19495-21ssh22:25:242
3058699813,36cyclictest31460-21cut20:50:202
3058699813,36cyclictest31460-21cut20:50:202
3058699813,35cyclictest22363-21grep20:21:512
30586998133,40cyclictest0-21swapper/221:24:582
30586998133,40cyclictest0-21swapper/221:24:582
3058699813,27cyclictest28029-21/usr/sbin/munin20:40:072
3058699813,23cyclictest2529-21ssh22:50:302
3058699812,26cyclictest27311-21ssh23:30:592
30586998114,31cyclictest5080-21sh23:48:252
30586998113,27cyclictest12229-21entropy00:00:192
30580998164,10cyclictest27341-21unixbench_singl20:35:331
30580998162,11cyclictest11558-21kworker/u17:4-nfsiod00:20:191
30580998144,30cyclictest29392-21sh23:35:111
30580998140,35cyclictest30541-21unixbench_multi20:45:321
30580998140,35cyclictest30541-21unixbench_multi20:45:321
3058099812,75cyclictest0-21swapper/120:25:071
30569998153,10cyclictest60-21kcompactd021:53:430
30586998050,21cyclictest26103-21grep22:35:312
3058699803,38cyclictest346-21runrttasks19:45:232
3058699803,36cyclictest18012-21/usr/sbin/munin00:10:192
3058699803,32cyclictest2774-21switchtime21:00:312
3058699803,30cyclictest3436-21grep19:25:002
3058699803,24cyclictest18084-21grep21:30:182
3058699802,37cyclictest16849-21latency_hist23:14:582
3058699802,37cyclictest16849-21latency_hist23:14:582
3058099802,60cyclictest18707-21ps20:10:301
3058099802,60cyclictest18707-21ps20:10:301
3058099802,48cyclictest0-21swapper/119:20:201
30569998063,11cyclictest5013-21diskmemload23:51:210
3058699793,38cyclictest14577-21modprobe19:59:572
3058699793,28cyclictest26220-21latency_hist20:34:592
3058699793,28cyclictest20407-21fschecks_time23:20:202
3058699793,21cyclictest16806-21/usr/sbin/munin20:05:212
3058699793,21cyclictest16806-21/usr/sbin/munin20:05:212
3058699792,37cyclictest13064-21ssh22:15:152
30580997936,33cyclictest0-21swapper/122:05:301
30569997963,10cyclictest5013-21diskmemload23:25:210
30569997963,10cyclictest5013-21diskmemload23:25:210
30569997963,10cyclictest5013-21diskmemload21:26:080
3058699783,35cyclictest29625-21/usr/sbin/munin20:45:072
3058699783,35cyclictest29625-21/usr/sbin/munin20:45:072
3058699783,25cyclictest19425-21modprobe20:14:562
3058699783,25cyclictest19425-21modprobe20:14:562
3058699783,24cyclictest22087-21sh22:30:122
3058699783,23cyclictest30920-21cut19:10:182
3058699783,22cyclictest24238-21taskset00:20:152
30580997856,16cyclictest21107-21apt-key20:19:571
30580997856,16cyclictest21107-21apt-key20:19:571
30580997839,28cyclictest0-21swapper/120:50:351
30580997839,28cyclictest0-21swapper/120:50:351
3058099782,70cyclictest0-21swapper/122:55:101
30569997862,10cyclictest10949-21/usr/sbin/munin23:05:240
30586997729,40cyclictest0-21swapper/222:20:152
3058099772,70cyclictest0-21swapper/123:25:081
3058099772,70cyclictest0-21swapper/123:25:081
3058099772,47cyclictest0-21swapper/120:00:141
3058099772,47cyclictest0-21swapper/120:00:141
30569997761,10cyclictest5013-21diskmemload00:08:350
30569997758,13cyclictest31826-21ssh22:45:290
30569997756,15cyclictest392-21cron19:39:570
30569997756,15cyclictest392-21cron19:39:570
3058699764,27cyclictest31218-21/usr/sbin/munin22:45:112
30580997656,13cyclictest30560-21cyclictest23:46:211
3058099763,68cyclictest0-21swapper/121:20:111
3058099763,68cyclictest0-21swapper/121:20:111
3058099763,64cyclictest0-21swapper/120:30:121
30580997635,34cyclictest0-21swapper/123:20:191
3058099762,66cyclictest27115-21ssh00:25:131
30569997659,10cyclictest7991-21ssh23:00:140
30569997656,14cyclictest24154-21ssh21:40:160
30569997656,13cyclictest25214-21ssh22:35:120
30569997655,15cyclictest22536-21sh22:30:220
30569997650,22cyclictest0-21swapper/021:39:140
30569997650,22cyclictest0-21swapper/021:39:140
3056999764,41cyclictest346-21runrttasks23:57:200
30569997643,9cyclictest0-21swapper/020:34:560
30569997621,8cyclictest181rcu_preempt21:45:320
30580997554,12cyclictest0-21swapper/119:40:311
3058099753,67cyclictest0-21swapper/119:45:111
3058099753,63cyclictest7948-21ssh23:00:131
3056999754,41cyclictest346-21runrttasks21:24:470
3056999754,41cyclictest346-21runrttasks21:24:470
3056999753,42cyclictest26857-21sort23:30:460
3056999753,41cyclictest346-21runrttasks22:15:260
30580997451,15cyclictest201-21systemd-journal23:53:061
30580997450,14cyclictest0-21swapper/123:55:341
3058099743,63cyclictest30581-21cat19:10:001
30580997417,50cyclictest15977-21cat22:20:091
3056999746,39cyclictest32058-21munin-run23:39:550
30569997459,10cyclictest30560-21cyclictest22:25:060
30569997457,12cyclictest346-21runrttasks20:50:210
30569997457,12cyclictest346-21runrttasks20:50:210
30569997456,11cyclictest1708-21ssh21:55:340
30569997456,11cyclictest1708-21ssh21:55:340
30569997455,13cyclictest21053-21latency_hist20:19:590
30569997455,13cyclictest21053-21latency_hist20:19:590
3056999745,33cyclictest10511-21ps22:10:280
30569997436,29cyclictest29188-21grep22:40:330
3058099733,60cyclictest0-21swapper/100:00:131
30580997334,30cyclictest0-21swapper/123:09:591
30580997315,49cyclictest32309-21cut19:15:001
30569997354,13cyclictest5013-21diskmemload00:15:140
30569997348,11cyclictest181rcu_preempt00:22:410
3056999733,38cyclictest27309-21fschecks_count00:25:190
3058099723,62cyclictest28999-21unixbench_singl20:40:331
3056999725,56cyclictest1921-21latency_hist20:59:590
3056999724,34cyclictest369-21sshd21:19:060
3056999724,33cyclictest16366-21if_eth022:20:230
30569997242,9cyclictest181rcu_preempt21:10:110
3056999723,45cyclictest32104-21kworker/0:1+pm00:39:590
3058099712,54cyclictest1693-21munin-run19:19:571
3056999718,52cyclictest21583-21apt-get20:20:100
30569997156,10cyclictest30560-21cyclictest00:00:030
3056999714,35cyclictest346-21runrttasks22:08:270
3056999714,33cyclictest20651-21latency_hist21:34:580
30569997138,11cyclictest181rcu_preempt19:29:570
3056999713,57cyclictest346-21runrttasks23:18:540
3056999713,38cyclictest346-21runrttasks23:22:480
30569997118,35cyclictest5764-21grep19:30:290
30580997055,9cyclictest201-21systemd-journal23:19:561
3058099704,61cyclictest0-21swapper/121:50:101
3058099702,60cyclictest14627-21sendmail-msp19:59:571
30569997055,9cyclictest31704-21unixbench_singl19:10:330
30569997052,13cyclictest0-21swapper/019:20:000
3056999704,34cyclictest346-21runrttasks23:49:450
3056999704,33cyclictest346-21runrttasks22:52:280
30569997040,13cyclictest30571-21grep20:45:320
30569997040,13cyclictest30571-21grep20:45:320
30580996950,11cyclictest0-21swapper/119:39:061
30580996950,11cyclictest0-21swapper/119:39:061
30569996948,15cyclictest3627-21latency_hist21:05:000
30569996937,23cyclictest18397-21sort20:10:210
30569996937,23cyclictest18397-21sort20:10:210
3056999693,35cyclictest13992-21/usr/sbin/munin23:10:190
3056999693,35cyclictest13992-21/usr/sbin/munin23:10:190
30569996923,35cyclictest27787-21sendmail-msp20:39:570
30569996846,12cyclictest0-21swapper/020:40:330
3056999684,43cyclictest346-21runrttasks00:10:450
3056999684,37cyclictest1959-21/usr/sbin/munin19:20:240
3056999674,40cyclictest13524-21fschecks_time19:55:210
3056999674,36cyclictest23251-21sort20:25:120
3056999673,34cyclictest15803-21tail20:00:330
3056999673,34cyclictest15803-21tail20:00:330
3056999663,37cyclictest3868-21fschecks_count21:05:170
3056999663,18cyclictest17834-21munin-plugin-st20:09:560
3056999663,18cyclictest17834-21munin-plugin-st20:09:560
30569996545,14cyclictest10988-21unixbench_singl19:45:330
30569996543,12cyclictest0-21swapper/019:40:170
3056999644,45cyclictest11622-21/usr/sbin/munin19:50:250
3059399633,43cyclictest3401-21taskset21:59:563
3059399633,43cyclictest3401-21taskset21:59:563
3059399586,35cyclictest21804-21chrt20:21:113
3059399584,38cyclictest2327-21chrt21:00:193
3059399582,38cyclictest27096-21taskset21:45:133
3059399573,37cyclictest11133-21chrt21:19:563
3059399573,36cyclictest20605-21chrt00:14:403
3059399573,36cyclictest1232-21taskset23:41:053
3059399564,38cyclictest9980-21chrt22:10:133
3059399563,36cyclictest15579-21taskset00:05:243
3059399562,37cyclictest18089-21taskset21:30:183
3059399554,36cyclictest11142-21chrt23:05:153
3059399554,35cyclictest718-21chrt20:55:213
3059399554,35cyclictest28281-21chrt20:40:193
3059399553,38cyclictest3359-21taskset22:51:463
3059399553,37cyclictest21152-21chrt20:19:573
3059399553,37cyclictest21152-21chrt20:19:573
3059399553,36cyclictest31418-21taskset20:50:193
3059399553,36cyclictest31418-21taskset20:50:193
3059399553,33cyclictest15858-21chrt20:00:343
3059399553,33cyclictest15858-21chrt20:00:343
3059399552,41cyclictest20210-21taskset23:20:143
3059399552,40cyclictest7603-21taskset22:05:323
3059399552,39cyclictest15284-21taskset23:11:033
3059399552,39cyclictest15284-21taskset23:11:033
3059399544,33cyclictest29947-21taskset00:29:593
3059399543,37cyclictest23195-21taskset22:30:403
3059399543,37cyclictest17482-21chrt23:15:243
3059399543,37cyclictest12973-21chrt19:54:563
3059399543,35cyclictest19670-21taskset20:15:003
3059399542,37cyclictest17047-21taskset21:28:403
3059399542,34cyclictest30993-21chrt19:10:183
3059399533,39cyclictest15943-21chrt22:20:063
3059399533,38cyclictest10991-21chrt19:45:333
3059399533,35cyclictest24238-21chrt00:20:153
3059399533,35cyclictest21088-21chrt21:35:163
3059399533,35cyclictest21088-21chrt21:35:163
3059399532,39cyclictest1999-21taskset19:20:143
3059399532,37cyclictest26357-21chrt20:35:003
3059399532,36cyclictest27816-21chrt22:39:553
3059399532,34cyclictest5610-21chrt22:55:313
3059399527,31cyclictest263782sleep323:30:123
3059399524,35cyclictest21836-21chrt22:29:593
3059399524,34cyclictest14720-21chrt00:04:553
3059399524,33cyclictest6628-21chrt22:04:593
3059399523,37cyclictest22334-21chrt00:16:063
3059399522,33cyclictest30538-21taskset21:50:243
3059399522,32cyclictest29506-21chrt23:35:143
3059399513,35cyclictest408-21taskset00:34:573
3059399513,35cyclictest30086-21chrt22:42:473
3059399513,33cyclictest4072-21chrt21:05:213
3059399513,33cyclictest23538-21chrt20:25:203
3059399513,30cyclictest31406-21chrt22:45:183
3059399512,38cyclictest8210-21chrt23:53:453
3059399504,34cyclictest31107-21chrt20:49:583
3059399504,34cyclictest31107-21chrt20:49:583
3059399503,35cyclictest10426-21chrt23:56:333
3059399503,34cyclictest1914-21chrt00:35:593
3059399502,36cyclictest24001-21taskset21:40:113
3059399502,35cyclictest23887-21chrt23:25:293
3059399502,35cyclictest23887-21chrt23:25:293
3059399502,34cyclictest4205-21chrt23:45:593
3059399502,34cyclictest12500-21chrt21:20:333
3059399502,34cyclictest12500-21chrt21:20:333
3059399493,35cyclictest5854-21chrt19:30:313
3059399492,36cyclictest872-21chrt19:15:293
3059399492,33cyclictest7217-21taskset21:12:383
3059399483,33cyclictest15706-21chrt22:19:583
3059399483,33cyclictest13380-21taskset19:55:163
3059399483,31cyclictest26910-21chrt20:35:253
3059399482,34cyclictest8740-21chrt19:40:183
3047624811,21sleep30-21swapper/319:09:593
3059399473,31cyclictest16460-21chrt20:05:003
3059399473,31cyclictest16460-21chrt20:05:003
3059399452,31cyclictest4096-21chrt19:25:253
3059399443,31cyclictest10048-21chrt23:03:213
3059399422,27cyclictest19066-21chrt20:10:343
3059399413,25cyclictest7786-21chrt19:35:373
3059399413,25cyclictest7786-21chrt19:35:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional