You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-02 - 11:39
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Jan 02, 2026 00:44:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28520216412,107sleep128570-21fschecks_count19:05:271
298139912441,8cyclictest0-21swapper/023:50:480
298139912441,8cyclictest0-21swapper/023:50:480
29828991175,59cyclictest27663-21ssh22:35:142
298139911535,14cyclictest0-21swapper/021:40:370
29813991153,46cyclictest5288-21/usr/sbin/munin21:10:260
298139911346,59cyclictest0-21swapper/023:05:230
298139911346,59cyclictest0-21swapper/023:05:230
298139911243,8cyclictest0-21swapper/022:59:570
298139911236,11cyclictest0-21swapper/019:45:330
298139911141,17cyclictest30911-21fschecks_time23:30:220
298139911137,13cyclictest0-21swapper/023:55:290
298139911137,13cyclictest0-21swapper/023:55:290
29813991112,72cyclictest24572-21cstates22:30:540
29813991104,48cyclictest352-21runrttasks21:56:520
298139911033,9cyclictest0-21swapper/021:45:360
29813991093,90cyclictest27643-21apt-get22:35:130
29813991093,21cyclictest4870-21/usr/sbin/munin00:30:280
29813991084,40cyclictest30674-21latency_hist20:50:020
298139910835,19cyclictest3054-21latency_hist21:05:010
298139910835,19cyclictest3054-21latency_hist21:05:010
29813991073,32cyclictest14602-21fschecks_time22:15:220
29813991073,32cyclictest14602-21fschecks_time22:15:220
29813991073,29cyclictest7695-21ssh00:35:130
29813991073,29cyclictest7695-21ssh00:35:130
28516210711,65sleep028333-21/usr/sbin/munin19:05:260
29813991063,71cyclictest956-21apt-get19:20:130
298139910637,11cyclictest0-21swapper/020:15:220
298139910637,11cyclictest0-21swapper/019:25:350
298139910631,12cyclictest0-21swapper/020:35:240
298139910627,27cyclictest22189-21apt-get20:25:120
29813991054,34cyclictest32025-21/usr/sbin/munin19:15:290
29813991053,24cyclictest352-21runrttasks22:28:590
29813991033,38cyclictest25200-21tail23:20:360
298139910234,9cyclictest0-21swapper/023:15:160
29813991023,39cyclictest3790-21wc21:05:320
29813991022,40cyclictest12141-21sort21:20:330
29813991022,38cyclictest18537-21ssh21:30:330
298139910215,33cyclictest30670-21sshd00:20:240
133321020,5sleep21336-21fschecks_time00:25:212
29813991013,33cyclictest21675-21timerwakeupswit20:20:340
2981399993,37cyclictest13689-21grep19:55:360
2981399993,37cyclictest13689-21grep19:55:360
29813999912,28cyclictest1955-21timerandwakeup00:25:350
2981399984,35cyclictest17582-21/usr/sbin/munin20:10:330
2981399984,33cyclictest23504-21latency_hist00:10:010
29828999749,14cyclictest181rcu_preempt19:25:112
2981399963,36cyclictest11608-21fschecks_time23:00:210
2981399963,31cyclictest1336-21latency_hist21:00:010
2982899953,47cyclictest7679-21apt-get00:35:132
2982899953,47cyclictest7679-21apt-get00:35:132
2981399954,38cyclictest352-21runrttasks23:49:010
2981399932,25cyclictest2352-21sed23:35:400
2981399913,37cyclictest15344-21head20:00:360
2982899908,67cyclictest1467-21apt-get19:20:132
29828999046,12cyclictest181rcu_preempt23:55:152
29828999046,12cyclictest181rcu_preempt23:55:152
2982899903,63cyclictest24218-21ssh22:30:132
29817999045,36cyclictest341-21cat19:15:361
29828998950,9cyclictest181rcu_preempt19:55:122
29828998950,9cyclictest181rcu_preempt19:55:122
29813998971,11cyclictest370-21snmpd22:45:360
29828998861,8cyclictest39-21ksoftirqd/221:35:412
29828998851,9cyclictest39-21ksoftirqd/223:54:262
29828998851,9cyclictest39-21ksoftirqd/223:54:262
29817998870,12cyclictest4526-21diskmemload21:19:411
29817998862,18cyclictest14759-21latency22:15:251
29817998862,18cyclictest14759-21latency22:15:251
29813998870,11cyclictest15660-21ssh21:25:380
2981399883,34cyclictest5709-21ssh22:00:360
2981799864,34cyclictest31-21ksoftirqd/120:44:541
29813998662,16cyclictest31465-21latency_hist19:15:010
29828998553,10cyclictest181rcu_preempt00:30:152
2981799852,34cyclictest31-21ksoftirqd/123:35:261
29828998461,16cyclictest29806-21cyclictest19:45:372
29817998457,18cyclictest4526-21diskmemload00:00:011
29817998457,18cyclictest4526-21diskmemload00:00:011
2981799843,31cyclictest31-21ksoftirqd/100:06:171
29813998459,17cyclictest21885-21timerandwakeup21:35:370
29817998355,20cyclictest10813-21latency_hist23:50:031
2981799832,69cyclictest277-21dbus-daemon22:39:591
29813998358,18cyclictest1-21systemd19:43:420
29828998239,12cyclictest181rcu_preempt22:24:232
29828998151,9cyclictest29413-21rm00:18:442
29817998156,19cyclictest29806-21cyclictest22:20:141
29813998158,16cyclictest29806-21cyclictest22:05:260
2982899806,53cyclictest22696-21apt-get20:25:122
29828998058,13cyclictest31595-21munin-node23:20:372
29828998057,15cyclictest200-21systemd-journal00:06:412
29817998063,9cyclictest0-21swapper/121:30:191
29817998056,15cyclictest22308-21apt-key20:25:011
2983499793,32cyclictest8471-21chrt22:05:313
29817997958,16cyclictest27729-21fschecks_time21:45:221
2836427911,32sleep20-21swapper/219:05:202
2983499783,36cyclictest7519-21chrt22:54:293
2983499783,36cyclictest7519-21chrt22:54:293
29828997846,9cyclictest7766-21/usr/sbin/munin23:45:242
29817997854,16cyclictest200-21systemd-journal23:29:411
29817997853,19cyclictest352-21runrttasks19:21:111
29817997822,13cyclictest301ktimers/119:55:151
29817997822,13cyclictest301ktimers/119:55:151
29834997711,31cyclictest27681-21chrt22:35:173
2981799776,56cyclictest7122-21apt-get00:35:131
2981799776,56cyclictest7122-21apt-get00:35:131
2981799775,29cyclictest31-21ksoftirqd/120:00:371
2981799773,27cyclictest31-21ksoftirqd/119:11:521
2981799772,39cyclictest31-21ksoftirqd/119:25:321
2981799772,38cyclictest31-21ksoftirqd/123:10:201
29813997748,21cyclictest1-21systemd00:16:400
2981399773,40cyclictest9138-21grep21:15:350
29813997728,12cyclictest0-21swapper/023:10:330
29828997655,16cyclictest29806-21cyclictest21:31:002
29828997650,8cyclictest181rcu_preempt23:10:122
29828997646,8cyclictest181rcu_preempt20:55:152
29828997638,15cyclictest181rcu_preempt00:25:012
29828997633,12cyclictest181rcu_preempt21:05:322
29828997622,9cyclictest0-21swapper/223:35:122
29817997658,12cyclictest29806-21cyclictest00:00:151
29817997624,10cyclictest301ktimers/100:15:091
29813997654,14cyclictest5983-21ssh23:41:190
29813997616,28cyclictest6408-21sh22:51:410
29813997616,28cyclictest6408-21sh22:51:410
2981799757,32cyclictest31-21ksoftirqd/120:10:281
2981799756,33cyclictest31-21ksoftirqd/121:29:031
29817997554,11cyclictest883-21latency_hist00:25:011
29817997551,10cyclictest181rcu_preempt21:51:211
29817997551,10cyclictest181rcu_preempt21:51:211
29813997558,11cyclictest30964-21fschecks_count20:50:190
2983499742,50cyclictest24471-21chrt22:30:513
29828997455,13cyclictest29806-21cyclictest23:05:392
29828997455,13cyclictest29806-21cyclictest23:05:392
29828997434,13cyclictest181rcu_preempt21:29:592
2982899742,35cyclictest26108-21cut20:35:232
29817997450,17cyclictest544-21cron22:54:581
29817997450,17cyclictest544-21cron22:54:581
2981799743,34cyclictest31-21ksoftirqd/100:10:221
2981799742,29cyclictest31-21ksoftirqd/120:55:321
143132742,21sleep114423-21latency_hist21:25:021
2983499733,31cyclictest15160-21chrt23:05:323
2983499733,31cyclictest15160-21chrt23:05:323
29828997344,9cyclictest181rcu_preempt19:10:112
29828997343,9cyclictest181rcu_preempt19:15:162
29817997357,10cyclictest23220-21ntp_offset20:25:301
29817997356,11cyclictest8992-21timerandwakeup22:55:361
29817997348,17cyclictest1-21systemd19:36:421
2981799734,31cyclictest31-21ksoftirqd/121:43:511
2981799733,50cyclictest27321-21apt-key22:35:011
2981799733,36cyclictest31-21ksoftirqd/120:39:521
29813997358,8cyclictest0-21swapper/023:25:390
2981399734,53cyclictest16092-21cstates20:05:160
2981399732,39cyclictest24141-21fschecks_time00:10:200
44822720,5sleep30-21swapper/319:30:013
29828997256,10cyclictest39-21ksoftirqd/222:15:022
29828997253,13cyclictest24786-21ntp_kernel_pll_21:40:282
29828997246,17cyclictest24710-21threads00:10:342
29828997244,20cyclictest1-21systemd19:54:412
29828997243,20cyclictest18353-21threads20:10:342
29828997236,9cyclictest181rcu_preempt21:10:112
2982899722,40cyclictest11634-21fschecks_time21:20:212
29828997214,39cyclictest300-21rs:main2
29817997253,13cyclictest1676-21sh00:25:291
29817997251,14cyclictest12169-21grep22:10:361
29817997249,12cyclictest1217-21latency_hist23:35:021
29817997247,7cyclictest181rcu_preempt23:15:551
2981799723,56cyclictest6033-21munin-run19:34:591
2981799723,22cyclictest31-21ksoftirqd/121:58:271
29813997236,29cyclictest0-21swapper/022:10:400
29813997234,28cyclictest0-21swapper/021:51:100
29813997234,28cyclictest0-21swapper/021:51:100
29817997149,14cyclictest370-21snmpd21:11:551
29817997149,13cyclictest0-21swapper/123:05:231
29817997149,13cyclictest0-21swapper/123:05:231
2981799713,47cyclictest4655-21ssh23:40:141
2981799713,18cyclictest31-21ksoftirqd/122:47:231
29817997122,9cyclictest291rcuc/120:30:321
29813997156,11cyclictest29806-21cyclictest00:00:400
29813997128,31cyclictest5645-21unixbench_singl19:30:350
2983499703,31cyclictest1688-21chrt21:55:223
29817997056,9cyclictest4183-21rm00:30:031
29817997053,11cyclictest24145-21/usr/sbin/munin23:20:371
2981799704,28cyclictest30836-21taskset22:40:061
2981799703,27cyclictest31-21ksoftirqd/120:54:591
29828996943,8cyclictest181rcu_preempt23:40:322
29828996941,10cyclictest181rcu_preempt23:03:482
29828996928,9cyclictest181rcu_preempt21:45:282
29817996952,11cyclictest8418-21rm22:05:291
29817996946,13cyclictest0-21swapper/122:00:361
2982899685,23cyclictest32479-21ssh21:52:242
2982899685,23cyclictest32479-21ssh21:52:242
29828996842,11cyclictest181rcu_preempt22:25:152
29828996841,16cyclictest14443-21cpuspeed20:00:152
2982899683,48cyclictest30387-21latency_hist23:30:012
29828996831,10cyclictest9182-21/usr/sbin/munin21:15:362
29828996826,11cyclictest181rcu_preempt22:15:312
29828996826,11cyclictest181rcu_preempt22:15:312
29817996850,13cyclictest29806-21cyclictest19:49:411
29817996848,14cyclictest7910-21/usr/sbin/munin19:40:231
29817996831,9cyclictest301ktimers/120:45:321
2981799682,26cyclictest31-21ksoftirqd/121:39:531
29813996824,14cyclictest7637-21sh19:39:590
29828996753,9cyclictest7475-21kworker/u17:3-rpciod00:00:202
29828996744,8cyclictest181rcu_preempt23:34:582
29828996741,10cyclictest181rcu_preempt23:15:372
2982899673,43cyclictest352-21runrttasks19:32:052
29828996723,32cyclictest27942-21latency20:40:242
2981799675,33cyclictest31-21ksoftirqd/121:05:561
2981399674,30cyclictest31177-21fschecks_time22:40:210
29813996735,12cyclictest181rcu_preempt22:24:580
124172670,7sleep30-21swapper/323:00:373
2983499669,31cyclictest0-21swapper/319:20:243
29828996646,12cyclictest4526-21diskmemload22:45:212
29828996643,14cyclictest0-21swapper/220:20:012
2982899663,53cyclictest1436-21ssh21:55:152
2982899663,45cyclictest30018-21timerwakeupswit20:45:362
2982899663,32cyclictest9202-21/usr/sbin/munin22:55:392
29828996631,11cyclictest29806-21cyclictest22:40:392
2982899662,31cyclictest6781-21ntpq19:35:282
29828996617,37cyclictest21546-21cat20:20:332
29817996645,15cyclictest22887-21sh22:28:081
29813996648,12cyclictest297-21systemd-logind19:54:590
2981399664,42cyclictest27929-21latency20:40:240
2981399663,26cyclictest25647-21idleruntime-cro20:34:590
2983499657,27cyclictest0-21swapper/320:25:123
29828996542,15cyclictest1-21systemd21:04:412
29828996542,15cyclictest1-21systemd21:04:412
29828996531,10cyclictest181rcu_preempt20:05:202
29828996515,10cyclictest181rcu_preempt22:00:172
2981799655,12cyclictest291rcuc/123:53:291
2981799655,12cyclictest291rcuc/123:53:291
29817996544,8cyclictest181rcu_preempt23:00:321
2981799652,22cyclictest31-21ksoftirqd/119:51:351
29817996444,7cyclictest181rcu_preempt21:02:261
29817996444,7cyclictest181rcu_preempt21:02:261
2981799643,43cyclictest20808-21latency_hist20:20:031
29817996422,28cyclictest16282-21sort20:05:221
2983499638,33cyclictest0-21swapper/300:35:133
2983499638,33cyclictest0-21swapper/300:35:133
2982899633,40cyclictest32150-21latency_hist20:55:012
29828996237,10cyclictest181rcu_preempt22:05:522
29828996224,9cyclictest181rcu_preempt20:30:242
2982899612,22cyclictest4887-21http22:50:162
2982899612,22cyclictest4887-21http22:50:162
2983499603,39cyclictest9994-21taskset19:45:283
2983499562,41cyclictest26868-21taskset21:44:283
2983499553,38cyclictest18747-21taskset20:12:413
2983499543,35cyclictest29136-21taskset20:45:093
2908525416,23sleep30-21swapper/319:05:463
2983499533,36cyclictest17712-21chrt23:10:093
2983499533,35cyclictest28027-21chrt21:45:293
2983499533,34cyclictest23875-21chrt00:10:143
2983499533,33cyclictest5920-21chrt23:41:093
2983499532,39cyclictest20171-21taskset22:24:233
2983499532,34cyclictest16904-21taskset21:28:433
2983499532,32cyclictest31842-21chrt23:30:563
2983499524,32cyclictest32276-21taskset21:51:533
2983499524,32cyclictest32276-21taskset21:51:533
2983499523,37cyclictest10504-21chrt21:18:513
2983499523,35cyclictest3110-21chrt21:05:013
2983499523,35cyclictest3110-21chrt21:05:013
2983499523,35cyclictest22868-21chrt21:37:353
2983499523,33cyclictest8405-21chrt19:40:263
2983499523,33cyclictest7960-21chrt21:15:023
2983499523,32cyclictest17743-21chrt21:30:093
2983499523,31cyclictest30142-21chrt19:10:173
29834995224,19cyclictest7734-21chrt22:05:023
2983499522,34cyclictest23798-21chrt23:19:583
2983499513,35cyclictest15216-21taskset20:00:343
2983499513,33cyclictest7814-21taskset23:45:143
2983499513,33cyclictest22343-21chrt20:25:003
2983499513,33cyclictest2170-21chrt23:35:383
2983499513,32cyclictest28130-21chrt20:40:323
2983499512,32cyclictest7733-21taskset19:40:013
2983499512,32cyclictest23096-21chrt22:28:403
2983499503,36cyclictest1949-21chrt22:45:223
2983499503,31cyclictest30286-21chrt23:29:593
2983499503,30cyclictest27561-21chrt00:15:253
2983499502,34cyclictest517-21chrt22:43:033
2983499502,33cyclictest24402-21chrt20:30:193
2983499502,31cyclictest2614-21chrt00:26:433
2983499493,31cyclictest19468-21taskset00:02:533
2983499493,30cyclictest26371-21chrt20:35:293
2983499493,27cyclictest19790-21taskset20:15:333
2983499492,31cyclictest11645-21taskset21:20:213
2983499483,33cyclictest11364-21chrt22:10:213
2983499483,32cyclictest30982-21chrt20:50:203
29834994825,14cyclictest44562sleep300:30:213
2983499482,32cyclictest14320-21taskset20:00:023
2983499482,32cyclictest14320-21taskset20:00:023
2983499473,32cyclictest30640-21chrt00:20:233
2983499473,32cyclictest20875-21chrt00:05:203
2983499473,32cyclictest16727-21chrt22:18:523
2983499473,32cyclictest16727-21chrt22:18:523
2983499473,28cyclictest32455-21chrt19:15:273
2983499463,28cyclictest26909-21chrt23:24:403
2983499453,29cyclictest3711-21chrt21:05:303
2983499433,30cyclictest4695-21chrt19:30:133
2983499433,28cyclictest8411-21taskset22:55:233
2983499423,30cyclictest15012-21taskset23:55:363
2983499423,30cyclictest15012-21taskset23:55:363
2983499413,26cyclictest11706-21chrt23:50:413
2983499413,26cyclictest11706-21chrt23:50:413
2983499393,25cyclictest11117-21taskset19:50:063
2983499392,25cyclictest32642-21taskset20:55:223
2983499383,26cyclictest17088-21taskset20:06:583
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional