You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-03 - 03:43
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed Jun 03, 2026 00:44:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17131212914,87sleep217173-21ls18:55:392
17131212914,87sleep217173-21ls18:55:392
17779212417,70sleep017815-21munin-run18:59:530
17779212417,70sleep017815-21munin-run18:59:530
179379911434,68cyclictest15690-21apt-get20:40:191
17924991133,22cyclictest352-21runrttasks22:06:160
17924991113,48cyclictest22833-21diskstats21:00:480
17924991113,48cyclictest22833-21diskstats21:00:480
17924991093,49cyclictest18550-21ntpq22:41:190
17924991093,49cyclictest18550-21ntpq22:41:190
17924991092,32cyclictest27157-21apt-get22:55:360
17924991084,34cyclictest31851-21fschecks_time21:15:150
17924991083,34cyclictest31245-21cron21:14:520
17924991073,29cyclictest12886-21fschecks_time00:20:170
17924991064,35cyclictest4966-21latency_hist21:24:560
17924991064,35cyclictest4966-21latency_hist21:24:560
179249910622,9cyclictest0-21swapper/023:05:120
17924991054,47cyclictest3877-21fschecks_time00:05:160
17924991054,47cyclictest3877-21fschecks_time00:05:160
17924991054,26cyclictest7946-21/usr/sbin/munin00:10:570
17924991044,28cyclictest9032-21ssh21:30:280
17924991044,27cyclictest5383-21fschecks_time21:25:140
17924991044,27cyclictest14881-21ls22:35:180
179249910441,56cyclictest0-21swapper/021:45:140
179249910441,56cyclictest0-21swapper/021:45:140
17924991043,42cyclictest27027-21apt-key22:54:560
17924991043,36cyclictest23749-21mailstats21:55:260
17924991043,36cyclictest23749-21mailstats21:55:260
17924991042,27cyclictest19041-21awk23:35:250
17924991034,21cyclictest26820-21modprobe23:49:520
17924991033,25cyclictest23439-21awk22:49:520
17924991033,25cyclictest23439-21awk22:49:520
17924991032,27cyclictest1180-21fschecks_time19:55:170
179249910319,11cyclictest0-21swapper/022:10:220
17924991023,34cyclictest466-21latency_hist23:59:550
17924991023,34cyclictest466-21latency_hist23:59:550
17924991023,24cyclictest12449-21taskset21:36:180
17924991022,38cyclictest22234-21ssh21:53:360
17924991022,34cyclictest18120-21sh23:34:550
17924991022,34cyclictest18120-21sh23:34:550
17924991022,22cyclictest20824-21cpu19:10:040
17924991013,22cyclictest15188-21latency_hist20:39:560
17924991013,22cyclictest15188-21latency_hist20:39:560
17924991004,33cyclictest5615-21latency_hist20:09:550
17924991004,32cyclictest15940-21cut20:41:110
179249910024,23cyclictest19282-21uname20:50:180
17924991002,32cyclictest5367-21taskset22:19:560
17924991002,32cyclictest5367-21taskset22:19:560
1792499994,32cyclictest32025-21grep19:50:150
17924999934,58cyclictest27684-21threads19:35:270
17924999934,58cyclictest27684-21threads19:35:270
1792499993,35cyclictest5997-21/usr/sbin/munin22:20:190
1792499993,34cyclictest16626-21sed00:25:300
1792499993,34cyclictest16626-21sed00:25:300
1792499993,29cyclictest17545-21tune2fs20:45:120
1792499993,29cyclictest17545-21tune2fs20:45:120
1792499992,35cyclictest16833-21latency_hist21:44:550
1792499992,33cyclictest21055-21ntp_kernel_pll_20:55:210
1792499992,33cyclictest21055-21ntp_kernel_pll_20:55:210
1792499992,21cyclictest3728-21sh23:10:110
1792499992,20cyclictest7147-21proc_pri23:15:240
1792499984,33cyclictest8980-21/usr/sbin/munin20:20:140
1792499984,33cyclictest8980-21/usr/sbin/munin20:20:140
1792499984,32cyclictest32696-21modprobe23:04:510
1792499983,39cyclictest25104-21apt-get21:05:070
1792499983,32cyclictest31059-21grep19:45:290
1792499983,30cyclictest352-21runrttasks00:01:250
1792499983,19cyclictest12745-21/usr/sbin/munin20:30:210
1792499982,37cyclictest8312-21awk20:15:280
1792499982,37cyclictest8312-21awk20:15:280
1792499982,37cyclictest26666-21latency_hist19:34:560
17944999759,13cyclictest181rcu_preempt21:45:202
17944999759,13cyclictest181rcu_preempt21:45:202
17924999740,50cyclictest0-21swapper/020:00:250
1792499973,35cyclictest30043-21latency_hist19:44:550
1792499973,35cyclictest30043-21latency_hist19:44:550
1792499973,31cyclictest6359-21ps20:10:240
1792499973,31cyclictest6359-21ps20:10:240
17924999728,9cyclictest0-21swapper/019:00:160
1792499972,32cyclictest21472-21grep23:40:140
1792499972,25cyclictest8636-21sh22:25:110
1792499972,25cyclictest8636-21sh22:25:110
1792499964,31cyclictest29919-21latency_hist23:54:540
1792499964,31cyclictest11261-21mailstats20:25:260
1792499964,22cyclictest23260-21fschecks_count19:20:100
1792499964,22cyclictest13123-21ssh22:32:070
1792499963,37cyclictest25930-21runrttasks21:59:560
1792499963,32cyclictest352-21runrttasks23:24:230
17944999539,20cyclictest181rcu_preempt22:41:282
17944999539,20cyclictest181rcu_preempt22:41:282
1792499953,32cyclictest10482-21switchtime00:15:280
1792499952,33cyclictest19767-21timerandwakeup19:04:580
17944999372,8cyclictest181rcu_preempt20:08:122
1792499933,25cyclictest12386-21runrttasks23:25:040
17937999136,29cyclictest0-21swapper/120:15:131
17937999136,29cyclictest0-21swapper/120:15:131
17937999130,25cyclictest12787-21ssh00:20:121
1792499913,33cyclictest22912-21latency_hist19:19:550
1792499909,13cyclictest250652chrt19:29:530
17937998939,44cyclictest0-21swapper/123:45:051
1794499883,48cyclictest2696-21ssh21:20:282
1794499883,48cyclictest2696-21ssh21:20:282
17944998834,14cyclictest181rcu_preempt23:10:142
17937998829,24cyclictest24268-21taskset19:23:231
17937998746,32cyclictest28288-21proc_pri22:56:521
17937998627,27cyclictest15233-21systemd-run20:39:531
17937998627,27cyclictest15233-21systemd-run20:39:531
17937998626,25cyclictest0-21swapper/120:05:281
17937998626,18cyclictest352-21runrttasks23:18:371
17937998623,30cyclictest4090-21uname00:05:211
17937998623,30cyclictest4090-21uname00:05:211
17944998562,8cyclictest181rcu_preempt22:35:262
17944998548,13cyclictest181rcu_preempt00:11:142
17944998548,12cyclictest181rcu_preempt22:30:312
17944998545,16cyclictest181rcu_preempt23:31:092
17944998545,16cyclictest181rcu_preempt23:31:092
17937998523,28cyclictest0-21swapper/123:30:181
17937998523,28cyclictest0-21swapper/123:30:181
1732828514,38sleep10-21swapper/118:55:421
1732828514,38sleep10-21swapper/118:55:421
17944998462,9cyclictest181rcu_preempt21:17:262
17944998452,17cyclictest181rcu_preempt22:05:302
17944998432,23cyclictest27465-21uniq23:50:172
17937998432,19cyclictest647-21taskset00:00:091
17937998426,35cyclictest0-21swapper/123:54:541
17937998425,23cyclictest30686-21ssh23:00:231
17944998362,15cyclictest32221-21fschecks_count22:10:152
17944998344,8cyclictest181rcu_preempt23:40:092
17944998335,12cyclictest181rcu_preempt22:20:552
17937998341,37cyclictest0-21swapper/123:55:121
17937998341,37cyclictest0-21swapper/123:55:121
17937998341,15cyclictest10731-21kworker/u17:2-nfsiod21:38:461
17937998326,22cyclictest0-21swapper/122:23:241
17944998261,15cyclictest25232-21apt-key21:04:552
17944998261,15cyclictest25232-21apt-key21:04:552
17944998259,17cyclictest21906-21diskmemload22:00:202
17944998256,9cyclictest181rcu_preempt21:29:542
17944998249,11cyclictest181rcu_preempt23:01:362
17944998246,17cyclictest181rcu_preempt23:26:582
17944998243,11cyclictest1058-21expr19:55:132
17944998242,13cyclictest181rcu_preempt21:30:152
17944998234,15cyclictest181rcu_preempt23:05:112
17937998263,12cyclictest22890-21latency_hist21:54:551
17937998227,22cyclictest0-21swapper/122:51:221
17937998223,24cyclictest607-21ssh22:10:311
17937998222,29cyclictest0-21swapper/121:40:141
17944998165,10cyclictest26901-21ssh23:49:542
17944998154,9cyclictest181rcu_preempt20:20:292
17944998154,9cyclictest181rcu_preempt20:20:292
17944998149,10cyclictest181rcu_preempt23:17:092
17937998159,8cyclictest181rcu_preempt21:19:541
17937998127,24cyclictest2231-21taskset19:59:301
17937998123,30cyclictest12264-21latency_hist23:24:551
17937998119,28cyclictest0-21swapper/121:06:341
17944998057,17cyclictest2766-21fschecks_time20:00:142
17944998055,9cyclictest181rcu_preempt22:51:152
17944998051,13cyclictest181rcu_preempt20:41:182
17944998051,11cyclictest181rcu_preempt19:45:062
17944997947,12cyclictest181rcu_preempt19:07:172
17944997944,16cyclictest181rcu_preempt20:58:482
17944997944,16cyclictest181rcu_preempt20:58:482
17944997943,15cyclictest181rcu_preempt21:11:002
17937997922,23cyclictest0-21swapper/122:46:051
17937997922,23cyclictest0-21swapper/122:46:051
267622780,4sleep226764-21cat21:06:452
17944997855,8cyclictest181rcu_preempt00:17:422
17944997848,8cyclictest181rcu_preempt19:01:052
17944997845,8cyclictest181rcu_preempt21:55:492
17944997845,8cyclictest181rcu_preempt21:55:492
17944997845,10cyclictest181rcu_preempt19:52:032
17944997844,12cyclictest181rcu_preempt19:23:292
17937997852,20cyclictest6509-21switchtime20:10:261
17937997852,20cyclictest6509-21switchtime20:10:261
17937997841,31cyclictest301ktimers/120:55:041
17937997841,31cyclictest301ktimers/120:55:041
17937997826,24cyclictest13193-21runrttasks22:32:191
17937997821,26cyclictest0-21swapper/121:30:271
17937997814,27cyclictest0-21swapper/122:04:491
17944997754,9cyclictest181rcu_preempt23:34:562
17944997753,18cyclictest16739-21sh21:44:532
17944997748,10cyclictest181rcu_preempt22:46:462
17944997748,10cyclictest181rcu_preempt22:46:462
17944997746,10cyclictest181rcu_preempt22:17:402
17944997746,10cyclictest181rcu_preempt22:17:402
1794499774,41cyclictest17306-21cpu20:45:062
1794499774,41cyclictest17306-21cpu20:45:062
17944997744,13cyclictest181rcu_preempt00:09:052
17944997744,13cyclictest181rcu_preempt00:09:052
17944997741,9cyclictest181rcu_preempt20:25:122
17944997734,17cyclictest0-21swapper/222:25:122
17944997734,17cyclictest0-21swapper/222:25:122
17937997739,29cyclictest0-21swapper/122:05:041
17937997720,31cyclictest5500-21runrttasks23:12:581
17944997652,10cyclictest181rcu_preempt21:38:432
17944997649,10cyclictest181rcu_preempt19:40:272
17944997649,10cyclictest181rcu_preempt19:40:272
17944997639,18cyclictest181rcu_preempt20:15:192
17944997639,18cyclictest181rcu_preempt20:15:192
17944997626,32cyclictest26913-21acpi19:35:022
17944997626,32cyclictest26913-21acpi19:35:022
17937997660,10cyclictest11276-21latency_hist22:29:551
17937997660,10cyclictest11276-21latency_hist22:29:551
17937997642,23cyclictest12369-21apt-get23:25:021
17937997636,34cyclictest0-21swapper/121:55:051
17937997636,34cyclictest0-21swapper/121:55:051
17937997632,32cyclictest0-21swapper/119:35:051
17937997632,32cyclictest0-21swapper/119:35:051
17937997629,17cyclictest24988-21taskset19:29:311
17937997625,20cyclictest3341-21ssh21:20:531
17937997625,20cyclictest3341-21ssh21:20:531
17937997619,24cyclictest0-21swapper/100:10:111
17944997546,7cyclictest181rcu_preempt21:53:592
17944997534,14cyclictest181rcu_preempt20:50:192
17937997542,25cyclictest5101-21ssh21:25:071
17937997541,24cyclictest20815-21apt-get19:10:041
17937997528,17cyclictest0-21swapper/122:36:031
17944997449,19cyclictest15188-21latency_hist20:39:552
17944997449,19cyclictest15188-21latency_hist20:39:552
1794499743,52cyclictest27157-21apt-get22:55:192
17944997432,12cyclictest181rcu_preempt00:00:072
17937997455,12cyclictest21131-21latency_hist23:39:551
17937997435,29cyclictest2701-21/usr/sbin/munin20:00:131
17944997345,9cyclictest181rcu_preempt23:20:212
17944997253,13cyclictest13239-21sh00:20:262
17944997241,9cyclictest181rcu_preempt20:30:162
17944997234,8cyclictest181rcu_preempt00:25:262
17944997234,8cyclictest181rcu_preempt00:25:262
17937997257,9cyclictest1237-21processes23:05:231
17937997251,14cyclictest28682-21/usr/sbin/munin21:11:341
17937997249,16cyclictest30090-21latency_hist19:44:551
17937997249,16cyclictest30090-21latency_hist19:44:551
17937997240,21cyclictest22459-21apt-get21:00:381
17937997240,21cyclictest22459-21apt-get21:00:381
17937997227,34cyclictest18632-21cron20:49:511
17937997227,34cyclictest18632-21cron20:49:511
1795099713,54cyclictest26636-21chrt22:54:523
17944997145,11cyclictest181rcu_preempt20:10:152
17944997145,11cyclictest181rcu_preempt20:10:152
17944997142,8cyclictest181rcu_preempt19:34:562
17937997140,26cyclictest0-21swapper/100:15:051
17937997138,26cyclictest21268-21ssh23:40:061
17937997135,31cyclictest0-21swapper/120:25:081
17937997133,29cyclictest19892-21latency_hist21:49:551
17937997133,29cyclictest19892-21latency_hist21:49:551
17937997131,31cyclictest20515-21cron19:09:521
17937997131,30cyclictest26670-21idleruntime-cro19:34:521
17937997130,29cyclictest15741-21/usr/sbin/munin00:25:131
17937997130,29cyclictest15741-21/usr/sbin/munin00:25:131
17937997123,19cyclictest5068-21ssh22:19:381
17937997123,19cyclictest5068-21ssh22:19:381
17944997048,8cyclictest181rcu_preempt23:55:282
17944997048,8cyclictest181rcu_preempt23:55:282
17937997054,12cyclictest14557-21kworker/u17:3-flush-179:019:52:211
17937997033,30cyclictest9258-21fschecks_time20:20:161
17937997033,30cyclictest9258-21fschecks_time20:20:161
1795099693,51cyclictest5436-21taskset21:25:153
1795099693,50cyclictest5388-21taskset00:07:353
1795099693,50cyclictest5388-21taskset00:07:353
17944996946,9cyclictest181rcu_preempt19:10:232
17944996913,21cyclictest25096-21latency_hist19:29:542
17937996940,20cyclictest31731-21latency_hist19:49:551
17937996933,29cyclictest22917-21sort19:19:551
17937996930,29cyclictest12682-21memory20:30:191
17937996924,34cyclictest18986-21cpu20:50:091
17937996917,23cyclictest0-21swapper/122:40:151
17937996917,23cyclictest0-21swapper/122:40:151
1717526912,38sleep30-21swapper/318:55:403
1717526912,38sleep30-21swapper/318:55:403
17937996836,26cyclictest0-21swapper/119:00:041
1795099673,50cyclictest12073-21taskset21:35:293
1795099673,49cyclictest0-21swapper/320:15:073
1795099673,49cyclictest0-21swapper/320:15:073
1795099672,45cyclictest2297-21chrt19:59:523
17944996747,13cyclictest17915-21cyclictest19:16:292
1795099663,50cyclictest765-21chrt23:05:123
1795099663,49cyclictest28280-21chrt21:09:543
1795099663,48cyclictest11719-21taskset20:27:103
1795099663,40cyclictest0-21swapper/322:55:363
1795099662,51cyclictest23320-21chrt21:55:153
1795099662,51cyclictest23320-21chrt21:55:153
1795099662,50cyclictest8268-21chrt22:24:543
1795099662,50cyclictest5977-21taskset23:14:203
1795099662,50cyclictest28673-21chrt21:10:023
1795099652,51cyclictest16776-21chrt22:39:013
1795099652,50cyclictest12556-21taskset23:25:103
1795099652,50cyclictest0-21swapper/322:00:033
1795099652,49cyclictest6811-21chrt00:10:103
1795099652,49cyclictest2871-21taskset00:03:503
1795099643,48cyclictest8562-21taskset23:18:173
1795099643,48cyclictest20823-21chrt19:10:033
1795099643,48cyclictest18890-21chrt23:35:203
1795099643,48cyclictest12605-21taskset00:20:043
1795099643,47cyclictest6316-21chrt20:10:213
1795099643,47cyclictest6316-21chrt20:10:213
1795099643,47cyclictest20830-21chrt22:45:013
1795099643,47cyclictest20830-21chrt22:45:013
1795099642,49cyclictest17877-21taskset22:40:433
1795099642,49cyclictest17877-21taskset22:40:433
1795099642,49cyclictest12433-21taskset20:30:143
1795099642,49cyclictest0-21swapper/300:15:053
1795099642,48cyclictest23052-21taskset19:19:593
1795099635,43cyclictest0-21swapper/319:40:073
1795099635,43cyclictest0-21swapper/319:40:073
1795099633,49cyclictest13058-21chrt22:31:563
1795099633,48cyclictest14068-21chrt21:40:073
1795099633,47cyclictest8977-21chrt22:25:193
1795099633,47cyclictest8977-21chrt22:25:193
1795099633,47cyclictest16433-21chrt23:30:293
1795099633,47cyclictest16433-21chrt23:30:293
1795099633,46cyclictest545-21taskset19:53:293
1795099632,52cyclictest0-21swapper/322:05:043
1795099632,49cyclictest4067-21chrt21:22:563
1795099632,49cyclictest4067-21chrt21:22:563
1795099632,48cyclictest22038-21chrt20:59:543
1795099632,48cyclictest22038-21chrt20:59:543
1795099632,47cyclictest24619-21chrt21:03:333
1795099632,47cyclictest24619-21chrt21:03:333
1795099632,47cyclictest0-21swapper/320:35:083
1795099632,47cyclictest0-21swapper/320:35:083
1795099632,46cyclictest5362-21taskset20:08:193
1795099632,45cyclictest9447-21taskset21:31:153
1795099622,47cyclictest27366-21chrt23:50:113
1795099622,47cyclictest2513-21taskset19:59:573
1795099622,47cyclictest22894-21chrt23:42:053
1795099622,47cyclictest20221-21chrt19:06:153
1795099622,46cyclictest18121-21chrt19:00:073
1795099622,46cyclictest1335-21chrt21:18:383
1795099622,45cyclictest30098-21chrt23:55:023
1795099622,45cyclictest30098-21chrt23:55:023
1795099613,46cyclictest26066-21chrt23:47:443
1795099613,45cyclictest3888-21chrt22:16:193
1795099613,45cyclictest3888-21chrt22:16:193
1795099613,45cyclictest22415-21chrt19:15:103
1795099613,45cyclictest18516-21chrt20:48:203
1795099613,45cyclictest18516-21chrt20:48:203
1795099612,48cyclictest27683-21taskset19:35:263
1795099612,48cyclictest27683-21taskset19:35:263
1795099612,47cyclictest11590-21taskset23:23:323
1795099612,46cyclictest31159-21chrt23:00:293
1795099612,46cyclictest19343-21chrt20:50:193
1795099612,46cyclictest10115-21chrt20:22:123
1795099612,46cyclictest10115-21chrt20:22:123
1795099612,45cyclictest16053-21taskset00:25:193
1795099612,45cyclictest16053-21taskset00:25:193
1795099603,45cyclictest21137-21chrt21:50:333
1795099603,44cyclictest26375-21taskset19:31:173
1795099602,46cyclictest24770-21chrt19:26:443
1795099602,44cyclictest31257-21chrt19:47:093
1795099592,45cyclictest16261-21chrt20:41:193
1795099592,44cyclictest18777-21chrt21:47:213
1795099592,44cyclictest18777-21chrt21:47:213
1795099582,46cyclictest0-21swapper/322:10:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional