You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-02 - 10:48
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Tue Jun 02, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
993721080,4sleep19941-21wc23:40:301
993721080,4sleep19941-21wc23:40:301
1638210113,47sleep00-21swapper/018:55:320
2607999818,51cyclictest39-21ksoftirqd/220:30:342
2599999738,30cyclictest0-21swapper/000:00:410
2603999470,15cyclictest25961-21ssh21:25:381
2603999468,19cyclictest2590-21cyclictest19:14:561
2603999458,12cyclictest181rcu_preempt23:20:501
2607999358,25cyclictest277-21dbus-daemon21:34:562
260399938,66cyclictest13882-21apt-get22:00:131
2603999375,10cyclictest0-21swapper/121:30:221
2603999220,54cyclictest12514-21sh23:45:181
2603999220,54cyclictest12514-21sh23:45:181
2599999174,11cyclictest2590-21cyclictest00:19:160
2599999119,34cyclictest28073-21sh22:23:320
2607999071,13cyclictest4596-21kworker/u19:0+flush-179:022:00:302
2599999011,56cyclictest27682-21/usr/sbin/munin20:15:330
2607998871,11cyclictest14933-21latency22:55:232
2599998825,31cyclictest0-21swapper/000:27:310
259999876,55cyclictest14606-21cstates22:55:140
2599998725,38cyclictest0-21swapper/022:44:580
2599998714,27cyclictest0-21swapper/020:20:120
2599998711,42cyclictest6180-21modprobe19:09:560
2607998651,12cyclictest181rcu_preempt20:08:542
2603998669,12cyclictest2590-21cyclictest20:40:321
259999867,47cyclictest29680-21ssh00:13:410
2599998637,30cyclictest0-21swapper/021:25:260
2599998628,23cyclictest0-21swapper/022:15:240
2599998620,47cyclictest0-21swapper/023:37:210
2603998558,13cyclictest0-21swapper/123:05:221
2599998523,22cyclictest5432-21ntpq20:45:280
260399843,34cyclictest1599-21ssh21:40:051
260399843,34cyclictest1599-21ssh21:40:051
2599998424,54cyclictest0-21swapper/021:15:120
2599998413,50cyclictest0-21swapper/022:46:540
2599998413,50cyclictest0-21swapper/022:46:540
122028411,38sleep10-21swapper/118:55:161
2607998317,37cyclictest39-21ksoftirqd/221:22:022
2607998314,42cyclictest39-21ksoftirqd/221:13:302
2603998358,16cyclictest20125-21latency_hist23:04:591
2603998340,11cyclictest181rcu_preempt22:35:081
2603998340,11cyclictest181rcu_preempt22:35:081
260799822,57cyclictest12484-21ssh23:45:142
260799822,57cyclictest12484-21ssh23:45:142
2603998261,15cyclictest15360-21proc_pri19:35:301
2603998261,15cyclictest15360-21proc_pri19:35:301
259999824,62cyclictest20405-21latency22:10:230
2599998219,26cyclictest1579-21runrttasks21:40:000
2599998219,26cyclictest1579-21runrttasks21:40:000
2599998216,41cyclictest13264-21ssh21:59:460
2599998212,28cyclictest13829-21grep19:30:300
2599998212,28cyclictest13829-21grep19:30:300
2607998150,11cyclictest181rcu_preempt20:48:272
260799814,45cyclictest39-21ksoftirqd/221:00:552
2603998157,16cyclictest13913-21unixbench_singl21:05:371
2603998156,9cyclictest24428-21ssh00:05:071
2603998132,12cyclictest18615-21sh23:55:131
2603998132,12cyclictest18615-21sh23:55:131
260399812,33cyclictest31-21ksoftirqd/123:35:121
259999818,35cyclictest14422-21fschecks_time22:00:540
2599998115,30cyclictest9006-21ssh21:51:120
2599998115,30cyclictest9006-21ssh21:51:120
2599998115,27cyclictest0-21swapper/022:35:100
2599998115,27cyclictest0-21swapper/022:35:100
2607998060,14cyclictest19717-21threads21:15:322
2603998048,9cyclictest181rcu_preempt22:08:421
2603998048,9cyclictest181rcu_preempt22:08:421
259999807,41cyclictest18282-21switchtime23:00:320
259999806,38cyclictest17403-21ssh21:11:470
2599998055,19cyclictest18133-21sh23:54:550
2599998021,18cyclictest7816-21modprobe19:14:560
2599998019,23cyclictest0-21swapper/023:49:570
2599998019,23cyclictest0-21swapper/023:49:570
2599998016,24cyclictest0-21swapper/021:08:320
2599998011,28cyclictest21496-21grep19:55:170
2599998011,28cyclictest21496-21grep19:55:170
260799797,30cyclictest39-21ksoftirqd/219:52:532
2607997960,14cyclictest544-21cron20:14:562
2607997952,18cyclictest1-21systemd22:23:152
2603997946,9cyclictest181rcu_preempt20:14:591
2603997925,30cyclictest0-21swapper/122:25:091
2603997912,32cyclictest0-21swapper/121:47:501
2599997917,22cyclictest1073-21seq20:30:350
2599997916,26cyclictest2236-21rm23:28:110
2599997916,23cyclictest24315-21sh23:10:260
2599997914,24cyclictest4105-21seq19:04:370
2599997912,25cyclictest1609-21/usr/sbin/munin00:20:280
2599997911,27cyclictest10023-21/usr/sbin/munin21:00:150
2599997911,25cyclictest18951-21grep19:45:330
2607997860,13cyclictest8930-21sh23:39:572
2607997860,12cyclictest3938-21mailstats20:40:282
2607997857,15cyclictest12881-21cpu21:05:162
2607997840,21cyclictest29846-21ntp_states20:20:272
260399783,48cyclictest31141-21fschecks_time20:25:191
260399783,48cyclictest31141-21fschecks_time20:25:191
259999788,42cyclictest2374-21systemd-journal23:23:150
2599997814,30cyclictest31893-21/usr/sbin/munin22:30:130
2607997761,10cyclictest9078-21awk19:15:342
2607997748,7cyclictest181rcu_preempt19:33:502
2607997748,7cyclictest181rcu_preempt19:33:502
2607997744,12cyclictest181rcu_preempt19:09:002
260799774,34cyclictest39-21ksoftirqd/223:59:502
260799774,34cyclictest39-21ksoftirqd/223:59:502
2607997738,17cyclictest181rcu_preempt21:50:292
2607997738,17cyclictest181rcu_preempt21:50:292
260799772,40cyclictest39-21ksoftirqd/223:53:342
2603997759,9cyclictest25490-21ssh22:19:431
2603997744,13cyclictest181rcu_preempt21:04:011
260399772,30cyclictest31-21ksoftirqd/121:35:391
260399771,24cyclictest26036-21latency_hist20:10:001
259999779,34cyclictest1889-21fschecks_count20:35:190
2599997759,12cyclictest352-21runrttasks21:20:240
259999775,31cyclictest0-21swapper/019:50:130
2599997714,28cyclictest1120-21ssh21:39:250
2599997713,28cyclictest26383-21ssh00:07:540
2607997658,8cyclictest0-21swapper/222:49:562
2607997658,8cyclictest0-21swapper/222:49:562
2607997641,10cyclictest28221-21ntpq20:15:222
260799762,35cyclictest39-21ksoftirqd/200:19:442
2603997652,9cyclictest181rcu_preempt23:27:311
2603997637,8cyclictest181rcu_preempt21:55:111
2603997634,12cyclictest181rcu_preempt19:25:541
2603997634,12cyclictest181rcu_preempt19:25:541
2603997629,8cyclictest181rcu_preempt23:15:151
2603997629,8cyclictest181rcu_preempt23:15:151
259999769,29cyclictest16131-21latency_hist19:39:590
259999765,36cyclictest9197-21/usr/sbin/munin23:40:260
259999765,36cyclictest9197-21/usr/sbin/munin23:40:260
259999764,62cyclictest11034-21apt-get19:25:110
259999764,62cyclictest11034-21apt-get19:25:110
2599997620,26cyclictest12828-21sshd22:51:160
2599997618,24cyclictest4981-21runrttasks21:45:190
2599997616,22cyclictest26665-21/usr/sbin/munin23:15:190
2599997616,22cyclictest26665-21/usr/sbin/munin23:15:190
2599997612,42cyclictest0-21swapper/020:50:290
2599997611,30cyclictest21347-21date23:59:590
2599997611,30cyclictest21347-21date23:59:590
2599997610,25cyclictest0-21swapper/023:34:550
2599997610,25cyclictest0-21swapper/023:34:550
2607997557,13cyclictest12171-21apt-key23:45:002
2607997557,13cyclictest12171-21apt-key23:45:002
2607997555,15cyclictest2590-21cyclictest00:08:152
2607997555,14cyclictest6914-21kernelversion20:50:212
2607997543,7cyclictest181rcu_preempt19:25:332
2607997543,7cyclictest181rcu_preempt19:25:332
2607997535,9cyclictest58-21kcompactd021:28:152
2607997525,24cyclictest39-21ksoftirqd/200:25:152
2603997559,8cyclictest0-21swapper/120:49:561
2603997557,11cyclictest23985-21fschecks_time23:10:171
2603997553,13cyclictest9496-21cron20:59:561
2603997536,11cyclictest181rcu_preempt23:51:381
260399753,32cyclictest2982-21fschecks_count19:00:181
2599997554,15cyclictest31117-21fschecks_count20:25:180
2599997554,15cyclictest31117-21fschecks_count20:25:180
259999755,25cyclictest0-21swapper/022:05:210
259999755,25cyclictest0-21swapper/022:05:210
259999754,31cyclictest26195-21cstates20:10:140
259999753,63cyclictest28125-21rm21:30:120
2599997526,17cyclictest8708-21ntp_states20:55:290
2599997517,21cyclictest352-21runrttasks19:15:080
2599997515,23cyclictest24749-21ls20:05:200
2599997515,22cyclictest28919-21ssh22:25:110
2599997513,30cyclictest15305-21awk19:35:260
2599997510,30cyclictest10170-21/usr/sbin/munin19:20:230
2607997446,8cyclictest181rcu_preempt23:34:512
2607997446,8cyclictest181rcu_preempt23:34:512
2607997436,21cyclictest22063-21ntp_states00:00:262
2607997421,33cyclictest39-21ksoftirqd/221:58:392
2607997417,25cyclictest39-21ksoftirqd/222:44:302
2603997449,8cyclictest24617-21ssh21:24:591
2603997441,11cyclictest181rcu_preempt00:29:231
2603997432,9cyclictest181rcu_preempt20:23:101
2599997417,26cyclictest16735-21awk19:40:210
2599997410,25cyclictest21684-21grep23:06:560
2607997355,13cyclictest22100-21threads23:07:062
2607997354,12cyclictest1-21systemd19:14:582
2607997343,8cyclictest181rcu_preempt20:29:022
2607997343,8cyclictest181rcu_preempt20:29:022
2607997338,9cyclictest2590-21cyclictest22:37:062
2607997338,9cyclictest2590-21cyclictest22:37:062
2607997334,11cyclictest2590-21cyclictest21:36:052
260799733,28cyclictest39-21ksoftirqd/223:00:212
2607997332,10cyclictest181rcu_preempt22:51:202
2607997330,12cyclictest181rcu_preempt19:43:312
2603997330,9cyclictest181rcu_preempt21:15:311
2603997330,23cyclictest0-21swapper/119:45:091
2603997329,12cyclictest181rcu_preempt21:10:311
2603997329,12cyclictest181rcu_preempt20:35:251
2603997315,9cyclictest3223-21apt-get23:30:061
2603997315,9cyclictest3223-21apt-get23:30:061
2599997316,45cyclictest0-21swapper/020:00:070
2599997316,45cyclictest0-21swapper/020:00:070
2607997259,9cyclictest2590-21cyclictest00:14:262
2607997257,9cyclictest21127-21kworker/u17:0-xprtiod00:20:322
2607997235,8cyclictest181rcu_preempt19:02:152
2607997233,10cyclictest9526-21diskmemload23:20:402
2607997232,11cyclictest181rcu_preempt23:26:292
2607997217,37cyclictest39-21ksoftirqd/220:35:172
2607997212,32cyclictest39-21ksoftirqd/222:13:572
2603997241,13cyclictest181rcu_preempt19:41:061
2603997237,12cyclictest181rcu_preempt00:23:011
2603997236,18cyclictest16908-21ssh22:59:191
260399723,49cyclictest31396-21ssh00:15:321
260399722,29cyclictest28598-21grep20:15:301
252027210,43sleep22532-21systemctl18:59:582
2607997149,15cyclictest2590-21cyclictest22:34:592
2607997134,10cyclictest181rcu_preempt21:44:192
2607997134,10cyclictest181rcu_preempt21:44:192
2607997132,8cyclictest181rcu_preempt22:26:022
260799712,37cyclictest39-21ksoftirqd/219:56:132
260799712,37cyclictest39-21ksoftirqd/219:56:132
260799712,32cyclictest39-21ksoftirqd/220:00:312
260799712,32cyclictest39-21ksoftirqd/220:00:312
2603997147,8cyclictest181rcu_preempt19:15:291
2603997136,12cyclictest181rcu_preempt22:40:551
2603997136,12cyclictest181rcu_preempt20:50:291
2603997134,8cyclictest181rcu_preempt22:30:181
2603997128,9cyclictest181rcu_preempt19:05:151
2603997126,8cyclictest181rcu_preempt22:51:131
2599997116,19cyclictest3319-21taskset20:40:100
2607997033,8cyclictest181rcu_preempt22:07:422
2607997033,8cyclictest181rcu_preempt22:07:422
260399708,30cyclictest24570-21kworker/1:0-events22:11:571
2603997036,8cyclictest181rcu_preempt00:04:551
260399703,43cyclictest22187-21head19:55:321
260399703,43cyclictest22187-21head19:55:321
2607996952,7cyclictest0-21swapper/222:19:152
2607996949,13cyclictest5110-21latency21:45:222
2607996935,8cyclictest181rcu_preempt20:55:362
2603996950,9cyclictest0-21swapper/119:20:321
2607996836,8cyclictest181rcu_preempt23:12:362
260799682,30cyclictest39-21ksoftirqd/219:20:192
2603996833,12cyclictest181rcu_preempt22:45:141
2603996833,12cyclictest181rcu_preempt22:45:141
2603996833,11cyclictest181rcu_preempt00:10:071
2603996817,26cyclictest0-21swapper/120:34:551
22882684,15sleep30-21swapper/318:57:353
2607996739,10cyclictest181rcu_preempt19:35:252
2607996739,10cyclictest181rcu_preempt19:35:252
2603996734,10cyclictest181rcu_preempt21:54:221
2603996734,10cyclictest181rcu_preempt21:54:221
260399672,24cyclictest19610-21/usr/sbin/munin19:50:341
2603996714,33cyclictest12106-21kworker/u18:2-rpciod22:20:481
2607996651,11cyclictest2590-21cyclictest23:15:332
2607996651,11cyclictest2590-21cyclictest23:15:332
2607996633,8cyclictest181rcu_preempt19:47:112
2603996531,12cyclictest181rcu_preempt20:00:241
2603996531,12cyclictest181rcu_preempt20:00:241
260399642,31cyclictest13607-21latency19:30:231
260399642,31cyclictest13607-21latency19:30:231
2612995744,9cyclictest451rcuc/323:25:093
2612995643,9cyclictest451rcuc/320:39:553
2612995538,10cyclictest451rcuc/321:29:513
2612995440,9cyclictest451rcuc/321:50:103
2612995440,9cyclictest451rcuc/321:50:103
261299543,41cyclictest20781-21taskset23:59:003
261299543,41cyclictest20781-21taskset23:59:003
2612995339,9cyclictest451rcuc/322:50:173
261299533,40cyclictest23378-21chrt23:09:563
261299532,41cyclictest23480-21taskset21:22:043
2612995240,8cyclictest451rcuc/320:08:033
2612995238,9cyclictest451rcuc/320:01:313
2612995238,9cyclictest451rcuc/320:01:313
261299523,39cyclictest508-21taskset22:30:333
261299523,35cyclictest26537-21chrt23:14:593
261299522,39cyclictest10010-21taskset22:46:503
261299522,39cyclictest10010-21taskset22:46:503
2612995138,8cyclictest451rcuc/320:43:073
2612995136,9cyclictest451rcuc/319:15:023
2612995134,12cyclictest451rcuc/320:25:453
2612995134,12cyclictest451rcuc/320:25:453
261299513,38cyclictest8475-21taskset20:55:213
261299513,38cyclictest8244-21chrt23:38:083
261299513,38cyclictest1603-21taskset00:20:283
261299513,38cyclictest15567-21chrt21:09:563
261299513,36cyclictest20404-21chrt22:10:233
261299512,38cyclictest21559-21chrt19:55:193
261299512,38cyclictest21559-21chrt19:55:193
261299512,36cyclictest27889-21chrt20:15:153
261299512,36cyclictest1527-21chrt21:39:593
175172515,11sleep30-21swapper/321:12:083
261299504,34cyclictest19160-21chrt21:15:193
2612995037,9cyclictest451rcuc/322:43:393
2612995036,8cyclictest451rcuc/323:45:003
2612995036,8cyclictest451rcuc/323:45:003
261299503,37cyclictest29010-21chrt00:11:483
261299503,37cyclictest17100-21chrt23:52:033
261299503,36cyclictest30133-21chrt23:20:223
261299503,36cyclictest19356-21chrt19:49:563
261299503,34cyclictest2289-21chrt21:40:283
261299503,34cyclictest2289-21chrt21:40:283
2612995029,13cyclictest451rcuc/322:26:533
261299494,35cyclictest31498-21taskset00:15:333
2612994937,8cyclictest451rcuc/322:20:093
2612994937,8cyclictest451rcuc/319:20:113
2612994936,9cyclictest451rcuc/319:42:033
2612994934,10cyclictest451rcuc/320:53:113
261299493,36cyclictest6526-21chrt21:47:463
261299493,36cyclictest3096-21chrt22:35:283
261299493,36cyclictest3096-21chrt22:35:283
261299493,35cyclictest13668-21chrt23:45:553
261299493,35cyclictest13668-21chrt23:45:553
261299493,33cyclictest27559-21chrt20:14:593
261299492,35cyclictest14807-21chrt19:35:143
261299492,35cyclictest14807-21chrt19:35:143
261299484,34cyclictest26678-21chrt23:15:073
261299484,34cyclictest26678-21chrt23:15:073
261299483,35cyclictest3053-21chrt19:00:193
261299483,35cyclictest20493-21taskset19:50:333
261299483,34cyclictest23186-21chrt00:02:203
261299483,33cyclictest16461-21chrt22:04:403
261299483,26cyclictest18423-21taskset22:06:463
261299483,26cyclictest18423-21taskset22:06:463
261299482,36cyclictest25683-21chrt00:05:573
261299482,36cyclictest19638-21chrt23:03:363
261299482,34cyclictest11319-21chrt21:55:283
261299474,31cyclictest11222-21taskset21:01:203
261299473,35cyclictest4090-21taskset23:30:313
261299473,35cyclictest4090-21taskset23:30:313
261299473,34cyclictest5125-21chrt00:25:423
2612994633,8cyclictest0-21swapper/319:25:293
2612994633,8cyclictest0-21swapper/319:25:293
261299463,32cyclictest1423-21chrt20:34:553
2612994533,8cyclictest451rcuc/320:47:403
261299452,34cyclictest28116-21chrt21:30:113
261299452,33cyclictest15784-21taskset22:56:103
261299444,30cyclictest13979-21taskset19:30:323
261299444,30cyclictest13979-21taskset19:30:323
2612994427,11cyclictest451rcuc/322:16:443
2612994329,10cyclictest451rcuc/320:20:593
261299403,27cyclictest6483-21chrt19:10:073
2612994028,8cyclictest451rcuc/319:05:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional