You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-16 - 04:09
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Sat May 16, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29318216814,23sleep00-21swapper/018:56:510
28747214310,94sleep228787-21sort18:56:322
2970321036,15sleep10-21swapper/118:58:311
30064999978,15cyclictest21415-21sh23:16:050
30064999874,14cyclictest0-21swapper/019:41:450
30069999772,19cyclictest5191-21diskmemload23:46:381
3006999969,67cyclictest7629-21apt-get19:31:231
30069999531,9cyclictest0-21swapper/123:51:201
3006999952,86cyclictest18470-21kworker/u18:3+ext4-rsv-conversion22:17:221
30069999469,19cyclictest12104-21sh21:11:361
30069999469,16cyclictest23186-21systemctl21:31:051
3006999943,40cyclictest2434-21grep19:11:431
3006999943,36cyclictest30145-21latency_hist22:36:081
30069999430,7cyclictest0-21swapper/120:56:381
30069999428,20cyclictest22310-21grep20:11:481
3006499944,69cyclictest9890-21apt-get23:51:230
3006999938,23cyclictest18413-21sendmail23:11:041
3006999934,36cyclictest8811-21fschecks_time21:06:281
3006999934,36cyclictest8811-21fschecks_time21:06:281
30069999333,8cyclictest0-21swapper/121:56:401
30076999255,15cyclictest181rcu_preempt23:21:162
30069999139,44cyclictest0-21swapper/100:01:061
3006999913,35cyclictest6571-21latency_hist22:51:081
3006999913,35cyclictest19697-21apt-config20:06:081
3006999913,35cyclictest19697-21apt-config20:06:081
30069999126,12cyclictest0-21swapper/119:30:401
3006999904,36cyclictest27674-21/usr/sbin/munin23:26:341
30069999037,46cyclictest1132-21sh00:30:161
3006999903,27cyclictest14997-21/usr/sbin/munin22:11:441
30069998934,47cyclictest0-21swapper/123:11:431
3006999893,41cyclictest32518-21cut19:06:291
3006999893,35cyclictest23382-21seq23:18:141
3006999893,34cyclictest15064-21awk21:16:321
3006999883,34cyclictest4852-21cut19:21:271
3006999883,30cyclictest31193-21idleruntime-cro20:41:051
3006999883,24cyclictest31114-21sort00:26:081
30069998832,45cyclictest0-21swapper/121:41:091
30069998827,7cyclictest0-21swapper/121:41:431
3006999878,34cyclictest5191-21diskmemload23:31:411
3006999873,40cyclictest19373-21apt-get20:06:191
3006999873,40cyclictest19373-21apt-get20:06:191
3006999873,34cyclictest17597-21cpuspeed21:21:211
3006999873,32cyclictest1467-21fschecks_count23:36:261
3006999873,32cyclictest1467-21fschecks_count23:36:261
3006999873,28cyclictest11552-21latency_hist19:41:101
30069998732,8cyclictest0-21swapper/123:01:381
3006999873,27cyclictest22327-21/usr/sbin/munin00:11:411
3006999873,27cyclictest22327-21/usr/sbin/munin00:11:411
3006999872,34cyclictest2998-21sort20:51:411
3006999872,34cyclictest2998-21sort20:51:411
3006999872,33cyclictest2997-21/usr/sbin/munin21:51:201
3006999872,33cyclictest2997-21/usr/sbin/munin21:51:201
3006999872,31cyclictest27563-21latency_hist23:26:071
3006999872,30cyclictest10465-21/usr/sbin/munin22:56:381
30076998664,13cyclictest26606-21taskset21:36:212
30076998658,19cyclictest7628-21latency_hist19:31:092
3006999864,37cyclictest14692-21latency_hist19:51:081
3006999863,36cyclictest5492-21/usr/sbin/munin21:01:321
3006999863,35cyclictest7613-21ssh22:51:401
3006999863,35cyclictest7613-21ssh22:51:401
3006999863,29cyclictest15225-21grep19:51:311
3006999863,29cyclictest15225-21grep19:51:311
3006999863,28cyclictest4460-21fschecks_count23:41:261
3006999863,28cyclictest31667-21fschecks_time20:41:281
3006999862,32cyclictest17778-21idleruntime-cro20:01:051
3006999854,32cyclictest25479-21ntp_kernel_pll_20:21:341
30069998535,42cyclictest0-21swapper/122:36:281
30069998535,42cyclictest0-21swapper/122:36:281
3006999853,34cyclictest4452-21latency_hist19:21:071
3006999853,31cyclictest9173-21cstates22:01:221
3006999853,31cyclictest9173-21cstates22:01:221
3006999853,29cyclictest23666-21idleruntime20:16:281
3006999853,29cyclictest23666-21idleruntime20:16:281
3006999853,28cyclictest19331-21/usr/sbin/munin00:06:331
3006999852,32cyclictest21277-21cpu22:21:211
3006999852,32cyclictest21277-21cpu22:21:211
3006999852,30cyclictest24190-21/usr/sbin/munin22:26:441
3006999852,29cyclictest27981-21modprobe00:21:041
30069998521,12cyclictest0-21swapper/120:46:371
3006999843,36cyclictest28794-21ps20:31:371
3006999843,29cyclictest23487-21/usr/sbin/munin21:31:301
3006999843,28cyclictest19181-21latency_hist00:06:071
3006999843,27cyclictest11737-21cpuspeed19:42:331
3006999842,34cyclictest26596-21cpu20:26:201
3006999842,32cyclictest31618-21latency_hist19:06:081
3006999842,32cyclictest31618-21latency_hist19:06:081
3006999842,32cyclictest13221-21ssh22:07:011
3007699834,55cyclictest39-21ksoftirqd/219:07:052
3006999834,28cyclictest32601-21cstates21:46:221
30064998366,11cyclictest14064-21unixbench_multi19:46:420
30064998363,13cyclictest26364-21latency_hist21:36:080
3007699825,39cyclictest39-21ksoftirqd/221:10:082
3007699825,39cyclictest39-21ksoftirqd/221:10:082
3007699822,32cyclictest39-21ksoftirqd/223:02:102
3006999823,31cyclictest1446-21processes22:41:371
30064998243,8cyclictest181rcu_preempt22:16:070
30076998149,22cyclictest19071-21fschecks_time23:11:282
30064998163,12cyclictest9943-40find23:51:060
30064998145,9cyclictest181rcu_preempt20:11:480
3006499813,46cyclictest8786-21fschecks_time21:06:280
3006499813,46cyclictest8786-21fschecks_time21:06:280
3008499803,30cyclictest18260-21chrt22:16:193
30076998058,7cyclictest181rcu_preempt23:10:202
30076998053,18cyclictest30055-21cyclictest22:11:202
30064998065,9cyclictest1935-21sendmail20:51:050
30076997956,17cyclictest2869-21sh21:51:082
30076997952,19cyclictest21268-21cpu22:21:212
30076997952,19cyclictest21268-21cpu22:21:212
30076997943,12cyclictest181rcu_preempt22:59:232
3007699793,30cyclictest39-21ksoftirqd/200:18:422
30064997951,11cyclictest181rcu_preempt22:46:190
3007699787,37cyclictest39-21ksoftirqd/222:19:242
30076997833,34cyclictest3841-21timerandwakeup19:16:402
3007699776,42cyclictest39-21ksoftirqd/223:26:242
30076997758,11cyclictest352-21runrttasks23:56:212
30076997755,15cyclictest19090-21latency_hist00:06:062
3007699772,30cyclictest39-21ksoftirqd/220:46:402
3007699772,30cyclictest39-21ksoftirqd/220:26:272
30064997752,18cyclictest28241-21ssh00:21:130
3006499774,63cyclictest87-21kswapd022:16:180
3008499762,39cyclictest8169-21chrt19:31:223
30076997655,12cyclictest0-21swapper/223:46:162
30076997652,11cyclictest181rcu_preempt19:02:312
30076997652,11cyclictest181rcu_preempt19:02:312
30064997622,32cyclictest7160-21kworker/u17:3+rpciod22:26:310
30064997614,32cyclictest20565-21cpu21:26:210
30076997555,15cyclictest30055-21cyclictest20:21:232
30076997552,8cyclictest181rcu_preempt23:41:222
30076997550,17cyclictest31935-21sh23:32:032
30076997548,8cyclictest181rcu_preempt22:50:162
30076997532,12cyclictest181rcu_preempt21:18:352
30076997530,8cyclictest181rcu_preempt22:10:442
30076997524,34cyclictest39-21ksoftirqd/220:04:502
30076997524,34cyclictest39-21ksoftirqd/220:04:502
3007699752,35cyclictest39-21ksoftirqd/219:46:382
30064997559,10cyclictest5552-21cpu21:01:220
30064997557,14cyclictest23577-21kworker/u18:1+events_unbound23:41:040
30064997557,14cyclictest23577-21kworker/u18:1+events_unbound23:41:040
30064997555,14cyclictest277-21dbus-daemon19:11:060
30064997548,19cyclictest30029-21fschecks_count20:36:250
3007699746,45cyclictest39-21ksoftirqd/220:40:462
30076997455,13cyclictest1-21systemd20:11:542
30076997450,17cyclictest4078-21latency_hist23:41:082
30076997450,17cyclictest4078-21latency_hist23:41:082
30076997448,11cyclictest181rcu_preempt21:11:292
30076997439,13cyclictest181rcu_preempt23:16:372
30076997435,12cyclictest181rcu_preempt19:21:242
3007699743,34cyclictest39-21ksoftirqd/220:46:082
30076997429,12cyclictest181rcu_preempt21:51:262
30076997429,12cyclictest181rcu_preempt21:51:262
30064997459,10cyclictest30055-21cyclictest23:43:350
3006499744,35cyclictest16608-21fschecks_count00:01:260
30076997358,11cyclictest30055-21cyclictest19:11:432
30076997340,11cyclictest181rcu_preempt19:43:292
30076997333,13cyclictest181rcu_preempt21:29:112
30076997331,13cyclictest181rcu_preempt21:03:122
3007699732,42cyclictest39-21ksoftirqd/222:38:012
3007699732,42cyclictest39-21ksoftirqd/222:38:012
30076997310,29cyclictest39-21ksoftirqd/222:29:072
30064997341,9cyclictest181rcu_preempt23:02:100
30064997334,8cyclictest181rcu_preempt23:21:340
30064997327,11cyclictest181rcu_preempt22:45:350
3007699728,35cyclictest39-21ksoftirqd/219:39:352
30076997255,11cyclictest22330-21acpi00:11:152
30076997255,11cyclictest22330-21acpi00:11:152
3007699725,36cyclictest39-21ksoftirqd/219:57:542
3007699722,35cyclictest39-21ksoftirqd/223:52:132
30076997213,34cyclictest39-21ksoftirqd/220:09:282
30076997213,34cyclictest39-21ksoftirqd/220:09:282
30064997257,9cyclictest28369-21ssh22:31:500
30064997253,14cyclictest28390-21fschecks_count20:31:260
30064997251,15cyclictest352-21runrttasks21:58:120
30064997249,8cyclictest181rcu_preempt19:31:180
30064997239,9cyclictest0-21swapper/022:02:590
30064997239,9cyclictest0-21swapper/022:02:590
30076997141,11cyclictest181rcu_preempt21:21:342
3007699712,26cyclictest39-21ksoftirqd/221:32:152
30064997155,10cyclictest15659-21timerwakeupswit19:51:410
30064997155,10cyclictest15659-21timerwakeupswit19:51:410
30064997154,10cyclictest0-21swapper/022:21:370
30064997154,10cyclictest0-21swapper/022:21:370
30064997151,13cyclictest16350-21ssh23:06:350
30064997148,15cyclictest15350irq/295-2188000.ethernet23:56:370
30076997051,10cyclictest0-21swapper/222:51:202
30076997051,10cyclictest0-21swapper/222:51:202
30076997045,11cyclictest181rcu_preempt22:00:292
3007699704,45cyclictest22284-21latency_hist00:11:082
3007699704,33cyclictest39-21ksoftirqd/219:56:082
3007699704,33cyclictest39-21ksoftirqd/219:56:082
30076997043,20cyclictest2684-21netstat20:51:332
30076997043,20cyclictest2684-21netstat20:51:332
30076997030,12cyclictest181rcu_preempt22:01:562
30076997030,12cyclictest181rcu_preempt22:01:562
30064997055,8cyclictest0-21swapper/000:16:440
30064997049,11cyclictest0-21swapper/019:31:050
30064997041,8cyclictest181rcu_preempt21:24:310
3007699692,31cyclictest39-21ksoftirqd/222:31:172
30064996955,10cyclictest30055-21cyclictest21:43:530
3006499694,33cyclictest22028-21fschecks_time23:16:270
30064996943,16cyclictest32613-21ssh21:46:230
30064996940,8cyclictest0-21swapper/023:30:490
3006499693,32cyclictest352-21runrttasks22:57:150
30076996834,12cyclictest181rcu_preempt00:26:412
30076996833,12cyclictest39-21ksoftirqd/220:31:422
30064996849,13cyclictest18291-21fschecks_time20:01:270
30064996849,13cyclictest18291-21fschecks_time20:01:270
30064996847,14cyclictest297-21systemd-logind21:56:060
30064996847,14cyclictest297-21systemd-logind21:56:060
30064996843,11cyclictest181rcu_preempt22:38:430
30064996843,11cyclictest181rcu_preempt22:38:430
3006499683,52cyclictest7004-21grep22:51:260
3006499683,52cyclictest7004-21grep22:51:260
3006499683,44cyclictest23026-21wc00:11:370
3006499683,44cyclictest23026-21wc00:11:370
3006499683,25cyclictest31696-21perl20:41:280
3007699673,42cyclictest28264-21runrttasks00:21:142
3007699673,36cyclictest39-21ksoftirqd/220:16:382
3007699673,36cyclictest39-21ksoftirqd/220:16:382
30076996733,10cyclictest181rcu_preempt22:42:362
30076996727,29cyclictest4533-21switchtime20:56:382
30064996753,8cyclictest0-21swapper/019:41:050
30064996742,17cyclictest13413-21ssh22:07:340
3006499673,54cyclictest352-21runrttasks20:51:160
3006499673,54cyclictest352-21runrttasks20:51:160
30064996735,13cyclictest0-21swapper/020:07:400
30064996735,13cyclictest0-21swapper/020:07:400
3006499673,26cyclictest1178-21cut23:36:080
30064996728,9cyclictest181rcu_preempt19:21:220
30064996722,19cyclictest15-21ksoftirqd/000:06:420
30064996647,13cyclictest3852-21cpu20:56:200
3006499663,47cyclictest2431-21chrt19:11:430
3006499663,33cyclictest14492-21/usr/sbin/munin21:16:350
30064996628,11cyclictest181rcu_preempt21:12:550
3007699653,45cyclictest39-21ksoftirqd/219:32:032
30076996514,22cyclictest39-21ksoftirqd/221:41:532
30064996550,9cyclictest16905-21ntp_kernel_err19:56:330
30064996544,15cyclictest30055-21cyclictest00:28:330
3006499654,24cyclictest31577-21idleruntime-cro19:06:050
3006499654,24cyclictest31577-21idleruntime-cro19:06:050
3006499653,46cyclictest352-21runrttasks20:26:580
3006499652,31cyclictest26937-21grep21:36:310
30064996438,11cyclictest181rcu_preempt20:25:490
30064996341,8cyclictest181rcu_preempt19:16:220
289342619,26sleep30-21swapper/318:56:373
3008499603,33cyclictest0-21swapper/320:06:233
3008499603,33cyclictest0-21swapper/320:06:233
30064996038,12cyclictest0-21swapper/020:16:400
30064996038,12cyclictest0-21swapper/020:16:400
30084995814,16cyclictest0-21swapper/323:51:253
3008499563,40cyclictest731-21chrt19:06:443
3008499563,39cyclictest22027-21chrt23:16:273
3008499563,37cyclictest23983-21chrt21:31:313
3008499554,40cyclictest0-21swapper/323:51:093
3008499553,34cyclictest17086-21chrt19:56:393
3008499552,38cyclictest12696-21chrt23:01:113
3008499552,36cyclictest19505-21chrt00:06:223
3008499543,39cyclictest26822-21chrt00:17:533
3008499543,38cyclictest14592-21chrt22:10:513
3008499543,37cyclictest17833-21chrt23:09:303
3008499543,37cyclictest10926-21chrt22:56:533
3008499543,36cyclictest17655-21chrt21:21:213
3008499543,35cyclictest18987-21chrt20:01:413
3008499543,35cyclictest18987-21chrt20:01:413
3008499543,35cyclictest18341-21chrt00:04:073
3008499542,35cyclictest917-21kworker/3:1+events22:11:463
3008499542,35cyclictest6933-21chrt19:26:363
3008499542,32cyclictest29085-21chrt21:40:553
3008499532,37cyclictest31878-21chrt00:26:373
3008499532,35cyclictest4691-21chrt19:21:223
3008499532,34cyclictest7403-21chrt21:03:543
3008499532,33cyclictest30740-21chrt21:42:093
3008499523,34cyclictest19381-21chrt23:11:373
3008499523,31cyclictest6771-21chrt23:45:463
3008499522,35cyclictest9432-21chrt21:06:423
3008499522,35cyclictest9432-21chrt21:06:423
3008499522,35cyclictest27958-21chrt22:31:393
3008499513,36cyclictest1997-21chrt22:42:053
3008499513,31cyclictest28572-21chrt23:26:413
3008499513,30cyclictest2334-21taskset23:36:423
3008499513,30cyclictest2334-21taskset23:36:423
3008499512,34cyclictest24053-21chrt22:26:083
3008499512,34cyclictest24053-21chrt22:26:083
3008499512,34cyclictest23308-21chrt00:11:413
3008499512,34cyclictest23308-21chrt00:11:413
3008499512,31cyclictest31463-21chrt20:41:223
3008499504,33cyclictest28607-21taskset20:31:303
3008499503,32cyclictest11614-21taskset19:41:183
3008499503,31cyclictest7754-21taskset21:58:273
3008499503,30cyclictest27081-21chrt20:26:323
3008499502,32cyclictest14656-21taskset19:51:083
3008499493,35cyclictest30737-21chrt23:31:193
3008499493,35cyclictest10058-21chrt19:36:283
3008499493,33cyclictest3052-21chrt19:16:153
3008499493,30cyclictest23634-21chrt20:16:273
3008499493,30cyclictest23634-21chrt20:16:273
3008499483,32cyclictest3040-21chrt21:51:173
3008499483,32cyclictest3040-21chrt21:51:173
3008499482,34cyclictest32634-21chrt21:46:233
3008499479,29cyclictest0-21swapper/320:11:243
3008499473,33cyclictest26075-21chrt22:28:523
3008499473,32cyclictest5031-21taskset22:47:123
3008499473,32cyclictest11827-21chrt21:11:283
3008499473,28cyclictest9603-21chrt22:56:083
3008499473,28cyclictest9603-21chrt22:56:083
3008499472,32cyclictest24764-21chrt23:21:213
3008499472,31cyclictest30216-21chrt00:24:053
3008499472,28cyclictest30599-21taskset22:36:303
3008499472,28cyclictest30599-21taskset22:36:303
3008499462,33cyclictest22286-21chrt21:28:393
3008499462,31cyclictest30359-21chrt20:36:323
3008499453,29cyclictest2757-21taskset20:51:353
3008499453,29cyclictest2757-21taskset20:51:353
3008499453,29cyclictest17219-21chrt21:21:053
3008499443,28cyclictest16119-21taskset00:01:073
3008499443,26cyclictest25489-21chrt20:21:343
3008499442,31cyclictest9041-21taskset22:01:193
3008499442,31cyclictest9041-21taskset22:01:193
3008499442,27cyclictest2868-21taskset19:16:073
3008499434,29cyclictest1990-21chrt20:51:063
3008499402,25cyclictest30240-21taskset19:01:173
3008499402,25cyclictest30240-21taskset19:01:173
3008499382,27cyclictest14810-21taskset19:51:163
3008499382,27cyclictest14810-21taskset19:51:163
3008499382,26cyclictest5236-21taskset21:01:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional