You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-21 - 07:32
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Sun Dec 21, 2025 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
261159911568,39cyclictest12717-21kworker/u17:2-rpciod21:55:011
261159911568,39cyclictest12717-21kworker/u17:2-rpciod21:55:011
26108991093,27cyclictest26113-21latency_hist19:10:040
26108991064,46cyclictest25135-21sed22:40:270
26108991064,29cyclictest27320-21fschecks_count00:30:210
261089910575,21cyclictest16129-21sh22:25:470
261089910525,11cyclictest0-21swapper/020:00:030
26108991043,39cyclictest8594-21grep21:20:370
26108991043,29cyclictest31977-21latency_hist21:05:020
26108991034,33cyclictest7876-21idleruntime23:05:240
26108991033,34cyclictest28125-21ls20:50:350
261089910318,13cyclictest0-21swapper/021:15:030
261089910316,31cyclictest29283-21grep19:15:380
26108991023,36cyclictest32696-21sh22:52:550
26108991023,35cyclictest16725-21/usr/sbin/munin21:35:300
26108991022,35cyclictest31984-21/usr/sbin/munin23:45:180
26108991022,35cyclictest31984-21/usr/sbin/munin23:45:180
261089910218,37cyclictest30177-21awk00:35:140
261089910149,46cyclictest24195-21sh22:39:560
26108991013,34cyclictest31743-21grep19:25:240
26108991013,33cyclictest29368-21ntp_kernel_err20:55:320
26108991013,27cyclictest23474-21sh23:30:340
261089910116,25cyclictest4257-21cat23:00:020
261089910044,47cyclictest2990-21kworker/u17:1+xprtiod21:30:030
26108991003,36cyclictest23230-21fschecks_time21:45:220
26108991003,36cyclictest14645-21ntpq20:10:320
26108991003,33cyclictest10554-21taskset23:10:100
26108991003,32cyclictest13967-21grep20:10:020
26108991003,29cyclictest7830-21grep19:51:080
26108991003,29cyclictest7830-21grep19:51:080
261089910026,8cyclictest0-21swapper/000:05:290
261089910026,8cyclictest0-21swapper/000:05:290
26108991002,37cyclictest5156-21sendmail_mailqu21:15:350
26108991002,37cyclictest5156-21sendmail_mailqu21:15:350
26108991002,35cyclictest14455-21xargs23:15:340
26108991002,27cyclictest28832-21apt-get23:40:100
2521421009,65sleep125236-21/usr/sbin/munin19:06:101
26108999934,58cyclictest2493-21sh22:04:590
2610899993,34cyclictest15725-21taskset20:15:150
26108999928,63cyclictest0-21swapper/021:59:290
2610899992,41cyclictest2527-21fschecks_count19:35:230
2610899992,41cyclictest2527-21fschecks_count19:35:230
2610899984,39cyclictest24031-21apt-get00:25:130
2610899984,31cyclictest21306-21latency_hist22:35:020
2610899983,34cyclictest23763-21latency_hist00:25:020
26115999756,34cyclictest198-21systemd-journal23:28:321
26115999756,34cyclictest198-21systemd-journal23:28:321
2610899973,36cyclictest22352-21grep20:35:020
2610899973,36cyclictest22352-21grep20:35:020
2610899973,35cyclictest4685-21cat19:40:340
2610899973,35cyclictest4685-21cat19:40:340
2610899973,35cyclictest12281-21latency_hist20:05:020
2610899973,35cyclictest12281-21latency_hist20:05:020
2610899973,34cyclictest20006-21/usr/sbin/munin23:25:220
2610899973,34cyclictest20006-21/usr/sbin/munin23:25:220
2610899973,32cyclictest20746-21latency_hist20:30:020
2610899973,32cyclictest20746-21latency_hist20:30:020
2610899973,27cyclictest28669-21grep22:45:360
26108999724,12cyclictest0-21swapper/023:20:380
2610899972,31cyclictest13933-21fschecks_count21:30:220
26108999721,15cyclictest0-21swapper/023:35:200
26108999718,22cyclictest4751-21fschecks_time23:00:240
2611599962,64cyclictest7620-21df_inode19:51:001
2611599962,64cyclictest7620-21df_inode19:51:001
2610899964,28cyclictest20649-21latency_hist00:20:020
2610899963,35cyclictest9417-21sh22:15:250
2610899963,35cyclictest24919-21timerwakeupswit20:40:350
2610899963,33cyclictest350-21runrttasks23:55:290
2610899963,32cyclictest7041-21latency_hist19:50:010
2610899963,32cyclictest12352-21fschecks_time22:20:210
2610899963,31cyclictest20846-21timerandwakeup21:40:510
26115999547,40cyclictest11089-21latency21:25:281
2610899953,35cyclictest22590-21fschecks_count20:35:220
2610899953,34cyclictest25926-21fschecks_time20:45:220
2610899953,22cyclictest6660-21mailstats22:10:340
2610899953,22cyclictest6660-21mailstats22:10:340
2610899952,35cyclictest2817-21acpi22:05:130
26115999453,34cyclictest26100-21cyclictest00:10:031
26115999453,34cyclictest26100-21cyclictest00:10:031
2611599942,56cyclictest350-21runrttasks20:05:201
2610899944,33cyclictest703-21/usr/sbin/munin19:30:250
2610899943,36cyclictest26713-21latency19:10:270
2610899943,34cyclictest17622-21apt-get20:20:270
2610899943,25cyclictest25981-21/usr/sbin/munin21:50:330
2610899943,25cyclictest25981-21/usr/sbin/munin21:50:330
2610899943,25cyclictest11283-21ssh00:04:590
26108999425,19cyclictest31303-21ls19:25:000
2610899933,30cyclictest2521-21sort23:50:140
2610899933,30cyclictest2521-21sort23:50:140
2610899933,30cyclictest15361-21find00:10:320
26117999276,10cyclictest0-21swapper/200:25:152
26117999267,8cyclictest181rcu_preempt23:45:072
26117999267,8cyclictest181rcu_preempt23:45:072
2610899913,28cyclictest569-21grep21:05:350
2610899913,28cyclictest569-21grep21:05:350
26117999062,21cyclictest9894-21sh22:15:352
26117998940,12cyclictest181rcu_preempt22:38:562
2611599893,25cyclictest27063-21ssh00:30:101
26117998838,29cyclictest0-21swapper/222:43:032
26115998839,42cyclictest13884-21fschecks_time23:15:211
26117998752,14cyclictest181rcu_preempt23:04:592
26115998765,15cyclictest31466-21kworker/u17:0-xprtiod22:00:021
26115998760,20cyclictest2448-21sh23:50:101
26115998760,20cyclictest2448-21sh23:50:101
26115998746,35cyclictest1725-21if_err_eth021:10:241
26115998737,43cyclictest1012-21latency_hist21:10:011
26115998737,43cyclictest1012-21latency_hist21:10:011
26117998655,22cyclictest16963-21sh22:26:572
26117998650,12cyclictest181rcu_preempt22:12:232
26117998650,12cyclictest181rcu_preempt22:12:232
26117998644,14cyclictest181rcu_preempt20:25:022
2611599863,27cyclictest29367-21ntp_kernel_err20:55:321
26117998570,8cyclictest181rcu_preempt19:24:052
26117998551,14cyclictest181rcu_preempt00:19:032
26117998548,13cyclictest181rcu_preempt21:40:272
26117998533,9cyclictest0-21swapper/221:50:362
26117998533,9cyclictest0-21swapper/221:50:362
26115998564,13cyclictest24417-21sh22:40:021
26115998534,44cyclictest27922-21fschecks_count22:45:221
2611599853,37cyclictest8542-21/usr/sbin/munin00:00:271
2611599853,31cyclictest30294-21interrupts19:20:231
2611599853,30cyclictest3611-21grep22:05:371
2611599852,38cyclictest8223-40ogrotate23:59:591
26117998449,11cyclictest181rcu_preempt23:37:222
26117998445,8cyclictest181rcu_preempt19:51:052
26117998445,8cyclictest181rcu_preempt19:51:052
26117998442,12cyclictest181rcu_preempt00:20:222
26117998441,13cyclictest181rcu_preempt22:00:242
26117998439,12cyclictest181rcu_preempt21:17:002
26117998439,12cyclictest181rcu_preempt21:17:002
26117998431,9cyclictest181rcu_preempt21:15:022
26117998412,40cyclictest39-21ksoftirqd/223:29:482
26117998412,40cyclictest39-21ksoftirqd/223:29:482
2611599844,37cyclictest371-21sshd22:10:311
2611599844,37cyclictest371-21sshd22:10:311
26115998434,41cyclictest0-21swapper/119:15:381
2611599843,39cyclictest6906-21latency_hist19:50:021
2611599843,28cyclictest350-21runrttasks20:43:221
2611599842,32cyclictest9669-21grep19:55:341
2611599842,25cyclictest28215-21unixbench_singl20:50:351
26117998361,16cyclictest26100-21cyclictest23:57:032
2611799833,70cyclictest12501-21apt-get20:05:142
26115998360,16cyclictest26100-21cyclictest21:40:231
26115998340,37cyclictest17188-21sh22:27:331
26115998340,36cyclictest5097-21sh23:00:331
26115998335,40cyclictest0-21swapper/122:59:591
2611599833,37cyclictest4700-21sed19:40:341
2611599833,37cyclictest4700-21sed19:40:341
2611599833,29cyclictest20731-21acpi00:20:091
2611599833,26cyclictest350-21runrttasks21:04:331
2611599832,29cyclictest21468-21mailstats20:30:341
2611599832,29cyclictest21468-21mailstats20:30:341
26117998254,11cyclictest181rcu_preempt21:48:482
26115998257,17cyclictest198-21systemd-journal00:10:321
2611599824,38cyclictest25635-21/usr/sbin/munin20:45:331
26115998239,37cyclictest13352-21sh23:15:011
26115998235,39cyclictest0-21swapper/120:05:031
26115998235,39cyclictest0-21swapper/120:05:031
26115998234,41cyclictest10693-21sh22:17:181
2611599823,27cyclictest23906-21latency_hist20:40:021
2611599823,26cyclictest32130-21sort23:45:241
2611599823,26cyclictest32130-21sort23:45:241
2611599822,27cyclictest27550-21grep22:45:021
2611799815,36cyclictest39-21ksoftirqd/222:53:182
26117998149,9cyclictest181rcu_preempt20:00:252
26117998149,9cyclictest181rcu_preempt20:00:252
26117998140,14cyclictest181rcu_preempt22:48:232
26115998162,14cyclictest17544-21timerandwakeup23:20:371
2611599814,38cyclictest23744-21unixbench_multi23:30:361
2611599814,33cyclictest17576-21/usr/sbin/munin20:20:421
2611599814,26cyclictest350-21runrttasks20:29:171
2611599814,26cyclictest350-21runrttasks20:29:171
26115998137,37cyclictest14600-21sh22:24:291
2611599812,24cyclictest21156-21cron22:34:591
26115998116,29cyclictest26302-21memory23:35:291
2463528111,44sleep224621-21apt-get19:05:212
26117998050,11cyclictest181rcu_preempt19:35:352
26117998050,11cyclictest181rcu_preempt19:35:352
26117998042,17cyclictest181rcu_preempt22:55:192
2611599808,27cyclictest14750-21mailstats20:10:351
2611599803,38cyclictest23529-21chrt21:45:291
26115998033,40cyclictest32761-21kworker/u17:1-nfsiod23:09:091
2611599803,32cyclictest7411-21ssh21:20:031
2611599803,32cyclictest7411-21ssh21:20:031
2611599802,31cyclictest16477-21latency_hist21:35:011
2611599802,28cyclictest2428-21cstates19:35:181
2611599802,28cyclictest2428-21cstates19:35:181
2611599802,27cyclictest1009-21diskmemload21:35:231
2553028016,41sleep30-21swapper/319:06:163
2611799798,33cyclictest39-21ksoftirqd/223:30:192
26117997953,20cyclictest11890-21fschecks_time00:05:232
26117997953,20cyclictest11890-21fschecks_time00:05:232
26117997946,9cyclictest181rcu_preempt00:10:242
2611799792,51cyclictest39-21ksoftirqd/223:42:472
26115997955,17cyclictest26100-21cyclictest22:50:381
2611599794,30cyclictest28812-21/usr/sbin/munin23:40:281
2611599793,29cyclictest465-21idleruntime-cro19:30:001
26115997928,46cyclictest0-21swapper/119:30:181
2611599792,26cyclictest15746-21cpu20:15:181
2611599792,22cyclictest26752-21/usr/sbin/munin19:10:281
26117997857,14cyclictest278-21systemd-logind19:20:012
26117997855,17cyclictest8363-21sh21:20:352
26117997849,8cyclictest181rcu_preempt19:10:152
26117997840,10cyclictest181rcu_preempt21:58:012
2611599783,33cyclictest30394-21fschecks_time00:35:201
2611599782,27cyclictest20413-21latency_hist00:20:011
26117997758,14cyclictest26100-21cyclictest23:08:282
26117997752,9cyclictest181rcu_preempt21:30:392
26117997751,9cyclictest181rcu_preempt23:21:372
26117997748,13cyclictest181rcu_preempt20:38:192
26117997742,11cyclictest181rcu_preempt21:38:222
26117997741,14cyclictest181rcu_preempt20:30:232
26117997741,14cyclictest181rcu_preempt20:30:232
26117997740,8cyclictest181rcu_preempt00:30:312
26117997739,19cyclictest0-21swapper/220:55:022
26117997735,13cyclictest181rcu_preempt21:00:282
2611599773,27cyclictest2488-21latency_hist22:05:021
26117997650,11cyclictest181rcu_preempt21:30:002
26117997646,9cyclictest181rcu_preempt19:25:202
26117997635,11cyclictest181rcu_preempt19:30:182
26117997631,16cyclictest181rcu_preempt20:15:382
26117997610,29cyclictest39-21ksoftirqd/223:51:062
26117997610,29cyclictest39-21ksoftirqd/223:51:062
2611599763,26cyclictest23472-21apt-get00:25:131
26117997562,9cyclictest58-21kcompactd020:40:512
26117997547,9cyclictest181rcu_preempt20:28:562
26117997547,9cyclictest181rcu_preempt20:28:562
26117997536,16cyclictest181rcu_preempt20:55:272
26117997449,9cyclictest181rcu_preempt22:20:382
26117997441,11cyclictest181rcu_preempt19:45:282
26117997434,9cyclictest181rcu_preempt20:45:292
2611599743,24cyclictest7886-21/usr/sbin/munin21:20:231
26117997352,14cyclictest13876-21fschecks_time23:15:212
26117997349,9cyclictest181rcu_preempt21:05:222
26117997349,9cyclictest181rcu_preempt21:05:222
26117997348,10cyclictest181rcu_preempt00:36:342
26117997343,8cyclictest181rcu_preempt19:55:342
26117997340,13cyclictest181rcu_preempt22:05:082
26117997246,10cyclictest181rcu_preempt20:10:232
26117997238,12cyclictest181rcu_preempt22:32:192
80612710,7sleep30-21swapper/323:05:283
26117997149,9cyclictest181rcu_preempt23:12:012
26124997010,48cyclictest24362-21chrt00:26:063
26117997041,8cyclictest181rcu_preempt00:02:102
26117996932,11cyclictest181rcu_preempt19:40:252
26117996932,11cyclictest181rcu_preempt19:40:252
26124996830,24cyclictest0-21swapper/322:25:333
26124996824,38cyclictest0-21swapper/319:51:003
26124996824,38cyclictest0-21swapper/319:51:003
26124996815,33cyclictest0-21swapper/320:05:483
26124996048,8cyclictest451rcuc/321:40:053
2612499604,37cyclictest8394-21chrt00:00:053
26124995945,9cyclictest451rcuc/322:57:313
26124995744,9cyclictest451rcuc/322:37:133
26124995741,9cyclictest451rcuc/319:42:583
26124995741,9cyclictest451rcuc/319:42:583
2612499572,37cyclictest9968-21chrt19:55:383
26124995643,9cyclictest451rcuc/323:40:413
2612499564,39cyclictest31462-21taskset00:36:193
26124995642,9cyclictest451rcuc/322:02:053
2612499563,44cyclictest23136-21taskset00:24:073
26124995541,9cyclictest451rcuc/321:35:513
26124995536,13cyclictest451rcuc/322:53:503
26124995536,12cyclictest451rcuc/320:50:423
2612499552,43cyclictest6803-21taskset23:03:583
2612499544,35cyclictest26400-21chrt19:10:183
26124995442,8cyclictest451rcuc/321:04:473
26124995441,9cyclictest451rcuc/322:20:033
26124995440,9cyclictest451rcuc/322:23:153
26124995440,9cyclictest451rcuc/320:37:343
26124995440,9cyclictest451rcuc/319:31:503
26124995439,9cyclictest451rcuc/320:02:153
26124995439,9cyclictest451rcuc/320:02:153
2612499542,40cyclictest31352-21chrt21:58:433
26124995341,8cyclictest451rcuc/321:47:383
26124995341,8cyclictest451rcuc/321:22:223
26124995340,9cyclictest451rcuc/300:33:363
26124995339,9cyclictest451rcuc/321:06:383
26124995339,9cyclictest451rcuc/321:06:383
2612499533,41cyclictest17831-21taskset00:15:163
2612499533,36cyclictest14310-21taskset20:10:243
26124995325,19cyclictest22397-21chrt23:29:593
26124995325,19cyclictest22397-21chrt23:29:593
2612499523,38cyclictest3288-21chrt19:35:373
2612499523,38cyclictest3288-21chrt19:35:373
2612499523,37cyclictest28777-21chrt21:54:593
2612499523,37cyclictest28777-21chrt21:54:593
2612499523,34cyclictest24788-21chrt22:40:203
2612499514,35cyclictest14953-21chrt00:10:223
2612499514,32cyclictest24231-21chrt20:40:213
26124995136,9cyclictest451rcuc/300:03:253
2612499513,35cyclictest32083-21chrt23:45:223
2612499513,35cyclictest32083-21chrt23:45:223
2612499513,33cyclictest29684-21chrt20:55:393
2612499513,33cyclictest24899-21chrt23:33:313
2612499513,31cyclictest11247-21chrt21:25:333
2612499512,39cyclictest16633-21chrt20:15:383
2612499503,34cyclictest29818-21chrt19:20:023
2612499503,33cyclictest31807-21chrt19:25:253
2612499502,39cyclictest7785-21taskset22:12:453
2612499502,39cyclictest7785-21taskset22:12:453
190102502,6sleep30-21swapper/320:25:033
2612499494,35cyclictest14013-21chrt23:15:253
26124994935,8cyclictest451rcuc/319:46:233
26124994935,10cyclictest451rcuc/320:29:363
26124994935,10cyclictest451rcuc/320:29:363
2612499493,34cyclictest3635-21taskset23:50:493
2612499493,34cyclictest3635-21taskset23:50:493
2612499492,34cyclictest25640-21taskset20:45:103
2612499492,34cyclictest16921-21taskset23:20:233
2612499492,33cyclictest20463-21taskset22:33:073
2612499492,32cyclictest25972-21chrt23:35:223
26124994834,9cyclictest451rcuc/323:12:383
2612499483,36cyclictest2948-21chrt22:05:203
2612499483,36cyclictest13107-21taskset00:06:413
2612499483,36cyclictest13107-21taskset00:06:413
2612499483,33cyclictest5244-21chrt21:15:363
2612499483,33cyclictest5244-21chrt21:15:363
2612499482,32cyclictest1781-21chrt21:10:253
26124994735,8cyclictest451rcuc/320:34:183
26124994735,8cyclictest451rcuc/320:34:183
26124994734,8cyclictest451rcuc/321:31:223
26124994631,10cyclictest451rcuc/319:20:243
26124994532,9cyclictest451rcuc/322:46:333
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional