You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 07:40
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sun Nov 23, 2025 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
315769911840,14cyclictest0-21swapper/022:09:560
315769911840,14cyclictest0-21swapper/022:09:560
315769911648,62cyclictest16747-21kworker/u17:0+xprtiod22:24:320
315769911140,10cyclictest0-21swapper/023:59:580
315769911140,10cyclictest0-21swapper/023:59:580
31576991104,46cyclictest28884-21unixbench_singl21:44:580
315769910936,11cyclictest0-21swapper/020:04:430
31576991063,35cyclictest25542-21latency_hist00:24:210
31576991062,31cyclictest2598-21/usr/sbin/munin22:49:490
31576991062,31cyclictest2598-21/usr/sbin/munin22:49:490
31576991043,39cyclictest355-21runrttasks19:44:450
315769910218,30cyclictest355-21runrttasks23:57:370
3157699993,29cyclictest957-21mailstats20:49:530
3143229515,54sleep031446-21modprobe19:09:190
31598999327,8cyclictest181rcu_preempt19:48:462
31598999271,14cyclictest32653-21latency_hist20:49:222
3011829213,41sleep10-21swapper/119:04:351
31588999138,47cyclictest0-21swapper/123:14:221
31588999138,47cyclictest0-21swapper/123:14:221
31588999044,38cyclictest0-21swapper/122:19:231
31598998973,10cyclictest14634-21latency_hist19:54:232
3158899892,43cyclictest19540-21apt-get22:24:321
3046728913,46sleep30-21swapper/319:04:543
3158899883,30cyclictest6975-21taskset19:29:441
3158899883,30cyclictest29705-21tail00:29:591
3158899882,34cyclictest11271-21/usr/sbin/munin23:04:411
3158899882,29cyclictest10453-21ssh21:14:521
31588998766,14cyclictest3240-21runrttasks19:19:341
31588998766,14cyclictest3240-21runrttasks19:19:341
3158899873,37cyclictest19814-21modprobe23:19:191
3158899873,28cyclictest28136-21cron22:39:191
3158899873,28cyclictest2012-21/usr/sbin/munin22:49:581
3158899873,28cyclictest2012-21/usr/sbin/munin22:49:581
31598998562,17cyclictest5407-21fschecks_time21:04:422
31588998534,45cyclictest18274-21sh00:10:081
31588998534,45cyclictest18274-21sh00:10:081
31588998534,43cyclictest0-21swapper/120:14:441
3158899853,29cyclictest2267-21grep19:14:581
3158899853,28cyclictest6467-21ssh22:56:031
3158899852,28cyclictest4165-21cat21:59:211
31576998560,19cyclictest32613-21latency_hist20:49:220
3159899842,36cyclictest39-21ksoftirqd/220:34:352
3158899843,30cyclictest4327-21/usr/sbin/munin21:59:401
3158899843,29cyclictest17750-21sh21:27:161
3158899843,28cyclictest30028-21latency20:40:371
3158899843,28cyclictest18514-21ntpq20:04:491
3158899842,28cyclictest860-21latency_hist19:14:231
31588998420,26cyclictest12962-21sh00:01:241
31588998420,26cyclictest12962-21sh00:01:241
31588998414,29cyclictest22148-21basename21:34:461
31588998414,29cyclictest22148-21basename21:34:461
31576998463,13cyclictest21779-21/usr/sbin/munin21:34:510
31576998463,13cyclictest21779-21/usr/sbin/munin21:34:510
31598998363,14cyclictest32631-21vmstat00:35:002
31598998362,15cyclictest316-21rs:main2
31598998362,15cyclictest316-21rs:main2
3158899834,26cyclictest19008-21fschecks_count21:29:401
3158899833,33cyclictest355-21runrttasks00:38:211
3158899833,30cyclictest27621-21latency_hist21:44:211
31588998332,44cyclictest0-21swapper/122:39:361
31588998329,45cyclictest0-21swapper/123:43:221
3158899832,32cyclictest23336-21awk20:19:491
3158899832,28cyclictest26414-21kernelversion20:29:451
3158899832,27cyclictest10671-21modprobe23:04:181
31576998346,29cyclictest0-21swapper/023:09:570
31576998346,29cyclictest0-21swapper/023:09:570
31588998238,38cyclictest27864-21cpu21:44:371
3158899823,29cyclictest9464-21head23:54:591
3158899823,29cyclictest20414-21fschecks_time23:19:421
3158899823,29cyclictest13363-21cut19:49:421
3158899823,28cyclictest8151-21switchtime22:04:531
3158899823,28cyclictest28221-21ssh23:33:051
31588998227,45cyclictest0-21swapper/100:19:211
31588998223,49cyclictest0-21swapper/121:54:221
3158899822,30cyclictest23434-21/usr/sbin/munin00:19:491
3158899822,30cyclictest10105-21cut19:39:431
31598998164,10cyclictest0-21swapper/223:29:182
31598998158,17cyclictest20707-21ntp_offset23:19:492
3158899813,32cyclictest32563-21unixbench_singl22:44:571
3158899813,32cyclictest14910-21/usr/sbin/munin00:04:511
3158899813,29cyclictest11942-21latency19:44:451
3158899813,28cyclictest2144-21cut20:54:431
3158899813,28cyclictest2144-21cut20:54:431
3158899813,27cyclictest8041-21modprobe23:54:181
3158899813,26cyclictest29759-21mailstats23:34:521
31588998129,45cyclictest0-21swapper/119:29:231
31588998128,47cyclictest0-21swapper/121:14:221
31588998118,9cyclictest0-21swapper/122:23:501
31598998066,10cyclictest31566-21cyclictest23:59:542
31598998066,10cyclictest31566-21cyclictest23:59:542
31598998064,10cyclictest20940-21sh20:14:192
31598998064,10cyclictest20940-21sh20:14:192
31598998045,14cyclictest12412-21ssh22:12:432
31598998045,14cyclictest12412-21ssh22:12:432
3159899802,35cyclictest39-21ksoftirqd/223:29:302
3158899803,31cyclictest6118-21grep21:04:571
3158899803,30cyclictest26010-21grep00:24:411
3158899803,30cyclictest25081-21sendmail_mailst20:24:551
3158899803,29cyclictest23106-21cpu23:24:371
3158899803,28cyclictest14839-21ssh21:22:281
3158899803,28cyclictest14839-21ssh21:22:281
3158899803,28cyclictest12990-21apt-get22:14:191
3158899803,28cyclictest12990-21apt-get22:14:191
3158899803,27cyclictest22534-21ssh22:29:341
3158899803,26cyclictest17733-21munin-run20:04:191
3158899803,19cyclictest5084-21modprobe23:49:191
3158899802,28cyclictest27933-21fschecks_time20:34:431
31598997946,13cyclictest181rcu_preempt23:49:402
3158899794,28cyclictest517-21fschecks_count20:49:421
3158899794,26cyclictest20497-21unixbench_multi20:09:561
3158899794,26cyclictest20497-21unixbench_multi20:09:561
3158899793,29cyclictest14991-21fschecks_time19:54:441
3158899793,29cyclictest14991-21fschecks_time19:54:441
31588997929,44cyclictest9156-21timerwakeupswit19:35:031
3158899792,32cyclictest31278-21expr20:44:371
31576997958,13cyclictest29385-21timerandwakeup22:39:550
31598997842,13cyclictest181rcu_preempt22:39:372
31598997840,16cyclictest181rcu_preempt00:04:412
3158899784,28cyclictest3842-21ls20:59:441
31576997854,17cyclictest31412-21latency_hist00:34:220
3157699784,41cyclictest17161-21apt-get23:14:400
31598997757,14cyclictest4015-21idleruntime-cro21:59:192
31598997740,29cyclictest8946-21wc22:59:552
31598997660,10cyclictest6511-21diskmemload22:35:492
31598997633,34cyclictest2274-21grep22:49:432
31598997633,34cyclictest2274-21grep22:49:432
31598997557,12cyclictest5240-21fschecks_count22:54:412
31598997542,9cyclictest181rcu_preempt00:24:202
31598997540,10cyclictest181rcu_preempt23:15:152
3157699753,35cyclictest6810-21/usr/sbin/munin21:09:400
31598997454,14cyclictest314-21in:imuxsock23:44:192
31598997454,12cyclictest0-21swapper/200:09:582
31598997454,12cyclictest0-21swapper/200:09:582
31598997451,9cyclictest181rcu_preempt23:11:162
31598997451,9cyclictest181rcu_preempt23:11:162
31598997445,19cyclictest22547-21modprobe20:19:192
31598997438,17cyclictest181rcu_preempt21:10:322
31598997437,26cyclictest16261-21systemctl22:19:202
31598997435,9cyclictest181rcu_preempt00:16:062
31598997425,16cyclictest31566-21cyclictest22:04:542
31576997455,13cyclictest7805-21latency_hist22:59:210
31576997443,7cyclictest181rcu_preempt21:34:210
3157699743,53cyclictest13545-21apt-get22:15:220
31598997360,9cyclictest3117-21kworker/u18:2-ext4-rsv-conversion19:24:442
31598997353,7cyclictest181rcu_preempt22:46:172
3070527311,38sleep230752-21awk19:05:052
31598997250,9cyclictest181rcu_preempt21:54:012
31598997248,16cyclictest31566-21cyclictest23:54:392
31598997242,9cyclictest181rcu_preempt21:25:532
3159899723,34cyclictest22413-21latency_hist22:29:232
31598997233,12cyclictest181rcu_preempt19:29:562
31598997232,12cyclictest181rcu_preempt21:04:222
3159899722,33cyclictest39-21ksoftirqd/219:14:332
31576997255,11cyclictest16962-21ssh22:19:460
31576997236,10cyclictest161ktimers/022:59:470
3159899718,53cyclictest3232-21apt-get19:19:312
3159899718,53cyclictest3232-21apt-get19:19:312
31598997156,10cyclictest58-21kcompactd022:33:442
31598997150,8cyclictest181rcu_preempt20:52:422
31598997150,8cyclictest181rcu_preempt20:43:512
31598997148,8cyclictest181rcu_preempt21:34:212
31576997153,13cyclictest15602-21latency_hist21:24:220
31576997153,13cyclictest15602-21latency_hist21:24:220
31576997142,21cyclictest13660-21systemd-run23:09:200
3157699713,55cyclictest20684-21ntpq23:19:480
31576997121,38cyclictest5012-21grep21:59:520
31598997052,13cyclictest20537-21kworker/u17:2-rpciod21:39:032
31598997052,13cyclictest20537-21kworker/u17:2-rpciod21:39:032
31598997049,15cyclictest31566-21cyclictest22:23:582
31598997048,8cyclictest181rcu_preempt21:17:222
3159899704,43cyclictest355-21runrttasks19:41:432
3159899703,42cyclictest25393-21head20:24:592
31576997057,8cyclictest23919-21rm22:30:490
31576997047,17cyclictest444-21ssh00:35:490
31576997023,15cyclictest181rcu_preempt20:59:430
3159899693,46cyclictest18219-21fschecks_time20:04:432
3159899692,44cyclictest11580-21ntp_states23:04:492
31598996912,31cyclictest5211-21timerandwakeup21:59:552
31576996954,11cyclictest31566-21cyclictest23:43:370
31576996941,22cyclictest3468-21fschecks_count19:20:020
31576996941,22cyclictest3468-21fschecks_count19:20:020
3157699693,30cyclictest26133-21ssh23:29:360
3157699693,30cyclictest20179-21ps20:09:520
3157699693,30cyclictest20179-21ps20:09:520
31598996842,11cyclictest181rcu_preempt19:59:492
3157699683,35cyclictest355-21runrttasks22:39:150
3157699683,21cyclictest16143-21grep21:24:440
31598996752,10cyclictest13787-21timerwakeupswit21:19:572
31598996752,10cyclictest13787-21timerwakeupswit21:19:572
31598996743,8cyclictest181rcu_preempt21:48:192
3159899673,41cyclictest23818-21grep20:19:582
31598996727,13cyclictest181rcu_preempt00:29:572
31598996722,9cyclictest181rcu_preempt00:27:232
3159899672,27cyclictest32024-21fschecks_time19:09:442
31576996750,9cyclictest0-21swapper/023:39:220
3157699674,28cyclictest20581-21ntp_kernel_pll_00:14:480
3157699673,46cyclictest8484-21fschecks_count19:34:490
31598996645,13cyclictest25184-21ssh21:39:442
31598996642,9cyclictest181rcu_preempt23:34:512
3159899662,44cyclictest3069-21ntp_offset23:44:492
31576996651,11cyclictest31566-21cyclictest20:54:490
31576996651,11cyclictest31566-21cyclictest20:54:490
31576996650,11cyclictest2706-21latency_hist19:19:240
31576996646,15cyclictest316-21rs:main0
31576996639,20cyclictest0-21swapper/000:04:520
31576996634,17cyclictest3117-21kworker/u18:2-writeback20:29:480
3157699663,21cyclictest11619-21chrt21:16:530
31598996546,14cyclictest355-21runrttasks19:54:392
31598996546,14cyclictest355-21runrttasks19:54:392
3159899653,44cyclictest8489-21fschecks_count19:34:492
31576996551,8cyclictest27511-21ssh21:44:190
31576996542,9cyclictest181rcu_preempt19:14:220
31576996541,11cyclictest181rcu_preempt20:19:370
3157699653,26cyclictest17458-21basename00:09:430
3157699653,26cyclictest17458-21basename00:09:430
3157699652,44cyclictest23265-21ssh23:24:400
3159899644,38cyclictest27401-21latency_hist20:34:222
31576996450,9cyclictest31566-21cyclictest23:51:200
31576996445,13cyclictest355-21runrttasks21:49:530
31576996443,14cyclictest13341-21fschecks_count19:49:410
3157699643,42cyclictest10883-21head19:39:580
3157699643,28cyclictest28452-21latency_hist00:29:210
3160699634,42cyclictest16984-21chrt00:09:223
31576996244,12cyclictest29397-21latency_hist20:39:230
3157699623,44cyclictest5178-21cpu21:04:360
3157699623,23cyclictest29580-21runrttasks20:40:010
3157699622,28cyclictest1319-21/usr/sbin/munin21:54:550
3160699615,41cyclictest17828-21chrt20:04:213
3157699614,43cyclictest15397-21mailstats19:54:550
3157699614,43cyclictest15397-21mailstats19:54:550
31576996143,11cyclictest0-21swapper/020:24:590
3157699614,28cyclictest355-21runrttasks22:46:380
3157699612,20cyclictest7571-21rm22:04:400
3157699603,34cyclictest6910-21fschecks_count19:29:430
3157699602,40cyclictest22036-21timerwakeupswit20:14:570
3160699593,45cyclictest13668-21chrt22:14:513
3160699584,38cyclictest24358-21chrt20:24:223
31606995811,39cyclictest29606-21chrt20:40:223
31576995834,13cyclictest0-21swapper/019:59:560
3160699574,39cyclictest6908-21taskset19:29:433
3160699574,38cyclictest14044-21chrt19:49:563
3160699573,41cyclictest9855-21taskset19:39:233
3160699572,40cyclictest17298-21chrt22:19:543
31576995738,13cyclictest5881-21threads19:25:260
3160699566,34cyclictest0-21swapper/323:59:223
3160699563,34cyclictest31395-21taskset22:44:223
3160699554,32cyclictest6612-21chrt19:29:223
3160699553,41cyclictest30087-21taskset23:34:563
3160699553,41cyclictest27285-21chrt00:26:123
3160699543,37cyclictest24688-21chrt21:39:253
3160699543,37cyclictest24688-21chrt21:39:253
3160699543,37cyclictest21867-21chrt22:28:253
3160699543,36cyclictest7281-21chrt22:58:203
31606995336,11cyclictest0-21swapper/300:19:183
3160699533,39cyclictest3416-21chrt19:19:593
3160699533,39cyclictest3416-21chrt19:19:593
3160699533,37cyclictest1747-21chrt21:54:453
3160699533,36cyclictest23670-21chrt22:30:073
3160699533,32cyclictest27588-21chrt20:34:233
3160699532,40cyclictest29291-21chrt21:46:023
3160699524,36cyclictest15240-21taskset19:54:503
3160699524,36cyclictest15240-21taskset19:54:503
3160699523,37cyclictest19305-21chrt00:13:153
3160699523,37cyclictest19305-21chrt00:13:153
3160699523,37cyclictest12565-21chrt22:13:093
3160699523,37cyclictest12565-21chrt22:13:093
3160699523,36cyclictest1755-21taskset22:49:213
3160699523,35cyclictest20178-21chrt20:09:523
3160699523,35cyclictest20178-21chrt20:09:523
3160699522,38cyclictest6769-21chrt23:50:413
3160699522,38cyclictest3921-21chrt20:59:463
3160699522,38cyclictest2749-21chrt20:54:563
3160699522,38cyclictest2749-21chrt20:54:563
3160699513,36cyclictest7947-21chrt22:59:263
3160699513,36cyclictest4398-21chrt23:47:213
3160699513,35cyclictest2425-21chrt22:49:453
3160699513,35cyclictest2425-21chrt22:49:453
3160699513,35cyclictest19594-21chrt21:29:533
3160699513,34cyclictest13093-21chrt19:49:233
3160699513,33cyclictest25061-21chrt20:24:553
3160699513,33cyclictest18318-21chrt23:15:393
3160699513,31cyclictest20055-21chrt23:19:243
3160699512,39cyclictest1054-21chrt23:41:093
3160699512,37cyclictest12870-21chrt23:07:123
3160699512,36cyclictest24263-21chrt00:20:453
3160699503,36cyclictest23685-21chrt23:24:513
3160699503,34cyclictest12810-21chrt00:00:593
3160699503,34cyclictest12810-21chrt00:00:593
3160699502,36cyclictest27183-21chrt22:36:403
3160699502,35cyclictest10368-21taskset19:39:473
3160699502,32cyclictest1813-21taskset00:39:213
3160699494,33cyclictest31676-21chrt21:49:553
3160699494,33cyclictest28098-21taskset20:34:473
3160699493,35cyclictest16716-21chrt21:24:593
3160699493,34cyclictest14864-21chrt23:09:563
3160699493,34cyclictest14864-21chrt23:09:563
3160699493,33cyclictest30712-21chrt00:32:533
3160699493,32cyclictest510-21chrt20:49:423
3160699493,32cyclictest25427-21taskset21:39:503
3160699493,31cyclictest6615-21chrt22:03:123
3160699493,22cyclictest0-21swapper/319:19:203
3160699492,36cyclictest15337-21chrt21:23:533
3160699492,36cyclictest15337-21chrt21:23:533
3160699492,32cyclictest31782-21taskset19:09:383
3160699484,34cyclictest9659-21taskset22:08:223
3160699484,31cyclictest32035-21chrt20:44:563
3160699483,35cyclictest28300-21chrt23:33:183
3160699483,34cyclictest10487-21chrt21:14:533
3160699483,33cyclictest21830-21chrt20:14:553
3160699472,32cyclictest6667-21chrt21:09:223
3160699462,32cyclictest8920-21taskset21:12:423
3160699434,29cyclictest18157-21taskset20:04:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional