You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 21:56
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Thu Nov 20, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12347991123,35cyclictest2417-21timerwakeupswit09:35:030
123479910951,51cyclictest0-21swapper/011:44:320
123479910951,51cyclictest0-21swapper/011:44:320
12347991092,35cyclictest26548-21uniq07:49:500
123479910734,66cyclictest0-21swapper/011:09:440
12347991073,28cyclictest7488-21fschecks_time09:44:540
12347991072,33cyclictest31205-21/usr/sbin/munin08:04:520
12347991063,22cyclictest31828-21ssh11:24:520
12347991063,22cyclictest31828-21ssh11:24:520
12347991054,27cyclictest12230-21awk10:49:580
10874210510,68sleep210891-21/usr/sbin/munin07:04:402
10874210510,68sleep210891-21/usr/sbin/munin07:04:402
12347991043,26cyclictest5251-21/usr/sbin/munin12:30:020
12347991043,26cyclictest5251-21/usr/sbin/munin12:30:020
12347991042,38cyclictest16532-21kworker/u18:1+flush-179:011:29:470
12347991034,35cyclictest3081-21latency_hist10:34:320
12347991033,38cyclictest22881-21latency_hist07:39:320
123479910328,70cyclictest0-21swapper/009:56:100
123479910317,31cyclictest15383-21unixbench_singl10:55:070
123559910162,33cyclictest22421-21/usr/sbin/munin12:04:472
12347991014,36cyclictest456-21latency_hist10:29:320
12347991014,35cyclictest22256-21ssh12:04:430
12347991014,31cyclictest355-21runrttasks12:09:510
12347991013,41cyclictest32020-21timerwakeupswit12:20:040
12347991013,34cyclictest14078-21latency_hist07:14:320
123479910123,21cyclictest26546-21grep09:20:020
123479910038,56cyclictest0-21swapper/011:49:300
123479910038,56cyclictest0-21swapper/011:49:300
12347991003,31cyclictest12232-21latency_hist07:09:330
12347991003,31cyclictest12232-21latency_hist07:09:330
123479910029,9cyclictest0-21swapper/009:49:480
12355999956,37cyclictest31730-21proc_pri12:20:012
1234799994,34cyclictest19021-21fschecks_time10:04:510
12347999941,52cyclictest18985-21kworker/u17:1+xprtiod11:17:580
1234799993,38cyclictest1280-21latency_hist09:34:310
1234799993,36cyclictest355-21runrttasks12:02:160
12347999933,58cyclictest15326-21latency_hist08:54:320
1234799992,35cyclictest22254-21sh09:12:550
1234799983,37cyclictest4119-21fschecks_time08:19:530
1234799983,36cyclictest13892-21cstates11:50:150
1234799983,34cyclictest3241-21acpi10:34:400
1234799983,32cyclictest29459-21grep09:25:070
1234799983,26cyclictest23296-21/usr/sbin/munin07:40:210
1234799983,24cyclictest19851-21/usr/sbin/munin07:29:500
1234799983,24cyclictest19851-21/usr/sbin/munin07:29:500
1234799982,35cyclictest18175-21cstates07:24:460
1234799982,35cyclictest18175-21cstates07:24:460
1234799982,35cyclictest17799-21modprobe07:24:290
1234799974,32cyclictest1506-21/usr/sbin/munin12:24:470
12347999725,10cyclictest0-21swapper/009:14:500
12347999725,10cyclictest0-21swapper/009:14:500
1234799972,37cyclictest17124-21modprobe10:00:280
1234799972,35cyclictest29570-21timerandwakeup11:20:060
1234799972,34cyclictest1572-21grep08:10:450
12347999719,23cyclictest30959-21cat12:19:320
1234799967,34cyclictest9919-21awk10:45:060
1234799965,31cyclictest27985-21runrttasks07:54:430
1234799964,38cyclictest19997-21ssh09:09:280
1234799963,36cyclictest355-21runrttasks10:12:050
1234799963,35cyclictest13261-21modprobe08:49:290
1234799963,35cyclictest13261-21modprobe08:49:290
1234799963,34cyclictest17407-21latency08:59:540
1234799963,33cyclictest20205-21runrttasks11:04:450
1234799963,30cyclictest11860-21latency_hist08:44:320
12347999627,61cyclictest0-21swapper/010:15:070
1234799954,37cyclictest17669-21latency10:59:560
1234799954,34cyclictest8541-21latency_hist10:44:320
1234799954,34cyclictest8541-21latency_hist10:44:320
1234799953,38cyclictest30439-21timerwakeupswit08:00:040
1234799953,36cyclictest5572-21cstates08:24:430
1234799953,34cyclictest16476-21/usr/sbin/munin11:54:470
1234799953,34cyclictest16476-21/usr/sbin/munin11:54:470
1234799953,31cyclictest2473-21fschecks_time08:14:540
12347999527,8cyclictest0-21swapper/007:15:270
12355999447,41cyclictest10201-21latency_hist08:39:322
1235199943,57cyclictest31793-21apt-get11:24:441
1235199943,57cyclictest31793-21apt-get11:24:441
1234799944,32cyclictest4570-21fschecks_count09:39:530
1234799944,32cyclictest4570-21fschecks_count09:39:530
1234799943,35cyclictest27319-21ls10:19:460
1234799942,32cyclictest355-21runrttasks12:34:550
1234799934,34cyclictest6818-21latency_hist08:29:330
1234799934,34cyclictest6818-21latency_hist08:29:330
1234799934,31cyclictest26108-21latency_hist07:49:330
1234799933,30cyclictest7398-21latency_hist11:39:310
1234799924,36cyclictest355-21runrttasks08:37:560
12347999217,20cyclictest16196-21sort08:55:090
12355999049,34cyclictest0-21swapper/210:09:532
12355999048,34cyclictest0-21swapper/209:04:502
12355999047,35cyclictest0-21swapper/210:04:542
1235599903,60cyclictest13638-21ssh11:49:442
153162870,8sleep10-21swapper/110:55:061
12355998765,16cyclictest27861-21latency_hist07:54:322
12355998748,32cyclictest0-21swapper/211:14:542
12355998743,37cyclictest2124-21sh12:24:552
1235599873,42cyclictest23344-21cstates11:10:102
1235599873,29cyclictest25486-21fschecks_time12:09:512
12355998647,34cyclictest12339-21cyclictest08:15:002
12355998544,35cyclictest5524-21netstat11:34:552
1235599853,39cyclictest5455-21/usr/sbin/munin08:25:042
1235599853,39cyclictest31570-21latency_hist11:24:322
1235599853,36cyclictest31746-21runrttasks08:04:572
12355998532,45cyclictest292-21dbus-daemon11:34:302
12355998521,54cyclictest21449-21sshd11:05:572
12351998556,22cyclictest9582-21kworker/u19:2+events_unbound07:19:301
12355998458,15cyclictest16378-21runrttasks07:19:332
1235599844,36cyclictest9445-21taskset12:37:152
12355998434,43cyclictest0-21swapper/210:55:042
1235599843,38cyclictest13516-21/usr/sbin/munin09:55:492
1235599843,35cyclictest27166-21/usr/sbin/munin10:19:582
1235599843,35cyclictest25335-21grep10:15:072
1235599842,36cyclictest355-21runrttasks08:10:042
1106628414,34sleep10-21swapper/107:04:561
1106628414,34sleep10-21swapper/107:04:561
12355998363,14cyclictest7175-21/usr/sbin/munin09:44:492
1235599834,40cyclictest18751-21mailstats07:25:022
1235599834,40cyclictest18751-21mailstats07:25:022
1235599833,37cyclictest25166-21latency07:44:562
1235599833,36cyclictest28742-21/usr/sbin/munin12:15:012
12351998357,17cyclictest2785-21sh09:35:411
12351998313,49cyclictest13058-21apt-get09:55:311
1235599824,33cyclictest8946-21cpuspeed10:44:452
1235599824,33cyclictest355-21runrttasks11:48:202
1235599824,33cyclictest355-21runrttasks11:48:202
1235599824,33cyclictest19935-21cut07:29:512
1235599824,33cyclictest19935-21cut07:29:512
12355998242,34cyclictest23343-21idleruntime09:14:512
12355998242,34cyclictest23343-21idleruntime09:14:512
12355998235,40cyclictest0-21swapper/209:14:322
1235599823,40cyclictest30589-21users08:00:062
1235599823,35cyclictest19430-21/usr/sbin/munin11:59:502
1235599823,33cyclictest5447-21vmstat09:40:102
1235599823,33cyclictest5447-21vmstat09:40:102
1235599823,25cyclictest30662-21/usr/sbin/munin10:25:072
1235599822,33cyclictest13915-21true07:14:302
1235599822,30cyclictest4001-21latency_hist09:39:322
1235599822,26cyclictest12539-21gpgv09:54:292
12355998161,14cyclictest8422-21latency_hist08:34:312
12355998161,14cyclictest8422-21latency_hist08:34:312
1235599813,39cyclictest6219-21fschecks_count10:39:522
1235599813,39cyclictest6219-21fschecks_count10:39:522
1235599813,38cyclictest16674-21/usr/sbin/munin10:00:032
1235599813,36cyclictest17238-21ssh10:59:452
1235599813,35cyclictest5064-21/usr/sbin/munin12:29:562
1235599813,35cyclictest5064-21/usr/sbin/munin12:29:562
1235599813,33cyclictest1956-21ssh11:29:312
1235599813,33cyclictest1956-21ssh11:29:312
1235599813,33cyclictest1220-21proc_pri10:30:022
1235599813,30cyclictest16457-21ssh11:54:342
1235599813,30cyclictest16457-21ssh11:54:342
1235599813,25cyclictest355-21runrttasks07:54:432
1235599812,37cyclictest8819-21/usr/sbin/munin11:40:082
1235599812,37cyclictest8819-21/usr/sbin/munin11:40:082
1235599812,36cyclictest3441-21ssh10:34:482
1235199812,35cyclictest23727-21mailstats09:15:091
1235199812,35cyclictest23727-21mailstats09:15:091
12355998061,14cyclictest15204-21latency_hist08:54:312
1235599804,30cyclictest16740-21fschecks_time07:19:512
1235599803,26cyclictest58-21kcompactd007:39:322
1235599803,26cyclictest10794-21seq08:39:542
1235599802,34cyclictest4014-21/usr/sbin/munin08:19:492
1235599802,34cyclictest15448-21grep08:54:462
12351998060,14cyclictest21142-21latency_hist07:34:311
12351998060,14cyclictest21142-21latency_hist07:34:311
12351998055,14cyclictest0-21swapper/109:44:321
12351998055,14cyclictest0-21swapper/109:44:321
12355997935,36cyclictest13548-21apt-key08:49:322
12355997935,36cyclictest13548-21apt-key08:49:322
1235599793,35cyclictest12495-21switchtime10:50:052
1235599793,33cyclictest28167-21date09:24:312
1235599793,32cyclictest355-21runrttasks09:33:422
1235599793,30cyclictest23463-21/usr/sbin/munin07:40:262
1235199792,51cyclictest13078-21apt-get11:49:431
1235599783,27cyclictest17306-21/usr/sbin/munin08:59:522
1235599783,22cyclictest29006-21seq09:24:592
12351997854,15cyclictest292-21dbus-daemon08:49:301
12351997854,15cyclictest292-21dbus-daemon08:49:301
12351997724,24cyclictest691-21cron08:29:331
12351997724,24cyclictest691-21cron08:29:331
12351997649,19cyclictest26044-21sh09:19:521
1235199758,57cyclictest30268-21apt-get10:24:531
12351997545,22cyclictest389-21apt-key08:09:341
1235199753,26cyclictest11669-21chrt11:45:111
1235199753,26cyclictest11669-21chrt11:45:111
12351997352,14cyclictest22613-21latency_hist11:09:311
12351997344,20cyclictest19734-21cut11:59:511
1235199732,48cyclictest28331-21modprobe11:19:281
1235199723,47cyclictest28593-21grep07:55:001
121522726,15sleep30-21swapper/307:08:413
121522726,15sleep30-21swapper/307:08:413
12351997155,10cyclictest13775-21apt-get08:49:431
12351997154,11cyclictest8063-21kworker/u17:1-writeback10:45:061
12351997054,10cyclictest29695-21gpgv10:24:321
12351997048,13cyclictest0-21swapper/112:09:511
12351996948,14cyclictest26594-21fschecks_time07:49:511
12351996945,18cyclictest22760-21munin-plugin-st09:14:311
12351996911,24cyclictest21718-21runrttasks10:09:501
1236099683,34cyclictest15017-21chrt09:57:453
12351996845,15cyclictest8541-21latency_hist10:44:311
12351996845,15cyclictest8541-21latency_hist10:44:311
1235199683,57cyclictest58-21kcompactd008:09:451
12351996745,14cyclictest17530-21fschecks_time10:59:531
12351996744,14cyclictest0-21swapper/112:04:501
12351996743,18cyclictest8752-21unixbench_multi11:40:061
12351996743,18cyclictest8752-21unixbench_multi11:40:061
12351996740,21cyclictest0-21swapper/112:39:311
12351996738,21cyclictest8699-21cpu08:34:481
39782660,6sleep30-21swapper/308:19:483
1236099663,30cyclictest20967-21taskset12:01:223
1236099662,32cyclictest8838-21taskset08:34:513
1236099662,30cyclictest15729-21taskset10:56:013
12351996651,10cyclictest12339-21cyclictest10:35:071
1235199663,46cyclictest28665-21/usr/sbin/munin11:20:061
12351996633,25cyclictest23150-21apt-get11:09:591
1236099653,25cyclictest10577-21chrt08:39:493
1236099653,24cyclictest21234-21chrt07:34:323
1236099653,24cyclictest21234-21chrt07:34:323
1236099652,27cyclictest9090-21chrt12:36:163
12351996546,15cyclictest16447-21kworker/u17:4+events_unbound07:44:291
12351996546,11cyclictest0-21swapper/110:05:051
1235199652,48cyclictest30036-21latency07:59:541
1236099643,27cyclictest30644-21chrt12:19:183
1236099643,24cyclictest10532-21chrt09:49:583
1236099642,30cyclictest650-21chrt08:10:043
1236099642,29cyclictest4022-21chrt09:39:293
1236099642,28cyclictest32743-21chrt10:29:313
1236099642,26cyclictest10179-21taskset10:45:483
1236099642,25cyclictest6002-21chrt10:39:443
1236099642,25cyclictest6002-21chrt10:39:443
12351996449,8cyclictest0-21swapper/109:54:301
12351996448,9cyclictest0-21swapper/109:25:061
1235199643,32cyclictest355-21runrttasks09:29:441
1236099638,46cyclictest0-21swapper/307:39:323
1236099633,29cyclictest21566-21chrt10:09:443
1236099633,29cyclictest19997-21taskset10:05:393
1236099633,28cyclictest9199-21taskset11:41:163
1236099633,28cyclictest9199-21taskset11:41:163
1236099633,24cyclictest28120-21chrt07:54:453
1236099632,26cyclictest27408-21chrt09:22:183
1236099632,26cyclictest1481-21chrt12:24:333
1236099632,25cyclictest3799-21chrt11:31:393
12351996349,10cyclictest12339-21cyclictest07:25:051
12351996349,10cyclictest12339-21cyclictest07:25:051
1235199634,44cyclictest22711-21munin-run07:39:301
12351996344,13cyclictest1-21systemd12:33:331
12351996344,13cyclictest1-21systemd12:33:331
1235199633,43cyclictest355-21runrttasks09:06:021
1235199632,40cyclictest5653-21fschecks_count08:24:471
1235199632,40cyclictest16156-21tune2fs09:59:511
1236099628,45cyclictest0-21swapper/312:09:513
1236099623,28cyclictest17671-21taskset09:00:023
1236099623,23cyclictest30750-21chrt08:01:583
1236099623,23cyclictest14267-21chrt08:50:383
1236099622,27cyclictest30700-21taskset09:29:233
1236099622,26cyclictest13250-21chrt10:51:353
1236099622,24cyclictest947-21taskset09:33:493
1236099622,24cyclictest30692-21chrt11:22:523
1236099622,23cyclictest25374-21chrt10:15:083
1236099622,22cyclictest24806-21chrt12:09:253
12351996239,15cyclictest0-21swapper/107:45:051
1235199623,42cyclictest28331-21fschecks_count12:14:511
1235199623,38cyclictest11716-21latency_hist08:44:311
1235199622,45cyclictest1539-21ssh12:24:391
1236099619,42cyclictest0-21swapper/307:19:513
1236099613,23cyclictest31082-21chrt12:19:423
1236099613,23cyclictest21003-21chrt09:10:023
1236099613,22cyclictest7559-21chrt09:44:563
1236099613,22cyclictest6569-21chrt09:43:383
1236099613,22cyclictest6569-21chrt09:43:383
1236099613,22cyclictest3287-21chrt10:34:453
1236099612,25cyclictest4538-21taskset12:29:393
1236099612,25cyclictest4538-21taskset12:29:393
1236099612,25cyclictest24390-21taskset07:43:153
1236099612,24cyclictest25528-21chrt11:14:213
1236099612,24cyclictest13720-21chrt11:49:593
1236099612,23cyclictest28959-21chrt10:22:133
1236099612,23cyclictest16775-21taskset08:59:293
1236099612,23cyclictest14543-21chrt07:14:393
1236099612,22cyclictest15881-21chrt09:59:393
12360996110,41cyclictest0-21swapper/308:24:513
12351996146,8cyclictest0-21swapper/110:50:031
12351996142,14cyclictest16845-21sh11:54:511
12351996142,14cyclictest16845-21sh11:54:511
1235199613,42cyclictest13313-21cat07:10:041
1235199613,38cyclictest2188-21/usr/sbin/munin11:30:041
1235199613,31cyclictest355-21runrttasks09:02:561
1236099609,43cyclictest0-21swapper/310:30:293
1236099604,25cyclictest18401-21taskset07:24:523
1236099604,25cyclictest18401-21taskset07:24:523
1236099604,23cyclictest27369-21chrt07:50:123
1236099603,23cyclictest3228-21chrt08:15:153
1236099602,24cyclictest6133-21taskset11:35:403
1236099602,24cyclictest27076-21chrt11:15:313
1236099602,24cyclictest12405-21chrt11:47:243
1236099602,24cyclictest12405-21chrt11:47:243
1236099602,23cyclictest583-21chrt11:25:393
1236099602,23cyclictest583-21chrt11:25:393
1236099602,23cyclictest26143-21chrt07:49:313
1236099602,23cyclictest18502-21chrt11:00:443
1236099602,22cyclictest19766-21chrt09:06:583
12360996018,34cyclictest0-21swapper/311:09:023
12351996043,11cyclictest534-21/usr/sbin/munin10:29:461
1235199603,38cyclictest24540-21fschecks_count10:14:511
1235199603,38cyclictest16738-21sh08:59:291
1235199602,30cyclictest5153-21cstates11:34:451
12351996013,36cyclictest1393-21sh12:24:311
1236099593,24cyclictest24138-21chrt09:15:253
1236099593,24cyclictest24138-21chrt09:15:253
1236099593,21cyclictest13854-21chrt07:14:153
1236099592,21cyclictest13132-21chrt08:47:513
1236099592,21cyclictest13132-21chrt08:47:513
1236099592,20cyclictest16474-21chrt11:54:373
1236099592,20cyclictest16474-21chrt11:54:373
12351995943,9cyclictest0-21swapper/108:20:071
1235199592,34cyclictest2213-21/usr/sbin/munin08:15:051
1236099582,24cyclictest7083-21chrt08:29:403
1236099582,24cyclictest7083-21chrt08:29:403
1236099582,21cyclictest32342-21chrt08:06:483
1235199583,42cyclictest16689-21fschecks_time07:19:501
1235199583,39cyclictest9732-21latency_hist09:49:301
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional