You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 10:55
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Tue Dec 02, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22570991123,28cyclictest355-21runrttasks22:59:010
22570991104,30cyclictest355-21runrttasks20:57:320
22570991082,29cyclictest18210-21sh00:28:550
22575991053,50cyclictest15833-21ssh23:29:111
22575991053,50cyclictest15833-21ssh23:29:111
22570991052,27cyclictest5501-21seq22:17:030
22570991033,32cyclictest15740-21apt-get23:28:570
22570991033,32cyclictest15740-21apt-get23:28:570
22575991023,34cyclictest22749-21apt-get19:08:551
22570991024,31cyclictest29702-21diskmemload21:18:560
22570991022,31cyclictest31255-21runrttasks22:04:440
22570991015,10cyclictest0-21swapper/019:08:480
22570991013,9cyclictest0-21swapper/020:34:000
22570991013,8cyclictest0-21swapper/023:58:590
22570991013,30cyclictest17907-21idleruntime-cro00:28:430
22570991012,30cyclictest355-21runrttasks21:09:310
22570991012,23cyclictest355-21runrttasks20:53:190
22570991012,22cyclictest22741-21runrttasks20:45:330
22570991003,85cyclictest15248-21ssh22:33:460
225709910032,61cyclictest0-21swapper/023:09:120
225709910032,61cyclictest0-21swapper/023:09:120
2257599993,6cyclictest0-21swapper/123:39:001
2257599993,6cyclictest0-21swapper/123:39:001
2257099994,29cyclictest27770-21cut23:49:050
2257099993,23cyclictest355-21runrttasks22:09:420
2257099992,24cyclictest1592-21seq19:41:380
2257099992,23cyclictest11173-21taskset20:10:220
2258599983,54cyclictest15740-21apt-get23:29:022
2258599983,54cyclictest15740-21apt-get23:29:022
22575999857,34cyclictest22557-21cyclictest19:54:111
22575999857,34cyclictest22557-21cyclictest19:54:111
2257099984,12cyclictest0-21swapper/022:23:560
22570999840,52cyclictest0-21swapper/019:44:210
2257099983,24cyclictest24535-21grep22:49:050
2257099982,28cyclictest355-21runrttasks21:18:190
2257099982,24cyclictest29374-21seq21:04:450
2257099982,24cyclictest29374-21seq21:04:450
2257099982,23cyclictest29151-21seq19:26:030
2257099982,23cyclictest29151-21seq19:26:030
2257099982,23cyclictest20080-21seq20:39:210
2257099974,7cyclictest0-21swapper/021:58:580
2257099973,29cyclictest355-21runrttasks20:07:060
2257099972,24cyclictest14429-21seq20:20:300
2257099972,22cyclictest355-21runrttasks23:05:500
2257099972,22cyclictest355-21runrttasks23:05:500
22570999645,45cyclictest9671-21/usr/sbin/munin23:19:170
22570999645,45cyclictest9671-21/usr/sbin/munin23:19:170
2257099964,26cyclictest22884-21runrttasks21:51:020
2257099964,26cyclictest22884-21runrttasks21:51:020
2257099963,28cyclictest355-21runrttasks23:54:250
2257099962,31cyclictest10465-21/usr/sbin/munin21:29:210
2257099962,25cyclictest355-21runrttasks19:09:300
22575999572,13cyclictest0-21swapper/120:48:471
2257099954,32cyclictest7087-21ntp_kernel_err21:24:150
2257099953,32cyclictest3861-21apt-get00:04:230
2257099952,19cyclictest355-21runrttasks20:02:560
22575999472,12cyclictest0-21swapper/123:38:461
22570999479,8cyclictest0-21swapper/023:15:540
22570999444,43cyclictest7871-21ssh22:20:210
2257099944,29cyclictest13433-21unixbench_singl21:34:210
2257099944,24cyclictest355-21runrttasks21:56:370
2257099943,8cyclictest0-21swapper/019:28:590
2257099943,26cyclictest18264-21/usr/sbin/munin21:43:560
2257099943,25cyclictest16718-21/usr/sbin/munin20:29:050
2257099943,25cyclictest16718-21/usr/sbin/munin20:29:050
2257099942,25cyclictest9896-21fschecks_time00:14:060
2257099942,23cyclictest355-21runrttasks22:47:060
22575999372,12cyclictest0-21swapper/119:43:461
22575999352,32cyclictest0-21swapper/120:03:471
2257099933,33cyclictest24540-21/usr/sbin/munin23:43:540
2257099932,21cyclictest355-21runrttasks22:37:300
22570999314,11cyclictest161ktimers/019:13:580
22570999314,11cyclictest161ktimers/019:13:580
22575999272,11cyclictest0-21swapper/120:18:461
22575999268,13cyclictest0-21swapper/122:13:461
22575999255,32cyclictest0-21swapper/122:24:191
2257099924,30cyclictest14864-21apt00:23:420
2257099924,30cyclictest14864-21apt00:23:420
2257099923,33cyclictest22658-21grep23:39:210
2257099923,33cyclictest22658-21grep23:39:210
2257099923,27cyclictest15455-21sed23:28:460
2257099923,22cyclictest5313-21cpuspeed19:54:020
2257099923,22cyclictest5313-21cpuspeed19:54:020
22585999140,26cyclictest14968-21ssh22:33:162
22575999172,9cyclictest0-21swapper/120:39:541
22575999151,34cyclictest29702-21diskmemload21:54:071
2257599913,75cyclictest0-21swapper/123:08:591
2257599913,75cyclictest0-21swapper/123:08:591
2257599913,26cyclictest355-21runrttasks21:36:071
2257599912,30cyclictest355-21runrttasks20:09:571
22570999166,14cyclictest0-21swapper/022:43:470
22570999166,14cyclictest0-21swapper/022:43:470
2257099913,31cyclictest6697-21/usr/sbin/munin00:08:560
2257099913,31cyclictest6697-21/usr/sbin/munin00:08:560
2257099913,27cyclictest15330-21/usr/sbin/munin21:39:240
217432919,55sleep121786-21sensors_temp19:04:271
22575999053,30cyclictest0-21swapper/121:18:441
22575999051,34cyclictest0-21swapper/121:47:571
2257599904,74cyclictest0-21swapper/120:34:001
2257599903,7cyclictest0-21swapper/121:48:561
2257599903,7cyclictest0-21swapper/121:48:561
22575999022,39cyclictest0-21swapper/121:01:451
2257099903,30cyclictest4980-21latency_hist19:53:470
2257099903,28cyclictest32145-21tail19:34:210
2257099902,30cyclictest26427-21apt-get19:18:560
2257099902,30cyclictest26427-21apt-get19:18:560
22575998972,10cyclictest0-21swapper/123:59:071
22575998971,8cyclictest0-21swapper/100:28:451
2257099892,26cyclictest27201-21/usr/sbin/munin22:54:060
2241228914,49sleep222453-21modprobe19:08:442
22575998873,8cyclictest0-21swapper/121:59:011
22575998869,10cyclictest0-21swapper/120:58:461
2257099883,29cyclictest18672-21/usr/sbin/munin23:34:030
2257099883,28cyclictest21138-21cstates00:34:010
2257099883,28cyclictest21138-21cstates00:34:010
2257099883,24cyclictest15907-21grep20:24:230
2257099882,29cyclictest28177-21latency_hist21:03:470
22585998761,19cyclictest455-40ogrotate23:58:442
22585998757,22cyclictest16049-21processes21:39:202
22575998770,7cyclictest0-21swapper/119:29:081
22575998761,16cyclictest22557-21cyclictest22:18:471
22575998747,33cyclictest0-21swapper/100:04:411
22575998669,7cyclictest0-21swapper/123:14:591
2257599862,6cyclictest0-21swapper/123:23:581
2258599853,60cyclictest15411-21apt-get00:24:172
2258599853,50cyclictest3861-21apt-get00:03:582
22585998516,42cyclictest3762-21kworker/u17:1+xprtiod22:44:422
22575998547,31cyclictest0-21swapper/122:04:011
22575998542,37cyclictest0-21swapper/121:04:281
22575998542,37cyclictest0-21swapper/121:04:281
2257099852,23cyclictest11767-21/usr/sbin/munin20:14:020
22575998446,31cyclictest0-21swapper/122:44:011
22575998443,34cyclictest0-21swapper/123:49:071
2257599843,6cyclictest0-21swapper/122:53:561
2257599843,38cyclictest30736-21/usr/sbin/munin21:09:171
2257599842,7cyclictest0-21swapper/100:33:551
2257599842,7cyclictest0-21swapper/100:33:551
2258599832,44cyclictest29868-21fschecks_time19:29:082
22575998369,7cyclictest0-21swapper/122:33:451
2257599834,8cyclictest0-21swapper/120:29:011
2257599834,8cyclictest0-21swapper/120:29:011
22575998344,33cyclictest31980-21sh23:56:031
2257599833,65cyclictest677-21ssh23:03:591
2257599833,65cyclictest677-21ssh23:03:591
2257599832,7cyclictest0-21swapper/122:48:571
2257599824,67cyclictest0-21swapper/120:49:011
22575998243,32cyclictest0-21swapper/123:19:201
22575998243,32cyclictest0-21swapper/123:19:201
22575998241,34cyclictest0-21swapper/121:29:171
2257599822,36cyclictest19284-21ssh00:29:201
2257599822,35cyclictest31754-21/usr/sbin/munin19:34:131
22585998135,36cyclictest9366-21apt-get22:23:562
2257599814,61cyclictest6537-21apt-get21:23:581
22575998144,30cyclictest0-21swapper/120:19:201
2257599813,8cyclictest0-21swapper/121:39:011
22585998057,15cyclictest22557-21cyclictest23:28:442
22585998054,17cyclictest26633-21fschecks_count19:19:062
22585998054,17cyclictest26633-21fschecks_count19:19:062
22585998018,28cyclictest23269-21runrttasks20:48:572
2257599804,8cyclictest0-21swapper/119:14:021
2257599804,8cyclictest0-21swapper/119:14:021
22575998047,26cyclictest0-21swapper/122:34:221
2257599804,66cyclictest0-21swapper/119:48:591
2257599802,37cyclictest5602-21chrt21:22:011
22585997958,14cyclictest1886-21latency_hist19:43:462
2257599793,32cyclictest30512-21/usr/sbin/munin22:59:091
2257599792,72cyclictest0-21swapper/120:24:021
2257599792,36cyclictest355-21runrttasks22:23:161
2257599783,7cyclictest0-21swapper/119:18:571
2257599783,7cyclictest0-21swapper/119:18:571
22575997835,36cyclictest0-21swapper/100:17:021
2257599783,32cyclictest6715-21apt-get00:08:571
2257599783,32cyclictest6715-21apt-get00:08:571
2257599783,32cyclictest28937-21unixbench_multi19:24:221
2257599783,32cyclictest28937-21unixbench_multi19:24:221
2245427826,36sleep30-21swapper/319:08:453
2257599774,8cyclictest0-21swapper/123:43:581
2257599773,36cyclictest13354-21/usr/sbin/munin00:19:171
2257599773,36cyclictest13354-21/usr/sbin/munin00:19:171
2257599772,7cyclictest0-21swapper/119:43:571
2257599772,34cyclictest20849-21ssh22:43:121
2257599772,34cyclictest20849-21ssh22:43:121
22585997660,10cyclictest24475-21df22:49:012
22585997653,17cyclictest309-21systemd-logind22:38:442
22585997534,27cyclictest2036-21apt-get19:43:562
2258599753,41cyclictest9962-21latency_hist20:08:462
2258599753,32cyclictest20038-21apt-get20:39:002
2257599754,28cyclictest8609-21cstates20:04:061
22585997459,9cyclictest22557-21cyclictest00:29:182
22585997451,17cyclictest9656-21sh23:18:522
22585997451,17cyclictest9656-21sh23:18:522
22585997451,16cyclictest31036-21grep22:04:202
2258599743,37cyclictest18258-21ssh21:43:562
2258599743,21cyclictest18804-21mailstats20:34:172
22585997414,28cyclictest12648-21fschecks_count21:34:062
2259099733,37cyclictest15199-21chrt23:28:433
22585997355,11cyclictest10039-21sh21:29:132
22585997349,18cyclictest6889-21kworker/u17:0-events_unbound23:43:432
22585997349,18cyclictest6889-21kworker/u17:0-events_unbound23:43:432
22585997344,19cyclictest13354-21timerandwakeup00:19:172
22585997344,19cyclictest13354-21timerandwakeup00:19:172
2258599733,56cyclictest7468-21/usr/sbin/munin23:15:382
22585997335,26cyclictest30023-21apt-get21:08:552
22585997335,23cyclictest5253-21apt-get19:53:572
22585997335,23cyclictest5253-21apt-get19:53:572
22585997334,29cyclictest3462-21ssh22:13:552
2258599733,30cyclictest10463-21chrt20:09:102
22585997330,37cyclictest0-21swapper/200:33:552
22585997330,37cyclictest0-21swapper/200:33:552
22585997252,13cyclictest9481-21latency_hist00:13:472
22585997252,13cyclictest9481-21latency_hist00:13:472
22585997242,23cyclictest7712-21timerwakeupswit19:59:222
22585997239,23cyclictest31247-21apt-get19:34:002
2258599723,37cyclictest6367-21ssh21:23:472
2259099713,34cyclictest9673-21taskset21:29:043
22585997152,13cyclictest309-21systemd-logind00:18:442
22585997146,17cyclictest27221-21apt-get22:53:592
22585997134,28cyclictest3586-21apt-get23:08:552
22585997134,28cyclictest3586-21apt-get23:08:552
2258599712,37cyclictest7228-21ssh22:19:192
2259099704,31cyclictest12909-21chrt22:29:303
2259099703,34cyclictest31177-21taskset21:09:303
22585997052,12cyclictest207-21systemd-journal20:33:452
22585997052,12cyclictest207-21systemd-journal20:33:452
22585997045,16cyclictest26749-21kworker/u19:221:58:432
2258599703,37cyclictest355-21runrttasks19:51:282
22585997016,27cyclictest0-21swapper/222:08:492
2259099694,28cyclictest9668-21chrt22:24:073
2259099692,30cyclictest29029-21chrt21:04:323
2259099692,30cyclictest29029-21chrt21:04:323
22585996950,14cyclictest15210-21fschecks_time20:24:082
22585996949,14cyclictest23079-21latency_hist20:48:462
22585996949,13cyclictest207-21systemd-journal21:16:572
22585996935,24cyclictest24554-21apt-get23:43:592
22585996932,28cyclictest27478-21apt-get23:48:592
22585996932,22cyclictest0-21swapper/219:14:022
22585996932,22cyclictest0-21swapper/219:14:022
2259099683,32cyclictest9672-21taskset23:18:543
2259099683,32cyclictest9672-21taskset23:18:543
2259099683,30cyclictest21214-21chrt21:48:583
2259099683,30cyclictest21214-21chrt21:48:583
22585996847,16cyclictest11614-21kworker/u18:3+events_unbound19:13:442
22585996845,16cyclictest26444-21latency_hist20:58:472
22585996828,29cyclictest28616-21cut21:04:202
22585996828,29cyclictest28616-21cut21:04:202
22585996811,24cyclictest0-21swapper/221:25:482
2259099673,26cyclictest15645-21chrt22:34:043
2259099672,30cyclictest6672-21chrt19:58:463
2259099672,30cyclictest6672-21chrt19:58:463
2259099672,29cyclictest18717-21chrt00:29:093
22585996739,10cyclictest21194-21ssh21:48:562
22585996739,10cyclictest21194-21ssh21:48:562
2258599672,31cyclictest1027-21fschecks_count23:59:052
2259099668,10cyclictest0-21swapper/322:58:573
2259099664,27cyclictest7471-21chrt00:09:203
2259099664,27cyclictest7471-21chrt00:09:203
2259099664,23cyclictest24841-21chrt19:14:033
2259099664,23cyclictest24841-21chrt19:14:033
2259099663,24cyclictest24009-21chrt22:48:433
2259099663,23cyclictest8864-21chrt21:27:553
2259099662,27cyclictest8181-21taskset22:21:163
2259099662,24cyclictest3825-21taskset23:09:053
2259099662,24cyclictest3825-21taskset23:09:053
22585996638,21cyclictest13825-21netstat20:19:132
22585996634,23cyclictest30179-21apt-get22:59:022
22585996632,24cyclictest28036-21apt-get19:24:022
22585996632,24cyclictest28036-21apt-get19:24:022
2259099654,28cyclictest9756-21chrt00:14:023
2259099653,28cyclictest1110-21chrt23:59:063
2259099653,26cyclictest26744-21chrt21:58:433
2259099653,25cyclictest22639-21chrt23:39:213
2259099653,25cyclictest22639-21chrt23:39:213
2259099653,24cyclictest27435-21chrt22:54:063
2259099653,23cyclictest25445-21chrt20:54:193
2259099652,26cyclictest28783-21chrt22:01:023
2259099652,26cyclictest14056-21chrt21:36:013
22585996546,15cyclictest18148-21kworker/u17:1-rpciod23:34:182
22585996534,21cyclictest27097-21apt-get21:58:552
22590996433,21cyclictest4925-21chrt21:20:123
2259099643,30cyclictest16490-21taskset20:28:473
2259099643,27cyclictest16988-21chrt23:29:493
2259099643,27cyclictest16988-21chrt23:29:493
2259099643,24cyclictest4071-21chrt19:49:103
2259099642,28cyclictest20830-21chrt20:39:533
2259099642,23cyclictest26659-21taskset20:59:053
22585996451,9cyclictest22557-21cyclictest22:41:002
22585996451,9cyclictest22557-21cyclictest22:41:002
22585996433,20cyclictest26504-21apt-get20:59:012
2259099636,10cyclictest0-21swapper/323:43:563
2259099634,15cyclictest538-21chrt21:13:563
2259099633,23cyclictest18440-21chrt22:38:583
2259099633,23cyclictest18440-21chrt22:38:583
2259099633,23cyclictest12646-21chrt20:14:223
2259099632,27cyclictest24301-21taskset20:49:583
2259099632,26cyclictest15353-21taskset21:38:573
2259099632,25cyclictest15471-21taskset00:24:073
2259099632,25cyclictest11176-21chrt20:10:253
2259099632,24cyclictest918-21taskset22:09:083
22585996330,22cyclictest663-21apt-get23:03:582
22585996330,22cyclictest663-21apt-get23:03:582
22585996327,25cyclictest12155-21/usr/sbin/munin20:14:112
2259099624,24cyclictest30644-21chrt23:54:083
2259099624,22cyclictest23303-21chrt00:38:413
2259099624,22cyclictest23303-21chrt00:38:413
2259099623,28cyclictest27992-21chrt19:23:473
2259099623,23cyclictest21014-21taskset23:37:553
2259099623,22cyclictest14580-21taskset20:22:283
2259099622,28cyclictest23381-21taskset19:09:173
2259099622,27cyclictest4033-21taskset00:04:253
2259099622,25cyclictest5759-21chrt22:17:473
2259099622,24cyclictest25853-21chrt22:50:423
2259099622,24cyclictest20688-21chrt21:48:133
2259099622,24cyclictest19318-21chrt20:36:253
2259099622,24cyclictest1702-21chrt23:04:233
2259099622,24cyclictest1702-21chrt23:04:233
2259099613,23cyclictest7951-21taskset23:15:503
2259099613,23cyclictest28564-21chrt19:24:163
2259099613,23cyclictest28564-21chrt19:24:163
2259099613,23cyclictest22832-21chrt20:46:463
2259099612,25cyclictest13865-21chrt00:20:183
2259099612,25cyclictest13865-21chrt00:20:183
2259099612,24cyclictest9674-21taskset20:06:483
2259099612,24cyclictest16785-21taskset20:29:083
2259099612,24cyclictest16785-21taskset20:29:083
2259099612,22cyclictest8203-21chrt20:03:323
2259099612,22cyclictest31906-21chrt22:06:373
2259099603,23cyclictest32288-21chrt19:35:013
2259099603,23cyclictest30702-21chrt19:30:193
2259099603,23cyclictest28921-21chrt23:50:233
2259099603,22cyclictest1646-21chrt19:42:183
2259099602,23cyclictest2008-21chrt19:43:483
22590995911,39cyclictest0-21swapper/319:19:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional