You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 11:38
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Thu Sep 18, 2025 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
62079911842,70cyclictest12681-21kworker/u18:1+events_unbound21:06:220
62079911842,70cyclictest12681-21kworker/u18:1+events_unbound21:06:220
6207991183,103cyclictest0-21swapper/022:31:360
62079911735,55cyclictest0-21swapper/019:06:380
62079911145,59cyclictest21418-21latency_hist21:16:280
6207991103,70cyclictest6286-21awk23:36:550
62079910847,54cyclictest0-21swapper/020:11:580
3025421080,8sleep06198-21cyclictest00:21:460
6207991052,74cyclictest0-21swapper/023:55:130
62079910442,56cyclictest24818-21ls00:11:400
6207991043,91cyclictest23504-21ssh22:16:340
6207991034,84cyclictest6081-21latency_hist19:01:260
62079910225,55cyclictest0-21swapper/021:26:400
6207991014,87cyclictest16275-21ntpq21:06:530
62079910038,55cyclictest0-21swapper/022:51:590
6207991002,90cyclictest16888-21/usr/sbin/munin23:56:350
6207991002,90cyclictest16888-21/usr/sbin/munin23:56:350
620799993,64cyclictest14674-21sh22:01:340
52252997,63sleep25255-21ntpq18:56:582
6207999842,49cyclictest12149-21sendmail_mailqu23:46:560
620799983,83cyclictest26396-21apt-get22:21:340
620799983,83cyclictest26396-21apt-get22:21:340
6218999781,9cyclictest11267-21kworker/u17:2+ext4-rsv-conversion20:16:382
6218999781,9cyclictest11267-21kworker/u17:2+ext4-rsv-conversion20:16:382
620799976,57cyclictest0-21swapper/022:46:530
620799976,57cyclictest0-21swapper/022:46:530
6207999640,46cyclictest16549-21cpu19:31:390
620799963,60cyclictest331-21dbus-daemon19:41:220
620799963,60cyclictest331-21dbus-daemon19:41:220
620799962,63cyclictest22622-21sh00:06:550
620799962,63cyclictest22622-21sh00:06:550
6214999567,21cyclictest8410-21latency_hist23:41:241
6214999564,22cyclictest19523-21kworker/u19:2+rpciod21:07:091
620799953,84cyclictest11462-21sh21:56:220
620799953,82cyclictest485-21runrttasks20:21:370
620799953,79cyclictest0-21swapper/021:56:390
620799953,79cyclictest0-21swapper/021:56:390
6207999532,58cyclictest0-21swapper/022:11:500
620799952,61cyclictest2978-21fschecks_count23:31:450
620799945,62cyclictest19592-21cat21:12:350
6207999440,47cyclictest0-21swapper/019:01:590
620799943,63cyclictest17634-21taskset22:06:340
620799943,63cyclictest17634-21taskset22:06:340
620799942,59cyclictest30830-21seq22:28:160
620799942,59cyclictest30830-21seq22:28:160
6218999373,10cyclictest14151-21kworker/u17:3-xprtiod23:56:252
621899933,68cyclictest26422-21ssh23:16:412
620799933,78cyclictest20320-21apt-get23:06:370
6207999333,53cyclictest0-21swapper/020:51:500
620799932,61cyclictest19739-21/usr/sbin/munin19:41:530
620799932,61cyclictest19739-21/usr/sbin/munin19:41:530
620799923,61cyclictest26384-21apt-get23:16:580
620799923,57cyclictest356-21runrttasks00:28:240
620799923,57cyclictest356-21runrttasks00:28:240
6207999232,51cyclictest0-21swapper/023:04:520
6207999232,51cyclictest0-21swapper/023:04:520
620799922,57cyclictest0-21swapper/021:21:390
620799922,57cyclictest0-21swapper/021:21:390
613129210,58sleep16150-21latency_hist19:01:251
620799914,53cyclictest6551-21mailstats21:46:520
620799914,53cyclictest6551-21mailstats21:46:520
620799913,80cyclictest31129-21awk21:31:550
620799913,80cyclictest31129-21awk21:31:550
6207999134,49cyclictest0-21swapper/019:16:530
620799912,56cyclictest0-21swapper/021:16:460
6207999122,41cyclictest356-21runrttasks22:41:190
6214999068,12cyclictest9174-21ntp_kernel_pll_23:41:511
621499906,44cyclictest3959-21apt-get20:32:071
620799907,55cyclictest0-21swapper/000:19:460
620799894,78cyclictest1338-21/usr/sbin/munin21:36:440
620799894,78cyclictest1338-21/usr/sbin/munin21:36:440
620799894,76cyclictest0-21swapper/019:11:340
620799893,74cyclictest24518-21taskset19:56:250
620799893,74cyclictest24518-21taskset19:56:250
6207998935,47cyclictest0-21swapper/021:01:230
621899884,60cyclictest3959-21apt-get20:31:582
620799884,76cyclictest5640-21apt-get22:41:370
620799883,78cyclictest21727-21modprobe00:06:210
620799883,77cyclictest21358-21apt-get19:46:330
6207998829,47cyclictest0-21swapper/020:36:410
6207998739,42cyclictest14882-21/usr/sbin/munin19:26:540
620799873,77cyclictest32181-21taskset23:26:330
620799872,76cyclictest23292-21rm23:11:370
620799872,76cyclictest23292-21rm23:11:370
620799872,54cyclictest11077-21ssh23:46:150
6214998667,12cyclictest4159-21ssh22:38:001
621499866,44cyclictest11337-21sed20:51:591
620799862,72cyclictest0-21swapper/020:41:420
6218998558,21cyclictest0-21swapper/223:47:002
620799853,72cyclictest29256-21/usr/sbin/munin23:21:550
6207998528,50cyclictest0-21swapper/021:41:560
620799852,72cyclictest2089-21apt-get20:26:340
6207998516,60cyclictest27966-21systemd-journal22:57:330
621499842,55cyclictest18851-21apt-get21:11:381
621499842,55cyclictest18851-21apt-get21:11:381
620799842,72cyclictest26217-21apt-get20:01:340
6218998352,23cyclictest25330-21sh21:21:582
6218998352,23cyclictest25330-21sh21:21:582
620799833,72cyclictest313-21latency_hist20:21:250
620799833,72cyclictest313-21latency_hist20:21:250
620799833,70cyclictest0-21swapper/020:46:340
620799833,70cyclictest0-21swapper/020:46:340
6207998328,47cyclictest0-21swapper/020:36:260
620799832,76cyclictest0-21swapper/019:21:520
6214998264,11cyclictest3686-21apt-key20:31:251
621499823,54cyclictest170-21jbd2/mmcblk2p2-800:27:241
621499823,54cyclictest170-21jbd2/mmcblk2p2-800:27:241
621899812,26cyclictest20825-21cut22:11:472
620799818,66cyclictest0-21swapper/020:06:570
6218998057,15cyclictest17824-21ssh23:57:012
6218998057,15cyclictest17824-21ssh23:57:012
6218998056,16cyclictest17858-21sh21:10:072
620799804,66cyclictest24611-21apt-get19:56:350
6218997957,16cyclictest29736-21latency_hist00:21:232
621899793,49cyclictest30392-21tail20:11:582
6218997858,14cyclictest5618-21/usr/sbin/munin22:41:512
6218997822,43cyclictest10206-21cat20:51:242
6218997822,43cyclictest10206-21cat20:51:242
6214997859,10cyclictest0-21swapper/120:51:221
6214997859,10cyclictest0-21swapper/120:51:221
6214997853,16cyclictest27351-21cpu00:16:381
621499783,21cyclictest30923-21kernelversion21:31:491
621499783,21cyclictest30923-21kernelversion21:31:491
6218997755,13cyclictest0-21swapper/222:16:332
6218997746,18cyclictest0-21swapper/200:26:252
6214997758,12cyclictest6198-21cyclictest22:18:281
6214997756,12cyclictest14428-21diskmemload22:01:241
6214997756,12cyclictest14428-21diskmemload22:01:241
621899763,54cyclictest32281-21apt-get00:26:362
621899763,54cyclictest32281-21apt-get00:26:362
6214997650,20cyclictest22280-21unixbench_multi19:46:581
6218997555,13cyclictest27966-21systemd-journal22:55:072
6218997553,16cyclictest376-21in:imuxsock00:06:222
6218997513,19cyclictest0-21swapper/222:26:452
6218997513,19cyclictest0-21swapper/222:26:452
621499759,34cyclictest31-21ksoftirqd/119:06:421
6214997550,18cyclictest25828-21latency_hist23:16:261
6214997550,18cyclictest25828-21latency_hist23:16:261
6214997543,25cyclictest6357-21sendmail_mailqu22:41:551
6218997452,14cyclictest31349-21ssh21:31:592
6218997452,14cyclictest31349-21ssh21:31:592
6214997449,17cyclictest0-21swapper/120:01:241
621499743,29cyclictest8836-21ssh21:51:371
6214997415,45cyclictest31311-21apt-get20:17:061
6214997415,45cyclictest31311-21apt-get20:17:061
6218997353,14cyclictest365-21systemd-logind22:11:232
6218997353,14cyclictest365-21systemd-logind22:11:232
6218997351,13cyclictest0-21swapper/220:26:592
6218997347,18cyclictest23031-21latency_hist23:11:242
621899734,32cyclictest23833-21ntp_kernel_pll_23:11:512
621899734,32cyclictest23833-21ntp_kernel_pll_23:11:512
6214997354,13cyclictest32438-21apt-key00:26:251
621499733,50cyclictest21235-21latency_hist19:46:251
621499733,50cyclictest21235-21latency_hist19:46:251
6214997332,14cyclictest19873-21ssh00:01:531
545827310,39sleep30-21swapper/318:57:073
6218997256,9cyclictest0-21swapper/223:36:382
6218997250,14cyclictest8918-21ntp_kernel_err19:06:532
6218997247,15cyclictest0-21swapper/219:26:252
6218997246,19cyclictest6198-21cyclictest23:21:552
6218997243,16cyclictest0-21swapper/220:26:252
6214997253,13cyclictest12432-21diskmemload23:27:001
6214997249,9cyclictest31-21ksoftirqd/123:59:081
6214997249,9cyclictest31-21ksoftirqd/123:59:081
621499724,43cyclictest16766-21fschecks_time19:31:441
6218997152,10cyclictest0-21swapper/222:23:552
6218997152,10cyclictest0-21swapper/222:23:552
6218997152,10cyclictest0-21swapper/220:42:002
621899712,32cyclictest15092-21ssh22:56:512
6214997152,11cyclictest21063-21mailstats23:06:551
622499703,54cyclictest23676-21taskset22:16:403
6218997057,9cyclictest6198-21cyclictest21:56:562
6218997057,9cyclictest6198-21cyclictest21:56:562
6218997054,12cyclictest6198-21cyclictest23:04:162
6218997054,12cyclictest6198-21cyclictest23:04:162
6214997055,9cyclictest0-21swapper/123:46:551
6214997052,12cyclictest24703-21/usr/sbin/munin00:11:561
6214997050,13cyclictest376-21in:imuxsock22:50:341
6214997050,13cyclictest376-21in:imuxsock22:50:341
6214997049,12cyclictest0-21swapper/122:21:581
6214997049,12cyclictest0-21swapper/122:21:581
6214997027,11cyclictest181rcu_preempt19:41:211
6214997027,11cyclictest181rcu_preempt19:41:211
622499692,52cyclictest3962-21taskset20:31:343
622499692,51cyclictest31179-21taskset00:22:293
6218996945,14cyclictest7775-21diskmemload21:49:232
6218996945,14cyclictest7775-21diskmemload21:49:232
621899693,47cyclictest9035-21fschecks_count21:51:442
621899693,40cyclictest20032-21timerwakeupswit21:12:452
621899693,40cyclictest20032-21timerwakeupswit21:12:452
6214996953,10cyclictest402-21ntp_offset22:31:521
621499694,51cyclictest21339-21mailstats22:11:591
6214996945,13cyclictest0-21swapper/122:56:521
6214996930,26cyclictest30348-21grep23:21:571
621499692,47cyclictest6790-21wc21:46:551
621499692,47cyclictest6790-21wc21:46:551
622499683,50cyclictest9324-21chrt21:51:483
6218996849,14cyclictest6198-21cyclictest19:26:392
6218996844,16cyclictest27966-21systemd-journal23:44:502
6218996844,15cyclictest0-21swapper/221:21:252
6218996843,14cyclictest0-21swapper/221:46:262
621899683,46cyclictest15569-21cat21:06:262
621899683,46cyclictest15569-21cat21:06:262
6218996815,17cyclictest30275-21latency_hist21:31:242
621499687,45cyclictest11905-21fschecks_time22:51:461
6214996849,12cyclictest356-21runrttasks23:18:191
6214996846,12cyclictest0-21swapper/122:01:491
621499684,47cyclictest30810-21apt-get20:16:271
6214996843,9cyclictest31-21ksoftirqd/121:04:541
6214996843,9cyclictest31-21ksoftirqd/121:04:541
6214996836,9cyclictest0-21swapper/119:16:311
6214996834,9cyclictest20160-21ssh22:11:121
6214996834,9cyclictest20160-21ssh22:11:121
6214996827,14cyclictest181rcu_preempt22:26:511
6214996827,14cyclictest181rcu_preempt22:26:511
622499673,48cyclictest10949-21chrt20:51:513
622499672,52cyclictest18836-21taskset21:11:323
622499672,52cyclictest18836-21taskset21:11:323
622499672,50cyclictest6929-21taskset20:41:223
622499672,50cyclictest23219-21taskset00:07:223
622499672,50cyclictest23219-21taskset00:07:223
6218996749,12cyclictest376-21in:imuxsock19:06:242
621899674,41cyclictest8810-21fschecks_count22:46:462
621899674,41cyclictest8810-21fschecks_count22:46:462
621899674,41cyclictest356-21runrttasks20:00:372
621899674,37cyclictest11458-21latency_hist19:16:242
6218996722,32cyclictest29416-21latency_hist20:11:262
6218996719,11cyclictest0-21swapper/223:32:302
6214996745,16cyclictest1805-21threads21:36:561
6214996745,16cyclictest1805-21threads21:36:561
621499673,44cyclictest27970-21irqstats21:26:491
622499663,49cyclictest27133-21taskset23:18:073
6218996640,14cyclictest0-21swapper/219:41:462
6218996640,14cyclictest0-21swapper/219:41:462
6218996633,23cyclictest0-21swapper/220:51:442
621899662,42cyclictest26489-21fschecks_time20:01:452
6218996614,12cyclictest12432-21diskmemload00:07:272
6218996614,12cyclictest12432-21diskmemload00:07:272
6214996646,14cyclictest364-21ntpd21:21:531
6214996646,14cyclictest364-21ntpd21:21:531
6214996646,12cyclictest0-21swapper/120:41:581
6214996643,14cyclictest5535-21latency_hist21:46:241
621499663,48cyclictest17467-21ssh23:01:391
621499663,48cyclictest17467-21ssh23:01:391
621499663,45cyclictest356-21runrttasks19:26:171
621499663,43cyclictest6652-21fschecks_time19:01:451
621499663,40cyclictest5668-21cstates20:36:401
264832660,4sleep2371rcuc/200:14:262
622499653,49cyclictest486-21chrt00:26:593
622499653,49cyclictest486-21chrt00:26:593
622499653,48cyclictest30320-21taskset23:21:563
622499653,48cyclictest0-21swapper/322:21:343
622499653,48cyclictest0-21swapper/322:21:343
622499653,46cyclictest23231-21chrt23:11:313
622499653,46cyclictest23231-21chrt23:11:313
6218996550,9cyclictest0-21swapper/223:26:572
6218996538,15cyclictest0-21swapper/220:36:392
6218996534,9cyclictest181rcu_preempt19:36:572
6218996534,9cyclictest181rcu_preempt19:36:572
621899653,43cyclictest16528-21acpi19:31:332
6218996510,13cyclictest181rcu_preempt19:16:522
621499658,42cyclictest3280-21ntp_kernel_pll_23:31:521
6214996545,15cyclictest22355-21idleruntime00:06:481
6214996545,15cyclictest22355-21idleruntime00:06:481
621499654,42cyclictest356-21runrttasks20:09:211
621499654,42cyclictest23571-21wc19:51:521
621499654,42cyclictest23571-21wc19:51:521
6214996544,12cyclictest0-21swapper/119:16:251
621499653,41cyclictest726-21fschecks_time20:21:441
6224996435,21cyclictest0-21swapper/323:51:223
622499643,49cyclictest9113-21chrt23:41:503
622499643,49cyclictest17071-21chrt19:31:523
622499643,48cyclictest20723-21chrt19:42:003
622499643,48cyclictest20723-21chrt19:42:003
622499643,48cyclictest19225-21chrt19:38:123
622499643,48cyclictest19225-21chrt19:38:123
622499643,48cyclictest11053-21chrt19:11:593
622499643,47cyclictest2198-21chrt20:26:403
622499643,47cyclictest16713-21chrt23:56:243
622499643,46cyclictest31677-21chrt21:32:163
622499643,46cyclictest31677-21chrt21:32:163
622499643,46cyclictest13454-21taskset19:21:433
622499642,49cyclictest3280-21chrt21:40:343
622499642,49cyclictest3280-21chrt21:40:343
622499642,49cyclictest29280-21chrt22:26:333
622499642,49cyclictest29280-21chrt22:26:333
622499642,49cyclictest19422-21chrt22:09:033
622499642,49cyclictest19422-21chrt22:09:033
622499642,48cyclictest5948-21chrt22:41:453
622499642,48cyclictest27714-21chrt00:16:463
622499642,48cyclictest27382-21taskset20:04:203
622499642,48cyclictest24314-21chrt21:21:213
622499642,47cyclictest9645-21taskset19:09:453
622499642,47cyclictest8384-21chrt20:44:423
622499642,47cyclictest28684-21chrt21:27:053
621899643,40cyclictest2413-21ssh22:36:212
621499644,38cyclictest21718-21/usr/sbin/munin21:16:511
621499642,40cyclictest16380-21ssh23:55:441
622499633,47cyclictest14863-21chrt22:01:413
622499632,49cyclictest24619-21chrt19:56:383
622499632,49cyclictest14736-21taskset22:56:443
622499632,48cyclictest9212-21taskset22:46:553
622499632,48cyclictest9212-21taskset22:46:553
622499632,48cyclictest485-21chrt20:21:373
622499632,48cyclictest18144-21taskset23:57:343
622499632,48cyclictest18144-21taskset23:57:343
622499632,48cyclictest1067-21taskset22:32:293
622499632,46cyclictest22004-21chrt19:46:553
6218996338,15cyclictest0-21swapper/219:56:222
6218996338,15cyclictest0-21swapper/219:56:222
6214996325,12cyclictest181rcu_preempt19:31:221
621499632,43cyclictest27023-21timerandwakeup20:01:581
622499623,47cyclictest13290-21chrt21:01:473
622499623,47cyclictest13290-21chrt21:01:473
622499623,47cyclictest12278-21chrt21:56:513
622499623,47cyclictest12278-21chrt21:56:513
622499623,46cyclictest14910-21chrt19:26:373
622499623,44cyclictest0-21swapper/322:36:373
622499622,48cyclictest7270-21taskset23:38:303
622499622,48cyclictest27057-21chrt21:25:513
622499622,48cyclictest27057-21chrt21:25:513
622499622,48cyclictest22948-21chrt22:15:433
622499622,47cyclictest5146-21chrt23:35:543
622499622,47cyclictest4009-21chrt21:42:003
622499622,46cyclictest32229-21chrt23:26:383
622499622,45cyclictest6988-21taskset21:47:063
622499622,45cyclictest6988-21taskset21:47:063
6218996232,18cyclictest0-21swapper/219:46:542
621899622,44cyclictest1177-21fschecks_count21:36:422
621899622,44cyclictest1177-21fschecks_count21:36:422
622499614,44cyclictest9080-21chrt20:46:453
622499614,44cyclictest9080-21chrt20:46:453
622499614,41cyclictest0-21swapper/320:06:383
622499613,46cyclictest0-21swapper/321:06:353
622499613,45cyclictest19788-21chrt23:05:433
622499613,45cyclictest19788-21chrt23:05:433
622499612,47cyclictest25963-21taskset00:12:333
622499612,47cyclictest12814-21taskset22:52:223
622499612,46cyclictest32041-21chrt20:17:533
622499612,46cyclictest32041-21chrt20:17:533
622499612,46cyclictest30532-21taskset20:12:593
622499612,46cyclictest20324-21chrt23:06:323
622499612,46cyclictest12715-21chrt19:17:303
621899614,26cyclictest14728-21grep22:01:392
6218996137,14cyclictest0-21swapper/222:36:552
622499602,46cyclictest24302-21chrt19:56:023
622499602,46cyclictest24302-21chrt19:56:023
622499602,46cyclictest12216-21chrt20:59:003
622499602,45cyclictest7453-21chrt19:02:523
622499602,45cyclictest21163-21chrt00:04:313
621899603,43cyclictest12423-21munin-run21:01:222
621499602,36cyclictest356-21runrttasks21:00:401
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional