You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-26 - 13:09

x86 Intel Core i7-3770K @3500 MHz, Linux 5.15.32-rt39 (Profile)

Latency plot of system in rack #0, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a4-7 -p99 -i200 -h400 -t4 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack0slot2.osadl.org (updated Sun Jun 26, 2022 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
326565126149,7sleep40-21swapper/419:05:004
326670425950,6sleep60-21swapper/619:05:426
326680923527,5sleep50-21swapper/519:07:015
326591723525,6sleep70-21swapper/719:05:037
326728099150,14cyclictest3381929-21ssh21:52:426
3267277991512,3cyclictest0-21swapper/523:16:545
3267284991413,1cyclictest0-21swapper/719:34:467
326727799130,13cyclictest0-21swapper/520:14:585
326728499129,2cyclictest3359869-21cat21:30:007
3267284991212,0cyclictest3500368-21ssh23:52:317
3267284991212,0cyclictest3500368-21ssh23:52:317
3267284991212,0cyclictest0-21swapper/723:37:427
3267284991212,0cyclictest0-21swapper/723:03:487
3267284991212,0cyclictest0-21swapper/722:58:557
3267284991212,0cyclictest0-21swapper/722:31:207
3267284991212,0cyclictest0-21swapper/722:25:587
3267284991212,0cyclictest0-21swapper/721:50:587
3267284991212,0cyclictest0-21swapper/721:35:417
3267284991212,0cyclictest0-21swapper/721:18:307
3267280991212,0cyclictest0-21swapper/623:01:536
3267280991212,0cyclictest0-21swapper/621:10:076
326728099120,12cyclictest0-21swapper/623:11:406
326728099120,12cyclictest0-21swapper/621:38:106
326728099120,12cyclictest0-21swapper/621:31:526
3267277991212,0cyclictest0-21swapper/522:46:005
3267274991212,0cyclictest0-21swapper/423:11:374
3267274991212,0cyclictest0-21swapper/422:31:344
3267274991212,0cyclictest0-21swapper/419:20:024
3267274991212,0cyclictest0-21swapper/419:20:024
326727499120,12cyclictest0-21swapper/420:35:134
326727499120,0cyclictest0-21swapper/422:59:094
3267284991111,0cyclictest0-21swapper/723:11:227
3267284991111,0cyclictest0-21swapper/720:11:307
326728499110,11cyclictest0-21swapper/720:56:257
3267280991111,0cyclictest0-21swapper/623:19:146
3267280991111,0cyclictest0-21swapper/622:54:486
3267280991111,0cyclictest0-21swapper/622:26:156
3267277991111,0cyclictest0-21swapper/522:58:465
3267277991111,0cyclictest0-21swapper/522:38:545
3267274991111,0cyclictest0-21swapper/421:50:444
326727499110,11cyclictest0-21swapper/420:55:154
326727499110,11cyclictest0-21swapper/420:12:314
326728499109,1cyclictest1017-21snmpd20:20:587
326728499108,1cyclictest3275011-21df19:19:577
326728499108,1cyclictest3275011-21df19:19:577
326728499108,1cyclictest1017-21snmpd23:17:297
326728499108,1cyclictest1017-21snmpd21:13:157
326728499104,2cyclictest19997-10wireplumber20:50:007
3267284991010,0cyclictest0-21swapper/723:32:207
3267284991010,0cyclictest0-21swapper/720:33:437
326728499100,10cyclictest0-21swapper/720:36:247
3267280991010,0cyclictest0-21swapper/620:28:586
3267280991010,0cyclictest0-21swapper/620:23:086
326728099100,10cyclictest0-21swapper/620:55:186
326727799109,1cyclictest0-21swapper/523:51:065
326727799109,1cyclictest0-21swapper/523:51:065
3267277991010,0cyclictest0-21swapper/523:38:115
3267277991010,0cyclictest0-21swapper/522:43:475
3267277991010,0cyclictest0-21swapper/521:17:435
326727799100,10cyclictest0-21swapper/520:55:195
326727799100,0cyclictest0-21swapper/520:12:465
3267274991010,0cyclictest0-21swapper/423:29:544
3267274991010,0cyclictest0-21swapper/420:29:524
326727499100,0cyclictest0-21swapper/420:50:214
32672849996,2cyclictest1017-21snmpd22:42:197
32672809999,0cyclictest0-21swapper/623:52:206
32672809999,0cyclictest0-21swapper/623:52:206
32672809999,0cyclictest0-21swapper/622:40:086
32672809999,0cyclictest0-21swapper/620:50:246
32672809990,9cyclictest0-21swapper/620:31:546
32672779999,0cyclictest0-21swapper/523:31:055
32672779999,0cyclictest0-21swapper/522:34:015
32672779999,0cyclictest0-21swapper/521:14:335
32672779999,0cyclictest0-21swapper/520:38:565
32672749999,0cyclictest0-21swapper/423:19:114
32672849986,1cyclictest1017-21snmpd20:19:017
32672849984,2cyclictest19997-10wireplumber22:44:587
32672849984,2cyclictest19997-10wireplumber22:35:017
32672809988,0cyclictest0-21swapper/623:29:576
32672809988,0cyclictest0-21swapper/622:47:426
32672809988,0cyclictest0-21swapper/620:13:506
32672809988,0cyclictest0-21swapper/619:20:236
32672809988,0cyclictest0-21swapper/619:20:236
32672779988,0cyclictest0-21swapper/523:10:165
32672779988,0cyclictest0-21swapper/520:34:035
32672779983,5cyclictest52-21ksoftirqd/522:25:165
32672779982,1cyclictest0-21swapper/521:53:355
32672779982,1cyclictest0-21swapper/521:31:325
32672779980,8cyclictest0-21swapper/523:01:025
32672749988,0cyclictest0-21swapper/423:52:174
32672749988,0cyclictest0-21swapper/423:52:174
32672749988,0cyclictest0-21swapper/423:01:504
32672749988,0cyclictest0-21swapper/422:47:394
32672749988,0cyclictest0-21swapper/422:40:054
32672749988,0cyclictest0-21swapper/422:26:124
32672749988,0cyclictest0-21swapper/421:13:414
32672749988,0cyclictest0-21swapper/420:27:114
32672849975,2cyclictest3313557-21cat20:24:597
32672809977,0cyclictest0-21swapper/621:17:506
32672779977,0cyclictest0-21swapper/520:21:355
32672779973,4cyclictest52-21ksoftirqd/521:37:165
32672779973,3cyclictest3413168-21sort22:24:465
32672779970,6cyclictest3403189-21sed22:14:465
32672779970,6cyclictest0-21swapper/521:24:455
32672779970,5cyclictest0-21swapper/522:04:465
32672749977,0cyclictest0-21swapper/423:35:154
32672749977,0cyclictest0-21swapper/421:35:274
32672849963,2cyclictest805-21in:imjournal19:19:467
32672849963,2cyclictest775-21dbus-daemon22:14:467
32672849963,1cyclictest2786393-21systemd-journal22:09:467
32672849962,2cyclictest2786393-21systemd-journal21:49:457
32672849962,2cyclictest19997-10wireplumber00:19:457
32672849962,2cyclictest19997-10wireplumber00:14:457
32672849960,3cyclictest2786393-21systemd-journal23:24:457
32672849960,3cyclictest2786393-21systemd-journal22:24:467
32672849960,3cyclictest2786393-21systemd-journal21:24:457
32672849960,3cyclictest2786393-21systemd-journal21:09:457
32672849960,3cyclictest2786393-21systemd-journal19:14:467
32672849960,3cyclictest2786393-21systemd-journal00:34:457
32672849960,2cyclictest3506823-40updatedb23:59:457
32672809966,0cyclictest0-21swapper/623:37:596
32672809966,0cyclictest0-21swapper/622:34:466
32672809966,0cyclictest0-21swapper/620:34:486
32672809960,5cyclictest0-21swapper/620:04:466
32672779966,0cyclictest0-21swapper/520:50:555
32672779966,0cyclictest0-21swapper/520:29:095
32672779964,1cyclictest3497031-21sed23:49:455
32672779963,1cyclictest795-21systemd-machine21:04:455
32672779963,1cyclictest0-21swapper/521:59:455
32672779962,3cyclictest52-21ksoftirqd/500:39:455
32672779960,6cyclictest0-21swapper/500:14:455
32672779960,1cyclictest3338950-21cat21:09:455
32672749962,3cyclictest3283113-21date19:34:464
32672849950,3cyclictest2786393-21systemd-journal23:44:457
32672849950,3cyclictest2786393-21systemd-journal23:29:457
32672849950,3cyclictest2786393-21systemd-journal23:09:457
32672849950,3cyclictest2786393-21systemd-journal22:19:457
32672849950,3cyclictest2786393-21systemd-journal21:29:457
32672849950,3cyclictest2786393-21systemd-journal21:04:457
32672849950,3cyclictest2786393-21systemd-journal20:49:457
32672849950,3cyclictest2786393-21systemd-journal20:44:457
32672849950,3cyclictest2786393-21systemd-journal00:39:457
32672849950,2cyclictest2786393-21systemd-journal22:04:467
32672849950,2cyclictest2786393-21systemd-journal21:59:457
32672849950,2cyclictest2786393-21systemd-journal21:44:457
32672849950,2cyclictest2786393-21systemd-journal21:44:457
32672849950,2cyclictest2786393-21systemd-journal20:09:457
32672849950,2cyclictest2786393-21systemd-journal20:04:467
32672849950,2cyclictest2786393-21systemd-journal19:54:467
32672849950,2cyclictest2786393-21systemd-journal19:49:457
32672849950,2cyclictest2786393-21systemd-journal19:39:467
32672849950,2cyclictest2786393-21systemd-journal19:29:467
32672849950,2cyclictest2786393-21systemd-journal19:29:467
32672849950,2cyclictest2786393-21systemd-journal00:29:457
32672849950,2cyclictest2786393-21systemd-journal00:09:457
32672849950,2cyclictest2786393-21systemd-journal00:04:457
32672809953,1cyclictest3353708-21latency_hist21:24:456
32672809952,1cyclictest751-17sedispatch21:59:456
32672809950,5cyclictest0-21swapper/623:59:456
32672809950,5cyclictest0-21swapper/623:44:456
32672809950,5cyclictest0-21swapper/623:24:456
32672809950,5cyclictest0-21swapper/622:29:466
32672809950,5cyclictest0-21swapper/622:19:456
32672809950,5cyclictest0-21swapper/622:14:466
32672809950,5cyclictest0-21swapper/622:04:466
32672809950,5cyclictest0-21swapper/621:04:456
32672809950,5cyclictest0-21swapper/620:44:456
32672809950,5cyclictest0-21swapper/620:14:466
32672809950,5cyclictest0-21swapper/620:09:466
32672809950,5cyclictest0-21swapper/619:59:466
32672809950,5cyclictest0-21swapper/619:54:466
32672809950,5cyclictest0-21swapper/619:49:456
32672809950,5cyclictest0-21swapper/619:34:466
32672809950,5cyclictest0-21swapper/619:29:466
32672809950,5cyclictest0-21swapper/619:29:466
32672809950,5cyclictest0-21swapper/619:14:466
32672809950,5cyclictest0-21swapper/600:34:456
32672809950,5cyclictest0-21swapper/600:29:456
32672809950,5cyclictest0-21swapper/600:24:456
32672809950,5cyclictest0-21swapper/600:19:456
32672809950,5cyclictest0-21swapper/600:14:456
32672809950,4cyclictest3373387-21munin-run21:44:456
32672809950,4cyclictest3373387-21munin-run21:44:456
32672809950,4cyclictest3285758-21/usr/sbin/munin19:39:466
32672809950,4cyclictest0-21swapper/623:09:456
32672779953,1cyclictest3511933-21latency_hist00:04:455
32672779953,1cyclictest3506779-21latency_hist23:59:455
32672779953,1cyclictest3373416-21sh21:44:455
32672779953,1cyclictest3373416-21sh21:44:455
32672779952,2cyclictest3280211-21cut19:29:465
32672779952,2cyclictest3280211-21cut19:29:465
32672779952,1cyclictest795-21systemd-machine20:09:465
32672779951,3cyclictest3536655-21date00:29:455
32672779950,5cyclictest0-21swapper/522:19:455
32672779950,5cyclictest0-21swapper/519:20:045
32672779950,5cyclictest0-21swapper/519:20:045
32672779950,1cyclictest3541358-21idleruntime-cro00:34:455
32672779950,1cyclictest3491912-21awk23:44:455
32672779950,1cyclictest0-21swapper/521:29:455
32672749955,0cyclictest0-21swapper/421:17:474
32672749953,2cyclictest3303497-21awk20:09:464
32672749953,1cyclictest3506722-21idleruntime-cro23:59:454
32672749953,1cyclictest3492154-21latency_hist23:44:454
32672749953,1cyclictest3324356-21sort20:44:454
32672749953,1cyclictest3270764-21latency_hist19:14:464
32672749952,2cyclictest3297912-21sed19:59:464
32672749951,3cyclictest3361594-21ssh21:31:344
32672749951,3cyclictest3335909-21cat21:04:454
32672749950,5cyclictest0-21swapper/422:35:504
32672749950,4cyclictest3457483-21date23:09:454
32672849940,3cyclictest3531105-21cron00:24:457
32672849940,2cyclictest749-17auditd19:59:467
32672849940,2cyclictest749-17auditd19:44:467
32672849940,2cyclictest2786393-21systemd-journal23:49:457
32672849940,2cyclictest2786393-21systemd-journal22:54:457
32672809942,1cyclictest3516314-21latency_hist00:09:456
32672809942,1cyclictest3339044-21latency_hist21:09:456
32672809941,2cyclictest3546258-21crond00:39:456
32672809940,3cyclictest3511796-21cat00:04:456
32672809940,3cyclictest3497063-21date23:49:456
32672809940,3cyclictest3477289-21cat23:29:456
32672809940,3cyclictest3412930-21cut22:24:466
32672809940,3cyclictest3398330-21latency_hist22:09:466
32672809940,3cyclictest3378240-21/usr/sbin/munin21:49:456
32672809940,3cyclictest3358619-21latency_hist21:29:456
32672809940,3cyclictest3327272-21latency_hist20:49:456
32672809940,3cyclictest3274227-21latency_hist19:19:466
32672779942,1cyclictest3531556-21latency_hist00:24:455
32672779940,4cyclictest0-21swapper/523:09:455
32672779940,3cyclictest3526515-21latency_hist00:19:455
32672779940,3cyclictest3516931-21cut00:09:455
32672779940,3cyclictest3477275-21latency_hist23:29:455
32672779940,3cyclictest3472130-21turbostat.cron23:24:455
32672779940,3cyclictest3442911-21cut22:54:455
32672779940,3cyclictest3398000-21rm22:09:465
32672779940,3cyclictest3378395-21cut21:49:455
32672779940,3cyclictest3327320-21cat20:49:455
32672779940,3cyclictest3324247-21sed20:44:455
32672779940,3cyclictest3300805-21cat20:04:465
32672779940,3cyclictest3297603-21perf19:59:465
32672779940,3cyclictest3294654-21crond19:54:465
32672779940,3cyclictest3291820-21cat19:49:465
32672779940,3cyclictest3286027-21sort19:39:465
32672779940,3cyclictest3282910-21sh19:34:465
32672779940,3cyclictest3274428-21latency_hist19:19:465
32672779940,2cyclictest795-21systemd-machine19:14:465
32672749942,1cyclictest3516950-21latency_hist00:09:454
32672749942,1cyclictest3511692-21idleruntime-cro00:04:454
32672749942,1cyclictest0-21swapper/419:29:464
32672749942,1cyclictest0-21swapper/419:29:464
32672749940,3cyclictest3546275-21basename00:39:454
32672749940,3cyclictest3541329-21sh00:34:454
32672749940,3cyclictest3536398-21awk00:29:454
32672749940,3cyclictest3531697-21cat00:24:454
32672749940,3cyclictest3526512-21sh00:19:454
32672749940,3cyclictest3521578-21turbostat00:14:454
32672749940,3cyclictest3497087-21date23:49:454
32672749940,3cyclictest3477414-21sed23:29:454
32672749940,3cyclictest3472137-21munin-plugin-st23:24:454
32672749940,3cyclictest3412914-21crond22:24:464
32672749940,3cyclictest3407962-21crond22:19:454
32672749940,3cyclictest3403029-21turbostat22:14:454
32672749940,3cyclictest3398423-21date22:09:464
32672749940,3cyclictest3393149-21munin-run22:04:454
32672749940,3cyclictest3388189-21crond21:59:454
32672749940,3cyclictest3378407-21latency_hist21:49:454
32672749940,3cyclictest3373376-21sh21:44:454
32672749940,3cyclictest3373376-21sh21:44:454
32672749940,3cyclictest3358610-21sed21:29:454
32672749940,3cyclictest3353581-21dump-pmu-power21:24:454
32672749940,3cyclictest3338921-21cat21:09:454
32672749940,3cyclictest3326637-21latency_hist20:49:454
32672749940,3cyclictest3309741-21chrt20:19:544
32672749940,3cyclictest3306798-21chrt20:14:504
32672749940,3cyclictest3300807-21latency_hist20:04:464
32672749940,3cyclictest3294663-21munin-plugin-st19:54:464
32672749940,3cyclictest3291715-21sh19:49:464
32672749940,3cyclictest3285880-21systemd-run19:39:464
32672749940,3cyclictest3274299-21latency_hist19:19:464
32672809930,2cyclictest3289129-21cat19:44:466
32672809930,1cyclictest0-21swapper/622:54:456
32672749930,2cyclictest3442913-21date22:54:454
32672749930,2cyclictest3289148-21cat19:44:464
32672779920,1cyclictest3289134-21cat19:44:465
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional