You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-27 - 04:39

x86 Intel Core i7-3770K @3500 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #0, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a4-7 -p99 -i200 -h400 -t4 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack0slot2.osadl.org (updated Mon Jun 27, 2022 00:43:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
266966626051,6sleep40-21swapper/419:09:354
266962125850,5sleep60-21swapper/619:08:576
266965824232,6sleep50-21swapper/519:09:285
2669996992914,15cyclictest0-21swapper/700:23:127
266959522011,6sleep70-21swapper/719:08:347
266999399170,1cyclictest0-21swapper/623:35:186
266999099160,15cyclictest0-21swapper/523:10:175
2669996991511,3cyclictest2787730-21ssh21:56:227
2669990991515,0cyclictest0-21swapper/523:22:145
2669990991515,0cyclictest0-21swapper/500:21:215
2669990991513,1cyclictest2762750-21ntpq21:30:185
2669990991513,1cyclictest0-21swapper/519:45:185
266999699140,13cyclictest2806009-21ssh22:16:057
266999399140,14cyclictest0-21swapper/622:17:456
2669990991414,0cyclictest0-21swapper/523:54:195
2669990991414,0cyclictest0-21swapper/523:38:415
2669990991414,0cyclictest0-21swapper/500:32:455
266999099140,13cyclictest0-21swapper/522:45:175
2669996991313,0cyclictest0-21swapper/723:23:347
2669996991313,0cyclictest0-21swapper/721:15:237
2669996991313,0cyclictest0-21swapper/700:25:227
266999699130,13cyclictest2778102-21ssh21:45:307
266999699130,13cyclictest0-21swapper/722:05:127
266999699130,0cyclictest0-21swapper/723:15:227
266999399130,13cyclictest0-21swapper/623:55:116
266999399130,0cyclictest0-21swapper/623:53:086
266999099139,3cyclictest2675106-21cat19:15:145
2669990991313,0cyclictest0-21swapper/500:10:015
2669990991312,1cyclictest0-21swapper/500:36:225
266999099130,0cyclictest0-21swapper/523:30:175
2669986991313,0cyclictest0-21swapper/423:53:494
2669986991313,0cyclictest0-21swapper/400:16:504
2669986991313,0cyclictest0-21swapper/400:00:174
266998699130,13cyclictest0-21swapper/421:22:564
2669996991212,0cyclictest0-21swapper/723:13:197
2669996991211,1cyclictest1010-21snmpd23:26:037
2669996991210,1cyclictest17506-21gnome-shell21:41:057
266999699120,12cyclictest0-21swapper/721:22:297
266999699120,12cyclictest0-21swapper/700:18:357
266999699120,0cyclictest0-21swapper/723:59:217
2669993991212,0cyclictest0-21swapper/623:21:276
2669993991212,0cyclictest0-21swapper/623:07:066
2669993991212,0cyclictest0-21swapper/622:49:456
266999399120,12cyclictest0-21swapper/623:32:016
266999399120,12cyclictest0-21swapper/622:58:356
266999399120,12cyclictest0-21swapper/600:04:396
266999399120,0cyclictest0-21swapper/623:42:346
266999099124,8cyclictest0-21swapper/522:51:095
266999099120,0cyclictest0-21swapper/522:25:585
266999099120,0cyclictest0-21swapper/522:25:585
2669986991212,0cyclictest0-21swapper/423:35:044
2669986991212,0cyclictest0-21swapper/423:08:444
2669986991212,0cyclictest0-21swapper/421:56:314
2669986991212,0cyclictest0-21swapper/421:39:114
2669986991212,0cyclictest0-21swapper/400:35:084
2669986991212,0cyclictest0-21swapper/400:25:594
266998699120,12cyclictest0-21swapper/423:55:524
266998699120,0cyclictest0-21swapper/422:47:174
266998699120,0cyclictest0-21swapper/422:39:054
266999699119,1cyclictest1010-21snmpd22:13:327
2669996991111,0cyclictest0-21swapper/723:35:517
2669996991111,0cyclictest0-21swapper/722:47:467
2669996991111,0cyclictest0-21swapper/719:16:497
266999699110,0cyclictest0-21swapper/722:22:337
2669993991111,0cyclictest0-21swapper/623:17:216
2669993991111,0cyclictest0-21swapper/623:13:156
2669993991111,0cyclictest0-21swapper/623:00:386
2669993991111,0cyclictest0-21swapper/622:32:446
2669993991111,0cyclictest0-21swapper/622:24:136
2669993991111,0cyclictest0-21swapper/621:21:096
2669993991111,0cyclictest0-21swapper/600:35:046
2669993991111,0cyclictest0-21swapper/600:28:186
2669993991111,0cyclictest0-21swapper/600:11:446
2669993991111,0cyclictest0-21swapper/600:09:236
266999399110,11cyclictest0-21swapper/622:54:106
266999399110,11cyclictest0-21swapper/620:02:196
2669990991111,0cyclictest0-21swapper/523:16:445
2669990991111,0cyclictest0-21swapper/522:56:055
2669990991111,0cyclictest0-21swapper/522:21:145
266999099110,11cyclictest0-21swapper/521:52:325
266999099110,0cyclictest0-21swapper/523:07:545
266999099110,0cyclictest0-21swapper/521:12:115
266999099110,0cyclictest0-21swapper/519:40:355
2669986991111,0cyclictest0-21swapper/422:58:104
2669986991111,0cyclictest0-21swapper/422:09:464
2669986991111,0cyclictest0-21swapper/400:05:014
266999699109,1cyclictest2844031-21ssh22:58:397
266999699108,2cyclictest1010-21snmpd22:34:117
266999699108,1cyclictest1010-21snmpd23:46:557
266999699108,1cyclictest1010-21snmpd22:02:347
266999699108,1cyclictest1010-21snmpd21:51:227
266999699108,1cyclictest1010-21snmpd21:35:347
2669996991010,0cyclictest0-21swapper/723:07:107
2669996991010,0cyclictest0-21swapper/723:03:047
2669996991010,0cyclictest0-21swapper/722:41:187
2669996991010,0cyclictest0-21swapper/722:26:397
2669996991010,0cyclictest0-21swapper/722:26:397
2669996991010,0cyclictest0-21swapper/719:41:157
2669993991010,0cyclictest0-21swapper/622:26:166
2669993991010,0cyclictest0-21swapper/622:26:166
2669993991010,0cyclictest0-21swapper/622:08:556
2669993991010,0cyclictest0-21swapper/621:40:046
2669993991010,0cyclictest0-21swapper/621:16:256
2669993991010,0cyclictest0-21swapper/619:18:586
266999399100,10cyclictest0-21swapper/619:23:236
266999099103,5cyclictest54-21ksoftirqd/522:14:245
266999099102,8cyclictest0-21swapper/500:15:125
2669990991010,0cyclictest0-21swapper/523:55:165
2669990991010,0cyclictest0-21swapper/523:45:225
2669990991010,0cyclictest0-21swapper/522:44:155
2669990991010,0cyclictest0-21swapper/522:35:065
2669990991010,0cyclictest0-21swapper/500:05:465
266999099100,10cyclictest0-21swapper/519:29:045
266998699107,1cyclictest0-21swapper/400:14:294
266998699106,3cyclictest2781618-21cat21:50:144
2669986991010,0cyclictest0-21swapper/421:41:144
2669986991010,0cyclictest0-21swapper/419:23:264
26699969999,0cyclictest0-21swapper/723:33:487
26699969998,1cyclictest1010-21snmpd22:51:117
26699969998,1cyclictest1010-21snmpd19:45:387
26699969997,1cyclictest1010-21snmpd23:50:167
26699969997,1cyclictest1010-21snmpd23:43:287
26699969997,1cyclictest1010-21snmpd22:39:597
26699969997,1cyclictest1010-21snmpd21:33:287
26699969997,1cyclictest1010-21snmpd19:24:017
26699969997,1cyclictest1010-21snmpd00:31:417
26699969997,1cyclictest1010-21snmpd00:11:017
26699969997,1cyclictest1010-21snmpd00:05:557
26699939999,0cyclictest0-21swapper/622:13:206
26699939997,1cyclictest2767757-21ntp_states21:35:206
26699939996,2cyclictest2704434-21cat20:05:136
26699909999,0cyclictest0-21swapper/522:04:415
26699909999,0cyclictest0-21swapper/500:03:055
26699909994,1cyclictest0-21swapper/522:31:095
26699869999,0cyclictest0-21swapper/423:30:584
26699869999,0cyclictest0-21swapper/422:22:234
26699869999,0cyclictest0-21swapper/421:47:424
26699869999,0cyclictest0-21swapper/421:18:134
26699869999,0cyclictest0-21swapper/420:00:014
26699869991,7cyclictest0-21swapper/419:45:144
26699969988,0cyclictest0-21swapper/700:39:147
26699969986,1cyclictest1010-21snmpd21:00:457
26699969986,1cyclictest1010-21snmpd20:06:387
26699969986,1cyclictest1010-21snmpd19:26:407
26699969986,1cyclictest1010-21snmpd00:02:287
26699969984,2cyclictest793-21systemd-logind20:30:007
26699939988,0cyclictest0-21swapper/622:36:506
26699939988,0cyclictest0-21swapper/622:04:306
26699939988,0cyclictest0-21swapper/619:27:486
26699939988,0cyclictest0-21swapper/600:21:126
26699939988,0cyclictest0-21swapper/600:18:506
26699909988,0cyclictest0-21swapper/523:41:295
26699909988,0cyclictest0-21swapper/523:25:105
26699909988,0cyclictest0-21swapper/520:00:165
26699909984,4cyclictest54-21ksoftirqd/521:20:115
26699909984,3cyclictest54-21ksoftirqd/521:40:165
26699909983,4cyclictest54-21ksoftirqd/521:15:075
26699909980,7cyclictest0-21swapper/522:15:305
26699909980,7cyclictest0-21swapper/521:55:135
26699909980,2cyclictest2846524-21tr23:00:195
26699869988,0cyclictest0-21swapper/423:45:194
26699869988,0cyclictest0-21swapper/423:00:134
26699869980,7cyclictest0-21swapper/421:10:134
26699969973,2cyclictest17584-10wireplumber20:02:107
26699969972,3cyclictest910-21lldpd21:10:177
26699939977,0cyclictest0-21swapper/623:46:406
26699939977,0cyclictest0-21swapper/623:25:336
26699939977,0cyclictest0-21swapper/621:48:546
26699909974,3cyclictest54-21ksoftirqd/519:20:115
26699909974,3cyclictest54-21ksoftirqd/500:25:285
26699909970,6cyclictest2776234-21grep21:45:115
26699909970,6cyclictest2705235-21sed20:05:215
26699909970,6cyclictest0-21swapper/522:05:215
26699909970,6cyclictest0-21swapper/521:35:195
26699869977,0cyclictest0-21swapper/423:41:134
26699869977,0cyclictest0-21swapper/422:53:454
26699869977,0cyclictest0-21swapper/422:11:494
26699869977,0cyclictest0-21swapper/419:27:514
26699869974,2cyclictest2675105-21cat19:15:144
26699969963,2cyclictest806-21in:imjournal20:45:007
26699969960,3cyclictest553-21systemd-journal20:55:007
26699969960,3cyclictest553-21systemd-journal20:40:007
26699969960,3cyclictest553-21systemd-journal19:55:007
26699939966,0cyclictest0-21swapper/621:30:376
26699939964,1cyclictest2929273-21ntpq00:30:166
26699939962,3cyclictest2781811-21cat21:50:156
26699939962,3cyclictest2738176-21date21:04:596
26699939960,5cyclictest2690808-21taskset19:41:236
26699939960,1cyclictest2827278-21diskstats22:40:126
26699909964,1cyclictest54-21ksoftirqd/520:15:005
26699909964,1cyclictest54-21ksoftirqd/520:15:005
26699869966,0cyclictest0-21swapper/423:16:374
26699869966,0cyclictest0-21swapper/422:26:294
26699869966,0cyclictest0-21swapper/422:26:294
26699869966,0cyclictest0-21swapper/420:06:474
26699869962,3cyclictest2918780-21/usr/sbin/munin00:20:204
26699869960,3cyclictest151rcu_preempt23:10:114
26699869960,1cyclictest0-21swapper/422:15:264
26699969950,5cyclictest0-21swapper/721:00:007
26699969950,5cyclictest0-21swapper/720:50:007
26699969950,5cyclictest0-21swapper/719:35:007
26699969950,2cyclictest553-21systemd-journal20:35:007
26699969950,2cyclictest553-21systemd-journal20:15:007
26699969950,2cyclictest553-21systemd-journal20:15:007
26699939953,1cyclictest2742295-21grep21:10:116
26699939951,3cyclictest2786624-21ntpq21:55:176
26699939951,2cyclictest17515-21llvmpipe-319:48:586
26699909955,0cyclictest0-21swapper/521:00:105
26699909951,3cyclictest2732750-21cut20:55:005
26699869952,2cyclictest2865290-21wc23:20:204
26699869952,2cyclictest2826878-21cpuspeed_turbos22:40:114
26699869952,2cyclictest2817658-21grep22:30:104
26699869952,2cyclictest2736467-21fschecks_time21:00:124
26699869952,2cyclictest2688955-21cpuspeed19:40:094
26699869951,3cyclictest2868945-21cut23:25:184
26699869951,3cyclictest2735672-21date21:00:004
26699869950,5cyclictest0-21swapper/422:01:234
26699869950,5cyclictest0-21swapper/421:30:154
26699869950,5cyclictest0-21swapper/400:30:034
26699869950,2cyclictest2709009-21idleruntime-cro20:15:004
26699869950,2cyclictest2709009-21idleruntime-cro20:15:004
26699969940,3cyclictest2760671-21ssh21:29:597
26699969940,3cyclictest2711596-21cron20:20:007
26699969940,3cyclictest2688167-21sh19:40:007
26699939944,0cyclictest0-21swapper/620:55:006
26699939944,0cyclictest0-21swapper/620:30:006
26699939940,3cyclictest2761244-21cat21:29:596
26699939940,3cyclictest2741606-21latency_hist21:09:596
26699939940,3cyclictest2735384-21crond21:00:006
26699939940,3cyclictest2729533-21dump-pmu-power20:50:006
26699939940,3cyclictest2726801-21latency_hist20:45:006
26699939940,3cyclictest2723678-21grep20:40:006
26699939940,3cyclictest2720736-21crond20:35:006
26699939940,3cyclictest2711879-21/usr/sbin/munin20:20:006
26699939940,3cyclictest2709012-21turbostat20:15:006
26699939940,3cyclictest2709012-21turbostat20:15:006
26699939940,3cyclictest2700427-21cut20:00:006
26699939940,3cyclictest2697294-21munin-plugin-st19:55:006
26699939940,3cyclictest2688767-21sed19:40:006
26699939940,3cyclictest2685596-21true19:35:006
26699939940,3cyclictest2673270-21latency_hist19:15:006
26699909940,4cyclictest0-21swapper/520:35:005
26699909940,4cyclictest0-21swapper/520:29:595
26699909940,4cyclictest0-21swapper/520:20:005
26699909940,3cyclictest2761267-21sed21:29:595
26699909940,3cyclictest2740919-21latency_hist21:09:595
26699909940,3cyclictest2735522-21latency_hist21:00:005
26699909940,3cyclictest2729519-21crond20:50:005
26699909940,3cyclictest2726784-21cat20:45:005
26699909940,3cyclictest2723668-21turbostat.cron20:40:005
26699909940,3cyclictest2715230-21latency_hist20:25:005
26699909940,3cyclictest2700314-21cut20:00:005
26699909940,3cyclictest2697295-21idleruntime-cro19:55:005
26699909940,3cyclictest2685594-21/usr/sbin/munin19:35:005
26699909940,3cyclictest2673249-21cat19:15:005
26699869940,3cyclictest2760744-21latency_hist21:29:594
26699869940,3cyclictest2741620-21cat21:09:594
26699869940,3cyclictest2729532-21ls20:50:004
26699869940,3cyclictest2726798-21latency_hist20:45:004
26699869940,3cyclictest2723664-21crond20:40:004
26699869940,3cyclictest2720749-21dump-pmu-power20:35:004
26699869940,3cyclictest2717808-21crond20:30:004
26699869940,3cyclictest2715215-21date20:25:004
26699869940,3cyclictest2711981-21systemd-run20:20:004
26699869940,3cyclictest2700347-21latency_hist20:00:004
26699869940,3cyclictest2697284-21sh19:55:004
26699869940,3cyclictest2688759-21latency_hist19:40:004
26699869940,3cyclictest2685615-21cut19:35:004
26699869940,3cyclictest2673389-21date19:15:004
26699869940,2cyclictest2732466-21grep20:55:004
26699969930,2cyclictest2741580-21latency_hist21:09:597
26699969930,2cyclictest2715198-21latency_hist20:25:007
26699969930,2cyclictest2700313-21latency_hist20:00:007
26699969930,2cyclictest2673193-21latency_hist19:15:007
26699939930,2cyclictest2715201-21sort20:25:006
26699909930,2cyclictest2688770-21date19:40:005
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional