You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-07-29 - 16:36

Intel(R) Atom(TM) Processor E3950 @ 1.60GHz, Linux 4.19.182-rt74 (Profile)

Latency plot of system in rack #d, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -p90 -i200 -h400 -a0-2 -t3 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot8.osadl.org (updated Thu Feb 06, 2020 10:49:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
121028690315,23cyclictest1284161-21munin-run08:40:011
121028690315,23cyclictest1284161-21munin-run08:40:001
121028590315,13cyclictest1428017-21perf11:35:010
1210286902514,6cyclictest1424479-21ptp4l-jitter11:30:031
121028690251,21cyclictest393-21systemd-journal12:40:011
121028690251,21cyclictest1612-21systemd-logind07:40:001
1210285902419,3cyclictest1358156-21munin-run10:10:010
121028590241,2cyclictest111rcu_preempt11:15:000
121028590241,11cyclictest1304765-21latency_hist09:05:010
1210287902319,2cyclictest0-21swapper/211:20:012
1210286902313,2cyclictest1382833-21taskset10:40:001
121028690231,19cyclictest393-21systemd-journal12:20:021
121028690231,15cyclictest1432254-21/usr/sbin/munin11:40:101
121028590230,11cyclictest1214348-21perf07:15:010
1210287902214,4cyclictest1464003-21netstat12:15:142
121028690226,3cyclictest14186252missed_timers11:20:151
121028690226,3cyclictest14186252missed_timers11:20:141
1210286902211,9cyclictest21-21ksoftirqd/108:55:041
121028590220,20cyclictest393-21systemd-journal09:00:020
121028590220,20cyclictest393-21systemd-journal07:57:470
121028590220,19cyclictest393-21systemd-journal09:40:110
1210287902116,3cyclictest1324380-21proc_pri09:25:172
1210287902115,4cyclictest1393769-21irqstats10:50:152
1210287902115,4cyclictest1338635-21sed09:45:052
1210287902115,4cyclictest1338635-21sed09:45:042
1210287902113,5cyclictest1304191-21grep09:00:212
121028690219,7cyclictest1509-21runrttasks11:49:331
1210286902116,3cyclictest21-21ksoftirqd/110:00:011
1210286902116,3cyclictest1274246-21grep08:25:111
1210286902116,3cyclictest1247232-21cut07:55:011
1210286902115,4cyclictest21-21ksoftirqd/111:15:011
1210286902115,4cyclictest1264491-21head08:15:041
1210286902114,4cyclictest1272077-21acpi08:25:021
1210286902113,4cyclictest1367705-21df_abs10:20:051
121028690211,2cyclictest111rcu_preempt10:25:181
121028690211,18cyclictest393-21systemd-journal09:44:441
1210286902110,7cyclictest1308827-21diskmemload09:30:091
121028690210,6cyclictest21-21ksoftirqd/111:25:081
121028590219,4cyclictest1431684-21unixbench_singl11:35:220
121028590219,4cyclictest1431684-21unixbench_singl11:35:210
121028590211,9cyclictest1391025-21munin-run10:50:010
1210285902115,4cyclictest1275645-21sshd08:26:360
121028590211,10cyclictest1478123-21acpi12:35:030
121028590210,9cyclictest13645972iostat10:15:100
121028590210,19cyclictest393-21systemd-journal10:25:000
121028590210,12cyclictest1465095-21systemd-udevd12:19:560
1210287902014,4cyclictest1274245-21unixbench_singl08:25:112
1210287902014,4cyclictest1229973-21sendmail_mailtr07:30:172
1210287902014,3cyclictest1358597-21open_files10:10:032
1210287902014,3cyclictest1297147-21cpu08:55:042
1210286902016,2cyclictest21-21ksoftirqd/110:05:011
1210286902015,3cyclictest21-21ksoftirqd/110:50:021
1210286902015,3cyclictest1465243-21/usr/sbin/munin12:20:051
1210286902015,3cyclictest1363371-21grep10:15:051
1210286902014,4cyclictest1330260-21cstates09:35:041
1210286902014,4cyclictest1254281-21wc08:00:161
1210286902014,4cyclictest1254281-21wc08:00:151
1210286902014,3cyclictest1262082-21wc08:10:121
1210286902013,4cyclictest1449625-21head12:00:041
1210286902013,4cyclictest1241001-21ls07:45:101
121028690201,16cyclictest1612-21systemd-logind11:00:001
121028690201,16cyclictest1612-21systemd-logind10:10:001
121028590203,15cyclictest1234850-21perf07:40:010
121028590201,16cyclictest1612-21systemd-logind12:00:010
121028590200,17cyclictest393-21systemd-journal07:25:010
121028790198,8cyclictest1449622-21cpuspeed_turbos12:00:052
1210287901914,3cyclictest1476702-21sendmail_mailst12:30:182
1210287901914,3cyclictest1308076-21sendmail_mailtr09:05:212
1210287901914,3cyclictest1308076-21sendmail_mailtr09:05:202
1210287901914,3cyclictest1244693-21cpuspeed07:50:082
1210287901913,4cyclictest1478778-21hwlatdetect12:35:052
1210287901913,4cyclictest1390177-21processes10:45:172
1210287901913,3cyclictest1409376-21grep11:10:092
1210287901912,4cyclictest1289707-21diskstats08:45:062
1210287901912,4cyclictest1281357-21cpuspeed_turbos08:35:042
1210287901912,4cyclictest1281357-21cpuspeed_turbos08:35:042
121028790190,8cyclictest0-21swapper/209:12:022
121028690198,6cyclictest157250irq/133-enp2s0-08:48:031
1210286901914,3cyclictest1457965-21wc12:10:061
1210286901914,3cyclictest1457965-21wc12:10:051
1210286901914,3cyclictest1448111-21grep11:55:211
1210286901914,3cyclictest1358241-21/usr/sbin/munin10:10:061
1210286901914,3cyclictest1235194-21/usr/sbin/munin07:40:161
1210286901913,4cyclictest1473436-21date12:30:011
1210286901913,4cyclictest1402447-21which11:00:171
1210286901913,4cyclictest1402447-21which11:00:161
1210286901913,4cyclictest1374714-21/usr/sbin/munin10:30:111
1210286901913,4cyclictest1324728-21sed09:25:201
1210286901913,3cyclictest1475524-21awk12:30:111
1210286901912,4cyclictest1441527-21head11:50:051
1210286901912,4cyclictest1339788-21sh09:45:111
1210286901912,4cyclictest1339788-21sh09:45:101
1210286901912,4cyclictest1280176-21cut08:35:011
121028690190,2cyclictest111rcu_preempt07:15:031
121028690190,16cyclictest393-21systemd-journal12:08:391
121028590197,7cyclictest412-21systemd-udevd12:20:120
1210285901915,2cyclictest1395119-21crond10:55:010
1210285901914,3cyclictest1452150-21ls12:00:200
1210285901914,3cyclictest1359706-21cut10:10:080
1210285901914,3cyclictest1329368-21sh09:35:010
1210285901914,3cyclictest1269505-21cat08:20:080
1210285901913,4cyclictest1473333-21seq12:29:530
1210285901913,4cyclictest1437490-21tr11:45:060
1210285901913,4cyclictest1348539-21cat09:55:150
1210285901913,4cyclictest1315138-21gltestperf09:15:100
1210285901912,4cyclictest1308588-21sshd09:07:110
1210285901912,4cyclictest1308588-21sshd09:07:100
121028590191,15cyclictest393-21systemd-journal11:01:010
121028590191,15cyclictest393-21systemd-journal11:01:010
121028590190,17cyclictest393-21systemd-journal08:40:010
121028590190,17cyclictest393-21systemd-journal08:40:000
121028790185,10cyclictest1350606-21cat10:00:032
1210287901813,3cyclictest1453473-21phc2sys-jitter12:05:052
1210287901813,3cyclictest1442353-21df_abs11:50:102
1210287901813,3cyclictest1437414-21df11:45:062
1210287901813,3cyclictest1424156-21sshd11:30:022
1210287901813,3cyclictest1419275-21grep11:20:192
1210287901813,3cyclictest1419275-21grep11:20:192
1210287901813,3cyclictest1365355-21open_files10:15:152
1210287901813,3cyclictest1349160-21sendmail_mailst09:55:232
1210287901813,3cyclictest1331096-21tune2fs09:35:082
1210287901813,3cyclictest1295618-21processes08:50:162
1210287901813,3cyclictest1270925-21proc_pri08:20:162
1210287901813,3cyclictest1265252-21idleruntime08:15:082
1210287901813,3cyclictest1254368-21nfs4_client08:00:172
1210287901813,3cyclictest1254368-21nfs4_client08:00:162
1210287901813,3cyclictest1248469-21fw_conntrack07:55:052
1210287901813,3cyclictest1241882-21nfs4_client07:45:142
1210287901813,3cyclictest1232396-21fschecks_time07:35:072
1210287901812,4cyclictest1402189-21cat11:00:152
1210287901812,4cyclictest1402189-21cat11:00:142
1210287901812,4cyclictest1215424-21fw_packets07:15:052
1210287901812,3cyclictest1373580-21open_inodes10:25:162
1210287901811,4cyclictest1385113-21unixbench_singl10:40:112
1210287901811,4cyclictest1375883-21cpuspeed_turbos10:30:052
1210287901810,5cyclictest1342601-21gltestperf09:50:052
121028790180,6cyclictest1460674-21systemd-udevd12:12:192
121028790180,6cyclictest1460674-21systemd-udevd12:12:192
121028790180,2cyclictest111rcu_preempt10:20:052
1210286901814,2cyclictest1342179-21/usr/sbin/munin09:50:171
1210286901813,3cyclictest1391848-21/usr/sbin/munin10:50:191
1210286901813,3cyclictest1313883-21unixbench_multi09:15:051
1210286901813,3cyclictest1216506-21awk07:15:091
121028690181,2cyclictest111rcu_preempt11:05:051
121028690181,15cyclictest1612-21systemd-logind07:30:011
1210286901811,4cyclictest21-21ksoftirqd/111:15:111
1210286901811,4cyclictest1476-21sssd_be09:00:151
1210286901811,4cyclictest1317839-21gltestperf09:20:031
1210286901810,5cyclictest1313103-21tune2fs09:15:021
121028690180,15cyclictest393-21systemd-journal07:34:141
121028590183,4cyclictest1292194-40dnf08:48:020
1210285901813,3cyclictest1423371-21ls11:25:190
1210285901813,3cyclictest1417665-21grep11:20:100
1210285901813,3cyclictest1417665-21grep11:20:090
1210285901813,3cyclictest1377579-21cut10:30:150
1210285901813,3cyclictest1342861-21df09:50:060
1210285901813,3cyclictest1265807-21which08:15:100
1210285901812,3cyclictest1381816-21cut10:35:170
121028590181,15cyclictest1229675-21needreboot07:30:140
121028590181,14cyclictest412-21systemd-udevd12:12:200
121028590181,14cyclictest412-21systemd-udevd12:12:200
121028590181,14cyclictest393-21systemd-journal08:45:020
1210285901810,5cyclictest1318350-21sh09:20:060
121028590180,16cyclictest393-21systemd-journal08:02:020
121028590180,16cyclictest393-21systemd-journal08:02:010
1210287901712,3cyclictest1470428-21pmu-power12:25:052
1210287901712,3cyclictest1447657-21pmu-power11:55:162
1210287901712,3cyclictest1436344-21/usr/sbin/munin11:45:032
1210287901712,3cyclictest1428702-21cpu11:35:042
1210287901712,3cyclictest1428702-21cpu11:35:042
1210287901712,3cyclictest1313633-21sendmail_mailtr09:15:032
1210287901712,3cyclictest1280432-21processes08:35:032
1210287901712,3cyclictest1257545-21hwlatdetect08:05:092
1210287901712,3cyclictest1236911-21df_abs07:40:092
1210287901712,3cyclictest1223843-21df_abs07:25:042
1210287901711,4cyclictest1380738-21sendmail_mailst10:35:102
1210287901711,4cyclictest1325577-21fschecks_time09:30:022
1210287901711,3cyclictest1395877-21grep10:55:032
1210287901710,5cyclictest1210575-21processes07:10:022
121028690171,2cyclictest111rcu_preempt08:50:031
1210286901712,3cyclictest1285150-21grep08:40:041
1210286901712,3cyclictest1248811-21grep07:55:071
1210286901711,4cyclictest1387095-21date10:45:011
1210286901710,4cyclictest1476-21sssd_be09:05:041
1210286901710,4cyclictest1476-21sssd_be09:05:031
1210286901710,4cyclictest0-21swapper/107:20:031
121028590174,10cyclictest0-21swapper/007:45:040
121028590173,11cyclictest1340512-21cstates09:45:140
121028590173,11cyclictest1340512-21cstates09:45:140
1210285901713,2cyclictest1243365-21/usr/sbin/munin07:50:200
1210285901712,3cyclictest1454846-21grep12:05:120
1210285901712,3cyclictest1441200-21cpuspeed_turbos11:50:040
1210285901712,3cyclictest1415801-21awk11:20:010
1210285901712,3cyclictest1396255-21grep10:55:050
1210285901712,3cyclictest1295734-21which08:50:170
1210285901712,3cyclictest1257677-21cat08:05:110
121028590171,14cyclictest393-21systemd-journal09:15:000
1210285901711,4cyclictest1322176-21tail09:25:050
1210285901711,4cyclictest1225378-21acpi07:25:130
121028590170,15cyclictest393-21systemd-journal11:43:320
121028790167,7cyclictest1318252-21cpuspeed_turbos09:20:052
1210287901611,3cyclictest1427950-21seq11:34:282
1210287901611,3cyclictest1355786-21fschecks_time10:05:092
1210287901611,3cyclictest1259958-21unixbench-2d08:10:022
1210287901610,4cyclictest1285686-21cpu08:40:072
1210287901610,4cyclictest1221742-21/usr/sbin/munin07:20:172
121028590168,6cyclictest1215071-21grep07:15:030
1210285901611,3cyclictest1403486-21/usr/sbin/munin11:05:080
1210285901611,3cyclictest1384864-21awk10:40:100
1210285901611,3cyclictest1276692-21sort08:30:050
1210285901611,3cyclictest1262010-21wc08:10:110
1210285901610,4cyclictest1370794-21perl10:25:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional