You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-06 - 14:55

x86 Intel Xeon E3-1220 @3100 MHz, Linux 4.19.1-rt3 (Profile)

Latency plot of system in rack #2, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rack2slot0.osadl.org (updated Wed Jul 06, 2022 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
185082580,0sleep20-21swapper/212:36:122
108992580,0sleep110905-21nfsd412:10:151
292272510,0sleep30-21swapper/312:22:553
24642510,0sleep00-21swapper/009:43:510
284382490,0sleep00-21swapper/009:34:180
7992480,0sleep20-21swapper/211:11:242
152532480,0sleep10-21swapper/109:46:531
68772470,0sleep0111rcuc/010:49:040
1003523119,7sleep20-21swapper/207:08:342
994622816,7sleep30-21swapper/307:07:313
1002022817,7sleep10-21swapper/107:08:251
1001922513,7sleep00-21swapper/007:08:240
3082170,0sleep10-21swapper/110:07:151
325782160,0sleep0111rcuc/012:07:370
152572150,0sleep30-21swapper/310:35:093
252912140,0sleep00-21swapper/012:13:580
81022130,0sleep30-21swapper/310:17:103
123312130,0sleep00-21swapper/010:10:150
1027099130,1cyclictest27637-21sshd10:54:123
1027099120,1cyclictest539-21lspci11:35:223
10269991210,1cyclictest13909-21sshd12:10:552
1026999120,1cyclictest15379-21sshd12:03:242
1026599120,10cyclictest23336-21bash09:16:471
10258991211,0cyclictest0-21swapper/011:16:510
10258991210,1cyclictest31641-21sshd11:51:110
10270991110,0cyclictest0-21swapper/312:13:023
1027099110,0cyclictest0-21swapper/311:44:323
10269991110,0cyclictest0-21swapper/211:20:252
1026999110,10cyclictest0-21swapper/211:35:242
1026999110,10cyclictest0-21swapper/211:03:142
1026999110,10cyclictest0-21swapper/209:50:442
1026599119,1cyclictest31598-21bash11:27:071
10265991110,0cyclictest0-21swapper/109:13:071
1026599110,10cyclictest0-21swapper/112:26:471
1026599110,10cyclictest0-21swapper/111:47:481
1026599110,10cyclictest0-21swapper/110:44:571
1026599110,0cyclictest0-21swapper/110:18:571
1025899113,4cyclictest8936-21munin-run08:25:000
10258991110,0cyclictest0-21swapper/012:03:120
10258991110,0cyclictest0-21swapper/010:44:010
1025899110,10cyclictest0-21swapper/011:02:010
1025899110,10cyclictest0-21swapper/007:25:210
1025899110,0cyclictest0-21swapper/010:24:520
1025899110,0cyclictest0-21swapper/009:37:110
211542100,0sleep30-21swapper/310:28:323
1027099109,0cyclictest0-21swapper/312:38:223
1027099109,0cyclictest0-21swapper/311:52:323
1027099109,0cyclictest0-21swapper/311:21:323
1027099109,0cyclictest0-21swapper/310:49:423
1027099109,0cyclictest0-21swapper/310:24:523
1027099109,0cyclictest0-21swapper/309:50:423
1027099109,0cyclictest0-21swapper/309:24:123
1027099100,0cyclictest0-21swapper/312:05:323
1027099100,0cyclictest0-21swapper/312:00:123
1027099100,0cyclictest0-21swapper/310:08:423
1027099100,0cyclictest0-21swapper/309:38:123
1026999109,0cyclictest0-21swapper/212:18:242
1026999109,0cyclictest0-21swapper/211:44:342
1026999109,0cyclictest0-21swapper/210:58:452
1026999109,0cyclictest0-21swapper/210:35:042
1026999109,0cyclictest0-21swapper/209:21:542
1026999109,0cyclictest0-21swapper/209:13:242
1026999108,1cyclictest22192-21lspci09:56:342
1026999100,9cyclictest0-21swapper/209:46:142
1026999100,0cyclictest0-21swapper/211:51:242
1026999100,0cyclictest0-21swapper/210:16:542
1026999100,0cyclictest0-21swapper/209:17:042
1026999100,0cyclictest0-21swapper/208:30:242
1026599109,0cyclictest0-21swapper/112:34:081
1026599109,0cyclictest0-21swapper/111:15:371
1026599109,0cyclictest0-21swapper/111:06:271
1026599109,0cyclictest0-21swapper/109:21:371
1026599100,9cyclictest0-21swapper/111:58:571
1026599100,9cyclictest0-21swapper/108:49:571
1026599100,0cyclictest0-21swapper/112:17:071
1026599100,0cyclictest0-21swapper/111:38:371
1026599100,0cyclictest0-21swapper/110:55:081
1026599100,0cyclictest0-21swapper/110:31:271
1025899109,0cyclictest0-21swapper/007:49:310
1025899109,0cyclictest0-21swapper/007:40:010
1025899100,9cyclictest0-21swapper/011:44:210
1025899100,9cyclictest0-21swapper/009:24:210
1025899100,0cyclictest0-21swapper/012:19:110
1025899100,0cyclictest0-21swapper/011:26:010
1025899100,0cyclictest0-21swapper/010:57:310
1025899100,0cyclictest0-21swapper/010:30:310
1025899100,0cyclictest0-21swapper/009:11:010
1025899100,0cyclictest0-21swapper/007:36:310
11600290,1sleep2619-21systemd-logind11:46:132
102709998,0cyclictest0-21swapper/311:14:423
102709998,0cyclictest0-21swapper/310:30:323
102709998,0cyclictest0-21swapper/309:46:123
102699998,0cyclictest0-21swapper/212:25:542
102699998,0cyclictest0-21swapper/210:48:342
102699998,0cyclictest0-21swapper/210:44:042
102659999,0cyclictest0-21swapper/112:09:371
102659998,0cyclictest0-21swapper/110:26:571
102659998,0cyclictest0-21swapper/110:20:081
102659998,0cyclictest0-21swapper/109:51:371
102659998,0cyclictest0-21swapper/109:41:271
102589998,0cyclictest0-21swapper/012:26:420
102589998,0cyclictest0-21swapper/011:37:320
102589998,0cyclictest0-21swapper/011:10:020
102589998,0cyclictest0-21swapper/010:06:210
102589998,0cyclictest0-21swapper/009:55:220
102589990,0cyclictest0-21swapper/010:19:110
102709987,0cyclictest0-21swapper/311:45:423
102709987,0cyclictest0-21swapper/311:05:333
102709986,1cyclictest598-21avahi-daemon11:28:303
102709986,1cyclictest1299-21diskstats08:05:123
102709985,2cyclictest598-21avahi-daemon12:16:233
102709985,2cyclictest598-21avahi-daemon11:19:563
102709985,2cyclictest598-21avahi-daemon10:10:373
102709985,2cyclictest598-21avahi-daemon10:04:533
102709985,2cyclictest598-21avahi-daemon09:33:143
102709985,2cyclictest598-21avahi-daemon09:27:053
102709985,2cyclictest598-21avahi-daemon09:12:483
102699987,0cyclictest0-21swapper/212:05:342
102699987,0cyclictest0-21swapper/210:54:142
102699987,0cyclictest0-21swapper/210:30:352
102699987,0cyclictest0-21swapper/210:24:542
102699987,0cyclictest0-21swapper/209:40:352
102699986,1cyclictest598-21avahi-daemon10:06:482
102699985,2cyclictest598-21avahi-daemon11:55:092
102659987,0cyclictest0-21swapper/110:50:371
102659985,2cyclictest598-21avahi-daemon11:33:461
102659985,2cyclictest598-21avahi-daemon11:22:551
102659985,2cyclictest598-21avahi-daemon10:37:301
102659985,2cyclictest598-21avahi-daemon09:56:001
102659985,2cyclictest598-21avahi-daemon09:26:031
102659980,0cyclictest0-21swapper/107:31:271
102589986,1cyclictest0-21swapper/012:35:010
102589986,1cyclictest0-21swapper/010:26:010
102589985,2cyclictest598-21avahi-daemon11:24:430
102589985,2cyclictest598-21avahi-daemon10:00:050
102589985,2cyclictest598-21avahi-daemon09:52:150
102589985,2cyclictest598-21avahi-daemon09:48:220
102589985,2cyclictest19688-21kworker/0:211:32:500
102589980,4cyclictest2816-21crond08:10:000
102709976,0cyclictest0-21swapper/310:55:223
102709976,0cyclictest0-21swapper/309:56:323
102709975,1cyclictest598-21avahi-daemon11:34:473
102709975,1cyclictest598-21avahi-daemon11:01:373
102709975,1cyclictest598-21avahi-daemon09:42:423
102709975,1cyclictest598-21avahi-daemon09:19:023
102709974,2cyclictest598-21avahi-daemon12:31:293
102709974,2cyclictest598-21avahi-daemon12:27:323
102709974,2cyclictest598-21avahi-daemon11:56:473
102709974,2cyclictest598-21avahi-daemon10:42:003
102699976,0cyclictest0-21swapper/208:50:142
102699976,0cyclictest0-21swapper/207:37:542
102699975,1cyclictest598-21avahi-daemon12:30:392
102699975,1cyclictest598-21avahi-daemon12:20:242
102699975,1cyclictest598-21avahi-daemon11:06:122
102699975,1cyclictest598-21avahi-daemon10:29:492
102699975,1cyclictest598-21avahi-daemon10:02:292
102699974,2cyclictest619-21systemd-logind11:15:552
102699974,2cyclictest598-21avahi-daemon11:30:012
102699974,2cyclictest598-21avahi-daemon09:36:472
102699974,2cyclictest598-21avahi-daemon09:32:062
102699973,1cyclictest33-21ksoftirqd/209:01:282
102699973,1cyclictest0-21swapper/211:28:002
102699972,3cyclictest30467-21sshd10:14:552
102659975,1cyclictest598-21avahi-daemon12:35:231
102659975,1cyclictest598-21avahi-daemon11:50:591
102659975,1cyclictest598-21avahi-daemon11:40:461
102659975,1cyclictest598-21avahi-daemon11:11:521
102659975,1cyclictest598-21avahi-daemon10:45:121
102659975,1cyclictest598-21avahi-daemon09:38:071
102659975,1cyclictest598-21avahi-daemon09:32:521
102659975,1cyclictest22973-21lspci11:00:471
102659974,2cyclictest598-21avahi-daemon12:21:201
102659974,2cyclictest598-21avahi-daemon10:11:541
102659974,2cyclictest598-21avahi-daemon10:01:091
102659970,6cyclictest0-21swapper/108:35:271
102589976,0cyclictest0-21swapper/012:21:210
102589975,1cyclictest598-21avahi-daemon11:06:290
102589975,1cyclictest598-21avahi-daemon10:51:160
102589975,1cyclictest598-21avahi-daemon10:35:090
102589975,1cyclictest598-21avahi-daemon09:25:400
102589975,1cyclictest598-21avahi-daemon09:16:000
102589975,1cyclictest598-21avahi-daemon07:20:010
102589974,2cyclictest598-21avahi-daemon12:32:590
102589974,2cyclictest598-21avahi-daemon11:59:230
102589974,2cyclictest26778-21sed09:05:220
102709963,2cyclictest970-21snmpd08:55:023
102709963,2cyclictest970-21snmpd07:45:063
102699964,1cyclictest643-21gdbus07:50:002
102699964,1cyclictest598-21avahi-daemon08:05:002
102699964,1cyclictest23676-21sshd09:25:072
102699963,2cyclictest970-21snmpd08:55:082
102699963,2cyclictest970-21snmpd08:46:192
102699963,2cyclictest970-21snmpd07:12:442
102699962,1cyclictest0-21swapper/207:31:282
102699960,5cyclictest0-21swapper/208:17:342
102699960,5cyclictest0-21swapper/207:55:542
102659964,1cyclictest8844-21sshd12:01:391
102659963,2cyclictest970-21snmpd08:27:071
102659963,2cyclictest970-21snmpd08:01:211
102659963,2cyclictest970-21snmpd07:37:231
102589965,1cyclictest23160-21kworker/0:208:59:580
102589965,0cyclictest0-21swapper/008:36:410
102589965,0cyclictest0-21swapper/007:55:310
102589964,1cyclictest614-21dbus-daemon11:45:240
102589963,2cyclictest970-21snmpd08:54:500
102589963,2cyclictest970-21snmpd08:34:190
102709954,0cyclictest0-21swapper/307:25:123
102709953,1cyclictest970-21snmpd09:07:323
102709953,1cyclictest970-21snmpd08:50:383
102709953,1cyclictest970-21snmpd08:46:103
102709953,1cyclictest970-21snmpd08:40:283
102709953,1cyclictest970-21snmpd08:30:493
102709953,1cyclictest970-21snmpd08:15:183
102709953,1cyclictest970-21snmpd08:11:153
102709953,1cyclictest970-21snmpd07:41:513
102709953,1cyclictest970-21snmpd07:30:023
102709953,1cyclictest970-21snmpd07:21:203
102709953,1cyclictest598-21avahi-daemon07:50:553
102709953,1cyclictest4421-21sendmail08:25:343
102709953,1cyclictest30613-21sed07:55:193
102709953,1cyclictest21615-21cat07:35:133
102709953,1cyclictest13498-21cat07:15:283
102709953,1cyclictest0-21swapper/308:20:303
102709952,2cyclictest970-21snmpd08:38:073
102709952,2cyclictest970-21snmpd08:01:573
102709950,4cyclictest24835-21sshd09:00:283
102709950,0cyclictest0-21swapper/307:11:323
102699953,1cyclictest970-21snmpd08:40:352
102699953,1cyclictest970-21snmpd08:36:372
102699953,1cyclictest970-21snmpd08:26:012
102699953,1cyclictest970-21snmpd08:21:342
102699953,1cyclictest970-21snmpd08:08:512
102699953,1cyclictest970-21snmpd07:50:062
102699953,1cyclictest970-21snmpd07:40:512
102699953,1cyclictest970-21snmpd07:24:202
102699953,1cyclictest3553-21cat08:10:162
102699953,1cyclictest25457-21/usr/sbin/munin09:05:142
102699953,1cyclictest18483-21uptime07:25:192
102699952,2cyclictest0-21swapper/207:16:452
102659953,1cyclictest970-21snmpd09:06:471
102659953,1cyclictest970-21snmpd09:03:111
102659953,1cyclictest970-21snmpd08:56:441
102659953,1cyclictest970-21snmpd08:51:591
102659953,1cyclictest970-21snmpd08:33:041
102659953,1cyclictest970-21snmpd08:13:001
102659953,1cyclictest970-21snmpd07:55:331
102659953,1cyclictest970-21snmpd07:52:421
102659953,1cyclictest970-21snmpd07:42:271
102659953,1cyclictest970-21snmpd07:27:081
102659953,1cyclictest970-21snmpd07:15:551
102659953,1cyclictest970-21snmpd07:11:291
102659953,1cyclictest27026-21sadc07:50:001
102659953,1cyclictest15553-21cat07:20:141
102659953,1cyclictest15549-21df_inode08:40:111
102659953,1cyclictest1499-21mii-tool08:05:141
102659950,4cyclictest0-21swapper/108:16:071
102589955,0cyclictest729-21kworker/0:008:16:060
102589955,0cyclictest729-21kworker/0:008:10:020
102589955,0cyclictest6879-21kworker/0:108:25:020
102589955,0cyclictest29283-21kworker/0:208:00:060
102589955,0cyclictest27011-21kworker/0:107:50:060
102589955,0cyclictest23160-21kworker/0:209:01:180
102589955,0cyclictest10952-21kworker/0:008:45:020
102589955,0cyclictest10952-21kworker/0:008:41:060
102589955,0cyclictest10149-21kworker/0:107:10:020
102589953,1cyclictest16420-21unixbench_singl07:20:200
102589951,2cyclictest20633-21sshd07:31:290
102659942,1cyclictest7177-21cpu08:20:101
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional