You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-17 - 21:25

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.1-rt3 (Profile)

Latency plot of system in rack #2, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack2slot0.osadl.org (updated Fri Sep 17, 2021 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
302592570,0sleep00-21swapper/012:05:000
64642550,0sleep20-21swapper/209:18:582
248742540,0sleep30-21swapper/310:03:383
156142520,0sleep20-21swapper/210:17:102
151122500,0sleep00-21swapper/011:20:560
9762490,0sleep10-21swapper/107:45:321
274132480,0sleep10-21swapper/110:52:041
255562480,0sleep30-21swapper/309:15:283
67192460,0sleep20-21swapper/209:59:112
263582410,2sleep31723999cyclictest07:30:173
9362380,0sleep10-21swapper/110:13:401
185982380,0sleep00-21swapper/010:57:570
1683723210,7sleep30-21swapper/307:06:283
1706223119,7sleep20-21swapper/207:09:152
1678823110,7sleep10-21swapper/107:05:501
1698222513,7sleep00-21swapper/007:08:160
3942190,2sleep11723699cyclictest09:05:211
59502180,0sleep30-21swapper/312:39:033
17238991716,0cyclictest4996-21lspci12:30:332
234462160,0sleep30-21swapper/311:39:123
316272150,0sleep20-21swapper/209:25:152
314702140,0sleep10-21swapper/112:21:081
290492140,0sleep029052-21pmu-power07:35:230
179692140,0sleep00-21swapper/010:01:450
15242140,0sleep10-21swapper/111:33:461
17238991312,0cyclictest33150irq/27-ahci[00010:09:232
17239991210,1cyclictest0-21swapper/309:45:273
17239991210,1cyclictest0-21swapper/309:45:273
1723999120,0cyclictest0-21swapper/311:03:573
1723899120,1cyclictest6467-21bash11:19:032
1723899120,11cyclictest1690-21sshd10:21:422
1722999120,1cyclictest10422-21bash10:07:580
17239991110,0cyclictest0-21swapper/312:12:173
1723999110,10cyclictest0-21swapper/311:59:383
1723999110,10cyclictest0-21swapper/311:22:373
1723999110,10cyclictest0-21swapper/311:12:173
17238991110,0cyclictest33150irq/27-ahci[00011:35:572
17238991110,0cyclictest0-21swapper/212:21:032
17238991110,0cyclictest0-21swapper/207:40:232
17238991110,0cyclictest0-21swapper/207:40:222
1723899110,10cyclictest0-21swapper/212:37:122
1723899110,10cyclictest0-21swapper/212:17:222
1723899110,10cyclictest0-21swapper/212:10:032
1723899110,10cyclictest0-21swapper/211:31:232
1723899110,10cyclictest0-21swapper/211:09:432
1723899110,10cyclictest0-21swapper/211:01:522
1723699110,10cyclictest0-21swapper/111:00:331
17229991110,0cyclictest0-21swapper/012:01:280
17229991110,0cyclictest0-21swapper/007:45:170
1722999110,10cyclictest0-21swapper/009:19:470
1723999109,0cyclictest0-21swapper/310:50:083
1723999109,0cyclictest0-21swapper/310:27:283
1723999109,0cyclictest0-21swapper/309:58:073
1723999109,0cyclictest0-21swapper/309:23:383
1723999100,0cyclictest0-21swapper/312:31:083
1723999100,0cyclictest0-21swapper/312:23:183
1723999100,0cyclictest0-21swapper/312:15:583
1723999100,0cyclictest0-21swapper/310:20:173
1723899109,0cyclictest33150irq/27-ahci[00009:10:282
1723899109,0cyclictest0-21swapper/212:09:332
1723899109,0cyclictest0-21swapper/211:42:232
1723899109,0cyclictest0-21swapper/210:33:132
1723899109,0cyclictest0-21swapper/209:30:532
17238991010,0cyclictest33150irq/27-ahci[00009:40:452
1723699109,0cyclictest0-21swapper/111:27:331
1723699109,0cyclictest0-21swapper/111:07:541
1723699109,0cyclictest0-21swapper/110:42:441
1723699109,0cyclictest0-21swapper/109:19:431
1723699108,1cyclictest7638-21bash11:51:031
1723699100,9cyclictest0-21swapper/109:23:441
1723699100,9cyclictest0-21swapper/107:24:441
1723699100,0cyclictest0-21swapper/112:26:531
1723699100,0cyclictest0-21swapper/109:43:531
1722999109,0cyclictest0-21swapper/012:37:080
1722999109,0cyclictest0-21swapper/011:58:580
1722999109,0cyclictest0-21swapper/010:24:280
1722999109,0cyclictest0-21swapper/009:38:580
1722999100,9cyclictest0-21swapper/012:32:580
1722999100,0cyclictest0-21swapper/011:17:070
1722999100,0cyclictest0-21swapper/010:35:170
1722999100,0cyclictest0-21swapper/009:46:080
1722999100,0cyclictest0-21swapper/009:46:070
172399998,0cyclictest0-21swapper/309:51:283
172389999,0cyclictest33150irq/27-ahci[00012:26:392
172389998,0cyclictest33150irq/27-ahci[00011:29:262
172389998,0cyclictest0-21swapper/210:44:332
172389998,0cyclictest0-21swapper/210:10:522
172389998,0cyclictest0-21swapper/209:21:132
172389998,0cyclictest0-21swapper/207:35:222
172369998,0cyclictest0-21swapper/112:19:031
172369998,0cyclictest0-21swapper/111:16:141
172369998,0cyclictest0-21swapper/110:21:231
172369998,0cyclictest0-21swapper/110:16:141
172369996,2cyclictest598-21avahi-daemon09:12:231
172369990,8cyclictest0-21swapper/107:29:431
172369990,0cyclictest0-21swapper/109:59:131
172299998,0cyclictest0-21swapper/012:20:280
172299998,0cyclictest0-21swapper/011:27:570
172299998,0cyclictest0-21swapper/011:09:180
172299997,1cyclictest26220-21sshd10:27:570
172299990,0cyclictest0-21swapper/010:13:370
172399987,0cyclictest0-21swapper/312:26:573
172399987,0cyclictest0-21swapper/310:36:173
172399987,0cyclictest0-21swapper/310:09:273
172399987,0cyclictest0-21swapper/308:35:173
172399985,2cyclictest598-21avahi-daemon11:32:263
172399985,2cyclictest598-21avahi-daemon11:18:243
172399985,2cyclictest598-21avahi-daemon10:42:343
172389987,0cyclictest0-21swapper/211:10:122
172389987,0cyclictest0-21swapper/208:56:132
172389987,0cyclictest0-21swapper/207:10:422
172389985,2cyclictest598-21avahi-daemon11:52:042
172389985,2cyclictest598-21avahi-daemon11:49:372
172389985,2cyclictest598-21avahi-daemon11:24:182
172389985,2cyclictest598-21avahi-daemon10:03:252
172389985,2cyclictest598-21avahi-daemon09:50:082
172389985,2cyclictest598-21avahi-daemon09:37:332
172389981,1cyclictest33-21ksoftirqd/210:52:152
172369987,0cyclictest0-21swapper/110:04:531
172369986,1cyclictest598-21avahi-daemon10:56:361
172369986,1cyclictest598-21avahi-daemon10:06:161
172369986,1cyclictest1-21systemd12:30:341
172369985,2cyclictest598-21avahi-daemon12:35:141
172369985,2cyclictest598-21avahi-daemon12:14:281
172369985,2cyclictest598-21avahi-daemon12:01:431
172369985,2cyclictest598-21avahi-daemon11:57:261
172369985,2cyclictest598-21avahi-daemon11:47:071
172369985,2cyclictest598-21avahi-daemon11:41:341
172369985,2cyclictest598-21avahi-daemon11:35:121
172369985,2cyclictest598-21avahi-daemon09:52:561
172369985,2cyclictest598-21avahi-daemon09:31:211
172299987,0cyclictest0-21swapper/010:47:080
172299987,0cyclictest0-21swapper/009:54:370
172299987,0cyclictest0-21swapper/009:40:370
172299986,2cyclictest5417-21kworker/0:010:52:400
172299986,2cyclictest2316-21kworker/0:011:51:400
172299985,2cyclictest598-21avahi-daemon12:13:470
172299985,2cyclictest598-21avahi-daemon11:04:160
172299985,2cyclictest598-21avahi-daemon10:33:360
17583270,1sleep017652-21sshd12:25:400
172399976,0cyclictest0-21swapper/309:30:283
172399976,0cyclictest0-21swapper/308:16:083
172399975,1cyclictest598-21avahi-daemon12:07:433
172399975,1cyclictest598-21avahi-daemon11:53:163
172399975,1cyclictest598-21avahi-daemon11:42:033
172399975,1cyclictest598-21avahi-daemon10:15:593
172399975,1cyclictest598-21avahi-daemon10:11:083
172399975,1cyclictest598-21avahi-daemon09:37:233
172399975,1cyclictest598-21avahi-daemon09:25:173
172399974,2cyclictest598-21avahi-daemon11:46:243
172399974,2cyclictest598-21avahi-daemon11:26:023
172399974,2cyclictest598-21avahi-daemon09:41:453
172399973,3cyclictest633-21polkitd09:11:183
172399973,1cyclictest598-21avahi-daemon11:06:593
172399972,2cyclictest616-21gdbus10:31:243
172389975,1cyclictest598-21avahi-daemon12:01:532
172389975,1cyclictest598-21avahi-daemon11:56:072
172389975,1cyclictest598-21avahi-daemon10:55:372
172389975,1cyclictest598-21avahi-daemon10:45:282
172389975,1cyclictest598-21avahi-daemon10:38:052
172389975,1cyclictest598-21avahi-daemon10:25:562
172389975,1cyclictest33-21ksoftirqd/209:05:002
172389971,0cyclictest131rcu_sched09:45:472
172389971,0cyclictest131rcu_sched09:45:462
172369976,0cyclictest0-21swapper/109:45:341
172369976,0cyclictest0-21swapper/109:45:331
172369975,1cyclictest598-21avahi-daemon11:21:361
172369975,1cyclictest598-21avahi-daemon10:46:531
172369975,1cyclictest598-21avahi-daemon10:35:591
172369975,1cyclictest598-21avahi-daemon10:30:361
172369975,1cyclictest598-21avahi-daemon10:25:091
172369975,1cyclictest598-21avahi-daemon09:36:491
172369975,1cyclictest598-21avahi-daemon09:25:141
172369974,2cyclictest598-21avahi-daemon12:05:441
172369971,3cyclictest573-13auditd11:10:431
172299975,1cyclictest614-21dbus-daemon12:16:140
172299975,1cyclictest614-21dbus-daemon11:14:420
172299975,1cyclictest598-21avahi-daemon11:45:120
172299975,1cyclictest598-21avahi-daemon11:35:050
172299975,1cyclictest598-21avahi-daemon10:42:250
172299975,1cyclictest598-21avahi-daemon10:15:320
172299975,1cyclictest598-21avahi-daemon09:34:540
172299975,1cyclictest598-21avahi-daemon09:29:110
172299975,1cyclictest598-21avahi-daemon09:21:000
172299974,2cyclictest598-21avahi-daemon09:59:140
14223274,1sleep3614-21dbus-daemon10:56:483
172399964,1cyclictest614-21dbus-daemon12:00:473
172399964,1cyclictest614-21dbus-daemon10:46:023
172399964,1cyclictest41-21ksoftirqd/308:55:013
172399963,2cyclictest970-21snmpd08:46:073
172399963,2cyclictest970-21snmpd08:41:083
172399963,2cyclictest970-21snmpd08:02:133
172399963,2cyclictest614-21dbus-daemon07:25:003
172389965,0cyclictest0-21swapper/207:45:222
172389963,2cyclictest970-21snmpd08:29:342
172389963,2cyclictest970-21snmpd08:00:092
172389963,2cyclictest970-21snmpd07:54:462
172389963,2cyclictest970-21snmpd07:24:032
172369964,1cyclictest14442-21grep08:20:131
172369963,2cyclictest970-21snmpd08:55:061
172369962,1cyclictest0-21swapper/107:34:431
172299964,1cyclictest643-21gdbus09:04:550
172299964,1cyclictest614-21dbus-daemon07:14:560
172299964,1cyclictest598-21avahi-daemon11:41:070
172299964,1cyclictest598-21avahi-daemon11:30:090
172299964,1cyclictest598-21avahi-daemon09:10:250
172299963,2cyclictest970-21snmpd07:31:230
172299963,2cyclictest643-21gdbus09:00:010
172299963,1cyclictest15912-21munin-run08:25:000
172399953,1cyclictest970-21snmpd09:02:593
172399953,1cyclictest970-21snmpd08:26:293
172399953,1cyclictest970-21snmpd07:59:333
172399953,1cyclictest970-21snmpd07:40:513
172399953,1cyclictest970-21snmpd07:40:503
172399953,1cyclictest970-21snmpd07:10:283
172399953,1cyclictest614-21dbus-daemon07:15:003
172399953,1cyclictest32745-21gltestperf09:05:203
172399953,1cyclictest22516-21mii-tool07:20:173
172399953,1cyclictest10505-21cat08:10:183
172399953,1cyclictest0-21swapper/308:30:173
172399952,2cyclictest970-21snmpd08:07:123
172399952,2cyclictest970-21snmpd07:50:343
172399952,2cyclictest970-21snmpd07:35:533
172399952,2cyclictest614-21dbus-daemon09:00:003
172399950,3cyclictest15584-21sshd08:21:053
172389954,0cyclictest0-21swapper/207:30:232
172389953,1cyclictest970-21snmpd09:05:212
172389953,1cyclictest970-21snmpd08:40:382
172389953,1cyclictest970-21snmpd08:35:492
172389953,1cyclictest970-21snmpd08:31:172
172389953,1cyclictest970-21snmpd08:15:232
172389953,1cyclictest970-21snmpd08:13:572
172389953,1cyclictest970-21snmpd08:06:452
172389953,1cyclictest970-21snmpd07:57:392
172389953,1cyclictest970-21snmpd07:25:202
172389953,1cyclictest970-21snmpd07:16:592
172389953,1cyclictest14587-21mii-tool08:20:142
172389953,1cyclictest0-21swapper/208:50:222
172389952,2cyclictest25505-21cat08:45:242
172369953,1cyclictest970-21snmpd09:04:201
172369953,1cyclictest970-21snmpd08:52:231
172369953,1cyclictest970-21snmpd08:40:111
172369953,1cyclictest970-21snmpd08:10:581
172369953,1cyclictest970-21snmpd08:06:391
172369953,1cyclictest970-21snmpd07:16:471
172369953,1cyclictest966-21runrttasks07:35:011
172369953,1cyclictest4341-21grep07:55:161
172369953,1cyclictest25402-21switchtime08:45:241
172369953,1cyclictest2387-21cat07:50:191
172369953,1cyclictest18619-21idleruntime08:30:181
172369953,1cyclictest17872-21cat07:10:151
172369953,1cyclictest12571-21cat08:15:201
172369953,1cyclictest1071-21lldpd08:26:471
172369952,2cyclictest0-21swapper/108:35:581
172369952,2cyclictest0-21swapper/107:42:081
172369952,2cyclictest0-21swapper/107:42:071
172369950,1cyclictest6130-21cstates08:00:131
172299953,1cyclictest970-21snmpd08:49:120
172299953,1cyclictest970-21snmpd08:41:520
172299953,1cyclictest970-21snmpd08:05:160
172299953,1cyclictest970-21snmpd08:03:420
172299953,1cyclictest970-21snmpd07:20:390
172299953,1cyclictest4420-21cat07:55:180
172299953,1cyclictest30540-21grep07:40:220
172299953,1cyclictest30540-21grep07:40:210
172299953,1cyclictest25010-21head07:25:200
172299953,1cyclictest19677-21taskset08:31:180
172299953,1cyclictest17526-21unixbench_singl08:25:260
172299953,1cyclictest1002-21rpcbind07:52:560
172299952,2cyclictest970-21snmpd08:10:270
172299952,2cyclictest970-21snmpd07:16:020
172299952,2cyclictest614-21dbus-daemon08:19:590
172299951,2cyclictest1769-21sshd09:08:450
172299950,0cyclictest0-21swapper/008:35:370
172399942,1cyclictest32285-21cat07:45:163
172299942,1cyclictest970-21snmpd08:50:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional