You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-26 - 18:51

x86 Intel Xeon E3-1220 @3100 MHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rack1slot0.osadl.org (updated Sun Jun 26, 2022 12:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
859121140,2sleep1114299cyclictest12:31:591
574021040,1sleep0113199cyclictest10:14:080
208822950,0sleep320836-21sshd11:33:573
26672870,0sleep30-21swapper/309:27:083
26672870,0sleep30-21swapper/309:27:083
23672850,0sleep10-21swapper/111:39:561
41732830,0sleep30-21swapper/312:24:213
194952830,0sleep10-21swapper/112:10:411
194952830,0sleep10-21swapper/112:10:411
28262660,0sleep20-21swapper/211:12:522
77826452,7sleep10-21swapper/107:08:191
247562630,0sleep30-21swapper/311:56:203
231192610,1sleep023109-21lspci11:16:510
18412610,1sleep31725-21sshd11:27:313
254792600,0sleep00-21swapper/012:20:560
90262590,0sleep10-21swapper/109:54:461
191422590,0sleep30-21swapper/312:18:043
51062580,0sleep00-21swapper/011:22:520
321372580,0sleep00-21swapper/012:28:550
86825746,7sleep30-21swapper/307:09:083
68925645,7sleep20-21swapper/207:07:322
178352560,0sleep20-21swapper/210:37:072
103792560,0sleep10-21swapper/112:29:441
90292550,0sleep09028-21bash10:24:150
81402550,0sleep00-21swapper/009:32:310
57525533,7sleep00-21swapper/007:06:160
229852550,0sleep00-21swapper/010:17:540
60112520,0sleep20-21swapper/209:47:062
139702430,1sleep313968-21sshd11:18:373
156942400,0sleep3401ktimersoftd/309:35:293
128292340,1sleep125-21ksoftirqd/111:06:131
44742220,0sleep00-21swapper/010:01:460
44242210,1sleep24422-21lspci11:32:402
272492210,1sleep027246-21sshd11:07:220
172692210,1sleep217272-21sshd09:38:082
113452210,1sleep30-21swapper/310:02:183
99182200,1sleep19922-21bash10:48:511
127432200,1sleep212742-21grep09:20:222
288072190,1sleep233-21ksoftirqd/209:58:452
177362190,0sleep10-21swapper/109:18:171
85132180,1sleep01-21systemd10:46:130
84202180,1sleep28380-21rm12:17:152
6342180,0sleep00-21swapper/012:01:540
13732170,0sleep10-21swapper/111:27:301
257692160,0sleep10-21swapper/110:18:071
227552160,0sleep122747-21lspci10:22:461
142162160,1sleep114212-21memory09:45:181
98712150,0sleep00-21swapper/012:14:560
98712150,0sleep00-21swapper/012:14:550
5522150,2sleep3115699cyclictest10:57:593
286902150,0sleep10-21swapper/111:24:411
225252150,1sleep322442-21sshd09:33:383
21002150,0sleep00-21swapper/010:06:280
106932150,0sleep010694-21id11:10:580
241622130,1sleep0111rcuc/011:51:210
114999130,12cyclictest32331-21gdbus11:40:222
1156991210,1cyclictest5760-21bash12:39:133
1156991210,1cyclictest31342-21bash11:07:393
115699120,1cyclictest27010-21sshd10:45:143
115699120,11cyclictest9814-21sshd12:29:423
114999120,1cyclictest13333-21grep10:56:262
114999120,11cyclictest0-21swapper/210:25:562
114999120,11cyclictest0-21swapper/209:54:272
1142991210,1cyclictest30854-21sshd12:04:151
1142991210,1cyclictest15412-21sshd10:05:061
114299120,1cyclictest21364-21lspci09:31:021
114299120,11cyclictest7877-21sshd10:58:311
114299120,11cyclictest22907-21bash12:06:021
113199120,1cyclictest801-21systemd-logind11:41:510
113199120,1cyclictest3141-21cp09:56:450
113199120,11cyclictest0-21swapper/010:28:530
106022120,2sleep1114299cyclictest11:47:531
312992110,0sleep20-21swapper/210:06:142
115699119,1cyclictest820-21dbus-daemon10:14:033
115699119,1cyclictest21120-21lspci11:14:183
1156991110,0cyclictest0-21swapper/312:06:073
1156991110,0cyclictest0-21swapper/311:47:043
1156991110,0cyclictest0-21swapper/311:24:573
115699110,10cyclictest0-21swapper/309:50:203
114999119,1cyclictest31906-21sshd09:19:252
114999119,1cyclictest14216-21lspci10:00:052
1149991110,0cyclictest0-21swapper/210:50:582
114999110,10cyclictest0-21swapper/212:36:242
114999110,10cyclictest0-21swapper/211:59:152
114299119,1cyclictest14173-21sshd10:51:361
1142991110,0cyclictest0-21swapper/111:54:331
114299110,10cyclictest5432-21sshd10:31:151
114299110,10cyclictest15440-21sshd09:23:061
114299110,10cyclictest0-21swapper/111:40:591
113199119,1cyclictest29189-21sshd10:33:050
1131991110,0cyclictest0-21swapper/011:32:390
1131991110,0cyclictest0-21swapper/011:26:120
113199110,10cyclictest16595-21sshd12:35:090
113199110,10cyclictest0-21swapper/012:06:230
113199110,10cyclictest0-21swapper/011:59:470
79872100,1sleep1787-21ModemManager10:11:491
115699109,0cyclictest0-21swapper/309:41:373
115699109,0cyclictest0-21swapper/308:33:393
115699109,0cyclictest0-21swapper/308:10:243
115699108,1cyclictest41-21ksoftirqd/311:04:533
115699108,1cyclictest18538-21lspci11:41:063
115699100,9cyclictest0-21swapper/307:45:213
115699100,0cyclictest0-21swapper/311:37:523
115699100,0cyclictest0-21swapper/310:17:473
115699100,0cyclictest0-21swapper/309:19:383
114999109,0cyclictest0-21swapper/212:07:392
114999109,0cyclictest0-21swapper/211:54:242
114999108,1cyclictest32470-21lspci11:49:342
114999108,1cyclictest19051-21lspci09:30:512
114999108,1cyclictest0-21swapper/210:30:362
114999108,1cyclictest0-21swapper/209:41:222
114299109,0cyclictest13141-21sshd09:27:581
114299109,0cyclictest13141-21sshd09:27:571
114299109,0cyclictest0-21swapper/108:25:291
114299108,1cyclictest28497-21lspci10:03:371
114299108,1cyclictest22266-21lspci10:42:241
114299108,1cyclictest14318-21sshd10:27:031
114299100,9cyclictest3451-21sshd11:32:351
114299100,9cyclictest17366-21sshd11:11:301
114299100,0cyclictest0-21swapper/107:12:391
113199109,0cyclictest0-21swapper/012:33:510
113199109,0cyclictest0-21swapper/011:45:060
113199109,0cyclictest0-21swapper/010:55:400
113199109,0cyclictest0-21swapper/009:49:490
113199109,0cyclictest0-21swapper/009:38:020
113199109,0cyclictest0-21swapper/007:14:190
113199100,9cyclictest0-21swapper/008:40:030
113199100,9cyclictest0-21swapper/007:20:370
113199100,0cyclictest26382-21sshd09:23:580
113199100,0cyclictest0-21swapper/011:03:170
11569998,0cyclictest0-21swapper/310:09:523
11569990,0cyclictest0-21swapper/308:43:383
11499998,0cyclictest0-21swapper/211:21:002
11499998,0cyclictest0-21swapper/210:13:192
11499990,8cyclictest4154-21sshd12:21:502
11499990,8cyclictest30471-21sshd10:20:522
11499990,8cyclictest24980-21sshd10:45:062
11499990,8cyclictest23879-21sshd12:25:452
11499990,8cyclictest15301-21sshd11:38:262
11499990,8cyclictest14336-21sshd11:28:302
11499990,8cyclictest10278-21sshd10:16:552
11429998,0cyclictest0-21swapper/107:35:251
11429990,8cyclictest22326-21sshd11:02:051
11429990,8cyclictest18882-21sshd09:38:161
11429990,8cyclictest17295-21sshd09:40:341
11429990,0cyclictest0-21swapper/107:31:221
11319998,0cyclictest0-21swapper/011:35:530
11319998,0cyclictest0-21swapper/010:52:570
11319997,1cyclictest30443-21id10:40:310
11319997,1cyclictest24033-21id09:41:050
11319996,2cyclictest11929-21kworker/0:109:13:280
11319996,2cyclictest11929-21kworker/0:109:13:270
11319995,3cyclictest16224-21kworker/0:309:25:200
11319995,3cyclictest16224-21kworker/0:309:25:190
11319995,3cyclictest16224-21kworker/0:309:17:240
11319990,0cyclictest0-21swapper/007:55:390
11569987,0cyclictest0-21swapper/311:54:383
11569987,0cyclictest0-21swapper/310:51:413
11569987,0cyclictest0-21swapper/310:31:203
11569987,0cyclictest0-21swapper/310:21:583
11569987,0cyclictest0-21swapper/309:13:023
11569987,0cyclictest0-21swapper/309:13:013
11569986,1cyclictest7533-21bash12:12:153
11569986,1cyclictest7533-21bash12:12:153
11569986,1cyclictest0-21swapper/310:27:093
11569983,1cyclictest32329-21polkitd09:56:303
11569980,7cyclictest6498-21sshd09:47:083
11569980,7cyclictest2842-21sshd09:22:073
11569980,7cyclictest24775-21sshd10:35:123
11569980,7cyclictest20950-21sshd12:30:273
11569980,7cyclictest12497-21sshd12:02:483
11569980,7cyclictest11970-21sshd10:41:353
11499987,0cyclictest0-21swapper/211:06:562
11499987,0cyclictest0-21swapper/209:11:002
11499987,0cyclictest0-21swapper/209:11:002
11499986,1cyclictest7821-21lspci09:27:332
11499986,1cyclictest7821-21lspci09:27:322
11499986,1cyclictest31864-21lspci12:11:382
11499986,1cyclictest31864-21lspci12:11:382
11499986,1cyclictest30535-21lspci10:43:022
11499986,1cyclictest23857-21lspci12:03:412
11429987,0cyclictest0-21swapper/112:15:141
11429986,1cyclictest23819-21lspci11:56:161
11429986,1cyclictest2351-21lspci11:17:451
11429980,7cyclictest29657-21sshd12:36:041
11429980,7cyclictest12878-21sshd10:36:441
11429980,0cyclictest0-21swapper/107:19:261
11319987,0cyclictest0-21swapper/012:16:540
11319986,1cyclictest32168-21lspci10:35:410
11319985,2cyclictest7327-21kworker/0:107:25:150
11319985,2cyclictest10502-21kworker/0:309:51:430
11319984,3cyclictest26775-21kworker/0:108:15:240
11499976,0cyclictest0-21swapper/208:40:012
11499975,1cyclictest25633-21lspci12:33:192
11499970,6cyclictest29382-21sshd11:00:122
11499970,6cyclictest2167-21kworker/2:111:15:252
11429976,0cyclictest0-21swapper/107:54:191
11429976,0cyclictest0-21swapper/107:25:141
11429972,2cyclictest22232-21sshd09:10:381
11429972,2cyclictest22232-21sshd09:10:371
11429970,6cyclictest19242-21sshd09:55:291
11429970,6cyclictest17763-21sshd12:20:201
11429970,6cyclictest0-21swapper/108:06:441
11429970,0cyclictest0-21swapper/108:40:011
11319976,0cyclictest0-21swapper/008:14:320
11319976,0cyclictest0-21swapper/007:35:380
11319974,2cyclictest19020-21kworker/0:008:05:150
11319970,6cyclictest15950-21sshd09:10:000
11569965,0cyclictest0-21swapper/308:37:303
11569963,2cyclictest1398-21snmpd08:45:173
11569963,2cyclictest1398-21snmpd07:53:593
11569963,2cyclictest1398-21snmpd07:23:223
11569963,2cyclictest1397-21snmpd08:52:553
11569963,2cyclictest12591-21if_err_eno109:00:133
11569960,5cyclictest0-21swapper/308:23:573
11499965,0cyclictest0-21swapper/207:40:352
11499965,0cyclictest0-21swapper/207:30:442
11499964,1cyclictest1219-21runrttasks07:15:142
11499963,2cyclictest1398-21snmpd08:40:582
11499963,2cyclictest1398-21snmpd08:01:162
11499963,2cyclictest1398-21snmpd07:46:382
11499963,2cyclictest1397-21snmpd07:29:182
11499963,2cyclictest1397-21snmpd07:11:552
11499960,5cyclictest0-21swapper/208:34:212
11499960,5cyclictest0-21swapper/208:21:462
11429965,0cyclictest0-21swapper/109:06:481
11429965,0cyclictest0-21swapper/108:47:551
11429964,1cyclictest1398-21snmpd08:02:131
11429964,1cyclictest1397-21snmpd08:11:061
11429963,2cyclictest1398-21snmpd07:43:271
11429963,2cyclictest1397-21snmpd08:55:561
11429960,0cyclictest0-21swapper/108:30:571
11319965,0cyclictest0-21swapper/007:34:020
11319964,1cyclictest9095-21ntp_states08:50:170
11319964,1cyclictest1397-21snmpd07:50:120
11319963,2cyclictest30659-21kworker/0:208:45:120
11319963,2cyclictest1398-21snmpd08:31:370
11319963,2cyclictest1397-21snmpd09:01:170
11319963,2cyclictest1397-21snmpd08:25:160
11319960,0cyclictest0-21swapper/008:21:190
11319960,0cyclictest0-21swapper/007:46:270
11319960,0cyclictest0-21swapper/007:16:340
11569953,1cyclictest24048-21rt-features08:05:163
11569953,1cyclictest21617-21cat08:00:153
11569953,1cyclictest19650-21cat07:55:133
11569953,1cyclictest1398-21snmpd09:05:093
11569953,1cyclictest1398-21snmpd08:55:173
11569953,1cyclictest1398-21snmpd08:15:183
11569953,1cyclictest1398-21snmpd07:25:083
11569953,1cyclictest1398-21snmpd07:15:113
11569953,1cyclictest1397-21snmpd08:25:033
11569953,1cyclictest1397-21snmpd07:10:013
11569953,1cyclictest12220-21ntpq07:35:163
11569952,2cyclictest14016-21tr07:40:163
11569952,2cyclictest10433-21gunzip07:30:183
11499953,1cyclictest6048-21cat07:20:122
11499953,1cyclictest31856-21sendmail_mailtr08:25:172
11499953,1cyclictest1398-21snmpd09:05:152
11499953,1cyclictest1398-21snmpd09:00:072
11499953,1cyclictest1398-21snmpd08:05:182
11499953,1cyclictest1397-21snmpd08:56:562
11499953,1cyclictest1397-21snmpd08:50:072
11499953,1cyclictest1397-21snmpd08:46:292
11499953,1cyclictest1397-21snmpd08:15:092
11499953,1cyclictest1397-21snmpd08:10:132
11499953,1cyclictest1397-21snmpd07:55:142
11499953,1cyclictest1397-21snmpd07:50:032
11499953,1cyclictest1397-21snmpd07:35:052
11429954,0cyclictest0-21swapper/107:55:061
11429953,1cyclictest8701-21cat08:50:141
11429953,1cyclictest6736-21timerwakeupswit07:20:181
11429953,1cyclictest15760-21cat07:45:131
11429953,1cyclictest1398-21snmpd08:16:241
11429952,2cyclictest29502-21cut08:20:161
11429950,1cyclictest4400-21/usr/sbin/munin08:40:171
11429950,1cyclictest0-21swapper/109:00:161
11319953,1cyclictest1398-21snmpd08:35:110
11319953,1cyclictest1398-21snmpd08:00:380
11319953,1cyclictest1397-21snmpd07:40:120
11319952,2cyclictest1398-21snmpd08:55:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional