You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-08 - 23:53

x86 Intel Xeon E5620 @2400 MHz, Linux 4.19.127-rt55 (Profile)

Latency plot of system in rack #c, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackcslot0.osadl.org (updated Mon Aug 08, 2022 00:43:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
799421830,1sleep21936499cyclictest21:34:198
170721760,6sleep91937499cyclictest00:12:1515
468321180,6sleep31936699cyclictest00:16:429
1873921140,1sleep90-21swapper/923:51:0715
18724211285,22sleep80-21swapper/819:15:3614
2775821100,7sleep61937099cyclictest22:03:3212
969121080,6sleep101937599cyclictest00:09:042
692321070,1sleep130-21swapper/1323:26:355
18719210682,19sleep30-21swapper/319:15:319
327321020,2sleep73271-21sshd22:16:2413
18546210294,4sleep20-21swapper/219:13:228
18891210184,12sleep100-21swapper/1019:17:152
1116021010,1sleep20-21swapper/200:44:468
1881521000,1sleep140-21swapper/1400:45:466
173002990,1sleep10-21swapper/100:45:331
1890529889,4sleep70-21swapper/719:17:2813
1870429886,4sleep50-21swapper/519:15:1711
13252980,0sleep130-21swapper/1321:52:575
270502970,0sleep100-21swapper/1023:17:102
270502970,0sleep100-21swapper/1023:17:102
253302970,1sleep40-21swapper/400:34:5410
63962940,1sleep01-21systemd22:47:340
21262940,0sleep130-21swapper/1321:37:585
1876429472,4sleep90-21swapper/919:16:1015
206892930,1sleep20-21swapper/200:03:028
126632890,23sleep71937299cyclictest21:04:5813
126632890,23sleep71937299cyclictest21:04:5813
47842880,0sleep10-21swapper/122:23:471
1856828878,3sleep40-21swapper/419:13:4010
107482880,1sleep130-21swapper/1322:32:495
203182870,1sleep50-21swapper/523:16:1711
203182870,1sleep50-21swapper/523:16:1711
167552870,1sleep100-21swapper/1023:47:072
1858228674,4sleep140-21swapper/1419:13:506
115242860,0sleep00-21swapper/022:40:050
84052840,0sleep00-21swapper/022:55:040
14852840,1sleep140-21swapper/1423:41:186
1613028272,4sleep150-21swapper/1519:13:037
1889828072,4sleep00-21swapper/019:17:210
1868527870,4sleep60-21swapper/619:15:0212
189152760,0sleep80-21swapper/821:31:4414
16292760,1sleep140-21swapper/1421:25:576
1864927358,4sleep110-21swapper/1119:14:353
1492927363,4sleep10-21swapper/119:12:491
1788027062,3sleep130-21swapper/1319:13:175
1873026856,4sleep120-21swapper/1219:15:404
320582620,0sleep70-21swapper/720:12:3313
71182600,0sleep150-21swapper/1519:42:117
44432600,0sleep120-21swapper/1221:45:274
117602570,0sleep60-21swapper/620:23:4612
59722550,5sleep15138-21ksoftirqd/1500:01:237
267872550,0sleep10-21swapper/123:36:331
19373995232,18cyclictest0-21swapper/820:13:0814
1936799524,39cyclictest512-21systemd-journal00:10:4510
1936499510,2cyclictest397-21md124_raid121:56:548
1936799500,3cyclictest1042-21NetworkManager22:42:1110
1936799490,2cyclictest0-21swapper/423:10:3910
1936799490,2cyclictest0-21swapper/423:10:3910
1936799480,2cyclictest1043-21ModemManager21:58:3110
1936499480,2cyclictest969-21dbus-daemon21:21:588
1936499480,2cyclictest7056-21sshd23:14:308
1936499480,2cyclictest7056-21sshd23:14:308
1937299470,44cyclictest0-21swapper/722:41:1113
1936799470,2cyclictest0-21swapper/422:48:2710
1936799470,2cyclictest0-21swapper/422:47:0310
1936799470,2cyclictest0-21swapper/421:55:2710
1936799470,2cyclictest0-21swapper/421:38:3910
1936799470,2cyclictest0-21swapper/400:02:5110
1936499470,2cyclictest0-21swapper/222:15:508
1937299460,42cyclictest8038-21sshd22:28:1313
1936799460,3cyclictest969-21dbus-daemon23:00:5510
1936799460,3cyclictest512-21systemd-journal23:19:0310
1936799460,3cyclictest512-21systemd-journal23:19:0310
1936799460,2cyclictest2035-36md124_resync21:19:5910
1936799460,2cyclictest0-21swapper/423:52:3910
1936799460,2cyclictest0-21swapper/422:19:5910
1936799460,2cyclictest0-21swapper/421:46:5110
1936799460,2cyclictest0-21swapper/400:43:5510
1936799460,2cyclictest0-21swapper/400:40:0710
1936799460,2cyclictest0-21swapper/400:20:0710
1936799460,1cyclictest0-21swapper/422:23:1910
1936499460,3cyclictest24295-21sshd21:59:288
1936499460,2cyclictest14048-21systemd-cgroups22:09:548
1936499460,2cyclictest0-21swapper/223:40:188
1936499460,2cyclictest0-21swapper/222:39:428
1936499460,2cyclictest0-21swapper/222:34:268
1936799451,42cyclictest512-21systemd-journal22:17:2210
1936799450,4cyclictest11465-2110-uname22:05:3910
1936799450,2cyclictest0-21swapper/423:38:4710
1936799450,2cyclictest0-21swapper/423:25:3510
1936799450,2cyclictest0-21swapper/423:03:0710
1936799450,2cyclictest0-21swapper/423:03:0710
1936799450,2cyclictest0-21swapper/422:53:2310
1936799450,2cyclictest0-21swapper/422:08:2010
1936799450,2cyclictest0-21swapper/421:48:5910
1936799450,2cyclictest0-21swapper/421:36:3910
1936799450,2cyclictest0-21swapper/421:14:0010
1936799450,1cyclictest0-21swapper/423:16:1910
1936799450,1cyclictest0-21swapper/423:16:1910
1936499450,2cyclictest969-21dbus-daemon23:46:148
1936499450,2cyclictest1-21systemd23:54:508
1936499450,2cyclictest0-21swapper/222:44:468
1936499450,2cyclictest0-21swapper/221:48:428
1936799444,37cyclictest969-21dbus-daemon22:28:0310
1936799441,36cyclictest26719-21sshd00:31:1410
1936799440,42cyclictest5395-21sh21:30:0310
1936799440,2cyclictest397-21md124_raid121:26:5810
1936799440,2cyclictest2035-36md124_resync20:58:4610
1936799440,2cyclictest2035-36md124_resync20:58:4610
1936799440,2cyclictest0-21swapper/423:52:5510
1936799440,2cyclictest0-21swapper/423:47:1910
1936799440,2cyclictest0-21swapper/423:34:0310
1936799440,2cyclictest0-21swapper/423:29:0710
1936799440,2cyclictest0-21swapper/422:33:1510
1936799440,2cyclictest0-21swapper/400:23:5510
1936799440,1cyclictest0-21swapper/421:09:0610
1936799440,1cyclictest0-21swapper/421:09:0610
1936799440,1cyclictest0-21swapper/420:22:2610
1936799440,1cyclictest0-21swapper/400:17:3110
1936499440,41cyclictest11975-21systemd00:37:308
1936499440,41cyclictest0-21swapper/200:13:218
1936499440,3cyclictest26910-21ystemctl21:25:068
1936499440,3cyclictest1-21systemd23:24:378
1936499440,2cyclictest11664-21sshd00:41:118
1936499440,2cyclictest0-21swapper/222:57:228
1936499440,2cyclictest0-21swapper/222:23:188
1936499440,2cyclictest0-21swapper/221:46:308
1936499440,2cyclictest0-21swapper/221:40:068
1936499440,2cyclictest0-21swapper/200:29:198
1936499440,2cyclictest0-21swapper/200:18:478
1936499440,2cyclictest0-21swapper/200:09:518
1936499440,1cyclictest0-21swapper/223:11:548
1936499440,1cyclictest0-21swapper/223:11:548
1937299430,41cyclictest0-21swapper/721:47:2013
1937299430,40cyclictest16950-21systemd22:17:5313
1936799430,3cyclictest2156-21sshd00:00:4310
1936799430,2cyclictest0-21swapper/420:33:4210
1936799430,2cyclictest0-21swapper/420:28:5810
1936799430,1cyclictest0-21swapper/419:58:1810
1936799430,1cyclictest0-21swapper/419:54:1810
1936799430,1cyclictest0-21swapper/419:48:5410
1936799430,1cyclictest0-21swapper/419:43:2610
1936499436,35cyclictest969-21dbus-daemon00:27:228
1936499436,35cyclictest2156-21sshd23:48:418
19364994315,26cyclictest23363-21sshd22:03:088
1936499430,2cyclictest1042-21NetworkManager00:02:228
1936499430,2cyclictest0-21swapper/222:29:108
1936499430,2cyclictest0-21swapper/219:43:108
1936499430,1cyclictest0-21swapper/221:30:068
1937299429,4cyclictest25334-2110-uname23:28:3213
1937299420,39cyclictest27547-21sshd00:00:0013
1937299420,12cyclictest1692-21sshd00:04:2713
1937299420,10cyclictest21811-21sshd23:04:4213
1937299420,10cyclictest21811-21sshd23:04:4213
1936799420,2cyclictest0-21swapper/420:57:0210
1936799420,2cyclictest0-21swapper/420:57:0210
1936799420,2cyclictest0-21swapper/420:05:4210
1936799420,2cyclictest0-21swapper/419:28:3010
1936799420,2cyclictest0-21swapper/419:23:3410
1936799420,1cyclictest0-21swapper/420:41:2210
1936799420,1cyclictest0-21swapper/420:23:4210
1936799420,1cyclictest0-21swapper/419:18:0210
1936499424,35cyclictest1353-21cat22:58:058
1936499421,3cyclictest8910-21sshd23:03:088
1936499421,3cyclictest8910-21sshd23:03:088
1936499421,3cyclictest24420-21apt-get20:42:498
1936499420,3cyclictest0-21swapper/223:35:148
1936499420,2cyclictest0-21swapper/223:19:028
1936499420,2cyclictest0-21swapper/223:19:028
1936499420,2cyclictest0-21swapper/222:19:058
1936499420,2cyclictest0-21swapper/219:28:148
1936499420,1cyclictest0-21swapper/222:49:028
1937299416,3cyclictest22078-21sh23:51:3213
19372994114,24cyclictest11373-21sshd21:19:5813
19372994110,3cyclictest12809-21sshd00:18:1013
1936799410,1cyclictest0-21swapper/420:14:2610
1936799410,1cyclictest0-21swapper/419:33:2610
1936499410,3cyclictest25836-21cp23:28:358
1936499410,2cyclictest0-21swapper/219:23:028
232822400,0sleep00-21swapper/023:55:310
1937299406,3cyclictest6876-21env00:08:4213
1937299400,38cyclictest0-21swapper/722:34:3113
1937299400,37cyclictest0-21swapper/722:53:3413
1936799400,1cyclictest0-21swapper/421:03:4210
1936799400,1cyclictest0-21swapper/421:03:4210
1936799400,1cyclictest0-21swapper/420:48:1410
1936499400,3cyclictest18342-21grep21:13:038
1936499400,2cyclictest397-21md124_raid120:27:388
1936499400,2cyclictest0-21swapper/220:53:028
1936499400,2cyclictest0-21swapper/220:53:028
1936499400,2cyclictest0-21swapper/220:03:168
1937299399,2cyclictest0-21swapper/722:59:5913
1937299399,1cyclictest0-21swapper/720:44:3713
1937299397,3cyclictest9905-2110-uname21:26:5513
1937299397,3cyclictest969-21dbus-daemon22:42:5013
19372993914,23cyclictest0-21swapper/721:53:0113
1937299390,10cyclictest9183-21sshd21:38:3313
1936799390,1cyclictest0-21swapper/420:42:5010
1936799390,1cyclictest0-21swapper/420:10:1810
1936799390,1cyclictest0-21swapper/419:39:5210
1936499391,2cyclictest0-21swapper/220:58:138
1936499391,2cyclictest0-21swapper/220:58:138
90712380,1sleep90-21swapper/921:23:1115
1937299389,3cyclictest0-21swapper/721:58:1113
1937299381,34cyclictest0-21swapper/700:23:5913
1937299380,36cyclictest0-21swapper/721:31:5913
1937299380,10cyclictest15513-21sshd00:29:4413
1936499380,2cyclictest2220-21sed19:33:288
1936499380,2cyclictest0-21swapper/221:07:508
1936499380,2cyclictest0-21swapper/221:07:508
1936499380,2cyclictest0-21swapper/220:28:108
1937299378,3cyclictest969-21dbus-daemon23:20:0613
1937299378,3cyclictest969-21dbus-daemon23:20:0613
1937299378,2cyclictest145-21kauditd00:43:0613
1937299376,3cyclictest693-21missed_timers23:33:1113
1937299376,3cyclictest23139-21latency_hist19:22:4813
1937299375,4cyclictest26475-21bash21:48:0413
19372993713,21cyclictest15695-21systemd22:25:1613
1936499370,2cyclictest0-21swapper/220:14:028
24862360,1sleep30-21swapper/300:00:559
1937299369,4cyclictest20269-2110-uname00:34:1313
1937299369,3cyclictest21396-21sh23:43:3213
1937299368,4cyclictest14792-21sh23:08:0313
1937299368,4cyclictest14792-21sh23:08:0313
1937299368,3cyclictest24353-21uname23:55:4013
1937299367,2cyclictest0-21swapper/719:55:3913
1937299366,3cyclictest512-21systemd-journal22:05:0013
1936499361,2cyclictest0-21swapper/219:18:238
1936499360,3cyclictest0-21swapper/220:18:228
1936499360,34cyclictest0-21swapper/221:03:318
1936499360,34cyclictest0-21swapper/221:03:318
163262360,0sleep90-21swapper/923:23:3415
19381993516,17cyclictest897-17auditd21:58:536
1937299359,3cyclictest709-21run-parts23:41:1213
1937299351,5cyclictest0-21swapper/700:13:0813
19372993510,1cyclictest0-21swapper/721:34:1913
1937299350,33cyclictest0-21swapper/720:07:4313
1937299350,32cyclictest16370-21sshd00:37:5913
1936499351,2cyclictest0-21swapper/220:07:518
1936499351,1cyclictest0-21swapper/219:58:188
1936499350,3cyclictest4287-21tr19:38:098
1936499350,3cyclictest31559-21nfsd420:48:108
1936499350,3cyclictest22629-21cat20:38:108
1936499350,2cyclictest0-21swapper/220:33:118
1936499350,2cyclictest0-21swapper/219:48:048
277272340,0sleep130-21swapper/1323:40:295
1937299349,2cyclictest1-21systemd23:25:2313
1937299349,2cyclictest0-21swapper/720:54:5213
1937299349,2cyclictest0-21swapper/720:54:5213
1937299345,3cyclictest6472-21gdbus22:48:1013
19372993413,19cyclictest0-21swapper/720:28:3313
1937299340,1cyclictest0-21swapper/722:08:0913
1936499340,2cyclictest0-21swapper/219:53:028
14102340,1sleep111041rcuc/1100:36:063
136072340,1sleep121121rcuc/1222:17:334
49392330,1sleep9881rcuc/921:49:0815
272662330,4sleep14130-21ksoftirqd/1400:11:166
218222330,1sleep90-21swapper/900:06:5515
1938199337,23cyclictest1042-21NetworkManager22:18:006
1938199333,24cyclictest1082-21gdbus23:03:016
1938199333,24cyclictest1082-21gdbus23:03:016
1938199332,29cyclictest0-21swapper/1419:33:416
1937299330,9cyclictest7962-21systemd23:14:3813
1937299330,9cyclictest7962-21systemd23:14:3813
1937299330,1cyclictest0-21swapper/720:34:1013
18122330,29sleep00-21swapper/022:27:480
121312330,2sleep012129-21sshd23:19:080
121312330,2sleep012129-21sshd23:19:080
87132320,1sleep110-21swapper/1123:26:493
35662320,27sleep120-21swapper/1220:14:314
318682320,0sleep00-21swapper/000:20:240
241052320,28sleep60-21swapper/600:46:2812
1938199323,27cyclictest22094-21systemctl23:28:126
1938199321,15cyclictest17359-21nfsd21:40:166
1937299329,2cyclictest0-21swapper/719:38:5613
19372993213,16cyclictest0-21swapper/720:41:0313
19372993211,2cyclictest0-21swapper/720:48:5413
1937299320,2cyclictest0-21swapper/719:29:5413
1937299320,29cyclictest0-21swapper/719:27:0613
1937299320,13cyclictest0-21swapper/719:45:0013
1938199314,21cyclictest28241-21systemd-cgroups21:28:486
1938199313,25cyclictest4121-21sshd22:31:566
19381993126,3cyclictest17356-21nfsd00:16:396
1938199312,27cyclictest16492-21cat20:32:496
1938199311,22cyclictest0-21swapper/1421:47:056
19375993114,4cyclictest17592-21sh22:10:232
1937599310,16cyclictest25053-21sshd21:36:482
56832300,2sleep1098-21ksoftirqd/1022:47:282
240342300,27sleep10-21swapper/121:59:271
22392300,2sleep00-21swapper/000:16:240
1938199309,18cyclictest29404-21systemctl21:22:056
1938199305,23cyclictest0-21swapper/1422:27:226
1938199303,24cyclictest12337-21ystemctl22:59:236
1938199302,26cyclictest0-21swapper/1400:02:386
1938199302,25cyclictest0-21swapper/1422:38:186
19381993013,15cyclictest0-21swapper/1421:48:296
1938199301,22cyclictest0-21swapper/1422:12:106
1937299300,28cyclictest0-21swapper/719:49:0513
64892290,25sleep110-21swapper/1122:32:153
315092290,26sleep90-21swapper/900:00:3015
304552290,1sleep80-21swapper/822:34:5214
1938199296,18cyclictest17360-21nfsd22:48:026
1938199292,25cyclictest0-21swapper/1400:36:506
1938199292,21cyclictest2391-21systemd21:33:306
19381992915,13cyclictest0-21swapper/1422:13:376
1938199291,20cyclictest31860-21systemd-cgroups00:04:126
1937299299,2cyclictest397-21md124_raid119:34:1413
159722290,3sleep1216082-21systemd22:25:194
128962290,1sleep30-21swapper/300:41:199
108272290,0sleep150-21swapper/1523:07:377
108272290,0sleep150-21swapper/1523:07:377
228072280,1sleep90-21swapper/922:37:5015
206062280,1sleep10-21swapper/121:36:111
1938199289,16cyclictest13238-21sh23:19:186
1938199289,16cyclictest13238-21sh23:19:186
1938199289,14cyclictest46750jbd2/md125p1-823:54:466
1938199288,18cyclictest0-21swapper/1423:43:386
1938199287,19cyclictest11999-21kill23:34:326
1938199285,21cyclictest825-21systemd23:13:466
1938199285,21cyclictest825-21systemd23:13:466
1938199284,21cyclictest29961-21sshd22:34:486
1938199283,23cyclictest0-21swapper/1419:23:436
19381992821,5cyclictest19248-21systemd21:54:586
1938199280,14cyclictest0-21swapper/1423:25:166
171642280,1sleep50-21swapper/522:37:1411
166452280,2sleep100-21swapper/1022:06:182
40772270,2sleep100-21swapper/1023:33:302
322712270,1sleep90-21swapper/922:12:2515
305222270,3sleep10-21swapper/122:15:461
284602270,3sleep12114-21ksoftirqd/1223:44:294
271492270,23sleep50-21swapper/500:03:3911
238932270,23sleep00-21swapper/021:59:260
1938199279,16cyclictest0-21swapper/1420:13:476
1938199277,17cyclictest968-21systemd-logind22:43:576
1938199276,18cyclictest1042-21NetworkManager22:55:406
1938199275,19cyclictest18589-21cut00:38:076
1938199275,16cyclictest253261kill22:03:186
19381992722,3cyclictest22726-21systemctl00:26:466
1938199272,23cyclictest0-21swapper/1421:14:016
19381992714,11cyclictest0-21swapper/1419:39:526
19381992712,13cyclictest12940-21uname23:07:486
19381992712,13cyclictest12940-21uname23:07:486
19381992710,15cyclictest32522-21sshd23:48:436
19372992711,2cyclictest0-21swapper/721:12:2513
19372992711,2cyclictest0-21swapper/721:12:2513
19372992711,2cyclictest0-21swapper/721:01:3713
19372992711,2cyclictest0-21swapper/721:01:3713
6842260,3sleep51-21systemd21:56:5111
52702260,2sleep50-21swapper/522:04:5211
4272260,23sleep60-21swapper/623:41:1112
1938199269,14cyclictest1082-21gdbus00:29:276
1938199267,3cyclictest0-21swapper/1420:01:556
1938199266,18cyclictest0-21swapper/1420:23:126
1938199266,18cyclictest0-21swapper/1420:05:336
1938199265,5cyclictest0-21swapper/1419:29:246
1938199264,20cyclictest0-21swapper/1420:58:266
1938199264,20cyclictest0-21swapper/1420:58:266
19381992612,7cyclictest46750jbd2/md125p1-800:20:246
19373992622,2cyclictest0-21swapper/822:52:5214
192532260,22sleep130-21swapper/1323:51:125
181122260,22sleep318230-21sshd23:12:159
181122260,22sleep318230-21sshd23:12:159
165822260,1sleep50-21swapper/522:59:5811
68442255,16sleep16472-21gdbus22:05:041
46272250,1sleep90-21swapper/922:12:5715
41872250,2sleep120-21swapper/1221:57:144
325922250,2sleep5397-21md124_raid120:48:2211
30762250,0sleep120-21swapper/1222:58:124
251082250,21sleep110-21swapper/1123:16:563
251082250,21sleep110-21swapper/1123:16:563
250542250,21sleep00-21swapper/023:28:310
214752252,19sleep139892-21rs:main5
214752252,19sleep139892-21rs:main5
206932250,2sleep150-21swapper/1523:35:437
1938199259,15cyclictest0-21swapper/1419:43:176
1938199259,14cyclictest397-21md124_raid120:35:436
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional