You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-31 - 01:27

x86 Intel Xeon E5620 @2400 MHz, Linux 4.19.127-rt55 (Profile)

Latency plot of system in rack #c, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackcslot0.osadl.org (updated Thu Mar 30, 2023 00:43:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1359421880,6sleep111239499cyclictest20:36:493
3010621600,3sleep151239899cyclictest23:16:517
2868721300,0sleep110-21swapper/1100:26:203
117672126107,14sleep60-21swapper/619:19:4412
117042119103,12sleep00-21swapper/019:18:500
2954321150,1sleep100-21swapper/1021:26:492
2954321150,1sleep100-21swapper/1021:26:492
117192114103,6sleep120-21swapper/1219:19:024
1948921090,5sleep31238299cyclictest21:42:459
1328221090,2sleep60-21swapper/622:42:5912
1627121080,1sleep150-21swapper/1500:23:277
11620210886,17sleep80-21swapper/819:17:3714
1440621060,1sleep00-21swapper/021:33:040
11896210685,17sleep70-21swapper/719:20:4913
1926921030,2sleep010-21ksoftirqd/000:04:120
11673210291,6sleep40-21swapper/419:18:2210
1956021010,3sleep21238199cyclictest22:26:208
1261621010,1sleep20-21swapper/223:23:038
1116821010,1sleep011166-21smartctl22:46:540
175922970,2sleep9969-21dbus-daemon23:10:5115
148222970,0sleep110-21swapper/1100:12:573
298652960,0sleep120-21swapper/1223:34:434
298652960,0sleep120-21swapper/1223:34:434
263232960,0sleep90-21swapper/922:26:5915
262972960,1sleep9891ktimersoftd/921:56:4515
112962960,5sleep51238499cyclictest22:29:2011
58032940,1sleep80-21swapper/822:15:4414
278882940,1sleep30-21swapper/322:45:059
228822940,0sleep20-21swapper/223:38:028
228822940,0sleep20-21swapper/223:38:028
202342940,1sleep3391rcuc/322:00:249
163832930,6sleep11237899cyclictest21:21:191
163832930,6sleep11237899cyclictest21:21:191
1186829369,19sleep130-21swapper/1319:20:235
283132920,2sleep70-21swapper/700:31:2113
230002910,1sleep60-21swapper/621:30:1312
230002910,1sleep60-21swapper/621:30:1312
12082910,5sleep101239299cyclictest22:14:582
1168029177,8sleep100-21swapper/1019:18:282
255792900,1sleep60-21swapper/623:29:3212
255792900,1sleep60-21swapper/623:29:3212
1176229070,15sleep10-21swapper/119:19:391
50602890,0sleep50-21swapper/519:51:1511
1156128981,4sleep30-21swapper/319:17:009
126132880,0sleep8811ktimersoftd/823:59:1614
1158328772,4sleep50-21swapper/519:17:1811
298462850,3sleep51238499cyclictest00:44:4111
190872840,0sleep100-21swapper/1022:17:382
1177728472,8sleep140-21swapper/1419:19:526
132432830,0sleep130-21swapper/1323:59:215
1172628375,4sleep20-21swapper/219:19:088
1165928375,4sleep90-21swapper/919:18:1115
1162728372,6sleep150-21swapper/1519:17:447
1162328375,4sleep110-21swapper/1119:17:403
54692820,5sleep51238499cyclictest22:41:5411
237612820,0sleep120-21swapper/1221:30:194
237612820,0sleep120-21swapper/1221:30:194
254022810,0sleep10-21swapper/122:01:101
242022780,1sleep40-21swapper/423:56:3110
39162750,0sleep110-21swapper/1123:21:513
70662740,0sleep120-21swapper/1222:02:424
217482690,1sleep150-21swapper/1522:17:597
125892640,0sleep80-21swapper/819:57:1114
203662630,0sleep30-21swapper/321:25:549
203662630,0sleep30-21swapper/321:25:549
56462540,1sleep90-21swapper/921:27:5215
56462540,1sleep90-21swapper/921:27:5215
1238499510,44cyclictest0-21swapper/521:51:3711
212902500,2sleep60-21swapper/600:39:0512
1238499500,3cyclictest1061-21gdbus23:59:3811
1238499480,2cyclictest0-21swapper/522:11:5011
1238499473,31cyclictest0-21swapper/523:52:5111
1238499470,3cyclictest5012-21sshd00:50:5011
1238499470,2cyclictest0-21swapper/523:17:4111
1238499470,2cyclictest0-21swapper/500:30:2211
231452460,7sleep41238399cyclictest23:38:0410
231452460,7sleep41238399cyclictest23:38:0410
1238499460,3cyclictest968-21systemd-logind23:22:1411
1238499460,3cyclictest1061-21gdbus00:19:0211
1238499460,2cyclictest0-21swapper/523:50:5811
1238499460,2cyclictest0-21swapper/523:42:2511
1238499460,2cyclictest0-21swapper/523:42:2511
1238499460,2cyclictest0-21swapper/523:26:4811
1238499460,2cyclictest0-21swapper/523:26:4811
1238499460,2cyclictest0-21swapper/523:07:2211
1238499460,2cyclictest0-21swapper/522:50:3311
1238499460,2cyclictest0-21swapper/521:58:0111
1238499453,39cyclictest0-21swapper/523:41:0611
1238499453,39cyclictest0-21swapper/523:41:0611
1238499450,39cyclictest6903-21sshd22:02:4011
1238499450,2cyclictest0-21swapper/523:15:0611
1238499450,2cyclictest0-21swapper/521:35:2111
1238499450,2cyclictest0-21swapper/500:06:5411
1238499450,1cyclictest0-21swapper/523:00:2211
1238499450,1cyclictest0-21swapper/522:40:3711
1238499450,1cyclictest0-21swapper/522:24:5311
1238499450,1cyclictest0-21swapper/500:01:5811
177322440,7sleep131239699cyclictest22:00:025
1239899441,19cyclictest2418-21ffmpeg00:11:167
1238499440,2cyclictest0-21swapper/522:20:5711
1238499440,2cyclictest0-21swapper/521:31:1311
1238499440,2cyclictest0-21swapper/521:31:1311
1238499440,2cyclictest0-21swapper/520:56:3711
1238499440,2cyclictest0-21swapper/500:23:3811
1238499440,1cyclictest0-21swapper/523:34:3011
1238499440,1cyclictest0-21swapper/523:34:3011
1238499440,1cyclictest0-21swapper/522:31:2111
1238499440,1cyclictest0-21swapper/521:22:1311
1238499440,1cyclictest0-21swapper/521:22:1311
1238499440,1cyclictest0-21swapper/500:38:2211
1238499430,2cyclictest968-21systemd-logind00:34:5411
1238499430,1cyclictest0-21swapper/523:02:3011
1238499430,1cyclictest0-21swapper/520:31:2111
1238499430,1cyclictest0-21swapper/520:16:0111
1238499430,1cyclictest0-21swapper/520:06:4911
1238499430,1cyclictest0-21swapper/500:13:5811
1238499420,3cyclictest5089-21sshd22:06:3711
1238499420,2cyclictest0-21swapper/522:51:2611
1238499420,2cyclictest0-21swapper/521:41:4911
1238499420,2cyclictest0-21swapper/521:38:0511
1238499420,2cyclictest0-21swapper/519:56:3711
1238499420,1cyclictest0-21swapper/521:46:1711
1238499420,1cyclictest0-21swapper/521:06:4111
1238499420,1cyclictest0-21swapper/520:36:3711
1238499410,2cyclictest0-21swapper/519:26:3711
1238499410,1cyclictest0-21swapper/520:47:1711
1238499410,1cyclictest0-21swapper/520:21:5311
1238499410,1cyclictest0-21swapper/520:16:4111
1238499410,1cyclictest0-21swapper/519:37:1311
1238499400,37cyclictest2779-21cat20:26:3611
1238499400,2cyclictest0-21swapper/521:16:3311
1238499400,2cyclictest0-21swapper/521:16:3311
1238499400,2cyclictest0-21swapper/519:31:2111
1238499400,1cyclictest0-21swapper/521:13:1711
1238499400,1cyclictest0-21swapper/519:51:3311
1238499400,1cyclictest0-21swapper/519:41:5711
1238499400,1cyclictest0-21swapper/519:22:0911
1238499390,1cyclictest0-21swapper/520:51:2511
1238499390,1cyclictest0-21swapper/520:41:5311
1238499390,1cyclictest0-21swapper/520:06:1311
314162380,0sleep120-21swapper/1222:49:464
1238499370,1cyclictest0-21swapper/521:01:2111
115332350,0sleep10-21swapper/122:46:551
305832340,0sleep80-21swapper/823:12:2714
229552340,1sleep150-21swapper/1500:04:447
220902340,2sleep20-21swapper/222:18:028
294122330,1sleep150-21swapper/1519:41:167
167982330,2sleep60-21swapper/600:47:2312
76922320,1sleep137691-21sshd00:32:475
44912320,28sleep150-21swapper/1523:58:037
1239599322,25cyclictest0-21swapper/1222:39:084
1239599320,24cyclictest1956-21systemd-cgroups21:44:514
83982310,1sleep80-21swapper/820:31:4114
41222310,1sleep120-21swapper/1200:50:414
129212310,1sleep150-21swapper/1519:21:287
1239599314,20cyclictest968-21systemd-logind23:17:544
265922300,1sleep10-21swapper/121:47:481
207952300,2sleep20-21swapper/200:39:028
157402300,2sleep449-21ksoftirqd/423:27:5410
157402300,2sleep449-21ksoftirqd/423:27:5410
146252300,0sleep90-21swapper/922:47:1915
13212300,1sleep10-21swapper/123:57:321
1239599301,26cyclictest1082-21gdbus23:43:054
1239599301,26cyclictest1082-21gdbus23:43:054
33372290,25sleep80-21swapper/822:50:3214
287782290,26sleep150-21swapper/1521:22:387
287782290,26sleep150-21swapper/1521:22:387
21702290,25sleep20-21swapper/222:23:578
17742290,2sleep150-21swapper/1500:45:237
12395992915,11cyclictest3985-21expr22:06:344
1239599290,26cyclictest2156-21sshd00:23:304
32302280,24sleep90-21swapper/923:04:1115
148752280,1sleep30-21swapper/323:45:499
148752280,1sleep30-21swapper/323:45:499
145902280,1sleep0121rcuc/022:21:290
141982280,22sleep140-21swapper/1421:33:026
1239599285,18cyclictest0-21swapper/1222:35:314
1239599284,22cyclictest0-21swapper/1222:30:124
1239599284,18cyclictest0-21swapper/1200:02:544
12395992823,2cyclictest968-21systemd-logind00:38:444
1239599282,20cyclictest1024-21sshd23:12:534
1239599281,25cyclictest0-21swapper/1223:53:184
1239599281,24cyclictest12921-21rm22:56:344
1239599281,24cyclictest0-21swapper/1223:21:424
1239599280,25cyclictest0-21swapper/1223:46:284
108032280,2sleep10-21swapper/123:45:151
108032280,2sleep10-21swapper/123:45:151
107962280,1sleep10-21swapper/100:12:161
99132270,23sleep80-21swapper/800:12:0714
31062270,2sleep80-21swapper/823:04:1014
297132270,0sleep140-21swapper/1400:19:416
291952270,1sleep80-21swapper/800:19:3514
291292270,1sleep151361rcuc/1500:36:027
231402270,1sleep30-21swapper/321:47:199
223782270,1sleep110-21swapper/1123:20:153
199892270,1sleep70-21swapper/721:42:5013
1239599279,15cyclictest0-21swapper/1221:40:584
1239599275,19cyclictest27422-21bash22:54:004
1239599273,21cyclictest15309-21gdbus23:38:024
1239599273,21cyclictest15309-21gdbus23:38:024
1239599272,22cyclictest0-21swapper/1200:16:454
12395992713,12cyclictest0-21swapper/1200:14:144
12395992713,11cyclictest12089-21avahi-daemon23:27:074
12395992713,11cyclictest12089-21avahi-daemon23:27:074
1239599271,24cyclictest0-21swapper/1200:26:564
1239599271,17cyclictest261211sshd23:07:074
1239599270,24cyclictest1-21systemd23:57:254
12394992724,2cyclictest1041rcuc/1122:36:343
315322260,2sleep70-21swapper/723:07:5413
257962260,2sleep80-21swapper/823:29:3514
257962260,2sleep80-21swapper/823:29:3514
22472260,22sleep150-21swapper/1522:37:167
193612260,2sleep9891ktimersoftd/922:52:4015
130092260,1sleep40-21swapper/400:46:4910
1239599268,15cyclictest29559-21sshd21:57:084
1239599267,8cyclictest1736250nfsd22:43:524
1239599267,11cyclictest0-21swapper/1200:33:444
1239599265,15cyclictest0-21swapper/1222:16:354
1239599264,20cyclictest0-21swapper/1200:08:504
1239599263,21cyclictest0-21swapper/1200:41:184
12395992613,11cyclictest17980-21sshd21:21:314
12395992613,11cyclictest17980-21sshd21:21:314
12395992612,8cyclictest0-21swapper/1223:02:154
1239499267,17cyclictest1051ktimersoftd/1123:56:293
1239499261,22cyclictest0-21swapper/1121:42:353
1238999263,21cyclictest19595-21kworker/9:200:13:5615
1238999260,23cyclictest12089-21avahi-daemon21:41:3315
1238399261,22cyclictest0-21swapper/421:40:4810
1237899261,23cyclictest24022-21gdbus21:42:191
285902254,16sleep20-21swapper/222:41:048
262602250,2sleep130-21swapper/1322:53:495
150832250,21sleep13969-21dbus-daemon22:47:245
142762250,1sleep40-21swapper/423:32:2610
142762250,1sleep40-21swapper/423:32:2610
128892250,21sleep80-21swapper/800:33:3114
1239599258,14cyclictest24532-21sshd21:47:314
1239599256,16cyclictest968-21systemd-logind22:14:374
1239599255,17cyclictest23073-21sh21:34:154
12395992519,4cyclictest126832chrt19:58:194
12395992511,11cyclictest16881-21sshd21:55:324
1239599250,2cyclictest0-21swapper/1219:31:184
1239599250,19cyclictest18101-21sshd22:21:424
1239499255,15cyclictest106-21ksoftirqd/1122:03:443
1238999251,22cyclictest24022-21gdbus21:40:3415
1238299251,21cyclictest20035-21sshd21:38:339
115112250,21sleep80-21swapper/821:37:1414
86972240,2sleep130-21swapper/1300:21:505
78562240,20sleep100-21swapper/1022:33:422
326852240,20sleep70-21swapper/700:11:0213
268022240,22sleep100-21swapper/1019:36:392
263632240,20sleep80-21swapper/800:09:5914
251682240,0sleep130-21swapper/1323:11:445
223042240,1sleep00-21swapper/022:35:570
209692240,0sleep70-21swapper/721:21:4113
209692240,0sleep70-21swapper/721:21:4113
209432240,1sleep320942-21systemd-cgroups00:39:029
179552240,1sleep40-21swapper/400:17:5610
175672240,21sleep00-21swapper/000:29:380
159332240,0sleep140-21swapper/1422:43:236
158442240,1sleep20-21swapper/221:37:568
145502240,20sleep100-21swapper/1000:23:072
134812240,1sleep70-21swapper/700:46:5313
1239699241,21cyclictest6994-2110-uname23:09:065
1239599245,17cyclictest0-21swapper/1219:47:204
1239599244,18cyclictest0-21swapper/1220:26:494
1239599244,17cyclictest2644-21latency_hist21:06:144
1239599243,19cyclictest0-21swapper/1221:06:574
1239599243,19cyclictest0-21swapper/1220:11:514
1239599243,19cyclictest0-21swapper/1220:02:324
1239599243,18cyclictest0-21swapper/1220:36:314
1239499247,8cyclictest106-21ksoftirqd/1100:18:563
1239499247,8cyclictest0-21swapper/1100:36:293
1239499246,11cyclictest106-21ksoftirqd/1123:45:283
1239499246,11cyclictest106-21ksoftirqd/1123:45:283
1239499245,8cyclictest0-21swapper/1100:51:003
1239499245,13cyclictest106-21ksoftirqd/1123:48:533
12394992412,7cyclictest0-21swapper/1123:41:023
12394992412,7cyclictest0-21swapper/1123:41:023
12394992412,6cyclictest131rcu_preempt22:47:503
1238699240,22cyclictest0-21swapper/721:50:0113
12385992420,2cyclictest0-21swapper/600:03:2312
1238299241,21cyclictest177-21kswapd000:25:389
109592240,2sleep211053-21sshd23:31:538
109592240,2sleep211053-21sshd23:31:538
106212240,2sleep100-21swapper/1022:46:482
75542230,1sleep60-21swapper/623:00:0712
318302230,1sleep8801rcuc/823:48:0114
314232230,1sleep40-21swapper/422:06:0310
27892230,1sleep150-21swapper/1500:27:137
259232230,19sleep10-21swapper/121:39:311
212132230,1sleep80-21swapper/821:30:0014
212132230,1sleep80-21swapper/821:30:0014
205172230,2sleep40-21swapper/423:24:1010
204142230,17sleep40-21swapper/422:17:4910
203892230,19sleep30-21swapper/323:24:099
169882230,1sleep20-21swapper/200:13:218
1239799230,21cyclictest0-21swapper/1421:41:056
1239599236,15cyclictest0-21swapper/1219:21:344
1239599234,17cyclictest0-21swapper/1221:18:034
1239599234,17cyclictest0-21swapper/1221:18:034
1239599233,18cyclictest0-21swapper/1220:41:274
1239599232,19cyclictest0-21swapper/1221:12:254
1239599232,19cyclictest0-21swapper/1219:51:374
1239599232,18cyclictest27208-21grep20:16:364
1239599231,19cyclictest0-21swapper/1220:31:524
1239499236,10cyclictest15792-21sshd00:33:593
12394992312,6cyclictest131rcu_preempt21:56:073
1239499230,21cyclictest1051ktimersoftd/1123:05:013
12389992319,2cyclictest0-21swapper/900:45:2715
12386992317,4cyclictest969-21dbus-daemon23:52:2013
65112220,18sleep140-21swapper/1423:22:126
320292220,16sleep100-21swapper/1021:31:232
317602220,1sleep40-21swapper/422:54:4410
313232220,1sleep130-21swapper/1322:58:545
300402220,2sleep7969-21dbus-daemon00:44:4413
299142220,1sleep10-21swapper/119:41:341
248222220,19sleep20-21swapper/223:42:358
248222220,19sleep20-21swapper/223:42:358
234692220,19sleep80-21swapper/822:40:2214
185812220,1sleep40-21swapper/421:25:4010
185812220,1sleep40-21swapper/421:25:4010
151072220,1sleep60-21swapper/623:23:2412
125842220,2sleep110-21swapper/1100:22:363
1239599225,15cyclictest0-21swapper/1219:27:174
1239599224,16cyclictest7781-21snmpd20:06:314
1239599224,16cyclictest0-21swapper/1220:56:444
1239599223,17cyclictest19831-21cstates20:46:334
1239599222,19cyclictest0-21swapper/1219:37:434
1239599222,18cyclictest0-21swapper/1220:51:474
1239599222,18cyclictest0-21swapper/1219:44:184
1239499226,8cyclictest106-21ksoftirqd/1121:56:363
1239499225,9cyclictest106-21ksoftirqd/1122:06:283
1239499225,9cyclictest0-21swapper/1122:16:583
1239499225,11cyclictest106-21ksoftirqd/1121:21:353
1239499225,11cyclictest106-21ksoftirqd/1121:21:353
12394992218,2cyclictest97550irq/38-eth0-TxR22:31:173
12394992212,5cyclictest141rcu_sched23:13:033
1239499221,19cyclictest1051ktimersoftd/1122:21:183
12394992211,6cyclictest131rcu_preempt21:39:093
12394992211,5cyclictest131rcu_preempt21:50:053
12389992218,3cyclictest0-21swapper/900:46:3115
12389992218,2cyclictest0-21swapper/922:06:0815
1238699221,3cyclictest969-21dbus-daemon23:41:4513
1238699221,3cyclictest969-21dbus-daemon23:41:4513
1238599221,2cyclictest0-21swapper/622:01:3312
12382992218,2cyclictest0-21swapper/321:54:099
12382992214,5cyclictest23641-21sshd23:20:289
5472210,2sleep40-21swapper/423:48:1410
317282210,1sleep70-21swapper/722:14:4013
303092210,1sleep150-21swapper/1523:43:217
303092210,1sleep150-21swapper/1523:43:217
254642210,19sleep20-21swapper/220:12:378
241652210,1sleep100-21swapper/1021:22:012
241652210,1sleep100-21swapper/1021:22:012
176722210,1sleep150-21swapper/1500:47:327
153102210,1sleep1515446-21sshd22:03:507
12398992116,3cyclictest18043-21sshd21:51:067
12398992115,4cyclictest25362-2110-uname22:09:177
12398992114,4cyclictest8311-21sh23:36:177
12398992114,4cyclictest8311-21sh23:36:177
12396992115,4cyclictest15110-21run-parts21:29:095
12396992115,4cyclictest15110-21run-parts21:29:095
1239599211,18cyclictest0-21swapper/1220:22:004
1239499218,8cyclictest106-21ksoftirqd/1121:28:393
1239499218,8cyclictest106-21ksoftirqd/1121:28:393
1239499218,7cyclictest106-21ksoftirqd/1100:01:233
1239499217,6cyclictest0-21swapper/1122:56:203
1239499216,10cyclictest106-21ksoftirqd/1100:42:383
1239499215,9cyclictest11378-21sshd00:07:243
1239499215,9cyclictest106-21ksoftirqd/1123:28:083
1239499215,9cyclictest106-21ksoftirqd/1123:28:083
1239499214,8cyclictest0-21swapper/1123:32:583
1239499214,8cyclictest0-21swapper/1123:32:583
1239499214,11cyclictest0-21swapper/1122:42:593
12394992116,4cyclictest1041rcuc/1122:12:433
12394992112,5cyclictest141rcu_sched21:31:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional