You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 11:23
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackcslot0.osadl.org (updated Sun Apr 28, 2024 00:43:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3234921810,5sleep133034499cyclictest23:29:015
296792130104,21sleep10-21swapper/119:10:301
296812122103,14sleep30-21swapper/319:10:329
2732321220,6sleep103034099cyclictest00:25:562
2237021190,0sleep00-21swapper/021:24:410
295502118103,10sleep130-21swapper/1319:08:495
708221140,2sleep47084-21sshd21:13:3110
29770210590,10sleep00-21swapper/019:11:490
29499210581,20sleep40-21swapper/419:08:0810
29544210391,8sleep70-21swapper/719:08:4313
2482421030,1sleep120-21swapper/1222:58:304
1311221020,2sleep50-21swapper/522:22:1711
2620021010,0sleep10-21swapper/121:54:241
234782980,1sleep110-21swapper/1121:43:523
2978029765,27sleep80-21swapper/819:11:5714
175442970,0sleep70-21swapper/700:29:0513
287952960,4sleep13032799cyclictest00:21:141
65332950,2sleep96528-21sshd23:55:0915
2949729577,13sleep20-21swapper/219:08:068
212142950,6sleep03032699cyclictest22:13:060
157292950,1sleep0111ktimersoftd/023:11:560
157292950,1sleep0111ktimersoftd/023:11:560
255492920,0sleep30-21swapper/319:47:259
134912910,0sleep70-21swapper/722:32:1213
266582900,0sleep10-21swapper/122:08:501
2956628981,4sleep110-21swapper/1119:09:033
2954728974,11sleep100-21swapper/1019:08:462
2954928872,12sleep120-21swapper/1219:08:484
2951428875,7sleep150-21swapper/1519:08:197
2977728776,7sleep60-21swapper/619:11:5512
259682870,2sleep225953-21sshd00:20:438
2953428672,7sleep140-21swapper/1419:08:346
276062860,1sleep927611-21cpuspeed_turbos20:32:2615
276062860,1sleep927611-21cpuspeed_turbos20:32:2615
54652850,1sleep120-21swapper/1223:45:024
135202850,5sleep43033199cyclictest21:27:5810
30012840,1sleep50-21swapper/522:30:1811
189202840,0sleep120-21swapper/1223:02:314
189202840,0sleep120-21swapper/1223:02:314
173592840,1sleep1231rcuc/122:57:311
9672830,1sleep20-21swapper/222:25:108
2639028375,4sleep50-21swapper/519:07:2111
32772820,1sleep20-21swapper/220:42:288
32772820,1sleep20-21swapper/220:42:288
2956428274,4sleep90-21swapper/919:09:0115
241742820,2sleep110-21swapper/1119:47:163
285422810,0sleep90-21swapper/921:54:4815
140452760,0sleep25040-21nfsd23:02:058
214382740,0sleep90-21swapper/921:48:0715
82482710,6sleep03032699cyclictest23:05:470
82482710,6sleep03032699cyclictest23:05:470
259752660,6sleep73033799cyclictest00:03:0913
322982650,6sleep03032699cyclictest21:31:230
3033499530,41cyclictest0-21swapper/623:28:5812
3033499521,2cyclictest0-21swapper/621:29:5812
280432520,1sleep00-21swapper/000:26:020
119482520,5sleep53033299cyclictest23:36:0611
3033499501,3cyclictest1-21systemd23:41:4312
3033499500,2cyclictest14690-21sshd23:16:5012
3033499500,2cyclictest14690-21sshd23:16:5012
3033499480,3cyclictest1029-21gdbus22:12:3412
3033499480,3cyclictest0-21swapper/623:32:1812
3033499480,3cyclictest0-21swapper/623:07:5012
3033499480,3cyclictest0-21swapper/623:07:5012
3033499480,2cyclictest5039-21nfsd00:08:1012
139162480,1sleep00-21swapper/022:27:170
3033499470,2cyclictest0-21swapper/622:35:1812
3033499470,2cyclictest0-21swapper/622:31:2612
3033499470,2cyclictest0-21swapper/621:41:4612
3033499470,2cyclictest0-21swapper/600:15:5412
3033499466,38cyclictest998-21dbus-daemon23:26:2412
3033499460,3cyclictest998-21dbus-daemon22:58:0612
3033499460,3cyclictest1076-21gdbus23:05:3012
3033499460,3cyclictest1076-21gdbus23:05:3012
3033499460,2cyclictest998-21dbus-daemon21:19:2212
3033499460,2cyclictest789-21jbd2/md125-821:52:3812
3033499460,2cyclictest1005-21NetworkManager22:43:3812
3033499460,2cyclictest0-21swapper/623:56:3812
3033499460,2cyclictest0-21swapper/623:20:4612
3033499460,2cyclictest0-21swapper/623:20:4612
3033499460,2cyclictest0-21swapper/622:49:2212
3033499460,2cyclictest0-21swapper/622:24:4612
3033499460,2cyclictest0-21swapper/621:49:4212
3033499460,2cyclictest0-21swapper/621:43:1412
3033499460,2cyclictest0-21swapper/600:30:0212
3033499455,37cyclictest1014-21systemd-logind21:25:2812
3033499450,4cyclictest29797-21sshd22:39:3912
3033499450,3cyclictest13022-21sshd22:57:0212
3033499450,2cyclictest5035-21nfsd22:03:3412
3033499450,2cyclictest27286-21sshd00:20:5712
3033499450,2cyclictest0-21swapper/623:45:1412
3033499450,2cyclictest0-21swapper/622:09:5012
3033499441,40cyclictest19725-21uname22:17:4812
3033499440,3cyclictest998-21dbus-daemon21:32:3012
3033499440,3cyclictest27217-21/usr/sbin/munin20:32:2612
3033499440,3cyclictest27217-21/usr/sbin/munin20:32:2612
3033499440,3cyclictest1-21systemd00:24:1012
3033499440,2cyclictest0-21swapper/623:57:2212
3033499440,2cyclictest0-21swapper/621:58:1812
153132440,1sleep130-21swapper/1300:38:175
89532430,1sleep141291ktimersoftd/1422:11:286
3033499431,3cyclictest998-21dbus-daemon21:16:0512
3033499430,3cyclictest9244-21sshd23:50:3012
3033499430,2cyclictest0-21swapper/600:33:3812
158502430,1sleep110-21swapper/1123:31:393
30340994236,4cyclictest10962-2110-uname21:42:182
3033499421,3cyclictest1-21systemd00:03:2012
3033499410,3cyclictest0-21swapper/619:27:3812
30345994018,20cyclictest0-21swapper/1420:07:266
3033499400,37cyclictest13657-21sshd00:38:0012
30332994010,28cyclictest0-21swapper/523:27:3811
15192390,2sleep70-21swapper/700:22:0813
3033499381,3cyclictest14797-21latency_hist20:57:1612
3033499380,3cyclictest7270-21apt-get20:47:2612
3033499380,3cyclictest7270-21apt-get20:47:2612
3033499380,3cyclictest3739-21expr20:42:2912
3033499380,3cyclictest3739-21expr20:42:2912
3033499380,3cyclictest19961-21grep21:02:2712
3033499380,2cyclictest0-21swapper/620:42:1612
3033499380,2cyclictest0-21swapper/620:42:1612
3033499375,29cyclictest10451-21/usr/sbin/munin20:12:3912
3033499370,3cyclictest27445-21cut21:12:1612
3033499370,2cyclictest6291-21apt-get20:07:2612
3033499370,2cyclictest0-21swapper/620:57:3812
3033499370,2cyclictest0-21swapper/619:57:1612
3033499370,2cyclictest0-21swapper/619:47:5412
3033499370,2cyclictest0-21swapper/619:37:2812
3033299378,28cyclictest0-21swapper/500:15:5611
3033299376,29cyclictest0-21swapper/522:04:1011
3033299376,24cyclictest0-21swapper/521:55:4611
3033299376,22cyclictest0-21swapper/522:23:0011
3033299375,25cyclictest289571sshd23:43:4011
30329993711,4cyclictest3291-21run-parts22:50:129
3033499360,3cyclictest5751-21date20:07:1712
3033499360,3cyclictest24554-21sed19:47:1712
3033499360,33cyclictest0-21swapper/619:12:4112
3033299369,24cyclictest7158-21systemd-cgroups00:32:2711
3033299369,24cyclictest13096-21sshd21:18:4311
3033299368,18cyclictest12947-21systemd-cgroups22:51:5211
30332993620,14cyclictest1014-21systemd-logind23:06:0411
30332993620,14cyclictest1014-21systemd-logind23:06:0411
30332993611,20cyclictest245161sshd23:23:0411
30332993610,23cyclictest0-21swapper/521:57:3811
30329993613,4cyclictest29815-21sshd21:35:499
3034499353,25cyclictest1014-21systemd-logind22:45:115
3033499351,2cyclictest0-21swapper/619:57:4612
3033499350,3cyclictest5496-21chrt19:18:0312
3033499350,3cyclictest27078-21date20:32:1612
3033499350,3cyclictest27078-21date20:32:1612
3033499350,33cyclictest18505-21latency_hist20:22:1612
3033499350,2cyclictest13382-21kworker/6:119:32:1812
3033299359,24cyclictest0-21swapper/521:15:5311
3033299358,25cyclictest0-21swapper/522:07:3411
3033299358,20cyclictest998-21dbus-daemon22:42:1111
3033299356,20cyclictest32752-21sshd21:26:1711
3033299355,28cyclictest0-21swapper/521:35:3711
3033299355,24cyclictest998-21dbus-daemon23:15:5311
3033299355,24cyclictest998-21dbus-daemon23:15:5311
3033299354,29cyclictest0-21swapper/500:18:2011
30329993512,3cyclictest26362-21sh22:14:079
3033499341,2cyclictest0-21swapper/619:23:1612
3033499340,3cyclictest5040-21nfsd20:22:1812
3033299349,22cyclictest31039-21sshd23:38:5411
3033299349,18cyclictest18290-21sshd21:43:0311
3033299348,17cyclictest267811uname00:25:5111
30332993414,16cyclictest9971-21cups-browsed23:07:3911
30332993414,16cyclictest9971-21cups-browsed23:07:3911
30332993413,19cyclictest1033-21gdbus21:40:2911
30332993413,18cyclictest5036-21nfsd00:03:0011
30332993412,20cyclictest12541-21env00:01:2511
3033299340,23cyclictest998-21dbus-daemon22:58:0111
122602340,31sleep20-21swapper/200:10:508
81392331,3sleep71005-21NetworkManager21:46:2913
3033299338,16cyclictest17690110-uname00:38:4211
30332993314,16cyclictest26523-21sshd23:53:0011
30332993313,18cyclictest0-21swapper/522:43:2611
30332993312,19cyclictest29378-21sshd22:54:0911
30332993312,19cyclictest0-21swapper/521:29:0911
30332993311,19cyclictest14849-21cp22:12:2711
30332993311,14cyclictest4636-21systemd-cgroups00:27:2411
30332993310,21cyclictest998-21dbus-daemon23:17:4611
30332993310,21cyclictest998-21dbus-daemon23:17:4611
15812330,30sleep150-21swapper/1522:35:187
8732320,1sleep70-21swapper/722:00:2313
61522320,28sleep120-21swapper/1221:41:454
50472320,2sleep40-21swapper/400:04:5910
3034499320,18cyclictest1033-21gdbus23:16:395
3034499320,18cyclictest1033-21gdbus23:16:395
3033299329,20cyclictest5039-21nfsd23:47:4811
30332993217,13cyclictest0-21swapper/521:48:5511
30332993212,18cyclictest0-21swapper/522:34:1911
30329993213,3cyclictest3134-21sshd22:40:389
282482320,1sleep150-21swapper/1521:16:397
148242320,1sleep121131ktimersoftd/1223:41:504
3034499315,18cyclictest137311sshd22:22:195
3033299311,21cyclictest0-21swapper/500:07:2711
30332993110,19cyclictest0-21swapper/519:12:5111
30329993113,4cyclictest26071-21run-parts23:23:209
192302313,2sleep140-21swapper/1421:34:026
161302310,27sleep30-21swapper/321:14:529
15692310,1sleep140-21swapper/1421:55:416
68432300,1sleep90-21swapper/923:40:2715
3034499303,24cyclictest21002-21sshd22:57:535
3034499300,22cyclictest21385-21systemd-cgroups00:34:155
3033299303,25cyclictest0-21swapper/519:27:4611
30332993011,17cyclictest0-21swapper/519:37:3511
30329993013,3cyclictest13150-21run-parts23:21:419
30329993013,3cyclictest13150-21run-parts23:21:419
262772300,4sleep233-21ksoftirqd/221:20:438
96202290,0sleep80-21swapper/820:47:3914
96202290,0sleep80-21swapper/820:47:3914
3034499296,20cyclictest998-21dbus-daemon22:41:075
30344992927,1cyclictest1201rcuc/1321:28:425
30344992919,6cyclictest47150jbd2/md124p1-800:03:205
3033299299,18cyclictest0-21swapper/519:49:5911
3033299299,18cyclictest0-21swapper/519:22:1911
3033299298,19cyclictest0-21swapper/520:52:3111
3033299297,20cyclictest0-21swapper/519:17:3711
30332992914,14cyclictest0-21swapper/519:42:2811
30332992912,15cyclictest20976-21fschecks_time21:02:3011
30332992911,16cyclictest0-21swapper/520:07:2611
30332992910,18cyclictest0-21swapper/519:32:2211
30332992910,17cyclictest14252-21cut20:17:1611
214572290,1sleep100-21swapper/1020:22:502
106792290,1sleep130-21swapper/1323:21:155
106792290,1sleep130-21swapper/1323:21:155
48252280,1sleep100-21swapper/1023:00:372
322512280,1sleep150-21swapper/1521:55:277
3034499288,18cyclictest0-21swapper/1322:30:315
3034499286,20cyclictest0-21swapper/1323:33:275
30344992824,3cyclictest1201rcuc/1322:37:045
30344992824,2cyclictest1201rcuc/1323:55:095
3034499282,24cyclictest23117-21sh21:24:485
3034499282,24cyclictest21450-21sshd00:07:265
3034499281,24cyclictest0-21swapper/1321:18:435
3033999283,2cyclictest90-21ksoftirqd/920:17:2715
3033299289,18cyclictest0-21swapper/520:24:5111
3033299289,17cyclictest4105-21nfsd20:02:4211
30332992814,12cyclictest0-21swapper/519:52:3111
30332992812,15cyclictest0-21swapper/521:11:4211
30332992812,15cyclictest0-21swapper/520:58:4211
30332992812,15cyclictest0-21swapper/520:47:1811
30332992812,15cyclictest0-21swapper/520:47:1811
30332992812,15cyclictest0-21swapper/520:37:3811
30332992812,15cyclictest0-21swapper/520:37:3811
30332992812,14cyclictest0-21swapper/520:42:2611
30332992812,14cyclictest0-21swapper/520:42:2611
30332992812,14cyclictest0-21swapper/520:27:5611
30332992812,14cyclictest0-21swapper/520:27:5611
30332992812,14cyclictest0-21swapper/519:57:3011
30332992811,15cyclictest0-21swapper/520:35:1111
30332992811,15cyclictest0-21swapper/520:35:1111
60202270,23sleep16084-21smart_sdd21:17:461
50992270,1sleep40-21swapper/400:37:0410
37392270,1sleep70-21swapper/721:13:0313
3034499279,16cyclictest10031-21gdbus22:56:305
3034499277,18cyclictest998-21dbus-daemon23:26:385
3034499276,19cyclictest5152-21sshd23:49:475
3034499276,19cyclictest24324-21sshd23:42:545
3034499276,18cyclictest998-21dbus-daemon21:14:365
3034499274,20cyclictest1-21systemd22:06:155
3034499274,20cyclictest0-21swapper/1300:01:255
3034499274,11cyclictest149261sshd22:17:265
3034499273,22cyclictest7994-21uname21:37:205
30344992723,2cyclictest1201rcuc/1321:45:135
30344992723,2cyclictest1201rcuc/1300:24:525
3034499272,23cyclictest0-21swapper/1323:38:545
3034499271,23cyclictest0-21swapper/1322:13:315
30344992712,13cyclictest0-21swapper/1321:54:595
30344992711,5cyclictest1-21systemd22:48:095
3034399270,2cyclictest153852sleep1219:32:204
3033299279,16cyclictest0-21swapper/520:17:2811
271052270,1sleep80-21swapper/800:25:5514
232942270,24sleep40-21swapper/422:48:1310
222772270,23sleep120-21swapper/1223:17:474
222772270,23sleep120-21swapper/1223:17:474
132362270,22sleep150-21swapper/1500:18:327
48982260,22sleep04953-21sshd22:10:430
321132260,22sleep40-21swapper/400:21:5010
3034499267,17cyclictest25316-21sshd22:08:355
3034499267,17cyclictest0-21swapper/1323:08:315
3034499267,17cyclictest0-21swapper/1323:08:315
3034499267,17cyclictest0-21swapper/1300:29:305
3034499266,12cyclictest0-21swapper/1323:02:335
3034499266,12cyclictest0-21swapper/1323:02:335
3034499265,19cyclictest17116-21run-parts00:19:115
3034499265,18cyclictest0-21swapper/1321:36:045
3034499263,20cyclictest0-21swapper/1300:12:225
30344992612,12cyclictest1029-21gdbus21:58:135
245762260,1sleep90-21swapper/900:02:5615
112442260,1sleep90-21swapper/900:37:4215
92732250,23sleep150-21swapper/1519:22:467
45502250,1sleep110-21swapper/1100:32:073
314432250,0sleep140-21swapper/1422:59:366
30344992520,4cyclictest1201rcuc/1321:51:045
30344992512,10cyclictest0-21swapper/1319:12:255
3033999258,7cyclictest90-21ksoftirqd/923:14:5615
3033999258,7cyclictest90-21ksoftirqd/923:14:5615
3033799250,19cyclictest998-21dbus-daemon22:45:3213
299032250,1sleep120-21swapper/1223:13:464
299032250,1sleep120-21swapper/1223:13:464
234352250,21sleep10-21swapper/122:18:291
169602250,21sleep20-21swapper/223:42:138
151802250,1sleep120-21swapper/1222:42:284
55922240,1sleep40-21swapper/420:42:4710
55922240,1sleep40-21swapper/420:42:4710
40902240,2sleep04179-21sshd23:44:510
37842240,0sleep110-21swapper/1100:27:173
313312240,1sleep110-21swapper/1120:37:183
313312240,1sleep110-21swapper/1120:37:183
309262240,0sleep00-21swapper/022:00:000
3034499246,16cyclictest0-21swapper/1319:37:235
3034499243,19cyclictest0-21swapper/1319:27:385
3034499242,20cyclictest0-21swapper/1319:36:455
3034499240,22cyclictest0-21swapper/1319:22:435
3034199240,7cyclictest0-21swapper/1100:11:203
3033999246,12cyclictest90-21ksoftirqd/923:03:0815
3033999246,12cyclictest90-21ksoftirqd/923:03:0815
30339992420,2cyclictest93050irq/38-eth0-TxR22:14:4915
253722240,1sleep90-21swapper/921:35:0115
178932241,19sleep1217999-21systemd-cgroups00:33:444
155872240,1sleep100-21swapper/1023:51:432
75372230,1sleep97534-21sshd21:17:5315
70572230,1sleep3401ktimersoftd/320:07:289
67522230,0sleep140-21swapper/1422:01:186
46102230,2sleep140-21swapper/1419:17:406
3034499234,17cyclictest0-21swapper/1320:07:385
30344992319,2cyclictest1201rcuc/1320:42:255
30344992319,2cyclictest1201rcuc/1320:42:255
30344992318,3cyclictest1201rcuc/1320:28:305
30344992318,3cyclictest1201rcuc/1320:28:305
30344992317,4cyclictest105442sleep1320:50:095
30344992317,4cyclictest105442sleep1320:50:095
3034499230,21cyclictest1201rcuc/1320:12:295
3034199235,9cyclictest106-21ksoftirqd/1121:20:163
30341992313,5cyclictest141rcu_sched22:01:163
30341992310,9cyclictest131rcu_preempt23:13:373
30341992310,9cyclictest131rcu_preempt23:13:373
3033999236,11cyclictest90-21ksoftirqd/922:20:4115
3033999236,10cyclictest90-21ksoftirqd/921:59:3515
3033999235,17cyclictest891ktimersoftd/900:29:5015
30339992316,5cyclictest891ktimersoftd/922:42:2815
30339992314,4cyclictest131rcu_preempt22:35:5315
30339992312,7cyclictest131rcu_preempt22:28:3715
30339992311,6cyclictest131rcu_preempt00:01:2315
30339992310,10cyclictest131rcu_preempt00:13:4615
3033999230,4cyclictest4676-21kworker/9:322:39:1015
30338992317,3cyclictest1033-21gdbus22:05:4414
3033799232,19cyclictest0-21swapper/721:53:4913
233742230,1sleep1231rcuc/123:37:381
224682230,0sleep40-21swapper/422:38:1910
52162220,0sleep10-21swapper/121:37:011
321682220,1sleep10-21swapper/123:09:181
321682220,1sleep10-21swapper/123:09:181
317352220,2sleep110-21swapper/1123:04:233
317352220,2sleep110-21swapper/1123:04:233
3142220,1sleep20-21swapper/222:44:598
30872220,2sleep153204-21sshd22:20:357
3034499227,14cyclictest0-21swapper/1320:19:055
3034499225,16cyclictest0-21swapper/1319:55:485
3034499225,15cyclictest0-21swapper/1319:47:375
3034499223,17cyclictest0-21swapper/1320:40:005
3034499223,17cyclictest0-21swapper/1320:40:005
3034499222,18cyclictest5815-21cron19:22:165
30344992219,1cyclictest1201rcuc/1321:02:235
30343992218,2cyclictest0-21swapper/1200:17:274
3034199227,10cyclictest141rcu_sched23:10:033
3034199227,10cyclictest141rcu_sched23:10:033
3034199226,9cyclictest106-21ksoftirqd/1122:59:523
3034199226,12cyclictest0-21swapper/1100:32:333
3034199224,16cyclictest106-21ksoftirqd/1123:26:443
3034199224,13cyclictest106-21ksoftirqd/1122:05:543
3034199224,12cyclictest106-21ksoftirqd/1123:17:593
3034199224,12cyclictest106-21ksoftirqd/1123:17:593
30341992218,2cyclictest92850irq/36-eth0-TxR23:47:503
30341992218,2cyclictest1041rcuc/1121:34:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional