You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 03:54
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackcslot0.osadl.org (updated Fri Jul 26, 2024 00:43:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1659621430,0sleep60-21swapper/622:59:0712
24676211892,22sleep150-21swapper/1519:10:397
24676211892,22sleep150-21swapper/1519:10:397
24824211596,14sleep130-21swapper/1319:12:455
24824211596,14sleep130-21swapper/1319:12:455
24675211092,13sleep140-21swapper/1419:10:386
24675211092,13sleep140-21swapper/1419:10:386
24600210980,24sleep120-21swapper/1219:09:324
24600210980,24sleep120-21swapper/1219:09:324
23665210995,6sleep60-21swapper/619:08:5312
23665210995,6sleep60-21swapper/619:08:5312
24687210591,9sleep80-21swapper/819:10:4814
24687210591,9sleep80-21swapper/819:10:4814
935221000,2sleep3998-21dbus-daemon00:23:519
272262980,0sleep40-21swapper/400:21:4410
2390529779,13sleep110-21swapper/1119:08:593
2390529779,13sleep110-21swapper/1119:08:593
180782960,2sleep62538199cyclictest22:37:1212
164802960,1sleep30-21swapper/321:38:029
101062950,0sleep140-21swapper/1421:55:046
101062950,0sleep140-21swapper/1421:55:046
58792930,8sleep12537299cyclictest23:30:041
275452930,0sleep10-21swapper/121:53:151
275452930,0sleep10-21swapper/121:53:151
92972920,1sleep20-21swapper/223:16:438
2457129274,13sleep00-21swapper/019:09:040
2457129274,13sleep00-21swapper/019:09:040
2118928972,11sleep20-21swapper/219:08:338
2118928972,11sleep20-21swapper/219:08:338
4222880,1sleep60-21swapper/620:38:4312
290672870,0sleep150-21swapper/1521:30:457
2459728779,4sleep90-21swapper/919:09:2915
2459728779,4sleep90-21swapper/919:09:2915
79872850,1sleep67989-21systemd-cgroups00:37:0112
79872850,1sleep67989-21systemd-cgroups00:37:0112
119032850,1sleep80-21swapper/822:17:5814
2461028472,7sleep50-21swapper/519:09:4111
2461028472,7sleep50-21swapper/519:09:4111
2476628372,7sleep100-21swapper/1019:11:542
2476628372,7sleep100-21swapper/1019:11:542
2475528375,4sleep30-21swapper/319:11:479
2475528375,4sleep30-21swapper/319:11:479
2464528372,7sleep10-21swapper/119:10:091
2464528372,7sleep10-21swapper/119:10:091
2459228375,4sleep40-21swapper/419:09:2410
2459228375,4sleep40-21swapper/419:09:2410
2466828072,4sleep70-21swapper/719:10:3113
2466828072,4sleep70-21swapper/719:10:3113
8592790,1sleep80-21swapper/822:20:3914
136592790,5sleep52537899cyclictest22:09:0611
285772780,1sleep12114-21ksoftirqd/1222:11:214
130412750,0sleep130-21swapper/1322:22:365
322362710,1sleep152539399cyclictest23:15:197
38562660,6sleep72538299cyclictest23:52:5413
104012600,2sleep14318-21cups-browsed21:41:291
321072560,1sleep70-21swapper/721:35:4213
77742510,1sleep60-21swapper/622:58:1912
2538899500,3cyclictest12828-21sshd23:26:404
2538899500,3cyclictest1-21systemd23:46:184
2538899470,2cyclictest28462-21sshd22:47:544
2538899466,35cyclictest5040-21nfsd22:57:394
2538899451,3cyclictest2337-21env22:39:074
2538899450,3cyclictest8610-21sshd23:12:184
2538899450,2cyclictest0-21swapper/1200:25:144
2538899450,2cyclictest0-21swapper/1200:13:584
2538899445,36cyclictest22003-21run-parts22:59:594
2538899440,4cyclictest3381-2110-uname00:23:124
2538899440,3cyclictest18980-21sh22:50:504
2538899440,3cyclictest15911-21sshd23:49:504
2538899440,3cyclictest11422-2110-uname21:15:124
2538899436,35cyclictest29539-2110-uname21:25:584
2538899435,35cyclictest6477-21sh21:50:034
2538899435,35cyclictest6477-21sh21:50:034
2538899430,2cyclictest0-21swapper/1200:05:184
2538899427,33cyclictest6764-21rm21:23:064
2538899427,33cyclictest21926-2110-uname23:55:114
2538899426,33cyclictest6822-21sshd00:13:034
2538899420,3cyclictest3600-21sshd22:21:064
2538899420,3cyclictest0-21swapper/1223:36:544
2538899418,31cyclictest145-21kauditd21:32:204
2538899416,33cyclictest6643-2110-uname21:40:564
2538899416,32cyclictest1066-21rm00:40:554
2538899416,32cyclictest1066-21rm00:40:554
2538899415,33cyclictest22070-21rm21:34:144
2538899410,3cyclictest23925-21sh22:14:544
246232410,2sleep70-21swapper/721:25:1613
2538899403,35cyclictest27763-21sshd23:33:174
2538899400,37cyclictest26693-21sshd22:33:454
296562390,1sleep40-21swapper/400:31:0610
296562390,1sleep40-21swapper/400:31:0610
2538899396,30cyclictest7684-21mailstats21:58:574
2538899396,30cyclictest7684-21mailstats21:58:574
2538899395,32cyclictest1618-21uname00:36:074
2538899395,32cyclictest1618-21uname00:36:074
2538899390,2cyclictest5039-21nfsd22:26:044
2538899390,2cyclictest27951-21sshd23:05:504
2538899390,2cyclictest0-21swapper/1223:13:464
2538899386,30cyclictest6666-21sshd23:38:564
2538899380,3cyclictest1-21systemd23:58:504
2538899380,2cyclictest0-21swapper/1221:55:424
2538899380,2cyclictest0-21swapper/1221:55:424
2538899380,2cyclictest0-21swapper/1221:44:264
2538899380,2cyclictest0-21swapper/1221:44:264
25387993815,21cyclictest0-21swapper/1119:23:423
2538899377,28cyclictest20599-21sshd22:05:184
2538899370,7cyclictest0-21swapper/1200:30:454
2538899370,7cyclictest0-21swapper/1200:30:454
2538899360,3cyclictest22274-21sshd23:18:374
2538899360,2cyclictest28120-21apt-get20:33:424
2538899360,29cyclictest3491-21sh22:30:394
2538899357,26cyclictest24162-21missed_timers19:48:514
2538899356,27cyclictest13344-21fschecks_count20:53:464
2538899354,29cyclictest11294-21fschecks_time19:33:464
25378993515,19cyclictest0-21swapper/522:58:5611
309722340,1sleep131211ktimersoftd/1323:56:295
2538899343,3cyclictest20187-21expr20:23:434
2538899340,2cyclictest32650-21timerwakeupswit19:19:014
2538899340,2cyclictest32650-21timerwakeupswit19:19:014
35832330,1sleep40-21swapper/422:39:1810
2538899337,24cyclictest23683-21/usr/sbin/munin20:28:514
2538899333,1cyclictest0-21swapper/1219:13:484
2538899333,1cyclictest0-21swapper/1219:13:484
87012320,1sleep50-21swapper/500:07:3611
2538899320,4cyclictest5038-21nfsd20:18:434
2538899320,4cyclictest0-21swapper/1219:43:454
2538899320,3cyclictest30540-21/usr/sbin/munin19:58:524
2538899320,2cyclictest0-21swapper/1220:09:014
25372993227,3cyclictest1-21systemd21:17:221
287252310,1sleep80-21swapper/823:42:3014
260652310,6sleep152539399cyclictest23:51:267
2538899315,23cyclictest17815-21latency20:58:524
2538899314,24cyclictest2593-21timerandwakeup20:39:004
2538899314,1cyclictest0-21swapper/1219:54:054
2538899312,3cyclictest4526-21latency20:03:514
2538899310,3cyclictest10997-21timerandwakeup20:50:044
2537299315,7cyclictest998-21dbus-daemon22:02:161
2537299315,7cyclictest998-21dbus-daemon22:02:161
25372993125,4cyclictest4342-21gdbus23:15:201
25372993125,4cyclictest0-21swapper/123:03:481
210002310,2sleep90-21swapper/900:34:0815
210002310,2sleep90-21swapper/900:34:0815
149982310,2sleep81010-21avahi-daemon23:58:4814
37662300,2sleep00-21swapper/022:16:470
254182300,1sleep70-21swapper/722:56:1213
2538899302,3cyclictest5705-21cpuspeed19:28:424
2538899302,2cyclictest789-21jbd2/md125-819:38:444
2538899302,2cyclictest25275-21cstates21:08:444
2538899302,2cyclictest20920-21cpuspeed_turbos21:03:424
2538899300,4cyclictest0-21swapper/1219:28:324
2538899300,3cyclictest5531-21if_eth020:44:024
25372993025,4cyclictest0-21swapper/122:57:541
25372993025,3cyclictest15180-21sshd22:09:221
166202300,5sleep52537899cyclictest21:42:2111
99352290,1sleep60-21swapper/622:17:4112
92672290,1sleep30-21swapper/323:53:389
88912290,25sleep140-21swapper/1421:59:066
88912290,25sleep140-21swapper/1421:59:066
56442290,5sleep558-21ksoftirqd/522:58:0211
2538899290,3cyclictest12645-21fschecks_time20:13:464
25372992926,2cyclictest0-21swapper/123:27:221
25372992926,2cyclictest0-21swapper/121:54:461
25372992926,2cyclictest0-21swapper/121:54:461
25372992925,2cyclictest0-21swapper/123:09:041
25372992924,4cyclictest0-21swapper/121:22:521
25372992923,4cyclictest31902-21packagekitd23:58:021
25372992923,4cyclictest0-21swapper/122:36:061
25372992923,3cyclictest0-21swapper/121:34:291
25372992922,5cyclictest6949-21sshd23:39:001
248342290,5sleep22537499cyclictest00:16:378
241812290,0sleep130-21swapper/1322:38:075
197422290,25sleep80-21swapper/823:32:0414
116812290,25sleep80-21swapper/800:42:4114
116812290,25sleep80-21swapper/800:42:4114
47402280,1sleep50-21swapper/521:22:5011
36822280,1sleep90-21swapper/900:41:2115
36822280,1sleep90-21swapper/900:41:2115
326252280,5sleep132538999cyclictest21:18:095
2537299287,5cyclictest141rcu_sched00:17:451
25372992823,3cyclictest0-21swapper/122:27:321
25372992822,4cyclictest5040-21nfsd22:59:201
25372992822,4cyclictest0-21swapper/123:20:431
244352280,24sleep1024552-21sshd22:51:432
16592280,24sleep90-21swapper/922:12:0715
127852281,22sleep100-21swapper/1021:19:272
88892270,1sleep70-21swapper/700:07:3713
33242270,1sleep10-21swapper/122:21:041
275082270,2sleep80-21swapper/800:05:2914
2537299272,6cyclictest1076-21gdbus00:05:321
25372992723,3cyclictest0-21swapper/123:43:521
25372992723,3cyclictest0-21swapper/122:30:251
25372992723,3cyclictest0-21swapper/121:44:271
25372992723,3cyclictest0-21swapper/121:44:271
25372992722,3cyclictest1014-21systemd-logind21:30:301
25372992721,4cyclictest998-21dbus-daemon23:59:051
25372992721,4cyclictest998-21dbus-daemon23:37:351
245762270,23sleep140-21swapper/1423:09:586
236512270,23sleep100-21swapper/1023:18:442
94842260,7sleep882-21ksoftirqd/823:44:0514
5642260,1sleep80-21swapper/821:44:1714
5642260,1sleep80-21swapper/821:44:1714
323962261,3sleep104342-21gdbus21:26:242
261232260,23sleep40-21swapper/421:53:0310
261232260,23sleep40-21swapper/421:53:0310
25372992622,3cyclictest0-21swapper/120:49:571
25372992622,3cyclictest0-21swapper/120:09:001
25372992622,3cyclictest0-21swapper/120:03:421
25372992621,4cyclictest0-21swapper/122:03:561
25372992621,3cyclictest1029-21gdbus23:48:351
25372992619,5cyclictest1014-21systemd-logind22:53:051
252982260,0sleep130-21swapper/1321:25:225
206002260,23sleep110-21swapper/1122:55:293
182822260,22sleep80-21swapper/821:33:5114
179712260,21sleep80-21swapper/823:27:3114
164652260,0sleep130-21swapper/1300:38:285
164652260,0sleep130-21swapper/1300:38:285
159092260,1sleep80-21swapper/821:51:3014
159092260,1sleep80-21swapper/821:51:3014
94292250,2sleep90-21swapper/900:28:3115
92642250,2sleep70-21swapper/723:44:0413
68582250,2sleep00-21swapper/021:32:230
37582250,2sleep90-21swapper/919:23:5515
36092250,1sleep140-21swapper/1400:32:056
36092250,1sleep140-21swapper/1400:32:056
275792250,1sleep90-21swapper/922:56:3115
25384992512,5cyclictest90-21ksoftirqd/921:18:2415
25384992511,6cyclictest90-21ksoftirqd/922:32:5515
25384992510,7cyclictest131rcu_preempt22:17:4215
25372992520,3cyclictest15850-21sshd21:24:021
236842250,2sleep20-21swapper/221:04:048
234392250,21sleep100-21swapper/1023:28:222
147652250,1sleep50-21swapper/521:37:4811
147412250,22sleep40-21swapper/421:42:0510
105662250,1sleep130-21swapper/1320:11:515
81772240,0sleep60-21swapper/623:34:4212
37302240,1sleep50-21swapper/523:25:0411
322642240,1sleep111041rcuc/1121:13:503
308112240,1sleep40-21swapper/423:38:0910
283952240,20sleep110-21swapper/1100:17:193
25384992414,5cyclictest90-21ksoftirqd/900:06:4815
2538299240,21cyclictest5038-21nfsd21:30:5413
2537299245,11cyclictest0-21swapper/100:21:131
25372992420,3cyclictest0-21swapper/122:14:411
25372992420,3cyclictest0-21swapper/120:24:061
2537299240,16cyclictest0-21swapper/122:44:281
253002240,1sleep150-21swapper/1500:34:467
253002240,1sleep150-21swapper/1500:34:467
214422240,20sleep110-21swapper/1121:24:493
207522240,2sleep1020847-21systemd-cgroups21:47:322
207522240,2sleep1020847-21systemd-cgroups21:47:322
200782240,1sleep30-21swapper/321:47:249
200782240,1sleep30-21swapper/321:47:249
167562240,1sleep150-21swapper/1522:45:557
100532240,1sleep50-21swapper/523:08:1311
94542230,19sleep90-21swapper/921:41:2215
60562230,19sleep70-21swapper/722:08:2013
301902230,19sleep20-21swapper/223:33:358
2538499237,5cyclictest90-21ksoftirqd/923:56:3515
2537299237,8cyclictest24143-21sshd00:30:091
2537299237,8cyclictest24143-21sshd00:30:091
2537299236,11cyclictest25-21ksoftirqd/100:09:501
2537299236,10cyclictest25-21ksoftirqd/100:35:061
2537299236,10cyclictest25-21ksoftirqd/100:35:061
25372992320,2cyclictest0-21swapper/120:43:491
25372992319,3cyclictest0-21swapper/121:03:511
25372992319,3cyclictest0-21swapper/120:54:011
224322230,1sleep20-21swapper/223:04:568
168352230,19sleep60-21swapper/622:09:4012
153982230,0sleep80-21swapper/822:00:0314
153982230,0sleep80-21swapper/822:00:0314
138602230,1sleep50-21swapper/500:33:3211
138602230,1sleep50-21swapper/500:33:3211
133642230,20sleep70-21swapper/720:13:5313
77652220,1sleep20-21swapper/222:17:218
45862220,2sleep00-21swapper/020:03:510
42412220,1sleep60-21swapper/622:25:5912
326762220,7sleep142539299cyclictest00:17:536
325132220,1sleep60-21swapper/622:02:4112
325132220,1sleep60-21swapper/622:02:4112
25387992217,3cyclictest1033-21gdbus23:16:463
2538499228,10cyclictest90-21ksoftirqd/923:36:5915
25384992210,5cyclictest131rcu_preempt21:19:5815
2538299221,2cyclictest0-21swapper/721:43:4313
2538299221,2cyclictest0-21swapper/721:43:4313
2537299224,9cyclictest25-21ksoftirqd/100:24:431
2537299223,11cyclictest25-21ksoftirqd/122:39:511
25372992219,2cyclictest0-21swapper/120:59:321
25372992219,2cyclictest0-21swapper/120:33:531
25372992218,2cyclictest0-21swapper/120:14:241
2537199228,5cyclictest10-21ksoftirqd/023:48:300
2537199226,8cyclictest131rcu_preempt23:34:340
2537199226,7cyclictest10-21ksoftirqd/022:39:140
25371992215,5cyclictest3285-21sshd00:12:100
242982220,18sleep90-21swapper/923:14:1015
210312220,19sleep20-21swapper/223:23:098
201562220,1sleep130-21swapper/1300:20:345
201162220,0sleep40-21swapper/400:08:5410
173322220,1sleep140-21swapper/1423:04:056
117502220,1sleep150-21swapper/1521:28:217
115102220,18sleep20-21swapper/221:50:528
115102220,18sleep20-21swapper/221:50:528
99832210,1sleep70-21swapper/723:34:5913
9542210,1sleep150-21swapper/1500:40:557
9542210,1sleep150-21swapper/1500:40:557
6112210,19sleep100-21swapper/1019:21:322
6112210,19sleep100-21swapper/1019:21:322
59202210,1sleep0121rcuc/000:23:320
321112210,2sleep110-21swapper/1100:31:303
321112210,2sleep110-21swapper/1100:31:303
318122210,1sleep50-21swapper/521:58:0911
318122210,1sleep50-21swapper/521:58:0911
292652210,1sleep150-21swapper/1523:33:327
28942210,1sleep5571ktimersoftd/500:01:5811
25393992114,4cyclictest5442-21run-parts23:11:527
25389992115,4cyclictest1761-21run-parts22:25:335
2538799213,3cyclictest1010-21avahi-daemon00:09:113
25387992114,4cyclictest10590-21sh22:49:393
2538499219,5cyclictest90-21ksoftirqd/900:00:3515
2538499217,7cyclictest90-21ksoftirqd/921:45:2415
2538499217,7cyclictest90-21ksoftirqd/921:45:2415
2538499214,8cyclictest90-21ksoftirqd/923:07:0515
25384992117,2cyclictest93050irq/38-eth0-TxR21:31:2515
25384992115,4cyclictest0-21swapper/922:48:2515
25384992114,4cyclictest1408-21sshd00:31:4515
25384992114,4cyclictest1408-21sshd00:31:4515
25384992111,4cyclictest131rcu_preempt23:21:5315
2538299210,4cyclictest0-21swapper/723:57:0913
2538299210,4cyclictest0-21swapper/721:21:3313
2538299210,3cyclictest0-21swapper/722:21:3313
25381992116,3cyclictest26831-21sshd21:25:3512
2537699210,19cyclictest0-21swapper/422:55:2210
25375992114,4cyclictest1894-21sh23:29:309
25375992114,4cyclictest10104-21sh22:26:579
2537299216,7cyclictest25-21ksoftirqd/100:39:061
2537299216,7cyclictest25-21ksoftirqd/100:39:061
25372992117,3cyclictest0-21swapper/120:31:521
2537199218,5cyclictest141rcu_sched21:27:570
25371992112,3cyclictest10-21ksoftirqd/000:06:230
25371992110,4cyclictest32482-21sshd22:02:410
25371992110,4cyclictest32482-21sshd22:02:410
217012210,1sleep13122-21ksoftirqd/1322:14:355
216882210,2sleep140-21swapper/1422:05:306
109562210,1sleep100-21swapper/1000:33:052
109562210,1sleep100-21swapper/1000:33:052
95422200,0sleep140-21swapper/1422:53:546
81902200,1sleep140-21swapper/1423:39:086
66472200,1sleep130-21swapper/1322:39:445
48432200,1sleep00-21swapper/019:26:590
46332200,2sleep104318-21cups-browsed23:11:462
267362200,1sleep70-21swapper/700:39:4313
267362200,1sleep70-21swapper/700:39:4313
2539399201,16cyclictest0-21swapper/1522:40:287
2539399200,19cyclictest0-21swapper/1500:13:477
25392992015,3cyclictest23950-21run-parts22:10:426
2539299200,2cyclictest0-21swapper/1400:06:556
2538999201,17cyclictest0-21swapper/1323:42:255
25387992015,3cyclictest1033-21gdbus23:43:523
25387992015,3cyclictest1033-21gdbus00:03:263
25387992013,4cyclictest31716-21sh21:22:033
2538499209,6cyclictest90-21ksoftirqd/921:24:2715
2538499208,7cyclictest90-21ksoftirqd/923:48:1315
2538499207,7cyclictest141rcu_sched21:34:4515
2538499206,7cyclictest141rcu_sched22:19:3415
2538499205,13cyclictest0-21swapper/921:53:4015
2538499205,13cyclictest0-21swapper/921:53:4015
25384992015,4cyclictest0-21swapper/922:39:4615
25384992015,3cyclictest15614-21sh00:19:4415
25384992014,5cyclictest90-21ksoftirqd/921:53:2915
25384992014,5cyclictest90-21ksoftirqd/921:53:2915
25384992014,4cyclictest0-21swapper/922:48:3515
25384992010,5cyclictest90-21ksoftirqd/922:07:0715
25383992014,4cyclictest5041-21nfsd22:05:3214
2538299201,3cyclictest22716-21sshd22:23:4913
2538299201,3cyclictest0-21swapper/700:27:5813
2538199200,17cyclictest0-21swapper/621:43:5512
2538199200,17cyclictest0-21swapper/621:43:5512
25378992014,4cyclictest26133-21run-parts22:42:2911
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional