You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 05:58
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack3slot0.osadl.org (updated Fri May 09, 2025 00:43:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16376997574,1cyclictest28105-21nfsd21:23:025
88942730,0sleep70-21swapper/723:06:497
219972490,0sleep10-21swapper/121:07:501
62392470,1sleep46241-21id22:52:214
298192470,0sleep50-21swapper/522:25:475
209742470,0sleep40-21swapper/400:12:304
96252460,0sleep00-21swapper/022:44:080
96252460,0sleep00-21swapper/022:44:080
126042460,0sleep70-21swapper/722:44:237
126042460,0sleep70-21swapper/722:44:237
19852450,0sleep3391rcuc/321:17:213
214332440,0sleep40-21swapper/400:06:374
1619324420,9sleep30-21swapper/319:00:533
98532430,0sleep20-21swapper/222:03:572
318022430,0sleep60-21swapper/621:20:046
317032430,0sleep50-21swapper/520:21:505
79452420,0sleep70-21swapper/723:18:117
230132420,0sleep60-21swapper/621:30:466
181842420,0sleep60-21swapper/623:01:526
1609924229,9sleep50-21swapper/518:59:355
1601624230,8sleep70-21swapper/718:58:257
37802410,0sleep20-21swapper/221:57:412
303942410,0sleep40-21swapper/421:14:184
1593924128,9sleep60-21swapper/618:57:206
134362410,0sleep20-21swapper/222:18:312
1337524127,9sleep40-21swapper/418:56:304
125482410,0sleep50-21swapper/500:18:075
324572400,0sleep30-21swapper/323:06:133
270302400,0sleep70-21swapper/723:57:197
1612324028,8sleep00-21swapper/018:59:540
1595324029,8sleep20-21swapper/218:57:322
247342390,0sleep00-21swapper/022:13:480
1371923927,8sleep10-21swapper/118:56:351
116692390,0sleep10-21swapper/122:01:241
233382370,0sleep00-21swapper/000:29:550
203762300,0sleep50-21swapper/523:10:565
1636499259,14cyclictest5762-21grepconf.sh21:34:551
1638499247,11cyclictest0-21swapper/723:02:187
1638499247,0cyclictest0-21swapper/720:45:477
1638499236,5cyclictest0-21swapper/722:09:527
16384992312,11cyclictest0-21swapper/721:02:497
165572200,0sleep30-21swapper/321:04:323
1638499208,6cyclictest0-21swapper/723:50:507
1638499208,5cyclictest0-21swapper/723:41:067
1638499208,5cyclictest0-21swapper/723:41:067
1637699204,15cyclictest8683-21sshd22:58:125
16366992015,3cyclictest33-21ksoftirqd/200:07:252
272882190,0sleep30-21swapper/321:59:503
1638499198,5cyclictest0-21swapper/720:12:397
1638499198,5cyclictest0-21swapper/720:11:057
1638499198,5cyclictest0-21swapper/720:11:057
1638499198,5cyclictest0-21swapper/719:42:077
1638499198,5cyclictest0-21swapper/719:01:577
1638499198,0cyclictest0-21swapper/723:21:367
1638499198,0cyclictest0-21swapper/723:21:367
1638499198,0cyclictest0-21swapper/722:21:147
1638499198,0cyclictest0-21swapper/722:02:527
1638499198,0cyclictest0-21swapper/721:55:307
1638499198,0cyclictest0-21swapper/721:23:337
1638499198,0cyclictest0-21swapper/720:50:277
1638499198,0cyclictest0-21swapper/720:29:357
1638499198,0cyclictest0-21swapper/720:24:377
1638499198,0cyclictest0-21swapper/719:26:117
1638499198,0cyclictest0-21swapper/719:08:017
1638499198,0cyclictest0-21swapper/719:08:017
1638499198,0cyclictest0-21swapper/700:07:307
1638499197,6cyclictest0-21swapper/723:53:287
1638499192,5cyclictest0-21swapper/720:57:297
1637599190,18cyclictest20965-21grepconf.sh21:05:004
16366991915,3cyclictest33-21ksoftirqd/222:55:222
16366991911,3cyclictest33-21ksoftirqd/222:41:482
16366991911,3cyclictest33-21ksoftirqd/222:41:482
251292180,0sleep50-21swapper/500:03:425
1638499187,5cyclictest0-21swapper/720:38:097
1638499187,5cyclictest0-21swapper/720:32:037
1638499187,0cyclictest0-21swapper/723:29:567
1638499187,0cyclictest0-21swapper/723:15:107
1638499187,0cyclictest0-21swapper/722:47:207
1638499187,0cyclictest0-21swapper/721:47:357
1638499187,0cyclictest0-21swapper/721:20:137
1638499187,0cyclictest0-21swapper/720:54:197
1638499187,0cyclictest0-21swapper/720:03:437
1638499187,0cyclictest0-21swapper/720:00:557
1638499187,0cyclictest0-21swapper/719:56:057
1638499187,0cyclictest0-21swapper/719:49:037
1638499187,0cyclictest0-21swapper/719:32:437
1638499187,0cyclictest0-21swapper/719:29:017
1638499187,0cyclictest0-21swapper/719:18:137
1638499187,0cyclictest0-21swapper/719:11:317
1638499186,6cyclictest0-21swapper/722:21:547
1638499180,17cyclictest18380-21sshd22:39:127
1637699188,0cyclictest0-21swapper/519:42:275
1637699187,5cyclictest0-21swapper/523:11:525
1636699186,1cyclictest131rcu_sched21:13:342
16366991813,3cyclictest33-21ksoftirqd/200:02:432
16366991811,1cyclictest33-21ksoftirqd/221:38:172
277982170,0sleep4763-21runrttasks23:11:354
1638499176,5cyclictest0-21swapper/723:46:047
1638499176,5cyclictest0-21swapper/720:18:477
1638499176,5cyclictest0-21swapper/719:40:197
1638499175,6cyclictest0-21swapper/700:29:047
1637699176,5cyclictest0-21swapper/520:30:355
1637699175,11cyclictest25229-21bash21:08:075
16366991715,1cyclictest33-21ksoftirqd/223:21:422
16366991715,1cyclictest33-21ksoftirqd/223:21:422
16366991715,1cyclictest33-21ksoftirqd/223:06:282
16366991712,3cyclictest33-21ksoftirqd/223:29:392
16366991712,3cyclictest33-21ksoftirqd/222:23:522
1636699170,2cyclictest121rcu_preempt21:51:452
1635699170,16cyclictest711-21dbus-daemon23:00:430
117342170,1sleep011731-21sshd00:18:020
1638499168,2cyclictest0-21swapper/721:36:397
1638499165,5cyclictest0-21swapper/721:15:197
1638499165,0cyclictest0-21swapper/700:11:447
1638499164,0cyclictest0-21swapper/700:25:407
1637699166,0cyclictest0-21swapper/523:16:365
1637699165,5cyclictest0-21swapper/519:59:575
1637699165,5cyclictest0-21swapper/519:02:595
1636699164,2cyclictest33-21ksoftirqd/221:10:492
1636699163,3cyclictest33-21ksoftirqd/223:57:152
16366991614,1cyclictest33-21ksoftirqd/223:35:342
16366991614,1cyclictest33-21ksoftirqd/223:35:342
16366991614,0cyclictest33-21ksoftirqd/219:01:362
16366991612,1cyclictest131rcu_sched23:06:412
16366991611,3cyclictest33-21ksoftirqd/223:52:232
16366991611,3cyclictest33-21ksoftirqd/223:47:532
16366991611,3cyclictest33-21ksoftirqd/223:18:062
16366991611,3cyclictest33-21ksoftirqd/223:15:092
1638499158,5cyclictest0-21swapper/723:31:407
1638499158,5cyclictest0-21swapper/723:31:407
1638499158,5cyclictest0-21swapper/721:07:597
1638499153,11cyclictest27343-21sshd21:42:237
16377991511,2cyclictest65-21ksoftirqd/621:52:376
1637699158,6cyclictest8955-21bash21:17:595
1637699156,0cyclictest0-21swapper/500:25:445
1637699154,4cyclictest2964-21bash23:55:165
1637699154,1cyclictest6520-21bash23:32:205
1637699154,1cyclictest6520-21bash23:32:205
1637699150,9cyclictest0-21swapper/523:43:285
1637699150,9cyclictest0-21swapper/520:42:555
1637699150,9cyclictest0-21swapper/520:33:575
1637699150,2cyclictest0-21swapper/522:17:125
1636699154,1cyclictest33-21ksoftirqd/221:22:232
16366991514,1cyclictest33-21ksoftirqd/222:59:002
16366991513,1cyclictest33-21ksoftirqd/222:51:092
16366991513,1cyclictest33-21ksoftirqd/222:38:322
16366991513,1cyclictest33-21ksoftirqd/222:07:002
16366991513,1cyclictest33-21ksoftirqd/221:21:192
16366991511,1cyclictest131rcu_sched23:40:382
16366991511,1cyclictest131rcu_sched23:40:382
16366991510,3cyclictest33-21ksoftirqd/221:06:232
1636699151,1cyclictest33-21ksoftirqd/221:28:512
1635699150,13cyclictest29369-21sshd23:40:140
1635699150,13cyclictest29369-21sshd23:40:140
120612150,0sleep30-21swapper/300:14:533
1638499148,5cyclictest0-21swapper/722:12:247
1638499148,5cyclictest0-21swapper/700:02:207
1637699148,6cyclictest0-21swapper/520:01:415
1637699143,0cyclictest0-21swapper/522:51:385
1637699142,12cyclictest0-21swapper/519:31:135
1637699142,0cyclictest0-21swapper/522:40:365
1637699142,0cyclictest0-21swapper/520:40:355
1637699140,8cyclictest0-21swapper/523:26:405
1637699140,8cyclictest0-21swapper/519:47:355
1637699140,8cyclictest0-21swapper/519:21:535
16367991412,1cyclictest41-21ksoftirqd/321:33:583
16366991413,1cyclictest33-21ksoftirqd/200:25:552
16366991412,1cyclictest33-21ksoftirqd/223:42:092
16366991412,1cyclictest33-21ksoftirqd/221:31:402
16366991411,3cyclictest33-21ksoftirqd/221:45:182
16366991410,3cyclictest33-21ksoftirqd/221:50:032
16366991410,2cyclictest33-21ksoftirqd/222:32:082
1636699141,1cyclictest33-21ksoftirqd/222:14:552
233762130,2sleep31636799cyclictest21:53:413
1638499138,5cyclictest0-21swapper/722:56:407
1638499138,5cyclictest0-21swapper/722:52:387
1638499138,5cyclictest0-21swapper/722:31:527
1638499138,5cyclictest0-21swapper/721:56:347
1638499138,5cyclictest0-21swapper/721:33:487
1638499138,5cyclictest0-21swapper/721:26:317
1638499138,5cyclictest0-21swapper/700:16:347
1638499138,0cyclictest0-21swapper/722:29:447
1637699138,5cyclictest22432-21bash21:59:245
1637699138,5cyclictest0-21swapper/522:49:485
1637699138,5cyclictest0-21swapper/522:36:285
1637699138,5cyclictest0-21swapper/522:11:385
1637699138,5cyclictest0-21swapper/522:07:385
1637699138,5cyclictest0-21swapper/522:02:505
1637699138,5cyclictest0-21swapper/521:52:315
1637699138,5cyclictest0-21swapper/520:17:555
1637699138,5cyclictest0-21swapper/519:52:155
1637699138,5cyclictest0-21swapper/519:11:055
1637699138,5cyclictest0-21swapper/519:11:055
1637699138,0cyclictest0-21swapper/520:50:555
1637699138,0cyclictest0-21swapper/520:12:515
1637699138,0cyclictest0-21swapper/519:37:195
1637699137,6cyclictest0-21swapper/523:21:525
1637699137,6cyclictest0-21swapper/523:21:525
1637699137,5cyclictest14187-21sshd00:28:585
1637699132,6cyclictest0-21swapper/521:16:135
1637699132,5cyclictest0-21swapper/523:47:085
1637699132,5cyclictest0-21swapper/522:44:445
1637699132,5cyclictest0-21swapper/522:44:445
1637699132,5cyclictest0-21swapper/521:46:585
1637699132,5cyclictest0-21swapper/521:42:345
1637699132,0cyclictest0-21swapper/523:57:265
1637699132,0cyclictest0-21swapper/521:31:035
1637699132,0cyclictest0-21swapper/520:55:155
1637699132,0cyclictest0-21swapper/520:08:475
1637699132,0cyclictest0-21swapper/520:08:475
1637699132,0cyclictest0-21swapper/519:12:295
1637699132,0cyclictest0-21swapper/500:12:165
1636699132,1cyclictest33-21ksoftirqd/200:12:042
16366991312,1cyclictest33-21ksoftirqd/200:29:212
16366991312,1cyclictest33-21ksoftirqd/200:17:372
16366991311,1cyclictest33-21ksoftirqd/222:27:332
16364991311,2cyclictest7000-21sshd21:55:111
288182120,0sleep10-21swapper/122:02:441
193322120,0sleep10-21swapper/122:13:171
188222120,1sleep665-21ksoftirqd/622:56:236
165272120,0sleep30-21swapper/320:51:503
1637799126,6cyclictest0-21swapper/622:17:066
16377991210,1cyclictest65-21ksoftirqd/622:49:126
16377991210,1cyclictest65-21ksoftirqd/622:33:596
1637699128,0cyclictest0-21swapper/521:00:015
1637699127,5cyclictest5164-21sshd23:38:005
1637699127,5cyclictest5164-21sshd23:38:005
1637699127,5cyclictest0-21swapper/519:31:475
1637699126,5cyclictest668-21systemd-logind21:40:035
1637699121,5cyclictest0-21swapper/521:03:455
1637699121,1cyclictest0-21swapper/522:27:345
1637699120,6cyclictest0-21swapper/500:06:505
16375991210,1cyclictest23696-21sshd21:50:564
1636799123,5cyclictest14691-21bash23:33:053
1636799123,5cyclictest14691-21bash23:33:053
16367991210,1cyclictest41-21ksoftirqd/321:43:343
1636799120,12cyclictest0-21swapper/322:26:233
16366991210,1cyclictest33-21ksoftirqd/220:01:292
16364991211,1cyclictest25-21ksoftirqd/122:56:491
1636499121,5cyclictest26272-21grep22:42:441
1636499121,5cyclictest26272-21grep22:42:441
1635699122,4cyclictest9718-21sshd23:32:370
1635699122,4cyclictest9718-21sshd23:32:370
1635699120,11cyclictest24832-21sshd23:05:320
103332120,0sleep00-21swapper/021:21:020
323752110,0sleep10-21swapper/123:40:311
323752110,0sleep10-21swapper/123:40:311
301412110,0sleep00-21swapper/022:02:510
249812110,0sleep40-21swapper/423:34:014
249812110,0sleep40-21swapper/423:34:014
1637799119,1cyclictest6833-21memory23:06:446
1637799116,4cyclictest65-21ksoftirqd/600:15:286
1637799112,6cyclictest8485-21cp21:43:356
1637799112,4cyclictest5437-21sshd23:55:296
1637799112,4cyclictest23258-21sshd00:22:406
1637799110,1cyclictest0-21swapper/622:43:246
1637799110,1cyclictest0-21swapper/622:43:246
1637699116,5cyclictest0-21swapper/523:02:525
1637699110,7cyclictest0-21swapper/521:34:255
1637699110,5cyclictest0-21swapper/519:19:595
1637599118,2cyclictest22908-21bash23:42:244
1637599112,5cyclictest711-21dbus-daemon23:00:374
16375991110,1cyclictest49-21ksoftirqd/400:02:594
1637599110,1cyclictest0-21swapper/400:28:544
1636799119,1cyclictest41-21ksoftirqd/300:30:593
1636799119,1cyclictest41-21ksoftirqd/300:06:173
1636799118,2cyclictest6042-21grepconf.sh23:43:493
1636799118,2cyclictest28586-21grepconf.sh21:48:163
1636799115,3cyclictest711-21dbus-daemon21:13:163
1636799112,5cyclictest28793-21bash21:08:263
1636799112,5cyclictest15265-21sshd22:41:463
1636799112,5cyclictest15265-21sshd22:41:463
1636699119,1cyclictest33-21ksoftirqd/220:31:292
1636499119,1cyclictest859-21polkitd23:03:041
1636499112,4cyclictest29957-21bash23:43:021
1636499112,4cyclictest25155-21sshd00:00:241
1636499112,4cyclictest2193-21sshd23:12:011
1636499112,4cyclictest0-21swapper/100:27:301
1636499110,5cyclictest31401-21bash23:31:451
1636499110,5cyclictest31401-21bash23:31:451
1636499110,10cyclictest4816-21sshd23:20:541
1636499110,10cyclictest0-21swapper/121:28:291
1636499110,1cyclictest0-21swapper/123:28:181
1636499110,1cyclictest0-21swapper/121:38:441
16356991110,1cyclictest0-21swapper/022:33:340
1635699111,4cyclictest27883-21bash21:28:120
1635699110,5cyclictest4707-21cp23:09:290
1635699110,1cyclictest0-21swapper/021:07:080
311372100,0sleep40-21swapper/423:51:474
19882100,0sleep10-21swapper/121:48:511
170322100,0sleep30-21swapper/323:30:323
1637799109,1cyclictest65-21ksoftirqd/623:21:086
1637799109,1cyclictest65-21ksoftirqd/621:22:556
1637799109,1cyclictest65-21ksoftirqd/600:19:366
1637799108,1cyclictest668-21systemd-logind21:07:386
1637799108,1cyclictest65-21ksoftirqd/623:57:236
1637799104,6cyclictest85350irq/25-em1-rx-022:37:146
1637799104,3cyclictest13784-21httpd21:32:086
1637799103,4cyclictest598-21bash23:26:176
1637799103,4cyclictest598-21bash23:26:176
1637799102,5cyclictest27218-21bash23:11:326
1637799102,4cyclictest0-21swapper/622:22:026
1637799101,8cyclictest16066-21sshd23:33:126
1637799101,8cyclictest16066-21sshd23:33:126
1637599109,1cyclictest28105-21nfsd22:33:304
1637599109,1cyclictest0-21swapper/422:20:064
1637599108,1cyclictest0-21swapper/423:28:034
1637599106,1cyclictest711-21dbus-daemon21:18:144
1637599102,4cyclictest10718-21sshd21:38:074
1637599100,9cyclictest20568-21kworker/4:122:24:584
1637599100,6cyclictest11878-21sshd00:25:084
1637599100,10cyclictest0-21swapper/421:59:274
1636799109,1cyclictest41-21ksoftirqd/323:26:253
1636799109,1cyclictest41-21ksoftirqd/323:26:253
1636799109,1cyclictest41-21ksoftirqd/322:20:533
1636799106,2cyclictest17199-21kworker/3:121:28:553
1636799106,1cyclictest27572-21kworker/3:222:05:383
1636799102,5cyclictest24828-21sshd00:22:523
1636799100,6cyclictest26731-21cp22:11:073
1636799100,10cyclictest0-21swapper/323:09:183
1636799100,10cyclictest0-21swapper/322:53:043
1636799100,1cyclictest0-21swapper/322:27:593
1636499109,1cyclictest25-21ksoftirqd/122:09:491
1636499109,1cyclictest0-21swapper/122:22:531
1636499109,1cyclictest0-21swapper/100:02:401
1636499107,2cyclictest25812-21bash23:22:381
1636499107,2cyclictest25812-21bash23:22:381
1636499103,6cyclictest568-21sshd21:17:141
1636499103,6cyclictest29250-21sshd21:05:491
1636499103,6cyclictest12337-21grepconf.sh00:14:541
1636499102,3cyclictest29546-21sshd23:08:471
1636499101,4cyclictest24934-21bash22:19:351
1636499101,4cyclictest1324-21sshd22:49:111
1636499101,4cyclictest0-21swapper/122:32:041
1636499100,4cyclictest31454-21sshd22:40:241
1636499100,1cyclictest0-21swapper/100:22:201
1635699108,1cyclictest9-21ksoftirqd/023:11:470
1635699108,1cyclictest9-21ksoftirqd/022:36:450
1635699107,2cyclictest21377-21latency_hist21:56:290
1635699104,3cyclictest20111-21perf21:01:280
1635699103,6cyclictest29839-21cat21:51:280
1635699102,7cyclictest8290-21bash22:55:240
1635699101,4cyclictest0-21swapper/000:13:190
1635699100,4cyclictest711-21dbus-daemon23:53:260
1635699100,10cyclictest0-21swapper/023:21:590
1635699100,10cyclictest0-21swapper/023:21:590
30487290,0sleep60-21swapper/623:46:056
163779997,1cyclictest85350irq/25-em1-rx-021:11:456
163779997,1cyclictest65-21ksoftirqd/600:28:356
163779996,1cyclictest0-21swapper/600:11:266
163779995,2cyclictest65-21ksoftirqd/623:46:406
163779994,4cyclictest6994-21grep22:06:396
163779994,3cyclictest0-21swapper/621:37:546
163779993,4cyclictest31940-21sshd21:48:386
163779993,3cyclictest3186-21grepconf.sh21:06:216
163779993,3cyclictest0-21swapper/600:03:216
163779992,4cyclictest7925-21systemd-cgroups22:03:486
163779992,3cyclictest1-21systemd22:13:586
163779991,5cyclictest2875-21bash21:57:366
163779991,2cyclictest0-21swapper/623:31:046
163779990,9cyclictest0-21swapper/622:26:326
163779990,3cyclictest3848-21sshd22:57:476
163759999,0cyclictest0-21swapper/423:56:444
163759999,0cyclictest0-21swapper/421:32:104
163759998,1cyclictest0-21swapper/423:06:464
163759997,1cyclictest49-21ksoftirqd/421:55:114
163759997,1cyclictest21994-21sshd21:24:554
163759993,3cyclictest1-21systemd23:51:144
163759992,4cyclictest7532-21sshd22:46:474
163759992,4cyclictest711-21dbus-daemon23:18:494
163759992,4cyclictest6562-21grep00:20:594
163759992,4cyclictest16858-21cp22:27:234
163759991,4cyclictest711-21dbus-daemon22:02:304
163759990,9cyclictest0-21swapper/421:07:064
163759990,9cyclictest0-21swapper/420:01:504
163759990,8cyclictest4722-21kworker/4:123:21:444
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional