You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 06:46
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack3slot0.osadl.org (updated Mon Nov 24, 2025 00:43:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1633225126,9sleep60-21swapper/618:52:376
1639124925,8sleep00-21swapper/018:53:270
1623224926,8sleep30-21swapper/318:51:143
104572490,0sleep20-21swapper/223:34:362
115452480,1sleep511526-21sshd21:07:115
20522450,0sleep60-21swapper/621:19:306
71012440,0sleep00-21swapper/023:16:240
131502440,0sleep00-21swapper/023:52:520
103922440,0sleep60-21swapper/622:40:596
194342430,0sleep00-21swapper/020:49:370
140352430,5sleep50-21swapper/518:49:305
123512430,0sleep50-21swapper/522:32:175
73722410,0sleep30-21swapper/323:34:213
246812410,0sleep10-21swapper/123:49:491
1625524129,8sleep70-21swapper/718:51:347
272162400,0sleep40-21swapper/422:20:464
251582400,0sleep00-21swapper/022:52:140
19072400,0sleep40-21swapper/421:32:414
18442400,0sleep20-21swapper/222:21:492
118752400,0sleep70-21swapper/723:43:547
76222390,0sleep50-21swapper/522:58:525
76222390,0sleep50-21swapper/522:58:525
55572390,2sleep61663599cyclictest22:26:496
42072390,0sleep60-21swapper/622:17:446
175292390,2sleep41662099cyclictest22:51:064
1624823927,9sleep10-21swapper/118:51:281
1624023925,10sleep20-21swapper/218:51:212
1613623928,8sleep40-21swapper/418:49:544
246222380,0sleep70-21swapper/722:02:397
1662799203,0cyclictest0-21swapper/520:37:235
1662799203,0cyclictest0-21swapper/520:37:235
1662799200,7cyclictest0-21swapper/521:10:215
72722190,0sleep40-21swapper/400:05:144
3352190,0sleep10-21swapper/123:02:161
1664399198,0cyclictest0-21swapper/722:57:417
1664399198,0cyclictest0-21swapper/722:57:417
1664399198,0cyclictest0-21swapper/720:45:257
1664399191,6cyclictest0-21swapper/723:19:237
1662799198,5cyclictest0-21swapper/523:34:375
1662799198,5cyclictest0-21swapper/522:40:475
1662799198,5cyclictest0-21swapper/521:59:475
1664399187,0cyclictest0-21swapper/720:41:337
1664399187,0cyclictest0-21swapper/720:14:477
1663599189,1cyclictest65-21ksoftirqd/622:51:116
1662799187,5cyclictest0-21swapper/523:09:355
1662799187,5cyclictest0-21swapper/520:53:235
1662799187,0cyclictest0-21swapper/519:54:475
1662799181,11cyclictest0-21swapper/523:53:155
227642170,2sleep21660099cyclictest21:04:202
1664399176,0cyclictest0-21swapper/720:14:157
16643991712,0cyclictest0-21swapper/722:12:037
1663599178,2cyclictest65-21ksoftirqd/623:08:036
1662799176,5cyclictest0-21swapper/520:41:235
1658699179,5cyclictest9-21ksoftirqd/023:05:290
288712160,0sleep30-21swapper/320:09:363
1664399164,9cyclictest27918-21sshd22:20:537
1664399164,9cyclictest15422-21sshd23:26:257
16643991612,0cyclictest0-21swapper/723:53:497
16635991613,1cyclictest65-21ksoftirqd/621:57:016
1663599160,2cyclictest65-21ksoftirqd/620:57:106
1662799165,6cyclictest28104-21nfsd00:22:255
320362150,1sleep4471rcuc/423:06:354
1664399158,6cyclictest18270-21grepconf.sh23:44:357
1664399152,7cyclictest0-21swapper/721:10:117
1664399150,9cyclictest0-21swapper/722:31:477
1664399150,9cyclictest0-21swapper/721:23:457
1664399150,9cyclictest0-21swapper/700:00:117
1663599157,4cyclictest65-21ksoftirqd/621:13:016
1663599156,3cyclictest65-21ksoftirqd/620:59:406
1663599156,1cyclictest65-21ksoftirqd/621:29:106
1663599155,3cyclictest65-21ksoftirqd/621:30:056
16635991514,1cyclictest65-21ksoftirqd/600:14:336
16635991514,1cyclictest65-21ksoftirqd/600:14:336
1663599151,3cyclictest65-21ksoftirqd/623:23:076
1663599151,3cyclictest0-21swapper/623:27:236
1663599151,2cyclictest65-21ksoftirqd/623:02:106
1663599151,2cyclictest0-21swapper/622:04:336
1663599151,2cyclictest0-21swapper/621:47:006
1663599151,2cyclictest0-21swapper/621:34:416
1663599151,1cyclictest65-21ksoftirqd/623:51:516
1663599150,3cyclictest0-21swapper/623:35:366
1663599150,3cyclictest0-21swapper/623:13:346
1663599150,3cyclictest0-21swapper/622:47:086
1663599150,13cyclictest30513-21sshd21:18:586
1663599150,1cyclictest65-21ksoftirqd/622:56:486
1663599150,1cyclictest65-21ksoftirqd/622:56:486
1663599150,1cyclictest3469-21kworker/6:122:13:186
1663599150,1cyclictest0-21swapper/600:00:406
1663599150,0cyclictest0-21swapper/622:24:046
1662799154,5cyclictest0-21swapper/523:54:335
1662799154,0cyclictest0-21swapper/521:47:575
16627991510,0cyclictest0-21swapper/523:26:395
1660899150,13cyclictest19103-21sed23:49:183
1659399150,9cyclictest13498-21systemd-cgroups22:54:541
1659399150,9cyclictest13498-21systemd-cgroups22:54:541
16586991514,1cyclictest9-21ksoftirqd/022:47:330
112152150,0sleep10-21swapper/121:42:551
91452140,2sleep71664399cyclictest22:49:497
236512140,0sleep10-21swapper/123:05:201
1664399149,0cyclictest0-21swapper/723:55:017
1664399148,5cyclictest4754-21grep21:50:517
1664399148,5cyclictest20709-21sshd21:04:057
1664399148,5cyclictest13545-21sshd21:47:437
1664399148,5cyclictest13232-21grepconf.sh21:38:477
1664399148,5cyclictest1-21systemd23:38:437
1664399148,1cyclictest711-21dbus-daemon23:03:417
1664399147,1cyclictest23884-21systemd-cgroups22:47:317
1664399142,6cyclictest0-21swapper/721:14:357
1664399142,6cyclictest0-21swapper/719:04:597
1663599146,1cyclictest65-21ksoftirqd/622:35:326
1663599142,3cyclictest21797-21sshd23:18:326
16635991413,1cyclictest65-21ksoftirqd/621:52:436
16635991413,1cyclictest65-21ksoftirqd/600:09:426
16635991413,1cyclictest65-21ksoftirqd/600:04:396
16635991411,2cyclictest711-21dbus-daemon23:41:186
16635991411,1cyclictest65-21ksoftirqd/623:56:336
1663599141,3cyclictest65-21ksoftirqd/621:59:326
1663599140,3cyclictest65-21ksoftirqd/622:32:096
1663599140,1cyclictest0-21swapper/623:48:166
1662799149,5cyclictest0-21swapper/519:16:545
1662799148,5cyclictest21763-21sshd22:06:375
1662799148,5cyclictest0-21swapper/520:58:475
1662799143,5cyclictest28105-21nfsd19:20:515
1662799143,5cyclictest28105-21nfsd19:20:515
1662799143,5cyclictest0-21swapper/521:27:355
1662799143,5cyclictest0-21swapper/520:47:575
1662799143,0cyclictest0-21swapper/520:33:375
1662799143,0cyclictest0-21swapper/519:41:255
1662799140,8cyclictest0-21swapper/522:29:035
1662799140,8cyclictest0-21swapper/521:39:135
1662799140,8cyclictest0-21swapper/521:17:015
1662799140,2cyclictest0-21swapper/523:03:075
16608991412,1cyclictest861-21gdbus22:56:013
16608991412,1cyclictest861-21gdbus22:56:013
16586991412,1cyclictest9-21ksoftirqd/023:38:210
157602140,0sleep00-21swapper/022:19:220
1664399138,5cyclictest0-21swapper/723:30:017
1664399138,5cyclictest0-21swapper/723:15:017
1664399138,5cyclictest0-21swapper/723:05:017
1664399138,5cyclictest0-21swapper/722:42:197
1664399138,5cyclictest0-21swapper/722:29:077
1664399138,5cyclictest0-21swapper/721:08:297
1664399138,5cyclictest0-21swapper/720:57:197
1664399138,5cyclictest0-21swapper/720:23:177
1664399138,5cyclictest0-21swapper/720:23:177
1664399138,5cyclictest0-21swapper/720:05:417
1664399138,5cyclictest0-21swapper/719:55:497
1664399138,5cyclictest0-21swapper/719:46:337
1664399138,5cyclictest0-21swapper/719:15:197
1664399138,5cyclictest0-21swapper/700:05:377
1664399138,0cyclictest0-21swapper/722:36:137
1664399138,0cyclictest0-21swapper/722:18:217
1664399138,0cyclictest0-21swapper/722:07:537
1664399138,0cyclictest0-21swapper/720:52:297
1664399138,0cyclictest0-21swapper/720:26:597
1664399138,0cyclictest0-21swapper/720:26:597
1664399138,0cyclictest0-21swapper/719:54:117
1664399138,0cyclictest0-21swapper/719:29:417
1664399138,0cyclictest0-21swapper/719:19:237
1664399138,0cyclictest0-21swapper/719:19:237
1664399138,0cyclictest0-21swapper/719:10:037
1664399138,0cyclictest0-21swapper/719:02:177
1664399138,0cyclictest0-21swapper/700:22:177
1664399138,0cyclictest0-21swapper/700:16:217
1664399138,0cyclictest0-21swapper/700:16:217
1664399137,6cyclictest0-21swapper/721:55:137
1664399137,6cyclictest0-21swapper/720:31:237
1664399137,1cyclictest21177-21sshd21:26:237
1664399132,5cyclictest32513-21sshd21:32:257
1664399131,6cyclictest0-21swapper/719:34:577
1664399130,7cyclictest0-21swapper/723:10:057
1663599137,3cyclictest65-21ksoftirqd/621:39:296
16635991311,1cyclictest65-21ksoftirqd/621:04:326
1663599130,2cyclictest0-21swapper/600:19:326
1663599130,1cyclictest0-21swapper/623:29:386
1662799138,5cyclictest0-21swapper/523:47:155
1662799138,5cyclictest0-21swapper/523:31:515
1662799138,5cyclictest0-21swapper/523:20:535
1662799138,5cyclictest0-21swapper/522:21:415
1662799138,5cyclictest0-21swapper/521:50:455
1662799138,5cyclictest0-21swapper/520:22:435
1662799138,5cyclictest0-21swapper/520:22:435
1662799138,5cyclictest0-21swapper/520:14:235
1662799138,5cyclictest0-21swapper/519:46:275
1662799138,5cyclictest0-21swapper/519:10:435
1662799138,5cyclictest0-21swapper/518:56:545
1662799138,5cyclictest0-21swapper/500:14:335
1662799138,5cyclictest0-21swapper/500:14:335
1662799138,5cyclictest0-21swapper/500:09:415
1662799138,5cyclictest0-21swapper/500:02:175
1662799138,0cyclictest0-21swapper/523:06:495
1662799138,0cyclictest0-21swapper/522:10:175
1662799138,0cyclictest0-21swapper/520:01:475
1662799138,0cyclictest0-21swapper/519:49:235
1662799138,0cyclictest0-21swapper/519:35:455
1662799132,5cyclictest31954-21sshd21:59:185
1662799132,5cyclictest0-21swapper/520:27:035
1662799132,5cyclictest0-21swapper/520:27:035
1662799132,5cyclictest0-21swapper/519:32:295
1662799132,11cyclictest0-21swapper/522:16:015
1662799132,0cyclictest0-21swapper/523:16:275
1662799132,0cyclictest0-21swapper/520:06:095
1662799131,6cyclictest0-21swapper/521:42:495
1662799130,0cyclictest0-21swapper/522:49:135
16586991312,1cyclictest9-21ksoftirqd/023:54:390
16586991311,1cyclictest9-21ksoftirqd/022:05:120
16586991310,1cyclictest9-21ksoftirqd/022:29:260
97802120,0sleep00-21swapper/022:59:110
97802120,0sleep00-21swapper/022:59:110
48622120,0sleep20-21swapper/219:29:312
1664399127,5cyclictest0-21swapper/719:59:577
1664399127,5cyclictest0-21swapper/700:12:237
1664399127,0cyclictest0-21swapper/720:35:177
1664399127,0cyclictest0-21swapper/720:35:177
1664399127,0cyclictest0-21swapper/719:24:597
1664399127,0cyclictest0-21swapper/719:24:597
1664399127,0cyclictest0-21swapper/718:57:027
16643991210,1cyclictest73-21ksoftirqd/721:39:417
1664399121,0cyclictest0-21swapper/719:40:097
1663599124,1cyclictest65-21ksoftirqd/620:49:096
1663599121,1cyclictest65-21ksoftirqd/619:24:376
1663599121,1cyclictest65-21ksoftirqd/619:24:376
1662799127,5cyclictest0-21swapper/523:43:235
1662799127,5cyclictest0-21swapper/522:38:035
1662799127,5cyclictest0-21swapper/520:59:575
1662799127,5cyclictest0-21swapper/520:12:355
1662799127,5cyclictest0-21swapper/500:04:515
1662799122,6cyclictest0-21swapper/518:59:555
1662799120,6cyclictest0-21swapper/521:22:575
1662799120,0cyclictest0-21swapper/522:53:295
16620991210,1cyclictest0-21swapper/420:55:584
1659399126,3cyclictest2671-21kworker/1:021:17:111
1659399126,1cyclictest0-21swapper/121:34:591
1659399125,2cyclictest14154-21sshd23:44:141
1659399124,2cyclictest711-21dbus-daemon22:29:311
1659399124,2cyclictest25047-21grepconf.sh23:23:271
1659399124,2cyclictest24972-21bash21:58:171
1659399124,2cyclictest1-21systemd23:13:001
1659399124,1cyclictest740-21gdbus23:34:241
1659399123,3cyclictest19876-21bash22:46:551
1658699129,2cyclictest9-21ksoftirqd/023:26:370
1658699129,2cyclictest9-21ksoftirqd/022:41:480
1658699128,1cyclictest9-21ksoftirqd/000:14:080
1658699127,3cyclictest9-21ksoftirqd/021:04:260
16586991211,1cyclictest9-21ksoftirqd/023:23:140
16586991211,1cyclictest9-21ksoftirqd/022:59:450
16586991211,1cyclictest9-21ksoftirqd/021:55:260
16586991211,1cyclictest9-21ksoftirqd/000:16:380
16586991211,1cyclictest9-21ksoftirqd/000:16:380
16586991210,1cyclictest9-21ksoftirqd/023:09:300
16586991210,1cyclictest9-21ksoftirqd/021:45:090
1658699120,1cyclictest131rcu_sched22:02:320
128092120,0sleep012812-21latency_hist19:44:190
1663599114,1cyclictest65-21ksoftirqd/620:30:516
1663599114,1cyclictest65-21ksoftirqd/619:15:226
1663599110,1cyclictest65-21ksoftirqd/620:49:356
1662799116,5cyclictest85450irq/26-em1-tx-019:04:325
1662799116,5cyclictest0-21swapper/521:32:095
1662799110,0cyclictest0-21swapper/519:24:495
1662799110,0cyclictest0-21swapper/519:24:495
1662099119,1cyclictest27918-21sshd21:04:484
1662099119,1cyclictest23673-21sshd21:58:054
1662099111,4cyclictest0-21swapper/400:13:264
1662099110,10cyclictest0-21swapper/423:35:124
1660899110,10cyclictest0-21swapper/321:57:453
1660899110,1cyclictest0-21swapper/321:07:553
1660099119,1cyclictest758-21sshd23:15:072
1660099119,1cyclictest33-21ksoftirqd/223:24:302
1660099110,10cyclictest0-21swapper/222:03:052
1659399118,3cyclictest629-21kworker/1:023:44:231
1659399114,2cyclictest711-21dbus-daemon23:59:271
1659399114,2cyclictest29702-21bash22:52:541
1659399114,2cyclictest29098-21rm22:12:081
1659399114,2cyclictest26898-21grepconf.sh22:25:101
1659399114,2cyclictest23404-21grepconf.sh23:54:201
1659399114,2cyclictest13282-21sshd22:00:591
1659399114,1cyclictest861-21gdbus00:17:491
1659399114,1cyclictest861-21gdbus00:17:491
1659399114,1cyclictest740-21gdbus21:02:001
1659399114,1cyclictest12192-21grepconf.sh21:11:471
1659399114,1cyclictest11215-21rm21:51:501
1659399114,1cyclictest10109-21bash22:23:021
1659399113,2cyclictest9456-21grepconf.sh23:16:451
1659399113,2cyclictest740-21gdbus22:39:431
1659399113,2cyclictest711-21dbus-daemon00:05:021
1659399113,2cyclictest5137-21bash21:24:221
1659399113,2cyclictest28523-21grepconf.sh00:22:351
1659399113,2cyclictest24468-21bash21:22:301
1659399113,2cyclictest16919-21cp23:31:031
1659399113,2cyclictest11810-21grepconf.sh22:14:191
1659399113,1cyclictest711-21dbus-daemon22:34:551
1659399113,1cyclictest711-21dbus-daemon21:04:341
1659399112,2cyclictest7841-21httpd19:14:381
1659399110,1cyclictest0-21swapper/100:10:021
1658699119,2cyclictest9-21ksoftirqd/022:14:380
1658699119,2cyclictest9-21ksoftirqd/022:10:260
1658699119,1cyclictest9-21ksoftirqd/021:30:070
1658699119,1cyclictest9-21ksoftirqd/021:02:290
1658699119,1cyclictest9-21ksoftirqd/000:01:190
16586991110,1cyclictest9-21ksoftirqd/023:45:360
16586991110,1cyclictest9-21ksoftirqd/021:53:220
16586991110,1cyclictest9-21ksoftirqd/021:39:300
16586991110,1cyclictest9-21ksoftirqd/021:34:390
16586991110,1cyclictest9-21ksoftirqd/021:18:560
16586991110,1cyclictest9-21ksoftirqd/020:57:170
1658699110,11cyclictest8103-21kworker/0:000:24:170
1658699110,1cyclictest0-21swapper/022:26:390
171502100,0sleep20-21swapper/223:57:512
1663599109,1cyclictest65-21ksoftirqd/620:14:186
1663599103,1cyclictest65-21ksoftirqd/620:41:556
1663599103,1cyclictest65-21ksoftirqd/620:19:206
1663599103,1cyclictest65-21ksoftirqd/620:19:206
1663599103,1cyclictest65-21ksoftirqd/620:14:336
1663599103,1cyclictest65-21ksoftirqd/620:08:056
1663599103,1cyclictest65-21ksoftirqd/620:01:426
1663599103,1cyclictest65-21ksoftirqd/619:37:096
1663599103,1cyclictest65-21ksoftirqd/618:59:366
1663599101,1cyclictest5337-21mailstats20:24:396
1663599101,1cyclictest5337-21mailstats20:24:396
1663599101,1cyclictest0-21swapper/619:39:336
1663599101,1cyclictest0-21swapper/619:29:366
1662099109,1cyclictest0-21swapper/400:15:434
1662099109,1cyclictest0-21swapper/400:15:434
1662099100,4cyclictest7716-21sshd22:45:064
1662099100,4cyclictest0-21swapper/422:42:174
1662099100,10cyclictest0-21swapper/423:13:364
1662099100,10cyclictest0-21swapper/422:04:434
1662099100,10cyclictest0-21swapper/421:13:384
1662099100,1cyclictest0-21swapper/423:15:204
1660899109,1cyclictest0-21swapper/323:14:413
1660899108,1cyclictest711-21dbus-daemon00:03:333
1660899108,1cyclictest3550-21diskstats00:09:323
1660899107,1cyclictest29304-21kworker/3:221:11:393
1660099108,1cyclictest33-21ksoftirqd/221:26:402
1660099104,3cyclictest31361-21sshd00:04:182
1660099100,9cyclictest0-21swapper/222:18:032
1659399104,1cyclictest861-21gdbus21:32:241
1659399104,1cyclictest1760-21grep23:24:341
1659399103,2cyclictest30834-21bash21:45:291
1659399103,2cyclictest25532-21grepconf.sh20:54:431
1659399103,1cyclictest20325-21grep22:06:251
1658699109,1cyclictest9-21ksoftirqd/021:22:220
1658699108,1cyclictest9-21ksoftirqd/023:29:350
1658699108,1cyclictest9-21ksoftirqd/022:35:160
1658699108,1cyclictest9-21ksoftirqd/021:29:010
1658699108,1cyclictest9-21ksoftirqd/021:09:370
1658699107,1cyclictest9-21ksoftirqd/023:44:160
166359991,1cyclictest65-21ksoftirqd/618:54:386
166359991,1cyclictest32600-21expr19:19:356
166359991,1cyclictest32600-21expr19:19:356
166359990,3cyclictest65-21ksoftirqd/619:09:366
166359990,1cyclictest0-21swapper/619:54:376
166359990,0cyclictest0-21swapper/619:48:336
166209998,1cyclictest0-21swapper/422:57:134
166209998,1cyclictest0-21swapper/422:57:134
166209990,9cyclictest0-21swapper/423:22:174
166209990,5cyclictest9635-21grep22:14:034
166209990,0cyclictest0-21swapper/422:28:314
166209990,0cyclictest0-21swapper/421:53:414
166209990,0cyclictest0-21swapper/421:02:364
166089999,0cyclictest0-21swapper/323:23:213
166089999,0cyclictest0-21swapper/323:06:003
166089999,0cyclictest0-21swapper/322:31:193
166089997,1cyclictest859-21polkitd21:00:193
166089992,6cyclictest24165-21sshd22:38:333
166089990,9cyclictest0-21swapper/322:04:223
166089990,9cyclictest0-21swapper/321:46:423
166089990,9cyclictest0-21swapper/321:18:293
166009999,0cyclictest0-21swapper/222:45:202
166009997,1cyclictest33-21ksoftirqd/223:31:582
166009997,1cyclictest33-21ksoftirqd/223:05:542
166009997,1cyclictest33-21ksoftirqd/222:59:312
166009997,1cyclictest33-21ksoftirqd/222:50:392
166009997,1cyclictest33-21ksoftirqd/222:34:142
166009997,1cyclictest33-21ksoftirqd/221:39:372
166009993,5cyclictest29192-21id21:18:462
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional