You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 10:14
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack3slot0.osadl.org (updated Fri Nov 21, 2025 00:43:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
298422900,2sleep61711099cyclictest22:57:416
236732680,0sleep20-21swapper/223:54:312
181142570,0sleep40-21swapper/423:35:434
1670325126,9sleep70-21swapper/718:51:587
1685424925,8sleep50-21swapper/518:54:045
153642470,1sleep449-21ksoftirqd/422:42:254
107152460,0sleep10-21swapper/122:28:281
90422450,0sleep40-21swapper/421:42:424
320002440,0sleep20-21swapper/221:00:202
25312430,0sleep00-21swapper/022:04:330
7022420,0sleep10-21swapper/121:23:251
1663024229,9sleep60-21swapper/618:51:016
119812420,0sleep40-21swapper/421:02:144
36692410,1sleep125-21ksoftirqd/122:54:151
1681724129,8sleep40-21swapper/418:53:314
161242410,1sleep125-21ksoftirqd/123:53:261
1433324027,9sleep00-21swapper/018:49:260
1683323928,8sleep20-21swapper/218:53:452
1660523928,8sleep30-21swapper/318:50:423
1441123928,8sleep10-21swapper/118:49:351
314442380,0sleep00-21swapper/021:54:480
233682380,0sleep40-21swapper/419:04:354
45012370,0sleep30-21swapper/323:34:063
45012370,0sleep30-21swapper/323:34:063
278162370,0sleep30-21swapper/323:41:183
240132370,0sleep324012-21sshd00:03:533
115342350,2sleep71712199cyclictest22:28:347
1712199258,5cyclictest0-21swapper/720:15:327
1712199258,5cyclictest0-21swapper/719:37:447
1712199208,6cyclictest0-21swapper/723:33:497
1712199208,6cyclictest0-21swapper/723:33:497
1710399200,5cyclictest0-21swapper/521:51:015
79162190,0sleep50-21swapper/522:23:365
28282190,0sleep50-21swapper/500:00:345
1712199198,5cyclictest0-21swapper/723:52:057
1712199198,5cyclictest0-21swapper/723:25:597
1712199198,5cyclictest0-21swapper/721:35:427
1712199198,5cyclictest0-21swapper/721:35:427
1712199198,5cyclictest0-21swapper/720:53:147
1712199198,0cyclictest0-21swapper/721:45:487
1712199197,6cyclictest0-21swapper/722:38:287
1712199197,6cyclictest0-21swapper/722:14:367
1712199191,6cyclictest0-21swapper/722:02:347
1712199191,6cyclictest0-21swapper/719:46:487
1710399194,7cyclictest6090-21bash21:55:455
1710399190,7cyclictest14160-21id21:38:595
1710399190,7cyclictest14160-21id21:38:595
17087991913,4cyclictest41-21ksoftirqd/322:27:253
17079991915,1cyclictest33-21ksoftirqd/223:29:372
17079991915,1cyclictest33-21ksoftirqd/223:29:372
1712199188,4cyclictest0-21swapper/720:42:327
1712199187,5cyclictest0-21swapper/723:34:357
1712199187,5cyclictest0-21swapper/722:59:057
1712199187,5cyclictest0-21swapper/721:32:187
1712199187,5cyclictest0-21swapper/720:33:347
1712199187,5cyclictest0-21swapper/719:33:347
1712199187,5cyclictest0-21swapper/719:18:547
1712199187,0cyclictest0-21swapper/719:04:147
1712199181,5cyclictest0-21swapper/723:17:497
1712199181,5cyclictest0-21swapper/723:17:497
1712199181,5cyclictest0-21swapper/720:37:367
1712199181,0cyclictest0-21swapper/722:10:067
1710399180,6cyclictest29286-21grepconf.sh22:26:275
1710399180,10cyclictest18528-21sshd23:00:265
1710399180,10cyclictest12674-21sshd22:55:135
1707999182,1cyclictest33-21ksoftirqd/221:17:262
1712199176,5cyclictest0-21swapper/720:59:127
1712199176,5cyclictest0-21swapper/720:44:407
1712199176,5cyclictest0-21swapper/719:53:447
1712199176,5cyclictest0-21swapper/718:55:287
1712199176,5cyclictest0-21swapper/700:05:597
1712199176,5cyclictest0-21swapper/700:00:557
1712199176,0cyclictest0-21swapper/722:33:327
1712199176,0cyclictest0-21swapper/721:22:327
1712199176,0cyclictest0-21swapper/720:08:307
1712199176,0cyclictest0-21swapper/719:55:027
1712199176,0cyclictest0-21swapper/719:24:167
1712199176,0cyclictest0-21swapper/700:16:577
17121991710,7cyclictest73-21ksoftirqd/723:39:477
1710399170,9cyclictest4809-21bash21:19:295
1710399170,8cyclictest10382-21sshd23:21:425
1710399170,3cyclictest0-21swapper/522:19:135
1710399170,14cyclictest27721-21sshd21:09:035
17087991714,1cyclictest41-21ksoftirqd/323:08:363
1707999179,2cyclictest33-21ksoftirqd/221:44:322
1707999178,2cyclictest33-21ksoftirqd/222:17:132
1707999177,3cyclictest33-21ksoftirqd/221:28:142
1707999175,2cyclictest33-21ksoftirqd/222:55:192
17079991714,1cyclictest33-21ksoftirqd/221:23:262
17079991713,1cyclictest33-21ksoftirqd/221:29:412
314672160,0sleep50-21swapper/521:32:105
1712199165,5cyclictest0-21swapper/700:13:097
1712199165,0cyclictest0-21swapper/722:40:527
1712199165,0cyclictest0-21swapper/721:09:007
1712199165,0cyclictest0-21swapper/719:39:347
1712199160,5cyclictest0-21swapper/720:11:427
1712199160,13cyclictest25337-21sshd21:58:397
17110991610,5cyclictest27072-21switchtime23:54:476
1710399160,8cyclictest7027-21grepconf.sh21:10:275
1710399160,7cyclictest0-21swapper/523:29:555
1710399160,7cyclictest0-21swapper/523:29:555
1710399160,6cyclictest29615-21systemd-cgroups23:50:295
1710399160,6cyclictest11751-21grepconf.sh21:43:075
1710399160,6cyclictest10923-21bash23:12:525
1710399160,5cyclictest5056-21date22:54:255
1710399160,5cyclictest18202-21kworker/5:023:55:545
1710399160,15cyclictest11971-21bash22:46:215
1710399160,10cyclictest4457-21grepconf.sh23:25:125
1710399160,10cyclictest26117-21grep23:36:505
17087991615,1cyclictest41-21ksoftirqd/300:18:343
17087991615,1cyclictest11801-21kworker/3:219:44:563
17087991614,1cyclictest41-21ksoftirqd/322:31:143
17087991611,3cyclictest41-21ksoftirqd/322:40:313
1708799160,1cyclictest41-21ksoftirqd/323:16:363
1708799160,1cyclictest41-21ksoftirqd/323:16:363
1707999165,2cyclictest33-21ksoftirqd/200:01:152
1707999165,1cyclictest33-21ksoftirqd/223:40:052
1707999164,3cyclictest33-21ksoftirqd/223:03:092
1707999160,1cyclictest0-21swapper/223:22:522
17063991611,3cyclictest9-21ksoftirqd/022:15:260
6162150,0sleep20-21swapper/223:11:222
295382150,0sleep10-21swapper/123:02:001
1712199154,0cyclictest0-21swapper/720:27:067
1712199154,0cyclictest0-21swapper/720:21:587
17121991513,1cyclictest740-21gdbus22:51:277
1712199150,9cyclictest0-21swapper/722:05:387
1712199150,9cyclictest0-21swapper/721:04:147
1712199150,3cyclictest0-21swapper/721:40:407
1712199150,3cyclictest0-21swapper/719:09:107
1710399154,8cyclictest31100-21sshd21:18:385
1710399150,8cyclictest0-21swapper/523:17:325
1710399150,8cyclictest0-21swapper/523:17:325
1710399150,7cyclictest15828-21bash22:01:445
1710399150,6cyclictest0-21swapper/521:45:005
1710399150,5cyclictest23013-21sshd23:44:595
1710399150,5cyclictest14673-21sshd23:04:295
1710399150,5cyclictest0-21swapper/522:04:465
17087991514,1cyclictest41-21ksoftirqd/323:24:433
17087991513,1cyclictest41-21ksoftirqd/320:59:253
17087991510,3cyclictest41-21ksoftirqd/322:57:273
1708799151,3cyclictest41-21ksoftirqd/323:51:513
1708799150,2cyclictest121rcu_preempt23:47:593
1707999157,5cyclictest33-21ksoftirqd/222:06:112
1707999156,2cyclictest33-21ksoftirqd/221:10:322
1707999155,2cyclictest33-21ksoftirqd/222:48:302
1707999155,2cyclictest33-21ksoftirqd/220:55:092
1707999154,3cyclictest33-21ksoftirqd/223:18:062
1707999154,3cyclictest33-21ksoftirqd/223:18:062
1707999154,1cyclictest33-21ksoftirqd/222:23:552
1707999154,1cyclictest33-21ksoftirqd/221:59:402
1707999152,3cyclictest33-21ksoftirqd/222:41:172
17079991514,1cyclictest33-21ksoftirqd/223:49:142
17079991513,1cyclictest33-21ksoftirqd/221:39:482
17079991512,3cyclictest33-21ksoftirqd/223:51:052
17079991512,3cyclictest33-21ksoftirqd/200:09:252
1707999150,1cyclictest121rcu_preempt22:28:032
17063991514,1cyclictest9-21ksoftirqd/021:49:510
17063991514,1cyclictest9-21ksoftirqd/021:13:360
17063991513,1cyclictest9-21ksoftirqd/021:59:420
17063991512,2cyclictest9-21ksoftirqd/021:41:190
17063991511,1cyclictest9-21ksoftirqd/022:58:130
15912150,0sleep60-21swapper/621:00:406
1712199148,6cyclictest0-21swapper/723:48:377
1712199148,5cyclictest2999-21sshd23:02:517
1712199143,5cyclictest0-21swapper/723:13:597
1712199143,5cyclictest0-21swapper/722:20:407
1712199143,5cyclictest0-21swapper/721:27:307
1712199143,5cyclictest0-21swapper/719:29:227
1712199143,5cyclictest0-21swapper/700:20:157
1712199143,0cyclictest0-21swapper/720:03:347
1712199142,6cyclictest0-21swapper/723:59:217
1712199142,6cyclictest0-21swapper/721:11:427
1711099140,1cyclictest0-21swapper/621:42:266
1710399140,8cyclictest26653-21sshd23:41:085
1710399140,7cyclictest4740-21bash22:09:275
1710399140,6cyclictest5877-21grepconf.sh00:05:425
1710399140,6cyclictest31391-21sshd21:27:395
1710399140,5cyclictest0-21swapper/522:40:575
1710399140,14cyclictest0-21swapper/500:14:445
1710399140,13cyclictest711-21dbus-daemon20:58:115
1710399140,13cyclictest4876-21sshd22:32:025
1708799144,2cyclictest41-21ksoftirqd/322:01:403
17087991414,0cyclictest11801-21kworker/3:220:26:123
17087991413,1cyclictest41-21ksoftirqd/321:57:293
17087991413,1cyclictest41-21ksoftirqd/321:10:223
17087991413,1cyclictest41-21ksoftirqd/300:19:413
17087991412,1cyclictest41-21ksoftirqd/323:56:363
17087991412,1cyclictest41-21ksoftirqd/323:36:033
17087991412,1cyclictest41-21ksoftirqd/322:34:413
17087991412,1cyclictest41-21ksoftirqd/321:49:003
17087991412,1cyclictest41-21ksoftirqd/321:34:473
17087991412,1cyclictest41-21ksoftirqd/321:34:473
17087991412,1cyclictest41-21ksoftirqd/321:04:413
17087991412,1cyclictest41-21ksoftirqd/320:59:313
17087991411,3cyclictest41-21ksoftirqd/323:09:363
1708799140,1cyclictest121rcu_preempt22:14:483
1707999145,1cyclictest33-21ksoftirqd/222:30:312
1707999144,4cyclictest33-21ksoftirqd/222:12:242
1707999144,3cyclictest33-21ksoftirqd/221:58:092
1707999143,1cyclictest33-21ksoftirqd/221:04:432
17079991412,1cyclictest33-21ksoftirqd/221:49:362
17079991412,1cyclictest33-21ksoftirqd/200:09:242
17079991411,1cyclictest33-21ksoftirqd/200:16:342
17079991410,1cyclictest33-21ksoftirqd/223:28:582
1707999140,1cyclictest33-21ksoftirqd/222:38:522
1706399149,2cyclictest9-21ksoftirqd/000:21:350
17063991413,1cyclictest9-21ksoftirqd/023:08:190
17063991412,2cyclictest9-21ksoftirqd/023:27:370
1712199138,5cyclictest0-21swapper/723:04:537
1712199138,5cyclictest0-21swapper/722:45:187
1712199138,5cyclictest0-21swapper/721:15:107
1712199138,5cyclictest0-21swapper/719:12:267
1712199138,0cyclictest2481-21sshd21:50:427
1712199138,0cyclictest0-21swapper/723:20:097
1711099130,12cyclictest11640-21kworker/6:021:17:566
1710399130,7cyclictest27703-21bash22:35:035
1710399130,7cyclictest13081-21cstates20:39:425
1710399130,5cyclictest0-21swapper/500:10:215
1710399130,4cyclictest0-21swapper/521:00:005
17095991311,1cyclictest49-21ksoftirqd/422:09:244
17095991311,1cyclictest49-21ksoftirqd/400:24:244
1709599130,0cyclictest131rcu_sched21:38:424
1709599130,0cyclictest131rcu_sched21:38:424
1708799137,3cyclictest41-21ksoftirqd/321:32:273
17087991312,1cyclictest41-21ksoftirqd/322:06:213
17087991312,1cyclictest41-21ksoftirqd/321:19:373
17087991311,1cyclictest41-21ksoftirqd/323:04:013
17087991311,1cyclictest41-21ksoftirqd/322:45:393
17087991311,1cyclictest41-21ksoftirqd/321:52:323
17087991311,1cyclictest41-21ksoftirqd/321:39:473
17087991311,1cyclictest41-21ksoftirqd/321:24:463
17087991311,1cyclictest41-21ksoftirqd/300:10:093
17087991311,1cyclictest41-21ksoftirqd/300:05:133
17087991310,3cyclictest41-21ksoftirqd/321:14:363
17087991310,2cyclictest41-21ksoftirqd/323:19:443
17087991310,2cyclictest41-21ksoftirqd/322:09:503
17087991310,1cyclictest41-21ksoftirqd/322:23:263
1708799130,2cyclictest131rcu_sched22:50:103
1707999138,3cyclictest33-21ksoftirqd/223:04:372
1707999138,2cyclictest33-21ksoftirqd/222:51:212
1707999138,1cyclictest33-21ksoftirqd/221:34:392
1707999138,1cyclictest33-21ksoftirqd/221:34:392
1707999135,1cyclictest33-21ksoftirqd/200:21:242
1707999133,2cyclictest33-21ksoftirqd/223:35:552
1707199137,2cyclictest10804-21kworker/1:322:12:181
1706399139,3cyclictest9-21ksoftirqd/022:32:400
1706399137,4cyclictest9-21ksoftirqd/023:03:000
17063991312,1cyclictest9-21ksoftirqd/022:27:290
17063991311,1cyclictest9-21ksoftirqd/023:11:530
17063991311,1cyclictest9-21ksoftirqd/022:47:170
17063991311,1cyclictest9-21ksoftirqd/021:05:230
17063991311,1cyclictest9-21ksoftirqd/000:09:450
17063991310,2cyclictest9-21ksoftirqd/021:24:370
17063991310,1cyclictest9-21ksoftirqd/000:08:150
87572120,0sleep60-21swapper/622:19:126
17110991210,2cyclictest85350irq/25-em1-rx-022:40:016
1710399120,5cyclictest0-21swapper/500:20:585
17087991211,1cyclictest25180-21kworker/3:019:29:103
17087991211,1cyclictest11801-21kworker/3:220:37:063
17087991211,1cyclictest11801-21kworker/3:219:55:203
1706399129,2cyclictest9-21ksoftirqd/023:39:450
17063991211,1cyclictest9-21ksoftirqd/023:21:450
17063991211,1cyclictest9-21ksoftirqd/022:39:430
17063991211,1cyclictest9-21ksoftirqd/022:19:450
17063991211,1cyclictest9-21ksoftirqd/021:31:340
17063991210,1cyclictest9-21ksoftirqd/023:59:480
17063991210,1cyclictest9-21ksoftirqd/023:36:290
17063991210,1cyclictest9-21ksoftirqd/022:14:240
17063991210,1cyclictest9-21ksoftirqd/021:45:140
17063991210,1cyclictest9-21ksoftirqd/021:00:040
111012120,0sleep10-21swapper/122:24:041
313452110,0sleep60-21swapper/620:14:436
1711099113,0cyclictest0-21swapper/619:01:396
1711099110,1cyclictest0-21swapper/621:45:036
1710399110,10cyclictest0-21swapper/520:34:465
17095991110,1cyclictest49-21ksoftirqd/423:32:324
17095991110,1cyclictest49-21ksoftirqd/423:32:324
17095991110,1cyclictest0-21swapper/421:31:024
1709599110,10cyclictest30377-21bash00:18:374
1708799119,1cyclictest41-21ksoftirqd/320:19:253
1708799119,1cyclictest41-21ksoftirqd/319:29:383
17087991111,0cyclictest25180-21kworker/3:019:22:343
17087991110,1cyclictest41-21ksoftirqd/320:14:253
17087991110,1cyclictest25180-21kworker/3:019:36:063
1707999119,1cyclictest33-21ksoftirqd/220:54:252
1707999119,1cyclictest33-21ksoftirqd/220:24:252
1707999117,4cyclictest33-21ksoftirqd/219:52:022
1707999114,2cyclictest33-21ksoftirqd/219:45:162
1707999111,1cyclictest33-21ksoftirqd/220:39:502
1706399119,1cyclictest9-21ksoftirqd/023:54:440
1706399119,1cyclictest9-21ksoftirqd/023:51:470
1706399119,1cyclictest9-21ksoftirqd/020:54:380
1706399119,1cyclictest9-21ksoftirqd/020:04:250
1706399117,1cyclictest9-21ksoftirqd/021:37:150
1706399117,1cyclictest9-21ksoftirqd/021:37:150
1706399113,4cyclictest13203-21/usr/sbin/munin19:44:240
17063991110,1cyclictest9-21ksoftirqd/023:44:330
17063991110,1cyclictest9-21ksoftirqd/023:31:530
17063991110,1cyclictest9-21ksoftirqd/023:31:530
17063991110,1cyclictest9-21ksoftirqd/022:52:030
17063991110,1cyclictest9-21ksoftirqd/022:34:310
17063991110,1cyclictest9-21ksoftirqd/021:15:490
285512100,0sleep60-21swapper/600:18:216
1711099109,1cyclictest0-21swapper/623:22:336
1711099100,10cyclictest0-21swapper/622:26:496
1711099100,10cyclictest0-21swapper/622:08:236
1711099100,10cyclictest0-21swapper/621:56:406
1710399100,9cyclictest18507-21tr18:54:435
1710399100,8cyclictest31285-21chrt19:19:175
1710399100,0cyclictest0-21swapper/519:09:485
1709599109,1cyclictest49-21ksoftirqd/423:02:574
1709599109,1cyclictest49-21ksoftirqd/422:36:044
1709599109,1cyclictest49-21ksoftirqd/422:14:414
1709599109,1cyclictest49-21ksoftirqd/421:54:424
1709599109,1cyclictest49-21ksoftirqd/421:20:294
1709599108,1cyclictest49-21ksoftirqd/423:57:444
1709599108,1cyclictest49-21ksoftirqd/423:40:134
1709599108,1cyclictest49-21ksoftirqd/423:22:384
1709599108,1cyclictest49-21ksoftirqd/423:11:494
1709599100,10cyclictest0-21swapper/421:28:584
1709599100,10cyclictest0-21swapper/420:57:084
1708799109,1cyclictest16805-21kworker/3:118:57:483
1708799109,1cyclictest11801-21kworker/3:220:42:183
1708799108,1cyclictest41-21ksoftirqd/320:49:253
1708799108,1cyclictest41-21ksoftirqd/319:19:253
17087991010,0cyclictest25180-21kworker/3:019:06:483
17087991010,0cyclictest11801-21kworker/3:220:05:343
17087991010,0cyclictest11801-21kworker/3:220:03:403
1707999109,1cyclictest33-21ksoftirqd/218:54:432
1707999108,1cyclictest33-21ksoftirqd/220:09:252
1707999108,1cyclictest33-21ksoftirqd/218:59:402
1707199109,1cyclictest29793-21sshd00:18:321
1707199109,1cyclictest0-21swapper/122:34:101
1707199107,1cyclictest6928-21kworker/1:023:43:471
1707199106,3cyclictest16377-21kworker/1:123:55:291
1707199105,1cyclictest6928-21kworker/1:023:10:531
1706399108,1cyclictest9-21ksoftirqd/023:14:420
1706399108,1cyclictest9-21ksoftirqd/023:14:420
1706399108,1cyclictest9-21ksoftirqd/021:19:470
1706399107,3cyclictest9-21ksoftirqd/000:14:400
32090290,1sleep10-21swapper/120:55:471
171109999,0cyclictest0-21swapper/623:31:286
171109999,0cyclictest0-21swapper/623:31:286
171109998,1cyclictest85350irq/25-em1-rx-023:19:176
171109998,1cyclictest85350irq/25-em1-rx-023:19:176
171109998,1cyclictest0-21swapper/621:25:116
171109997,1cyclictest65-21ksoftirqd/623:27:156
171109997,1cyclictest65-21ksoftirqd/623:07:266
171109996,2cyclictest21639-21sshd21:30:426
171109993,3cyclictest15902-21cpuspeed_turbos21:34:376
171109993,3cyclictest15902-21cpuspeed_turbos21:34:376
171109993,3cyclictest14583-21sshd23:48:326
171109991,4cyclictest4437-21bash21:10:036
171109990,9cyclictest0-21swapper/623:52:456
171109990,9cyclictest0-21swapper/623:09:426
171109990,9cyclictest0-21swapper/621:53:196
171109990,9cyclictest0-21swapper/620:58:046
171109990,9cyclictest0-21swapper/619:14:486
171109990,5cyclictest23070-21bash21:21:566
171039994,5cyclictest0-21swapper/519:45:005
171039994,0cyclictest0-21swapper/520:24:445
171039993,5cyclictest757-21snmpd20:21:385
171039993,5cyclictest757-21snmpd20:15:025
171039993,5cyclictest757-21snmpd20:10:385
171039993,5cyclictest757-21snmpd20:05:595
171039993,5cyclictest757-21snmpd19:00:245
171039992,6cyclictest18580-21grep20:49:365
171039990,9cyclictest0-21swapper/519:05:165
171039990,8cyclictest2293-21cpuspeed_turbos19:24:335
171039990,8cyclictest16352-21sed20:44:395
171039990,4cyclictest0-21swapper/519:59:565
171039990,4cyclictest0-21swapper/519:56:185
171039990,4cyclictest0-21swapper/519:49:305
171039990,4cyclictest0-21swapper/519:43:265
171039990,4cyclictest0-21swapper/519:35:155
171039990,4cyclictest0-21swapper/519:30:325
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional