You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 12:06
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack1slot1.osadl.org (updated Wed Sep 17, 2025 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
225682214152,21sleep20-21swapper/219:07:022
227852209147,22sleep00-21swapper/019:09:490
227882202165,24sleep30-21swapper/319:09:523
226872200163,24sleep10-21swapper/119:08:341
221492600,0sleep30-21swapper/322:48:393
41452590,0sleep10-21swapper/122:24:421
22908991814,1cyclictest28-21ksoftirqd/223:48:252
231712150,0sleep30-21swapper/322:02:443
70892140,0sleep30-21swapper/323:11:063
2292199130,5cyclictest254-21systemd-journal19:13:463
2292199130,1cyclictest13528-21latency19:55:203
22908991312,1cyclictest28-21ksoftirqd/221:40:202
22908991311,1cyclictest28-21ksoftirqd/200:25:002
22908991211,1cyclictest28-21ksoftirqd/223:55:142
22908991211,1cyclictest28-21ksoftirqd/223:43:502
22908991211,1cyclictest28-21ksoftirqd/223:13:232
22908991211,1cyclictest28-21ksoftirqd/223:05:092
22908991210,1cyclictest28-21ksoftirqd/221:15:142
319042110,0sleep10-21swapper/120:35:111
2292199113,5cyclictest254-21systemd-journal22:30:003
2292199112,1cyclictest254-21systemd-journal22:20:003
2292199111,5cyclictest27821-21open_inodes22:55:203
2292199110,3cyclictest0-21swapper/300:34:213
2290899119,1cyclictest28-21ksoftirqd/223:25:212
2290899119,1cyclictest28-21ksoftirqd/222:40:162
2290899119,1cyclictest28-21ksoftirqd/222:00:182
22908991110,1cyclictest28-21ksoftirqd/223:50:002
22908991110,1cyclictest28-21ksoftirqd/223:37:512
22908991110,1cyclictest28-21ksoftirqd/222:20:112
22908991110,1cyclictest28-21ksoftirqd/222:10:432
22908991110,1cyclictest28-21ksoftirqd/221:50:282
22908991110,1cyclictest28-21ksoftirqd/200:35:152
2290899110,7cyclictest5524-21apt-get19:40:122
22891991110,1cyclictest9-21ksoftirqd/021:36:550
2289199111,5cyclictest19496-21perf20:10:000
2289199110,10cyclictest6102-21ssh21:40:130
2289199110,10cyclictest0-21swapper/021:00:150
2292199104,1cyclictest32042-21ntp_states23:45:163
2292199103,0cyclictest15539-21diskmemload00:27:543
2292199102,1cyclictest10019-21grep21:45:153
2292199101,3cyclictest15539-21diskmemload00:03:143
2292199100,3cyclictest0-21swapper/323:56:323
2292199100,3cyclictest0-21swapper/321:18:393
2290899109,1cyclictest28-21ksoftirqd/223:00:192
2290899109,1cyclictest28-21ksoftirqd/222:59:262
2290899109,1cyclictest28-21ksoftirqd/222:32:232
2290899109,1cyclictest28-21ksoftirqd/222:25:162
2290899109,1cyclictest28-21ksoftirqd/222:18:432
2290899109,1cyclictest28-21ksoftirqd/221:36:172
2290899109,1cyclictest28-21ksoftirqd/221:20:182
2290899109,1cyclictest28-21ksoftirqd/220:40:152
2290899109,1cyclictest28-21ksoftirqd/200:30:172
2290899109,1cyclictest28-21ksoftirqd/200:19:452
2290899109,1cyclictest28-21ksoftirqd/200:10:122
2290899109,1cyclictest28-21ksoftirqd/200:05:172
2290899109,1cyclictest28-21ksoftirqd/200:05:162
2290899109,1cyclictest28-21ksoftirqd/200:00:182
2290899108,1cyclictest28-21ksoftirqd/223:15:152
2290899108,1cyclictest28-21ksoftirqd/222:35:222
2290899108,1cyclictest28-21ksoftirqd/222:35:222
2290899108,1cyclictest28-21ksoftirqd/221:32:172
2290899108,1cyclictest28-21ksoftirqd/221:25:192
2290899108,1cyclictest28-21ksoftirqd/220:50:002
2290899108,1cyclictest28-21ksoftirqd/200:29:492
22908991010,0cyclictest28-21ksoftirqd/222:45:412
22908991010,0cyclictest28-21ksoftirqd/221:45:202
2290899100,4cyclictest19909-21apt-get23:30:012
2289199102,4cyclictest6159-21perf20:50:000
2289199100,5cyclictest17082-21latency_hist20:04:590
229219995,4cyclictest989-21ssh23:04:343
229219995,4cyclictest1839-21cat22:20:163
229219994,1cyclictest4548-21ssh23:09:213
229219993,1cyclictest6183-21ssh00:38:043
229219993,1cyclictest3650-21expr19:35:173
229219993,0cyclictest15539-21diskmemload23:36:193
229219992,1cyclictest13574-21/usr/sbin/munin23:20:203
229219992,0cyclictest15539-21diskmemload21:57:293
229219990,1cyclictest0-21swapper/300:16:063
229089998,1cyclictest28-21ksoftirqd/222:50:202
229089998,1cyclictest28-21ksoftirqd/222:05:092
229089998,1cyclictest28-21ksoftirqd/221:59:242
229089998,1cyclictest28-21ksoftirqd/220:00:172
229089998,1cyclictest28-21ksoftirqd/219:35:152
229089997,1cyclictest28-21ksoftirqd/221:09:592
229009998,1cyclictest0-21swapper/123:39:221
228919993,1cyclictest10229-21/usr/sbin/munin19:50:130
228919992,5cyclictest15058-21kworker/u8:021:25:140
228919990,8cyclictest0-21swapper/020:40:020
229219986,1cyclictest554-21in:imuxsock20:11:013
229219983,1cyclictest31427-21expr21:30:153
229219981,1cyclictest254-21systemd-journal21:25:003
229219981,1cyclictest254-21systemd-journal21:07:483
229219981,1cyclictest254-21systemd-journal20:54:033
229219980,4cyclictest6097-21apt-get21:40:133
229219980,1cyclictest254-21systemd-journal22:45:013
229219980,1cyclictest0-21swapper/322:37:483
229219980,1cyclictest0-21swapper/322:37:483
229089987,1cyclictest28-21ksoftirqd/221:10:332
229089987,1cyclictest28-21ksoftirqd/221:00:202
229089987,1cyclictest28-21ksoftirqd/220:55:162
229089987,1cyclictest28-21ksoftirqd/220:20:122
229089987,1cyclictest28-21ksoftirqd/220:10:182
229089987,1cyclictest28-21ksoftirqd/219:20:142
229089986,1cyclictest28-21ksoftirqd/223:20:142
229089986,1cyclictest28-21ksoftirqd/220:05:192
228919987,1cyclictest9-21ksoftirqd/022:23:080
228919986,1cyclictest22583-21ssh00:15:300
228919984,2cyclictest254-21systemd-journal00:10:000
228919984,2cyclictest254-21systemd-journal00:09:590
228919984,2cyclictest15539-21diskmemload23:03:180
228919983,2cyclictest254-21systemd-journal23:17:490
228919981,6cyclictest28376-21ssh22:55:220
228919981,6cyclictest19301-1kworker/u9:022:10:120
228919981,6cyclictest0-21swapper/023:40:400
228919981,3cyclictest0-21swapper/021:30:190
228919980,7cyclictest5002-21ssh23:10:010
228919980,7cyclictest32109-21ssh22:17:150
228919980,7cyclictest25878-21ssh22:53:440
228919980,7cyclictest18978-21ssh00:11:200
228919980,7cyclictest18238-21ssh23:26:160
228919980,5cyclictest22324-21grep22:00:220
228919980,3cyclictest16941-21/usr/sbin/munin21:55:190
27985270,0sleep30-21swapper/319:16:453
229219975,1cyclictest15808-21apt-get21:10:113
229219973,1cyclictest254-21systemd-journal00:13:203
229219972,1cyclictest2879-21seq19:34:443
229219971,1cyclictest0-21swapper/319:40:113
229219970,3cyclictest0-21swapper/300:20:143
229219970,2cyclictest0-21swapper/322:10:113
229219970,2cyclictest0-21swapper/300:08:523
229219970,2cyclictest0-21swapper/300:08:523
229089976,1cyclictest28-21ksoftirqd/220:50:142
229089976,1cyclictest28-21ksoftirqd/219:30:172
229089976,1cyclictest28-21ksoftirqd/219:25:152
229089976,1cyclictest28-21ksoftirqd/219:10:162
229089976,1cyclictest25240-21kworker/2:020:30:192
229089975,1cyclictest28-21ksoftirqd/219:55:162
229089973,2cyclictest254-21systemd-journal20:40:012
229009970,6cyclictest0-21swapper/121:56:501
229009970,4cyclictest0-21swapper/121:45:071
229009970,4cyclictest0-21swapper/121:22:191
228919975,1cyclictest16908-21needreboot21:10:180
228919974,1cyclictest15539-21diskmemload00:04:200
228919973,2cyclictest3444-21apt-get00:35:120
228919972,1cyclictest31466-21/usr/sbin/munin20:35:210
228919971,6cyclictest3549-21ssh23:06:200
228919971,5cyclictest9-21ksoftirqd/019:35:210
228919971,5cyclictest803-21sshd20:52:360
228919971,1cyclictest15539-21diskmemload21:15:130
228919971,1cyclictest10138-21ssh21:45:150
228919970,7cyclictest4065-21ssh23:50:250
228919970,6cyclictest8246-21ssh23:57:050
228919970,6cyclictest25494-21ssh22:09:070
228919970,6cyclictest24162-21tune2fs23:35:140
228919970,6cyclictest23668-21cstates21:20:140
228919970,6cyclictest22956-21ssh23:33:570
228919970,6cyclictest22030-21ssh22:48:180
228919970,6cyclictest15539-21diskmemload21:50:290
228919970,6cyclictest11822-21ssh22:35:150
228919970,6cyclictest11822-21ssh22:35:140
228919970,6cyclictest0-21swapper/023:20:140
228919970,6cyclictest0-21swapper/019:25:110
228919970,6cyclictest0-21swapper/000:25:020
228919970,4cyclictest0-21swapper/021:05:120
228919970,4cyclictest0-21swapper/020:10:110
228919970,3cyclictest0-21swapper/020:20:120
228919970,1cyclictest5407-21cat22:25:190
228919970,1cyclictest24489-21apt-get00:20:130
228919970,1cyclictest1997-21/usr/sbin/munin19:30:220
228919970,1cyclictest1780-21ssh00:31:090
228919970,1cyclictest1088-21ssh23:48:000
228919970,1cyclictest10315-21ssh22:31:290
229219965,1cyclictest5097-21ssh23:53:293
229219965,1cyclictest4933-21ssh21:37:413
229219965,1cyclictest35-21ksoftirqd/320:10:003
229219965,1cyclictest29735-21ssh23:42:303
229219965,1cyclictest25069-21ssh22:07:503
229219965,1cyclictest24950-21ssh22:51:003
229219965,1cyclictest22152-21ssh23:31:343
229219965,1cyclictest19054-21ssh23:28:443
229219965,1cyclictest16354-21ssh21:54:063
229219964,1cyclictest3982-21apt-get20:45:113
229219964,1cyclictest35-21ksoftirqd/320:35:183
229219963,1cyclictest254-21systemd-journal19:49:463
229219962,2cyclictest12071-21memory21:00:193
229219961,3cyclictest1630-21apt-get20:40:133
229219961,3cyclictest0-21swapper/321:25:103
229219960,4cyclictest0-21swapper/323:17:113
229089966,0cyclictest60950irq/123-eno119:46:192
229089965,1cyclictest60950irq/123-eno120:25:132
229089964,1cyclictest28-21ksoftirqd/220:15:162
229009963,2cyclictest0-21swapper/121:50:191
229009963,2cyclictest0-21swapper/121:18:161
229009963,1cyclictest0-21swapper/123:48:551
229009961,2cyclictest0-21swapper/123:16:421
229009960,6cyclictest0-21swapper/122:40:161
229009960,6cyclictest0-21swapper/119:35:201
229009960,4cyclictest0-21swapper/123:54:441
229009960,3cyclictest0-21swapper/122:05:021
229009960,3cyclictest0-21swapper/121:30:131
229009960,2cyclictest0-21swapper/123:05:101
229009960,2cyclictest0-21swapper/100:10:351
229009960,1cyclictest0-21swapper/120:10:161
228919965,1cyclictest9-21ksoftirqd/019:42:080
228919965,1cyclictest29400-21cat19:20:170
228919963,1cyclictest254-21systemd-journal20:59:550
228919960,6cyclictest0-21swapper/019:55:210
228919960,3cyclictest0-21swapper/020:30:100
228919960,1cyclictest17357-21ssh22:40:510
228919960,0cyclictest0-21swapper/020:25:540
10597264,0sleep335-21ksoftirqd/322:32:213
229219955,0cyclictest0-21swapper/320:57:163
229219953,1cyclictest10521-21tr19:50:133
229219952,1cyclictest35-21ksoftirqd/320:31:173
229219951,2cyclictest254-21systemd-journal20:23:473
229219950,5cyclictest0-21swapper/320:28:103
229219950,2cyclictest557-21nscd20:00:173
229219950,1cyclictest254-21systemd-journal20:15:173
229089954,1cyclictest60950irq/123-eno119:50:112
229089954,1cyclictest25240-21kworker/2:019:15:142
229009955,0cyclictest0-21swapper/122:52:311
229009955,0cyclictest0-21swapper/100:04:491
229009954,1cyclictest0-21swapper/100:20:141
229009953,2cyclictest0-21swapper/123:22:251
229009953,1cyclictest21805-21chrt23:30:311
229009951,2cyclictest0-21swapper/121:25:171
229009951,1cyclictest0-21swapper/100:19:231
229009950,5cyclictest0-21swapper/120:40:171
229009950,5cyclictest0-21swapper/120:30:191
229009950,4cyclictest0-21swapper/123:25:161
229009950,4cyclictest0-21swapper/123:00:151
229009950,3cyclictest0-21swapper/122:56:231
229009950,3cyclictest0-21swapper/122:47:041
229009950,3cyclictest0-21swapper/122:25:171
229009950,3cyclictest0-21swapper/122:16:001
229009950,3cyclictest0-21swapper/122:10:221
229009950,3cyclictest0-21swapper/120:15:141
229009950,3cyclictest0-21swapper/100:25:441
229009950,2cyclictest0-21swapper/123:44:301
229009950,2cyclictest0-21swapper/121:35:131
229009950,2cyclictest0-21swapper/100:35:221
229009950,2cyclictest0-21swapper/100:05:141
229009950,2cyclictest0-21swapper/100:05:141
229009950,1cyclictest0-21swapper/122:00:211
229009950,1cyclictest0-21swapper/121:10:101
229009950,1cyclictest0-21swapper/100:30:401
228919952,1cyclictest254-21systemd-journal19:45:010
228919950,5cyclictest0-21swapper/019:10:140
228919950,4cyclictest0-21swapper/020:15:150
228919950,1cyclictest0-21swapper/019:15:130
10366250,1sleep10-21swapper/122:31:401
229219942,1cyclictest28511-21apt-get19:20:133
229219940,1cyclictest254-21systemd-journal19:25:163
229009944,0cyclictest0-21swapper/120:55:151
229009942,1cyclictest7648-21turbostat19:45:011
229009942,1cyclictest0-21swapper/121:00:231
229009941,2cyclictest0-21swapper/120:26:011
229009941,1cyclictest0-21swapper/120:00:141
229009940,4cyclictest0-21swapper/121:05:201
229009940,3cyclictest0-21swapper/119:25:101
229009940,2cyclictest0-21swapper/123:55:211
229009940,2cyclictest0-21swapper/122:36:451
229009940,2cyclictest0-21swapper/122:36:441
229009940,2cyclictest0-21swapper/119:35:001
229009940,1cyclictest0-21swapper/123:10:151
229009940,1cyclictest0-21swapper/121:40:191
229009940,1cyclictest0-21swapper/120:05:161
229009940,1cyclictest0-21swapper/119:55:161
229009940,1cyclictest0-21swapper/119:10:201
229009940,0cyclictest0-21swapper/120:20:201
229009940,0cyclictest0-21swapper/119:45:311
229009931,1cyclictest0-21swapper/120:45:121
229009931,1cyclictest0-21swapper/119:50:111
229009930,3cyclictest0-21swapper/119:20:141
229009930,1cyclictest0-21swapper/120:50:131
229009930,0cyclictest0-21swapper/119:15:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional