You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-02-17 - 14:46
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rack1slot1.osadl.org (updated Mon Feb 17, 2025 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
251112229154,22sleep30-21swapper/307:07:273
252452218153,22sleep00-21swapper/007:09:120
252682202150,41sleep10-21swapper/107:09:301
250482202162,27sleep20-21swapper/207:06:372
2525421100,0sleep00-21swapper/011:25:280
188142700,0sleep10-21swapper/108:00:191
73792530,1sleep12539499cyclictest11:45:111
257542170,0sleep30-21swapper/311:26:393
2539499175,11cyclictest0-21swapper/107:15:131
25401991614,1cyclictest28-21ksoftirqd/207:20:172
25401991614,1cyclictest28-21ksoftirqd/207:20:172
2540199130,13cyclictest0-21swapper/210:48:182
25391991311,1cyclictest9-21ksoftirqd/010:20:000
155212130,0sleep00-21swapper/009:45:230
25409991210,1cyclictest35-21ksoftirqd/308:10:013
2540199129,0cyclictest0-21swapper/210:25:522
25391991210,1cyclictest9-21ksoftirqd/012:35:000
2540199119,1cyclictest28-21ksoftirqd/207:30:002
2540199117,4cyclictest29229-1kworker/dying10:59:272
25401991110,1cyclictest28-21ksoftirqd/211:35:332
25401991110,1cyclictest28-21ksoftirqd/210:31:552
2539499111,1cyclictest21-21ksoftirqd/110:20:151
2539199110,11cyclictest0-21swapper/010:40:000
2540199109,1cyclictest28-21ksoftirqd/210:35:162
2540199108,1cyclictest28-21ksoftirqd/211:40:142
2539499108,1cyclictest111rcu_preempt07:30:241
2539499103,6cyclictest201rcuc/110:19:041
2539499102,7cyclictest111rcu_preempt09:27:011
2539499102,7cyclictest111rcu_preempt09:18:231
2539499102,7cyclictest111rcu_preempt09:18:231
2539499102,4cyclictest111rcu_preempt11:00:191
25394991010,0cyclictest21-21ksoftirqd/112:28:411
25394991010,0cyclictest21-21ksoftirqd/110:30:471
2539199100,5cyclictest29267-21apt-get08:25:110
2539199100,5cyclictest29086-21latency_hist08:25:010
115962100,0sleep20-21swapper/210:22:542
254099998,1cyclictest35-21ksoftirqd/311:20:003
254019998,1cyclictest28-21ksoftirqd/212:15:172
254019998,1cyclictest28-21ksoftirqd/211:55:192
254019998,1cyclictest28-21ksoftirqd/210:15:122
254019998,1cyclictest28-21ksoftirqd/210:11:092
254019998,1cyclictest28-21ksoftirqd/209:55:132
254019998,1cyclictest28-21ksoftirqd/209:40:142
254019998,1cyclictest28-21ksoftirqd/208:34:592
254019997,1cyclictest28-21ksoftirqd/211:30:082
254019997,1cyclictest28-21ksoftirqd/210:40:162
254019990,6cyclictest28020-21sh10:00:212
253949999,0cyclictest21-21ksoftirqd/111:35:101
253949999,0cyclictest21-21ksoftirqd/111:10:181
253949999,0cyclictest21-21ksoftirqd/111:10:181
253949999,0cyclictest21-21ksoftirqd/110:00:411
253949999,0cyclictest21-21ksoftirqd/108:45:161
253949998,1cyclictest21-21ksoftirqd/112:39:301
253949998,1cyclictest21-21ksoftirqd/112:22:241
253949998,1cyclictest21-21ksoftirqd/112:11:081
253949998,0cyclictest21-21ksoftirqd/112:19:041
253949998,0cyclictest21-21ksoftirqd/111:15:151
253949998,0cyclictest21-21ksoftirqd/110:25:081
253949998,0cyclictest111rcu_preempt11:50:091
253949993,4cyclictest21-21ksoftirqd/109:46:031
253949992,0cyclictest111rcu_preempt10:42:021
253949992,0cyclictest111rcu_preempt09:43:281
253949992,0cyclictest111rcu_preempt09:10:131
253949991,8cyclictest111rcu_preempt10:35:241
253949991,8cyclictest0-21swapper/109:58:431
253949991,7cyclictest111rcu_preempt07:45:191
253949991,1cyclictest21-21ksoftirqd/110:09:481
253949991,0cyclictest21-21ksoftirqd/112:05:141
253949991,0cyclictest111rcu_preempt11:30:031
253949990,8cyclictest21-21ksoftirqd/111:20:121
253949990,8cyclictest21-21ksoftirqd/108:55:141
253949990,8cyclictest21-21ksoftirqd/108:25:191
253949990,8cyclictest21-21ksoftirqd/108:20:091
253949990,8cyclictest111rcu_preempt12:00:181
253949990,8cyclictest111rcu_preempt10:50:091
253949990,7cyclictest21-21ksoftirqd/108:30:221
253949990,1cyclictest21-21ksoftirqd/111:41:281
253949990,1cyclictest21-21ksoftirqd/109:36:281
253919997,1cyclictest9-21ksoftirqd/008:40:000
253919997,1cyclictest24915-21ssh11:25:000
254099988,0cyclictest35-21ksoftirqd/311:55:173
254099987,1cyclictest557-21nscd12:21:513
254099983,0cyclictest18040-21diskmemload10:59:553
254099980,5cyclictest0-21swapper/310:37:543
254099980,4cyclictest0-21swapper/309:35:123
254099980,2cyclictest254-21systemd-journal07:54:273
254019988,0cyclictest28-21ksoftirqd/209:49:002
254019987,1cyclictest28-21ksoftirqd/211:29:142
254019987,1cyclictest28-21ksoftirqd/211:20:312
254019987,1cyclictest28-21ksoftirqd/211:15:122
254019987,1cyclictest28-21ksoftirqd/209:30:072
254019987,0cyclictest28-21ksoftirqd/209:05:172
254019986,1cyclictest60950irq/123-eno109:35:132
254019986,1cyclictest60950irq/123-eno107:49:592
254019986,1cyclictest28-21ksoftirqd/212:25:002
254019986,1cyclictest28-21ksoftirqd/209:28:132
254019986,1cyclictest28-21ksoftirqd/209:00:142
254019986,1cyclictest28-21ksoftirqd/208:49:592
254019986,1cyclictest28-21ksoftirqd/208:10:002
254019986,1cyclictest11679-21kworker/2:012:00:192
254019985,2cyclictest30783-21perf07:20:012
254019982,3cyclictest28-21ksoftirqd/209:22:522
254019981,5cyclictest28-21ksoftirqd/212:10:092
253949988,0cyclictest21-21ksoftirqd/111:55:021
253949988,0cyclictest21-21ksoftirqd/110:55:481
253949988,0cyclictest21-21ksoftirqd/110:45:181
253949988,0cyclictest21-21ksoftirqd/109:20:011
253949988,0cyclictest21-21ksoftirqd/108:15:011
253949988,0cyclictest0-21swapper/111:25:101
253949987,1cyclictest111rcu_preempt08:05:191
253949981,7cyclictest111rcu_preempt09:50:171
253949980,8cyclictest211ksoftirqd/107:20:211
253949980,8cyclictest211ksoftirqd/107:20:211
253949980,8cyclictest21-21ksoftirqd/112:30:021
253949980,8cyclictest21-21ksoftirqd/111:05:181
253949980,8cyclictest21-21ksoftirqd/110:10:161
253949980,8cyclictest21-21ksoftirqd/109:30:251
253949980,8cyclictest21-21ksoftirqd/107:55:181
253949980,8cyclictest21-21ksoftirqd/107:35:181
253949980,8cyclictest21-21ksoftirqd/107:11:361
253949980,8cyclictest111rcu_preempt08:10:171
253949980,8cyclictest0-21swapper/108:51:171
253949980,8cyclictest0-21swapper/108:40:041
253949980,8cyclictest0-21swapper/107:25:031
253949980,7cyclictest21-21ksoftirqd/108:36:221
253949980,1cyclictest21-21ksoftirqd/109:05:161
253949980,1cyclictest21-21ksoftirqd/109:00:181
253949980,0cyclictest21-21ksoftirqd/107:50:011
253949980,0cyclictest0-21swapper/107:40:181
253919987,1cyclictest9-21ksoftirqd/012:05:010
253919987,1cyclictest9-21ksoftirqd/007:35:010
253919987,1cyclictest16111-21ssh11:56:260
253919986,1cyclictest9-21ksoftirqd/010:00:000
253919986,1cyclictest9-21ksoftirqd/008:00:000
253919982,4cyclictest28490-21/usr/sbin/munin12:15:100
253919980,7cyclictest9044-21ssh11:49:070
253919980,7cyclictest6002-21ssh11:42:290
253919980,7cyclictest571-21ssh09:28:110
253919980,7cyclictest10512-21ssh09:39:230
253919980,7cyclictest10274-21ssh11:05:270
253919980,1cyclictest0-21swapper/010:03:460
4344270,0sleep30-21swapper/311:40:163
32555270,0sleep30-21swapper/310:52:003
254099975,1cyclictest21067-21ssh09:12:503
254099975,1cyclictest19819-21ssh12:00:283
254099974,2cyclictest29663-21ntpq12:15:173
254099972,2cyclictest254-21systemd-journal09:25:003
254099972,1cyclictest26234-21/usr/sbin/munin10:00:173
254099972,1cyclictest0-21swapper/309:05:133
254099971,3cyclictest0-21swapper/310:10:083
254099971,3cyclictest0-21swapper/308:55:163
254099970,2cyclictest0-21swapper/312:39:303
254099970,2cyclictest0-21swapper/311:50:053
254099970,1cyclictest0-21swapper/310:30:143
254019977,0cyclictest28-21ksoftirqd/211:53:052
254019976,1cyclictest60950irq/123-eno109:18:382
254019976,1cyclictest60950irq/123-eno109:18:372
254019976,1cyclictest60950irq/123-eno108:10:252
254019976,1cyclictest28-21ksoftirqd/211:06:482
254019976,1cyclictest28-21ksoftirqd/211:00:242
254019976,1cyclictest28-21ksoftirqd/210:05:072
254019976,1cyclictest28-21ksoftirqd/209:10:212
254019976,1cyclictest28-21ksoftirqd/207:40:222
254019976,0cyclictest60950irq/123-eno112:32:402
254019976,0cyclictest28-21ksoftirqd/211:10:062
254019976,0cyclictest28-21ksoftirqd/211:10:062
254019975,1cyclictest60950irq/123-eno108:00:172
254019975,1cyclictest32139-21kworker/2:208:55:132
254019975,1cyclictest28-21ksoftirqd/212:26:312
254019975,1cyclictest28-21ksoftirqd/209:50:092
254019975,1cyclictest28-21ksoftirqd/207:55:152
254019975,1cyclictest11679-21kworker/2:012:35:142
254019974,1cyclictest28-21ksoftirqd/208:25:012
253919976,1cyclictest9-21ksoftirqd/007:45:000
253919976,1cyclictest5764-21ssh12:28:460
253919976,1cyclictest11758-21ssh11:50:510
253919975,2cyclictest12543-21/usr/sbin/munin11:10:110
253919975,2cyclictest12543-21/usr/sbin/munin11:10:100
253919975,1cyclictest9-21ksoftirqd/011:15:020
253919975,1cyclictest9-21ksoftirqd/008:49:540
253919975,1cyclictest854-21df_inode10:10:100
253919975,1cyclictest22251-21ssh12:05:400
253919975,1cyclictest18040-21diskmemload12:24:190
253919973,2cyclictest254-21systemd-journal08:57:130
253919971,2cyclictest31808-21ssh10:08:270
253919971,2cyclictest0-21swapper/007:45:110
253919970,7cyclictest0-21swapper/010:32:440
253919970,6cyclictest30158-21ssh11:32:230
253919970,6cyclictest28358-21df07:15:010
253919970,6cyclictest25920-21ssh09:18:320
253919970,6cyclictest25920-21ssh09:18:320
253919970,6cyclictest25811-21ssh10:41:320
253919970,6cyclictest1712-21ssh10:55:040
253919970,6cyclictest0-21swapper/010:27:070
253919970,6cyclictest0-21swapper/007:40:000
253919970,1cyclictest9185-21cstates08:50:160
253919970,1cyclictest908-21ssh10:53:470
253919970,1cyclictest31584-21ssh11:35:110
253919970,1cyclictest28207-21ssh10:46:300
253919970,1cyclictest18852-21ls09:50:110
253919970,1cyclictest18291-21ssh09:10:090
253919970,1cyclictest12970-21ssh12:36:390
253919970,1cyclictest0-21swapper/012:10:100
253919970,1cyclictest0-21swapper/009:30:110
253919970,1cyclictest0-21swapper/009:20:520
254099965,1cyclictest8491-21ssh11:01:573
254099965,1cyclictest7787-21ssh11:46:103
254099965,1cyclictest31697-21ssh10:08:113
254099965,1cyclictest29688-21ssh11:31:183
254099965,1cyclictest25474-21ssh09:17:443
254099965,1cyclictest25474-21ssh09:17:433
254099965,1cyclictest25322-21ssh10:40:243
254099965,1cyclictest24132-21ssh09:55:563
254099965,1cyclictest22104-21ssh12:05:183
254099965,1cyclictest17124-21ssh09:49:003
254099965,1cyclictest1449-21ntpd08:44:223
254099965,1cyclictest1346-21ssh11:36:163
254099965,1cyclictest11421-21ssh10:22:313
254099964,1cyclictest947-21apt-get07:25:123
254099964,1cyclictest35-21ksoftirqd/308:24:563
254099964,1cyclictest35-21ksoftirqd/307:35:003
254099964,1cyclictest20965-21ssh11:20:033
254099964,1cyclictest12879-21ssh10:25:183
254099964,1cyclictest10175-21ssh11:05:123
254099963,1cyclictest254-21systemd-journal09:03:583
254099963,1cyclictest254-21systemd-journal08:19:273
254099961,3cyclictest0-21swapper/309:40:213
254099961,2cyclictest18040-21diskmemload09:51:073
254099960,4cyclictest0-21swapper/311:10:493
254099960,4cyclictest0-21swapper/311:10:493
254099960,3cyclictest88912sleep312:31:373
254099960,3cyclictest0-21swapper/310:45:373
254099960,3cyclictest0-21swapper/309:34:573
254099960,2cyclictest0-21swapper/310:15:223
254099960,2cyclictest0-21swapper/309:25:483
254099960,1cyclictest29448-21hddtemp_smartct07:15:143
254099960,1cyclictest0-21swapper/312:10:493
254019966,0cyclictest60950irq/123-eno112:06:212
254019965,1cyclictest60950irq/123-eno108:40:132
254019965,1cyclictest28-21ksoftirqd/211:45:062
254019965,1cyclictest28-21ksoftirqd/210:50:072
254019965,1cyclictest28-21ksoftirqd/208:15:162
254019965,1cyclictest28-21ksoftirqd/207:50:112
254019965,1cyclictest28-21ksoftirqd/207:10:162
254019964,1cyclictest32139-21kworker/2:208:35:132
254019964,1cyclictest28-21ksoftirqd/207:35:172
254019960,6cyclictest0-21swapper/208:25:152
253919965,1cyclictest7023-21ssh11:00:170
253919964,1cyclictest9-21ksoftirqd/008:05:130
253919964,1cyclictest13411-21grep07:50:140
253919962,3cyclictest0-21swapper/010:20:000
253919962,1cyclictest254-21systemd-journal08:42:580
253919960,6cyclictest0-21swapper/008:15:140
253919960,5cyclictest0-21swapper/008:33:270
253919960,2cyclictest0-21swapper/008:10:150
253919960,1cyclictest0-21swapper/009:05:210
253919960,0cyclictest8943-1kworker/u9:009:40:380
253919960,0cyclictest0-21swapper/007:10:160
254099955,0cyclictest18040-21diskmemload12:25:353
254099953,1cyclictest32020-21expr08:30:163
254099952,1cyclictest570-21systemd-logind08:00:193
254099952,1cyclictest35-21ksoftirqd/308:25:433
254099951,2cyclictest254-21systemd-journal07:40:413
254099950,5cyclictest0-21swapper/307:20:183
254099950,5cyclictest0-21swapper/307:20:183
254019954,1cyclictest28-21ksoftirqd/207:30:172
254019950,1cyclictest254-21systemd-journal08:50:432
253919954,1cyclictest9-21ksoftirqd/009:05:010
253919951,1cyclictest0-21swapper/008:00:120
253919950,5cyclictest0-21swapper/007:25:120
254099944,0cyclictest0-21swapper/307:35:063
254099942,1cyclictest23093-21kernelversion08:10:203
254099941,2cyclictest16638-21/usr/sbin/munin07:55:203
254099941,2cyclictest10079-21tr08:50:223
254099941,1cyclictest6485-21apt-get08:45:123
254099940,4cyclictest0-21swapper/307:45:033
254099940,4cyclictest0-21swapper/307:11:163
254099940,2cyclictest254-21systemd-journal08:36:273
253919940,3cyclictest31017-21/usr/sbin/munin07:20:200
253919940,3cyclictest31017-21/usr/sbin/munin07:20:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional