You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 04:54
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rack1slot1.osadl.org (updated Mon May 05, 2025 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
37042207157,25sleep00-21swapper/019:09:090
34552203162,27sleep20-21swapper/219:05:532
36452199155,22sleep30-21swapper/319:08:243
37002198161,24sleep10-21swapper/119:09:071
106532710,0sleep20-21swapper/220:30:002
305042680,0sleep00-21swapper/020:00:230
174452610,1sleep335-21ksoftirqd/323:40:033
3858991818,0cyclictest28-21ksoftirqd/223:34:352
3858991818,0cyclictest28-21ksoftirqd/222:36:092
3858991818,0cyclictest28-21ksoftirqd/222:02:242
385899175,12cyclictest28-21ksoftirqd/200:16:422
3858991615,1cyclictest28-21ksoftirqd/222:09:042
3858991515,0cyclictest28-21ksoftirqd/222:28:242
3858991414,0cyclictest28-21ksoftirqd/222:32:292
3858991414,0cyclictest28-21ksoftirqd/200:11:252
3858991413,0cyclictest28-21ksoftirqd/219:45:202
385899141,0cyclictest28-21ksoftirqd/219:59:102
170712140,0sleep30-21swapper/322:15:193
385899139,3cyclictest0-21swapper/222:58:132
3858991313,0cyclictest28-21ksoftirqd/223:59:352
3858991313,0cyclictest28-21ksoftirqd/223:48:502
3858991313,0cyclictest28-21ksoftirqd/222:18:322
3858991313,0cyclictest28-21ksoftirqd/221:48:332
3858991313,0cyclictest28-21ksoftirqd/221:33:192
3858991313,0cyclictest28-21ksoftirqd/221:26:432
3845991313,0cyclictest9-21ksoftirqd/023:06:220
3845991312,1cyclictest9-21ksoftirqd/023:40:000
3845991312,1cyclictest18607-21ssh22:17:160
385899129,2cyclictest28828-21latency_hist23:55:002
385899126,5cyclictest28-21ksoftirqd/223:02:082
3858991212,0cyclictest28-21ksoftirqd/223:23:442
3858991212,0cyclictest28-21ksoftirqd/221:23:142
3858991212,0cyclictest28-21ksoftirqd/200:31:242
3858991211,1cyclictest29483-21df21:50:172
385399120,12cyclictest0-21swapper/121:40:151
3845991211,1cyclictest9-21ksoftirqd/021:25:010
388099110,7cyclictest0-21swapper/323:15:123
388099110,7cyclictest0-21swapper/323:15:123
385899119,1cyclictest4285-21cut19:10:142
385899119,1cyclictest0-21swapper/200:20:192
3858991110,1cyclictest28-21ksoftirqd/221:58:162
3853991111,0cyclictest21-21ksoftirqd/122:16:161
384599119,1cyclictest9-21ksoftirqd/023:45:150
384599119,1cyclictest9-21ksoftirqd/019:40:000
3845991110,1cyclictest9-21ksoftirqd/023:55:000
3845991110,1cyclictest9-21ksoftirqd/022:55:190
3845991110,1cyclictest9-21ksoftirqd/022:55:000
3845991110,1cyclictest9-21ksoftirqd/022:40:000
3845991110,1cyclictest9-21ksoftirqd/021:10:140
3845991110,1cyclictest9-21ksoftirqd/019:15:010
3845991110,1cyclictest9-21ksoftirqd/000:40:000
3845991110,1cyclictest9-21ksoftirqd/000:15:110
44572100,0sleep10-21swapper/122:00:081
385899109,1cyclictest14065-21expr23:35:142
385899109,1cyclictest0-21swapper/220:00:152
385899108,1cyclictest7283-21ps20:20:222
385899108,1cyclictest0-21swapper/221:40:182
385899108,1cyclictest0-21swapper/221:15:202
385899102,8cyclictest28-21ksoftirqd/222:40:002
385899102,8cyclictest28-21ksoftirqd/220:18:462
385899102,7cyclictest28-21ksoftirqd/219:29:162
385899102,7cyclictest271rcuc/220:35:202
3858991010,0cyclictest28-21ksoftirqd/223:19:462
3858991010,0cyclictest28-21ksoftirqd/223:19:462
3858991010,0cyclictest28-21ksoftirqd/222:52:532
385399108,1cyclictest111rcu_preempt20:00:201
385399102,7cyclictest21-21ksoftirqd/119:50:121
385399100,3cyclictest111rcu_preempt21:30:241
384599109,1cyclictest9-21ksoftirqd/023:20:000
384599109,1cyclictest9-21ksoftirqd/023:20:000
384599109,1cyclictest9-21ksoftirqd/022:25:000
384599109,1cyclictest9-21ksoftirqd/022:10:020
384599109,1cyclictest9-21ksoftirqd/021:40:000
384599109,1cyclictest9-21ksoftirqd/021:30:000
384599109,1cyclictest9-21ksoftirqd/019:25:000
384599108,1cyclictest6571-21sh22:44:540
3845991010,0cyclictest9-21ksoftirqd/023:55:010
3845991010,0cyclictest9-21ksoftirqd/023:30:160
3845991010,0cyclictest9-21ksoftirqd/021:55:150
38809991,7cyclictest0-21swapper/323:30:503
38809991,5cyclictest5410-21kworker/u8:021:26:403
38809990,7cyclictest6414-21munin-node23:00:373
38809990,6cyclictest28826-21apt-get23:55:013
38589999,0cyclictest28-21ksoftirqd/223:42:122
38589999,0cyclictest28-21ksoftirqd/222:20:182
38589999,0cyclictest28-21ksoftirqd/221:36:572
38589999,0cyclictest28-21ksoftirqd/220:53:112
38589999,0cyclictest28-21ksoftirqd/200:27:122
38589999,0cyclictest28-21ksoftirqd/200:06:462
38589998,1cyclictest777-21apt-get00:00:132
38589997,1cyclictest25161-21grep21:00:192
38589997,1cyclictest16741-21apt-get19:35:132
38589994,3cyclictest60950irq/123-eno119:50:212
38589993,5cyclictest0-21swapper/220:05:202
38589992,7cyclictest28-21ksoftirqd/222:45:162
38589992,7cyclictest28-21ksoftirqd/222:10:182
38589991,8cyclictest28-21ksoftirqd/223:25:362
38589991,8cyclictest28-21ksoftirqd/221:10:482
38589991,8cyclictest28-21ksoftirqd/221:05:442
38589991,8cyclictest28-21ksoftirqd/219:40:182
38589991,8cyclictest28-21ksoftirqd/219:23:322
38589991,8cyclictest28-21ksoftirqd/219:15:302
38589991,7cyclictest28-21ksoftirqd/200:35:062
38589991,0cyclictest28-21ksoftirqd/220:12:562
38589990,8cyclictest28-21ksoftirqd/223:06:102
38589990,8cyclictest0-21swapper/220:40:182
38589990,4cyclictest26479-21apt-get23:10:142
38589990,1cyclictest28-21ksoftirqd/220:30:182
38539999,0cyclictest21-21ksoftirqd/123:00:191
38539999,0cyclictest21-21ksoftirqd/122:54:571
38539999,0cyclictest21-21ksoftirqd/100:23:111
38539998,1cyclictest21-21ksoftirqd/123:26:501
38539998,1cyclictest21-21ksoftirqd/122:55:231
38539998,1cyclictest21-21ksoftirqd/121:54:551
38539998,1cyclictest21-21ksoftirqd/121:25:151
38539998,1cyclictest21-21ksoftirqd/121:22:521
38539998,1cyclictest21-21ksoftirqd/121:15:111
38539998,1cyclictest21-21ksoftirqd/121:07:241
38539998,1cyclictest21-21ksoftirqd/120:55:151
38539998,1cyclictest21-21ksoftirqd/120:34:121
38539997,1cyclictest0-21swapper/122:35:141
38539994,1cyclictest21-21ksoftirqd/123:59:371
38539993,5cyclictest111rcu_preempt19:45:201
38539991,8cyclictest21-21ksoftirqd/122:08:461
38539991,1cyclictest21-21ksoftirqd/121:45:191
38539990,8cyclictest21-21ksoftirqd/122:45:181
38539990,8cyclictest111rcu_preempt22:25:141
38539990,8cyclictest111rcu_preempt22:13:471
38539990,8cyclictest111rcu_preempt21:11:201
38539990,8cyclictest111rcu_preempt19:20:221
38539990,8cyclictest0-21swapper/119:10:201
38539990,1cyclictest21-21ksoftirqd/100:05:221
38539990,1cyclictest111rcu_preempt23:15:181
38539990,1cyclictest111rcu_preempt23:15:181
38539990,1cyclictest111rcu_preempt22:30:181
38539990,1cyclictest111rcu_preempt22:20:141
38539990,1cyclictest111rcu_preempt20:05:261
38459999,0cyclictest9-21ksoftirqd/019:40:010
38459998,1cyclictest9-21ksoftirqd/023:00:130
38459998,1cyclictest9-21ksoftirqd/021:50:190
38459998,1cyclictest9-21ksoftirqd/021:50:000
38459998,1cyclictest9-21ksoftirqd/019:30:000
38459998,1cyclictest9-21ksoftirqd/000:30:000
38459998,1cyclictest9-21ksoftirqd/000:10:150
38459998,1cyclictest9-21ksoftirqd/000:05:200
38459997,1cyclictest9-21ksoftirqd/020:35:010
38459997,1cyclictest9-21ksoftirqd/020:25:170
38459997,1cyclictest9-21ksoftirqd/019:50:140
38459997,1cyclictest29333-21diskmemload22:11:280
38459992,6cyclictest5813-21perf20:20:010
38809986,1cyclictest35-21ksoftirqd/320:08:343
38809983,4cyclictest29333-21diskmemload00:37:563
38809982,3cyclictest31601-21apt-get22:35:153
38809982,3cyclictest29333-21diskmemload21:40:463
38809981,4cyclictest3841-21cyclictest23:37:223
38809981,4cyclictest29333-21diskmemload22:04:503
38589988,0cyclictest28-21ksoftirqd/220:55:212
38589988,0cyclictest28-21ksoftirqd/220:45:332
38589981,7cyclictest28-21ksoftirqd/219:30:012
38589981,7cyclictest28-21ksoftirqd/219:30:002
38539988,0cyclictest21-21ksoftirqd/123:50:191
38539988,0cyclictest21-21ksoftirqd/123:31:171
38539988,0cyclictest21-21ksoftirqd/123:20:471
38539988,0cyclictest21-21ksoftirqd/123:05:051
38539988,0cyclictest21-21ksoftirqd/121:36:511
38539988,0cyclictest21-21ksoftirqd/120:41:491
38539988,0cyclictest21-21ksoftirqd/120:25:151
38539988,0cyclictest21-21ksoftirqd/120:23:491
38539988,0cyclictest21-21ksoftirqd/120:17:091
38539988,0cyclictest21-21ksoftirqd/119:27:151
38539988,0cyclictest21-21ksoftirqd/119:16:451
38539988,0cyclictest111rcu_preempt23:45:041
38539988,0cyclictest111rcu_preempt00:35:111
38539988,0cyclictest0-21swapper/123:10:281
38539988,0cyclictest0-21swapper/100:25:241
38539987,1cyclictest21-21ksoftirqd/122:40:121
38539987,1cyclictest21-21ksoftirqd/121:56:241
38539987,1cyclictest21-21ksoftirqd/119:35:191
38539987,1cyclictest21-21ksoftirqd/100:30:461
38539987,1cyclictest21-21ksoftirqd/100:15:211
38539987,1cyclictest21-21ksoftirqd/100:10:441
38539980,8cyclictest21-21ksoftirqd/119:40:141
38539980,8cyclictest111rcu_preempt23:35:121
38539980,7cyclictest0-21swapper/123:40:281
38539980,1cyclictest21-21ksoftirqd/100:00:181
38459988,0cyclictest9-21ksoftirqd/000:00:140
38459987,1cyclictest9-21ksoftirqd/023:25:000
38459987,1cyclictest9-21ksoftirqd/022:45:250
38459987,1cyclictest9-21ksoftirqd/022:35:000
38459987,1cyclictest9-21ksoftirqd/022:00:160
38459987,1cyclictest9-21ksoftirqd/021:30:170
38459987,1cyclictest9-21ksoftirqd/020:15:000
38459986,1cyclictest9-21ksoftirqd/023:25:130
38459986,1cyclictest9-21ksoftirqd/019:45:180
38459986,1cyclictest9-21ksoftirqd/000:20:010
38459983,4cyclictest21266-21apt-get21:40:150
38809975,1cyclictest2750-21ssh21:56:203
38809975,1cyclictest20202-21cat19:40:163
38809971,4cyclictest0-21swapper/321:20:173
38809971,2cyclictest29333-21diskmemload22:05:143
38809971,2cyclictest29333-21diskmemload00:00:223
38809971,2cyclictest0-21swapper/323:25:143
38809971,1cyclictest0-21swapper/300:20:593
38809970,4cyclictest0-21swapper/300:07:593
38809970,3cyclictest0-21swapper/321:00:233
38809970,2cyclictest0-21swapper/323:11:253
38539977,0cyclictest21-21ksoftirqd/120:50:131
38539977,0cyclictest21-21ksoftirqd/119:55:111
38539976,1cyclictest21-21ksoftirqd/120:35:131
38459976,1cyclictest9-21ksoftirqd/023:10:150
38459976,1cyclictest9-21ksoftirqd/021:05:160
38459976,1cyclictest9-21ksoftirqd/020:25:000
38459976,1cyclictest9-21ksoftirqd/019:30:230
38459976,1cyclictest9-21ksoftirqd/019:30:230
38459976,1cyclictest9-21ksoftirqd/019:10:130
38459976,1cyclictest9-21ksoftirqd/000:30:180
38459975,1cyclictest9-21ksoftirqd/019:55:140
38459970,6cyclictest19418-21/usr/sbin/munin20:45:230
38459970,6cyclictest13281-21grep20:35:140
38459970,6cyclictest0-21swapper/023:40:050
38459970,6cyclictest0-21swapper/021:15:110
38459970,2cyclictest0-21swapper/020:40:140
38459970,1cyclictest27357-21ssh22:29:510
14567270,1sleep329333-21diskmemload21:30:213
38809965,1cyclictest6396-21perf19:15:003
38809965,1cyclictest27912-21ssh23:52:413
38809965,1cyclictest25613-21ssh21:45:143
38809965,1cyclictest23801-21ssh22:25:053
38809965,1cyclictest23334-21ssh23:45:473
38809965,1cyclictest19701-21sshd20:47:343
38809965,1cyclictest13593-21ntpq00:15:183
38809965,1cyclictest10377-21ssh00:10:313
38809964,1cyclictest35-21ksoftirqd/320:05:003
38809964,1cyclictest29499-21diskstats21:50:183
38809964,1cyclictest22730-21cpuspeed_turbos23:05:143
38809964,1cyclictest16592-21ssh22:55:453
38809963,1cyclictest29333-21diskmemload21:37:163
38809963,1cyclictest254-21systemd-journal21:10:103
38809962,2cyclictest20730-21gltestperf22:20:153
38809962,1cyclictest16515-21awk19:35:003
38809962,1cyclictest16515-21awk19:35:003
38809962,0cyclictest29333-21diskmemload00:31:443
38809961,1cyclictest12295-21tr19:25:153
38809961,1cyclictest0-21swapper/322:30:173
38809960,4cyclictest21878-21/usr/sbin/munin00:25:203
38809960,2cyclictest1999-21/usr/sbin/munin21:15:183
38809960,2cyclictest1968-21ssh23:20:153
38809960,2cyclictest0-21swapper/322:45:213
38809960,2cyclictest0-21swapper/322:43:153
38809960,1cyclictest9426-21apt19:20:003
38809960,1cyclictest0-21swapper/322:50:223
38809960,1cyclictest0-21swapper/319:50:133
38539966,0cyclictest111rcu_preempt21:00:201
38539966,0cyclictest111rcu_preempt20:45:161
38539966,0cyclictest111rcu_preempt20:10:161
38539965,1cyclictest21-21ksoftirqd/119:30:501
38539965,1cyclictest21-21ksoftirqd/119:30:501
38459965,1cyclictest22999-21df_inode20:55:170
38459960,6cyclictest0-21swapper/020:08:120
38459960,0cyclictest0-21swapper/020:50:130
38809953,1cyclictest16602-21ntp_states20:40:203
38809952,1cyclictest4805-21awk20:15:233
38809952,1cyclictest254-21systemd-journal20:20:013
38809952,1cyclictest11906-21awk20:30:223
38809952,0cyclictest20140-21apt-get20:50:133
38809951,2cyclictest9683-21/usr/sbin/munin19:20:193
38809951,2cyclictest9042-21fschecks_time20:25:193
38809951,2cyclictest0-21swapper/322:11:413
38809951,1cyclictest3502-21latency_hist20:15:003
38809951,1cyclictest254-21systemd-journal21:05:003
38809950,4cyclictest24031-21wc19:50:003
38809950,4cyclictest22799-21cpuspeed_turbos20:55:173
38809950,2cyclictest254-21systemd-journal20:40:003
38459950,1cyclictest24864-21/usr/sbin/munin21:00:250
38809943,1cyclictest16786-21cut19:35:133
38809941,2cyclictest26615-21apt-get19:55:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional