You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 03:47
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot2.osadl.org (updated Tue Nov 25, 2025 00:44:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
36682210194,11sleep10-21swapper/119:08:551
35242193177,12sleep30-21swapper/319:07:063
37092187170,12sleep00-21swapper/019:09:280
15292163144,14sleep20-21swapper/219:05:072
103122490,0sleep10-21swapper/122:35:211
95902430,0sleep10-21swapper/119:20:001
26422420,0sleep30-21swapper/322:31:473
145252420,0sleep10-21swapper/123:59:521
31332410,0sleep10-21swapper/100:10:131
259282410,0sleep30-21swapper/321:38:513
50012390,0sleep10-21swapper/122:16:401
391299130,6cyclictest4154-21cut00:10:190
391299120,9cyclictest32234-21apt-get20:05:120
391299110,7cyclictest15000-21ssh22:05:200
391299110,6cyclictest4554-21if_eth023:05:140
391299110,6cyclictest31862-21cat21:25:170
391299110,6cyclictest25159-21ssh21:54:570
391299110,6cyclictest15973-21ssh00:00:140
391299110,5cyclictest32210-21diskmemload23:31:530
391299110,5cyclictest2828-21latency_hist23:05:000
285482110,0sleep10-21swapper/100:22:541
391299100,8cyclictest32210-21diskmemload22:52:230
391299100,7cyclictest21769-21grep20:45:210
391299100,6cyclictest32210-21diskmemload22:19:010
391299100,6cyclictest32210-21diskmemload21:35:540
391299100,6cyclictest32210-21diskmemload00:08:320
391299100,6cyclictest31542-21awk20:00:210
391299100,6cyclictest15886-21apt22:55:130
391299100,5cyclictest32210-21diskmemload23:56:170
391299100,5cyclictest32210-21diskmemload22:11:040
391299100,5cyclictest32210-21diskmemload00:24:080
391299100,5cyclictest12397-21apt-get20:30:120
391299100,5cyclictest11445-21python20:25:190
39129990,8cyclictest9768-21apt-get19:20:120
39129990,8cyclictest326742sleep021:41:470
39129990,8cyclictest29973-21apt-get21:05:120
39129990,8cyclictest24968-21apt-get20:55:110
39129990,8cyclictest22497-21apt-get20:50:110
39129990,8cyclictest10132-21apt-get00:30:130
39129990,7cyclictest29290-21nfs4_client23:50:170
39129990,7cyclictest27798-21df_inode19:55:120
39129990,7cyclictest0-21swapper/000:25:120
39129990,6cyclictest32210-21diskmemload22:45:080
39129990,6cyclictest32210-21diskmemload22:01:050
39129990,6cyclictest13324-21memory19:25:160
39129990,5cyclictest9179-21hddtemp_smartct22:35:140
39129990,5cyclictest32210-21diskmemload22:32:030
39129990,5cyclictest32210-21diskmemload21:11:230
39129990,5cyclictest32210-21diskmemload00:18:080
39129990,5cyclictest20504-21ssh21:20:060
39129990,5cyclictest14294-21ssh21:49:210
39129990,5cyclictest11524-21fschecks_count23:25:150
39129990,5cyclictest10303-21ssh21:15:020
39129990,4cyclictest32210-21diskmemload23:40:560
39129990,4cyclictest32210-21diskmemload23:11:300
39129990,4cyclictest32210-21diskmemload22:20:440
2036290,0sleep10-21swapper/121:59:301
39189983,2cyclictest31-21ksoftirqd/221:50:192
39189981,1cyclictest7204-21latency_hist20:20:012
39129980,7cyclictest7461-21apt-get20:20:120
39129980,7cyclictest7276-21apt-get19:15:120
39129980,7cyclictest23448-21apt-get23:15:130
39129980,7cyclictest22148-21tr22:25:180
39129980,6cyclictest6853-21chrt20:18:160
39129980,6cyclictest32210-21diskmemload21:55:340
39129980,6cyclictest32210-21diskmemload21:31:170
39129980,6cyclictest30435-21ssh23:35:030
39129980,6cyclictest18346-21irqstats20:40:160
39129980,5cyclictest32210-21diskmemload23:46:150
39129980,5cyclictest32210-21diskmemload22:40:030
39129980,5cyclictest24800-21taskset19:50:120
39129980,5cyclictest191082sleep019:36:210
39129980,5cyclictest14979-21cpu20:35:130
39129980,5cyclictest0-21swapper/023:20:540
39129980,5cyclictest0-21swapper/000:35:070
39129980,4cyclictest22863-21diskstats19:45:130
39189976,0cyclictest31-21ksoftirqd/200:40:012
39189973,2cyclictest31-21ksoftirqd/222:35:012
39189973,2cyclictest31-21ksoftirqd/222:25:002
39189973,2cyclictest31-21ksoftirqd/200:20:142
39189973,0cyclictest31-21ksoftirqd/223:12:472
39189972,2cyclictest31-21ksoftirqd/222:25:202
39189972,2cyclictest31-21ksoftirqd/220:10:142
39189972,1cyclictest31-21ksoftirqd/220:55:012
39189972,1cyclictest31-21ksoftirqd/200:15:572
39189971,2cyclictest31-21ksoftirqd/223:55:132
39189971,2cyclictest31-21ksoftirqd/221:13:282
39189971,2cyclictest18033-21apt-get23:45:112
39189971,1cyclictest31-21ksoftirqd/223:21:572
39189971,1cyclictest31-21ksoftirqd/221:40:142
39189971,0cyclictest81rcu_preempt23:40:172
39189970,6cyclictest0-21swapper/221:30:192
39189970,2cyclictest31-21ksoftirqd/223:25:202
39189970,2cyclictest31-21ksoftirqd/222:05:192
39189970,1cyclictest31-21ksoftirqd/223:16:282
39189970,1cyclictest15366-21perf21:50:022
39189970,1cyclictest0-21swapper/219:15:182
39129970,7cyclictest0-21swapper/019:10:130
39129970,6cyclictest27503-21apt-get21:00:110
39129970,4cyclictest15978-21ntp_states19:30:190
39189965,0cyclictest31-21ksoftirqd/221:35:212
39189965,0cyclictest31-21ksoftirqd/221:25:212
39189963,3cyclictest31-21ksoftirqd/219:40:012
39189962,2cyclictest81rcu_preempt22:46:572
39189962,2cyclictest31-21ksoftirqd/221:00:172
39189962,2cyclictest31-21ksoftirqd/220:40:152
39189962,1cyclictest31-21ksoftirqd/220:05:112
39189962,0cyclictest81rcu_preempt23:50:142
39189962,0cyclictest81rcu_preempt23:36:562
39189962,0cyclictest81rcu_preempt23:00:232
39189962,0cyclictest31-21ksoftirqd/220:30:132
39189961,2cyclictest31-21ksoftirqd/220:25:122
39189961,2cyclictest31-21ksoftirqd/219:55:202
39189961,2cyclictest0-21swapper/222:15:132
39189961,1cyclictest31-21ksoftirqd/222:55:132
39189961,1cyclictest31-21ksoftirqd/222:10:132
39189961,1cyclictest31-21ksoftirqd/220:55:152
39189961,1cyclictest31-21ksoftirqd/220:35:112
39189961,1cyclictest31-21ksoftirqd/219:30:192
39189961,1cyclictest31-21ksoftirqd/219:20:132
39189961,1cyclictest31-21ksoftirqd/200:10:172
39189960,1cyclictest31-21ksoftirqd/223:30:182
39189960,1cyclictest31-21ksoftirqd/221:24:502
39189960,1cyclictest31-21ksoftirqd/221:05:112
39189960,1cyclictest31-21ksoftirqd/200:25:192
39189960,1cyclictest0-21swapper/222:00:122
39189960,0cyclictest31-21ksoftirqd/222:40:122
39189960,0cyclictest31-21ksoftirqd/200:00:132
39129960,6cyclictest19420-21apt-get19:40:020
39129960,4cyclictest4470-21runrttasks20:14:260
39229954,0cyclictest0-21swapper/323:30:393
39229950,5cyclictest0-21swapper/323:40:323
39229950,3cyclictest6309-21ssh00:28:043
39189954,0cyclictest31-21ksoftirqd/223:05:022
39189954,0cyclictest31-21ksoftirqd/219:10:142
39189953,0cyclictest31-21ksoftirqd/222:50:112
39189953,0cyclictest31-21ksoftirqd/219:25:132
39189951,2cyclictest31-21ksoftirqd/219:50:132
39189951,2cyclictest31-21ksoftirqd/200:30:142
39189951,1cyclictest31-21ksoftirqd/200:05:152
39189950,2cyclictest31-21ksoftirqd/219:35:152
39189950,2cyclictest0-21swapper/221:55:122
39189950,2cyclictest0-21swapper/220:45:132
39189950,1cyclictest23450-21sed19:45:152
39189950,0cyclictest0-21swapper/222:35:152
39139951,2cyclictest0-21swapper/123:35:141
39139951,2cyclictest0-21swapper/123:15:141
39139950,5cyclictest0-21swapper/120:59:531
39229944,0cyclictest26931-21kworker/3:222:00:323
39229944,0cyclictest0-21swapper/323:55:413
39229944,0cyclictest0-21swapper/322:25:023
39229944,0cyclictest0-21swapper/322:15:083
39229944,0cyclictest0-21swapper/321:23:023
39229942,1cyclictest11553-21ssh22:52:363
39229941,1cyclictest0-21swapper/300:30:133
39229940,4cyclictest28406-21ssh22:45:133
39229940,4cyclictest27945-21/usr/sbin/munin23:50:143
39229940,4cyclictest23448-21apt-get23:15:143
39229940,4cyclictest15037-21ssh00:15:373
39229940,4cyclictest14038-21ssh22:21:053
39229940,4cyclictest1113-21diskstats21:10:163
39229940,4cyclictest10583-21/usr/sbin/munin21:15:163
39229940,4cyclictest0-21swapper/323:10:313
39229940,4cyclictest0-21swapper/322:35:123
39229940,4cyclictest0-21swapper/322:05:463
39229940,4cyclictest0-21swapper/321:25:123
39229940,3cyclictest6367-21sh23:05:483
39229940,3cyclictest5627-21ssh21:45:023
39229940,3cyclictest31326-21sh21:40:553
39229940,3cyclictest2873-21grep23:20:213
39229940,3cyclictest28216-21ssh00:05:583
39229940,3cyclictest24503-21cat00:04:553
39229940,3cyclictest2311-21sh23:04:413
39229940,3cyclictest2252-21cut20:10:013
39229940,3cyclictest17996-21sh22:40:023
39229940,3cyclictest16636-21sed20:35:203
39229940,3cyclictest13229-21hddtemp_smartct20:30:143
39229940,3cyclictest11449-21diskstats23:25:143
39229940,1cyclictest29970-21/usr/sbin/munin21:05:163
39229940,1cyclictest27500-21/usr/sbin/munin21:00:153
39229940,1cyclictest24965-21/usr/sbin/munin20:55:163
39229940,1cyclictest17174-21/usr/sbin/munin19:35:193
39229940,1cyclictest12234-21/usr/sbin/munin19:25:213
39229940,0cyclictest0-21swapper/320:15:213
39229940,0cyclictest0-21swapper/319:55:173
39189943,0cyclictest31-21ksoftirqd/220:20:012
39189941,0cyclictest31-21ksoftirqd/220:00:082
39189940,1cyclictest10929-21tr21:15:152
39139944,0cyclictest0-21swapper/123:42:161
39139944,0cyclictest0-21swapper/123:29:181
39139944,0cyclictest0-21swapper/121:35:051
39139944,0cyclictest0-21swapper/119:32:351
39139941,2cyclictest0-21swapper/123:45:121
39139941,2cyclictest0-21swapper/123:20:111
39139941,2cyclictest0-21swapper/123:10:121
39139941,2cyclictest0-21swapper/122:40:121
39139941,2cyclictest0-21swapper/122:00:111
39139941,2cyclictest0-21swapper/121:15:131
39139941,2cyclictest0-21swapper/120:50:121
39139941,2cyclictest0-21swapper/120:45:121
39139941,2cyclictest0-21swapper/120:40:111
39139941,2cyclictest0-21swapper/120:30:121
39139941,2cyclictest0-21swapper/120:00:131
39139941,2cyclictest0-21swapper/119:45:111
39139941,2cyclictest0-21swapper/119:10:121
39139941,2cyclictest0-21swapper/100:30:141
39139941,2cyclictest0-21swapper/100:15:131
39139940,4cyclictest0-21swapper/123:52:531
39139940,4cyclictest0-21swapper/121:47:551
39139940,4cyclictest0-21swapper/121:14:521
39139940,4cyclictest0-21swapper/120:23:221
39139940,4cyclictest0-21swapper/120:14:281
39139940,0cyclictest0-21swapper/122:34:191
39139940,0cyclictest0-21swapper/100:00:281
39229932,1cyclictest714-21snmpd00:10:023
39229931,1cyclictest0-21swapper/319:40:143
39229930,3cyclictest8873-21df21:30:153
39229930,3cyclictest25708-21ssh21:55:093
39229930,3cyclictest23212-21grep20:50:133
39229930,3cyclictest17924-21ssh23:45:073
39229930,3cyclictest15887-21apt-get22:55:123
39229930,3cyclictest15225-21df19:30:143
39229930,3cyclictest0-21swapper/322:10:113
39229930,3cyclictest0-21swapper/320:45:013
39229930,3cyclictest0-21swapper/320:40:123
39229930,3cyclictest0-21swapper/320:20:013
39229930,3cyclictest0-21swapper/320:10:123
39229930,3cyclictest0-21swapper/320:00:123
39229930,3cyclictest0-21swapper/319:45:053
39229930,3cyclictest0-21swapper/319:20:123
39229930,3cyclictest0-21swapper/319:15:013
39229930,3cyclictest0-21swapper/319:10:043
39229930,3cyclictest0-21swapper/300:35:003
39229930,3cyclictest0-21swapper/300:20:073
39229930,1cyclictest15582-21ssh21:50:013
39229930,1cyclictest0-21swapper/323:35:003
39229930,0cyclictest9929-21apt-get20:25:113
39229930,0cyclictest0-21swapper/319:50:143
39139933,0cyclictest0-21swapper/122:05:211
39139931,2cyclictest0-21swapper/121:00:111
39139931,1cyclictest0-21swapper/123:05:121
39139931,1cyclictest0-21swapper/122:50:121
39139931,1cyclictest0-21swapper/122:45:131
39139931,1cyclictest0-21swapper/122:25:001
39139931,1cyclictest0-21swapper/121:30:141
39139931,1cyclictest0-21swapper/121:25:131
39139931,1cyclictest0-21swapper/121:20:131
39139931,1cyclictest0-21swapper/121:05:111
39139931,1cyclictest0-21swapper/120:35:121
39139931,1cyclictest0-21swapper/120:25:121
39139931,1cyclictest0-21swapper/120:15:131
39139931,1cyclictest0-21swapper/120:05:121
39139931,1cyclictest0-21swapper/119:55:111
39139931,1cyclictest0-21swapper/119:40:021
39139931,1cyclictest0-21swapper/119:35:141
39139931,1cyclictest0-21swapper/119:25:121
39139931,1cyclictest0-21swapper/119:20:111
39139931,1cyclictest0-21swapper/100:35:111
39139931,1cyclictest0-21swapper/100:30:001
39139930,2cyclictest31039-21chrt21:40:441
39139930,2cyclictest25261-21chrt22:59:591
39139930,2cyclictest0-21swapper/122:20:111
39139930,2cyclictest0-21swapper/100:05:131
39139930,1cyclictest0-21swapper/123:30:131
39139921,1cyclictest0-21swapper/121:50:111
39139920,1cyclictest0-21swapper/123:00:131
39139920,1cyclictest0-21swapper/122:10:051
39139920,1cyclictest0-21swapper/119:50:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional