You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-29 - 10:43
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot2.osadl.org (updated Wed Oct 29, 2025 00:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
263492199182,12sleep00-21swapper/019:08:590
261262191175,11sleep30-21swapper/319:06:063
262732189174,10sleep10-21swapper/119:08:001
264162181164,12sleep20-21swapper/219:09:532
290412740,0sleep00-21swapper/022:17:180
278422450,0sleep00-21swapper/023:38:170
113202430,0sleep30-21swapper/300:35:143
295572400,1sleep02660899cyclictest21:29:390
7133280,0sleep10-21swapper/100:32:591
7133280,0sleep10-21swapper/100:32:591
16777280,0sleep10-21swapper/123:49:161
266089971,3cyclictest25408-21awk00:10:010
9045260,0sleep00-21swapper/021:35:150
3807260,0sleep00-21swapper/022:05:140
266179960,6cyclictest6969-21perl21:50:162
266209955,0cyclictest55950irq/130-eth022:44:343
266209950,5cyclictest0-21swapper/323:48:543
266209950,4cyclictest6346-21latency_hist22:55:013
266179950,5cyclictest6452-21ssh23:27:192
266179950,5cyclictest460-21ssh23:24:492
266179950,5cyclictest0-21swapper/223:56:252
266179950,5cyclictest0-21swapper/223:16:502
266179950,5cyclictest0-21swapper/222:12:142
266179950,5cyclictest0-21swapper/220:00:162
266179950,5cyclictest0-21swapper/219:31:532
266179950,5cyclictest0-21swapper/219:19:572
266179950,4cyclictest22165-21diskstats22:30:162
266179950,4cyclictest20435-21ssh21:25:042
266129951,2cyclictest0-21swapper/121:35:121
266129950,5cyclictest0-21swapper/123:22:241
266129950,5cyclictest0-21swapper/122:13:511
266129950,5cyclictest0-21swapper/100:25:151
266089950,5cyclictest0-21swapper/022:25:030
266089950,5cyclictest0-21swapper/021:21:200
266089950,5cyclictest0-21swapper/000:32:500
266089950,5cyclictest0-21swapper/000:32:490
266089950,5cyclictest0-21swapper/000:10:020
266089950,4cyclictest0-21swapper/021:50:120
266089950,4cyclictest0-21swapper/021:45:020
20194250,0sleep00-21swapper/022:45:170
266209944,0cyclictest29064-21kworker/3:222:20:073
266209944,0cyclictest29064-21kworker/3:221:13:003
266209944,0cyclictest29064-21kworker/3:221:01:473
266209944,0cyclictest29064-21kworker/3:220:39:213
266209944,0cyclictest29064-21kworker/3:200:12:073
266209942,1cyclictest4324-21df_inode22:05:153
266209941,3cyclictest0-21swapper/320:05:233
266209941,3cyclictest0-21swapper/300:00:033
266209940,4cyclictest28112-21ssh21:28:443
266209940,4cyclictest0-21swapper/323:00:123
266209940,4cyclictest0-21swapper/322:33:373
266209940,4cyclictest0-21swapper/322:25:033
266209940,4cyclictest0-21swapper/321:30:133
266209940,4cyclictest0-21swapper/321:20:173
266209940,4cyclictest0-21swapper/321:15:153
266209940,4cyclictest0-21swapper/320:15:183
266209940,4cyclictest0-21swapper/300:30:323
266209940,4cyclictest0-21swapper/300:30:313
266209940,3cyclictest6480-21sh23:27:203
266209940,3cyclictest3791-21sh23:58:533
266209940,3cyclictest32315-21memory23:40:173
266209940,3cyclictest31804-21sh22:02:363
266209940,3cyclictest31735-21/usr/sbin/munin22:35:173
266209940,3cyclictest31686-21sh00:29:193
266209940,3cyclictest31199-21ntpq19:15:193
266209940,3cyclictest27061-21df_inode23:05:133
266209940,3cyclictest25318-21ntp_states23:20:223
266209940,3cyclictest24574-21ssh23:52:553
266209940,3cyclictest21478-21ssh21:40:583
266209940,3cyclictest20847-21ssh23:35:003
266209940,3cyclictest20020-21cut21:05:143
266209940,3cyclictest16678-21sh23:32:303
266209940,3cyclictest16655-21diskstats00:05:163
266209940,3cyclictest15576-21users00:20:223
266209940,3cyclictest14512-21sh22:10:163
266209940,3cyclictest13197-21hddtemp_smartct20:50:183
266209940,3cyclictest10735-21ssh00:18:363
266209940,3cyclictest10417-21df_inode20:45:163
266209940,3cyclictest10013-21ssh23:12:523
266209940,2cyclictest0-21swapper/323:17:123
266209940,2cyclictest0-21swapper/321:35:213
266209940,1cyclictest32450-21/usr/sbin/munin19:20:233
266209940,1cyclictest2495-21/usr/sbin/munin19:25:143
266209940,1cyclictest20030-21/usr/sbin/munin20:00:203
266209940,1cyclictest16096-21basename23:00:013
266209940,0cyclictest0-21swapper/321:50:193
266179944,0cyclictest31-21ksoftirqd/219:55:002
266179944,0cyclictest0-21swapper/223:06:222
266179944,0cyclictest0-21swapper/220:39:572
266179944,0cyclictest0-21swapper/200:15:022
266179943,1cyclictest6088-21unixbench_singl23:10:232
266179943,0cyclictest0-21swapper/219:25:122
266179942,2cyclictest24957-21ssh23:53:102
266179941,1cyclictest0-21swapper/222:00:132
266179940,4cyclictest31-21ksoftirqd/200:10:112
266179940,4cyclictest11431-21ssh22:25:052
266179940,4cyclictest11270-21apt-get23:30:142
266179940,4cyclictest10131-21apt-get19:40:122
266179940,4cyclictest0-21swapper/223:35:112
266179940,4cyclictest0-21swapper/223:00:132
266179940,4cyclictest0-21swapper/222:50:132
266179940,4cyclictest0-21swapper/222:45:142
266179940,4cyclictest0-21swapper/222:40:142
266179940,4cyclictest0-21swapper/222:35:132
266179940,4cyclictest0-21swapper/221:45:142
266179940,4cyclictest0-21swapper/221:44:072
266179940,4cyclictest0-21swapper/221:35:322
266179940,4cyclictest0-21swapper/221:30:122
266179940,4cyclictest0-21swapper/221:20:132
266179940,4cyclictest0-21swapper/221:17:052
266179940,4cyclictest0-21swapper/221:05:012
266179940,4cyclictest0-21swapper/219:10:112
266179940,4cyclictest0-21swapper/200:22:022
266179940,4cyclictest0-21swapper/200:01:192
266179940,3cyclictest8081-21latency_hist23:45:012
266179940,3cyclictest7178-21ssh00:33:012
266179940,3cyclictest7178-21ssh00:33:012
266179940,3cyclictest27340-21cat20:15:002
266179940,3cyclictest23506-21ssh21:58:412
266179940,3cyclictest22230-21diskmemload00:25:142
266179940,3cyclictest1485-21/usr/sbin/munin22:20:172
266179940,3cyclictest12768-21ssh22:09:472
266179940,1cyclictest8476-21hddtemp_smartct19:35:152
266179940,0cyclictest0-21swapper/222:55:082
266179940,0cyclictest0-21swapper/220:51:022
266179940,0cyclictest0-21swapper/219:45:592
266129944,0cyclictest0-21swapper/123:43:211
266129944,0cyclictest0-21swapper/123:38:061
266129944,0cyclictest0-21swapper/122:44:161
266129944,0cyclictest0-21swapper/122:30:091
266129944,0cyclictest0-21swapper/119:35:151
266129941,2cyclictest0-21swapper/123:55:131
266129941,2cyclictest0-21swapper/122:50:141
266129941,2cyclictest0-21swapper/122:35:121
266129941,2cyclictest0-21swapper/122:00:141
266129941,2cyclictest0-21swapper/121:40:131
266129941,2cyclictest0-21swapper/121:00:131
266129941,2cyclictest0-21swapper/120:25:131
266129940,4cyclictest0-21swapper/123:32:521
266129940,0cyclictest0-21swapper/123:27:371
266129940,0cyclictest0-21swapper/123:12:391
266129940,0cyclictest0-21swapper/122:58:231
266129940,0cyclictest0-21swapper/120:41:521
266129940,0cyclictest0-21swapper/100:35:441
266089943,1cyclictest23030-21diskstats20:05:170
266089942,1cyclictest714-21snmpd22:22:220
266089941,3cyclictest714-21snmpd22:51:130
266089941,2cyclictest0-21swapper/020:55:150
266089941,2cyclictest0-21swapper/020:35:130
266089940,4cyclictest3486-21ssh00:15:090
266089940,4cyclictest22230-21diskmemload23:57:030
266089940,4cyclictest22230-21diskmemload23:45:140
266089940,4cyclictest14598-21latency_hist20:55:000
266089940,4cyclictest0-21swapper/023:50:140
266089940,4cyclictest0-21swapper/023:43:250
266089940,4cyclictest0-21swapper/022:35:060
266089940,4cyclictest0-21swapper/022:10:010
266089940,4cyclictest0-21swapper/022:00:140
266089940,4cyclictest0-21swapper/021:33:150
266089940,4cyclictest0-21swapper/019:35:120
266089940,4cyclictest0-21swapper/019:30:120
266089940,4cyclictest0-21swapper/000:35:120
266089940,4cyclictest0-21swapper/000:25:130
266089940,4cyclictest0-21swapper/000:20:120
266089940,3cyclictest32406-21/usr/sbin/munin20:25:200
266089940,3cyclictest27060-21ssh23:05:120
266089940,3cyclictest24965-21/usr/sbin/munin20:10:150
266089940,3cyclictest24906-21ssh21:10:260
266089940,3cyclictest23674-21users22:30:240
266089940,3cyclictest22993-21ssh21:41:560
266089940,3cyclictest21298-21ntp_states20:00:210
266089940,3cyclictest17169-21diskstats23:00:140
266089940,3cyclictest16749-21df21:55:150
266089940,3cyclictest14470-21df_inode23:15:150
266089940,3cyclictest12711-21ssh00:03:130
266089940,3cyclictest11675-21ssh23:13:560
266209932,1cyclictest714-21snmpd19:30:083
266209931,1cyclictest0-21swapper/320:40:123
266209930,3cyclictest7495-21apt-config19:35:013
266209930,3cyclictest28674-21apt-get21:45:143
266209930,3cyclictest19265-21apt-get22:45:133
266209930,3cyclictest15237-21sort19:50:143
266209930,3cyclictest0-21swapper/322:15:033
266209930,3cyclictest0-21swapper/320:55:003
266209930,3cyclictest0-21swapper/320:30:123
266209930,3cyclictest0-21swapper/320:25:123
266209930,3cyclictest0-21swapper/320:10:133
266209930,3cyclictest0-21swapper/319:55:143
266209930,3cyclictest0-21swapper/319:45:143
266209930,3cyclictest0-21swapper/319:10:013
266209930,1cyclictest29941-21/usr/sbin/munin20:20:153
266209930,1cyclictest17286-21irqstats21:55:173
266209930,0cyclictest10128-21/usr/sbin/munin19:40:143
266179931,1cyclictest0-21swapper/223:45:012
266179930,3cyclictest32607-21expr20:25:142
266179930,3cyclictest30249-21grep20:20:172
266179930,3cyclictest23697-21ssh22:15:062
266179930,3cyclictest22582-21ssh21:10:072
266179930,3cyclictest18240-21cut19:55:162
266179930,3cyclictest0-21swapper/221:00:132
266179930,3cyclictest0-21swapper/220:55:012
266179930,3cyclictest0-21swapper/220:45:142
266179930,3cyclictest0-21swapper/220:40:122
266179930,3cyclictest0-21swapper/220:30:132
266179930,3cyclictest0-21swapper/220:15:062
266179930,3cyclictest0-21swapper/220:05:142
266179930,3cyclictest0-21swapper/219:20:012
266179930,3cyclictest0-21swapper/200:35:002
266179930,3cyclictest0-21swapper/200:05:032
266129933,0cyclictest0-21swapper/123:08:081
266129933,0cyclictest0-21swapper/120:35:121
266129933,0cyclictest0-21swapper/100:05:011
266129931,2cyclictest0-21swapper/119:40:131
266129931,1cyclictest0-21swapper/123:50:131
266129931,1cyclictest0-21swapper/122:45:141
266129931,1cyclictest0-21swapper/122:25:031
266129931,1cyclictest0-21swapper/121:55:131
266129931,1cyclictest0-21swapper/121:45:141
266129931,1cyclictest0-21swapper/121:30:131
266129931,1cyclictest0-21swapper/121:25:041
266129931,1cyclictest0-21swapper/121:20:131
266129931,1cyclictest0-21swapper/121:15:141
266129931,1cyclictest0-21swapper/121:10:111
266129931,1cyclictest0-21swapper/121:05:131
266129931,1cyclictest0-21swapper/121:00:011
266129931,1cyclictest0-21swapper/120:45:131
266129931,1cyclictest0-21swapper/120:20:141
266129931,1cyclictest0-21swapper/120:00:151
266129931,1cyclictest0-21swapper/119:50:121
266129931,1cyclictest0-21swapper/119:30:121
266129931,1cyclictest0-21swapper/119:25:121
266129931,1cyclictest0-21swapper/119:15:131
266129931,1cyclictest0-21swapper/100:15:121
266129931,1cyclictest0-21swapper/100:10:021
266129931,1cyclictest0-21swapper/100:00:131
266129930,3cyclictest0-21swapper/120:54:331
266129930,2cyclictest0-21swapper/122:05:131
266129930,2cyclictest0-21swapper/119:20:151
266089932,1cyclictest714-21snmpd23:20:040
266089931,1cyclictest711-21runrttasks22:40:000
266089931,1cyclictest0-21swapper/023:30:140
266089930,3cyclictest30787-21hddtemp_smartct19:15:160
266089930,3cyclictest2980-21df20:30:140
266089930,3cyclictest22230-21diskmemload21:15:140
266089930,3cyclictest17733-21cut21:00:140
266089930,3cyclictest1225-21ssh23:25:090
266089930,3cyclictest0-21swapper/021:05:130
266089930,3cyclictest0-21swapper/020:45:060
266089930,3cyclictest0-21swapper/020:40:070
266089930,3cyclictest0-21swapper/020:20:150
266089930,3cyclictest0-21swapper/020:15:140
266089930,3cyclictest0-21swapper/019:55:130
266089930,3cyclictest0-21swapper/019:50:130
266089930,3cyclictest0-21swapper/019:45:140
266089930,3cyclictest0-21swapper/019:40:130
266089930,3cyclictest0-21swapper/019:25:100
266089930,3cyclictest0-21swapper/019:20:160
266089930,3cyclictest0-21swapper/019:10:130
266089930,2cyclictest7217-21expr22:55:140
266129921,1cyclictest0-21swapper/123:00:111
266129921,1cyclictest0-21swapper/121:50:131
266129921,1cyclictest0-21swapper/100:20:111
266129920,1cyclictest0-21swapper/123:15:131
266129920,1cyclictest0-21swapper/122:20:141
266129920,1cyclictest0-21swapper/122:15:141
266129920,1cyclictest0-21swapper/120:30:131
266129920,1cyclictest0-21swapper/120:15:131
266129920,1cyclictest0-21swapper/120:10:131
266129920,1cyclictest0-21swapper/120:05:151
266129920,1cyclictest0-21swapper/119:55:141
266129920,1cyclictest0-21swapper/119:45:131
266129920,1cyclictest0-21swapper/119:10:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional