You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 19:09
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot2.osadl.org (updated Sun Nov 16, 2025 12:44:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
251962205190,10sleep10-21swapper/107:08:121
252252194161,28sleep00-21swapper/007:08:350
250642178161,12sleep30-21swapper/307:06:303
252592173156,12sleep20-21swapper/207:09:012
242852460,0sleep20-21swapper/209:59:552
241462430,0sleep20-21swapper/210:31:302
232932380,1sleep32551199cyclictest11:35:563
236272140,0sleep30-21swapper/309:10:243
10082130,0sleep00-21swapper/010:36:040
188982110,0sleep10-21swapper/109:24:591
227432100,0sleep10-21swapper/109:42:331
255019991,4cyclictest4260-21awk09:50:000
10287290,0sleep10-21swapper/111:30:091
255019971,3cyclictest29722-21perf10:35:010
255019961,4cyclictest0-21swapper/012:10:010
255019960,6cyclictest0-21swapper/010:55:130
255019960,5cyclictest0-21swapper/010:25:130
255019960,3cyclictest31986-21munin-run10:20:010
255019960,3cyclictest28433-21sendmail-msp08:20:000
255019960,3cyclictest15854-21sendmail-msp09:00:000
255119955,0cyclictest8373-21kworker/3:012:17:553
255119955,0cyclictest8373-21kworker/3:011:55:323
255119955,0cyclictest8373-21kworker/3:010:48:223
255119955,0cyclictest8373-21kworker/3:009:41:123
255119955,0cyclictest28086-21kworker/3:108:22:513
255119955,0cyclictest28086-21kworker/3:108:11:393
255119955,0cyclictest28086-21kworker/3:108:00:273
255119955,0cyclictest28086-21kworker/3:107:49:133
255119955,0cyclictest28086-21kworker/3:107:38:033
255119955,0cyclictest28086-21kworker/3:107:15:403
255119950,4cyclictest16412-21ntp_states09:55:193
255079950,5cyclictest3285-21ssh09:32:572
255079950,5cyclictest0-21swapper/212:33:402
255079950,5cyclictest0-21swapper/212:14:192
255079950,5cyclictest0-21swapper/211:39:412
255079950,5cyclictest0-21swapper/210:45:332
255079950,5cyclictest0-21swapper/209:42:512
255079950,5cyclictest0-21swapper/209:29:542
255079950,5cyclictest0-21swapper/209:20:442
255079950,0cyclictest0-21swapper/211:10:002
255029950,5cyclictest0-21swapper/112:08:231
255029950,5cyclictest0-21swapper/109:14:131
255019955,0cyclictest0-21swapper/010:51:000
255019955,0cyclictest0-21swapper/010:05:450
255019954,1cyclictest8419-21munin-run08:45:000
255019952,2cyclictest0-21swapper/008:00:150
255019950,5cyclictest0-21swapper/009:43:380
255019950,4cyclictest0-21swapper/011:20:020
255019950,0cyclictest20780-21diskmemload09:30:410
8751240,0sleep10-21swapper/112:18:181
28019240,0sleep30-21swapper/312:28:063
255119943,1cyclictest4570-21grep07:30:163
255119943,1cyclictest14844-21grep11:15:233
255119942,1cyclictest19908-21cpuspeed_turbos09:25:173
255119941,2cyclictest0-21swapper/310:00:143
255119940,4cyclictest0-21swapper/311:40:123
255119940,4cyclictest0-21swapper/310:10:133
255119940,4cyclictest0-21swapper/309:15:163
255119940,4cyclictest0-21swapper/308:05:153
255119940,4cyclictest0-21swapper/307:25:133
255119940,3cyclictest9195-21ntpq11:45:183
255119940,3cyclictest8934-21/usr/sbin/munin12:35:223
255119940,3cyclictest7506-21sh10:40:033
255119940,3cyclictest7350-21users10:55:233
255119940,3cyclictest6168-21memory12:00:173
255119940,3cyclictest5269-21sh11:27:073
255119940,3cyclictest32578-21ssh10:35:323
255119940,3cyclictest32329-21sh10:20:053
255119940,3cyclictest32079-21perl07:20:153
255119940,3cyclictest31122-21/usr/sbin/munin08:25:173
255119940,3cyclictest29798-21users10:50:233
255119940,3cyclictest27941-21perl09:45:153
255119940,3cyclictest27248-21memory08:15:163
255119940,3cyclictest25354-21rm11:05:123
255119940,3cyclictest25101-21/usr/sbin/munin12:10:143
255119940,3cyclictest23488-21/usr/sbin/munin11:20:143
255119940,3cyclictest22530-21ssh10:15:133
255119940,3cyclictest19955-21ssh10:30:013
255119940,3cyclictest1974-21ssh12:30:523
255119940,3cyclictest1579-21cstates08:30:153
255119940,3cyclictest15376-21sh12:05:153
255119940,3cyclictest13470-21sed08:55:013
255119940,3cyclictest11127-21perl10:25:163
255119940,3cyclictest10018-21df09:20:153
255119940,3cyclictest0-21swapper/308:35:183
255119940,1cyclictest3297-21perl10:05:163
255119940,1cyclictest29978-21expr09:30:143
255119940,1cyclictest16070-21/usr/sbin/munin09:00:173
255119940,1cyclictest15219-21/usr/sbin/munin11:00:183
255079944,0cyclictest0-21swapper/210:05:412
255079944,0cyclictest0-21swapper/208:15:582
255079943,1cyclictest9540-21perl07:40:182
255079943,1cyclictest804-21cut10:20:162
255079943,1cyclictest28748-21ssh10:50:172
255079942,2cyclictest714-21snmpd07:14:192
255079942,1cyclictest7123-21ssh12:00:242
255079942,1cyclictest5150-21ntpq07:30:202
255079942,1cyclictest28421-21ssh10:01:122
255079942,1cyclictest20010-21ssh11:50:332
255079942,1cyclictest12279-21ssh10:10:022
255079941,2cyclictest0-21swapper/208:25:132
255079941,1cyclictest0-21swapper/212:25:162
255079941,1cyclictest0-21swapper/208:45:122
255079940,4cyclictest2892-21apt-get11:10:132
255079940,4cyclictest0-21swapper/212:35:152
255079940,4cyclictest0-21swapper/212:05:132
255079940,4cyclictest0-21swapper/211:30:132
255079940,4cyclictest0-21swapper/211:25:152
255079940,4cyclictest0-21swapper/211:20:122
255079940,4cyclictest0-21swapper/211:15:132
255079940,4cyclictest0-21swapper/211:00:132
255079940,4cyclictest0-21swapper/210:35:122
255079940,4cyclictest0-21swapper/210:25:192
255079940,4cyclictest0-21swapper/210:15:152
255079940,4cyclictest0-21swapper/209:35:132
255079940,4cyclictest0-21swapper/207:56:112
255079940,4cyclictest0-21swapper/207:23:352
255079940,3cyclictest8532-21perl11:45:142
255079940,3cyclictest6730-21cat07:35:162
255079940,3cyclictest5127-21sshd10:55:082
255079940,3cyclictest4939-21ssh09:50:112
255079940,3cyclictest29239-21python11:55:202
255079940,3cyclictest29181-21df_inode08:20:152
255079940,3cyclictest28608-21ssh09:45:192
255079940,3cyclictest2814-21ssh11:42:122
255079940,3cyclictest2698-21df12:15:152
255079940,3cyclictest22095-21ssh09:10:152
255079940,3cyclictest21880-21perl08:05:172
255079940,3cyclictest16812-21perl09:00:142
255079940,3cyclictest16014-21latency_hist07:55:012
255079940,3cyclictest13734-21cut07:50:012
255079940,3cyclictest12740-21ssh12:20:142
255079940,3cyclictest10247-21ssh10:40:362
255079940,3cyclictest1013-21ssh09:15:212
255029944,0cyclictest0-21swapper/111:40:271
255029944,0cyclictest0-21swapper/111:15:171
255029944,0cyclictest0-21swapper/110:57:401
255029944,0cyclictest0-21swapper/110:40:481
255029944,0cyclictest0-21swapper/110:23:551
255029941,2cyclictest0-21swapper/109:55:131
255029941,2cyclictest0-21swapper/108:45:121
255029941,2cyclictest0-21swapper/108:15:121
255029941,2cyclictest0-21swapper/107:35:151
255029940,4cyclictest0-21swapper/111:51:481
255029940,4cyclictest0-21swapper/111:12:131
255029940,2cyclictest0-21swapper/109:45:131
255019944,0cyclictest3-21ksoftirqd/008:50:000
255019942,2cyclictest30987-21kworker/0:110:49:520
255019942,1cyclictest0-21swapper/012:35:150
255019941,2cyclictest0-21swapper/011:30:130
255019941,2cyclictest0-21swapper/011:05:130
255019941,2cyclictest0-21swapper/007:30:140
255019941,2cyclictest0-21swapper/007:25:130
255019941,1cyclictest0-21swapper/009:10:140
255019941,1cyclictest0-21swapper/007:20:130
255019940,4cyclictest8350-21munin-run12:35:000
255019940,4cyclictest28645-21irqstats11:55:160
255019940,4cyclictest20780-21diskmemload11:38:360
255019940,4cyclictest0-21swapper/011:45:120
255019940,4cyclictest0-21swapper/011:40:130
255019940,4cyclictest0-21swapper/010:20:130
255019940,4cyclictest0-21swapper/009:58:590
255019940,4cyclictest0-21swapper/009:25:160
255019940,4cyclictest0-21swapper/009:15:120
255019940,4cyclictest0-21swapper/009:05:160
255019940,4cyclictest0-21swapper/007:50:120
255019940,3cyclictest4363-21perl08:35:160
255019940,3cyclictest28402-21ntp_states07:15:010
255019940,3cyclictest25198-21python08:10:210
255019940,3cyclictest23540-21sh12:25:240
255019940,3cyclictest16841-21cat09:40:010
255019940,3cyclictest16603-21diskstats09:00:140
255019940,3cyclictest15809-21diskstats11:00:140
255019940,3cyclictest12048-21perl07:45:160
255019940,3cyclictest0-21swapper/007:35:150
255019940,1cyclictest11072-21latency_hist07:45:000
255019940,0cyclictest0-21swapper/009:50:290
255119932,1cyclictest6454-21expr08:40:153
255119932,1cyclictest2620-21ssh11:10:023
255119931,2cyclictest0-21swapper/312:20:033
255119931,1cyclictest714-21snmpd11:50:033
255119931,1cyclictest714-21snmpd11:30:033
255119931,1cyclictest13626-21aten2_r5power_e08:55:123
255119930,3cyclictest25882-21grep07:10:123
255119930,3cyclictest16992-21fw_packets07:55:163
255119930,3cyclictest0-21swapper/309:05:163
255119930,3cyclictest0-21swapper/308:45:123
255119930,3cyclictest0-21swapper/307:50:123
255119930,3cyclictest0-21swapper/307:40:153
255119930,2cyclictest4614-21apt-config09:50:013
255119930,1cyclictest0-21swapper/309:35:103
255079932,1cyclictest714-21snmpd08:30:042
255079932,1cyclictest14023-21cat08:55:132
255079932,1cyclictest11563-21expr08:50:132
255079930,3cyclictest4379-21tune2fs08:35:162
255079930,3cyclictest18775-21grep09:05:172
255079930,3cyclictest0-21swapper/208:40:012
255079930,3cyclictest0-21swapper/208:10:132
255079930,3cyclictest0-21swapper/208:00:072
255079930,3cyclictest0-21swapper/207:25:072
255079930,3cyclictest0-21swapper/207:15:152
255029933,0cyclictest0-21swapper/112:04:361
255029933,0cyclictest0-21swapper/110:18:321
255029933,0cyclictest0-21swapper/110:05:251
255029933,0cyclictest0-21swapper/107:58:591
255029931,1cyclictest0-21swapper/111:35:141
255029931,1cyclictest0-21swapper/110:45:131
255029931,1cyclictest0-21swapper/110:25:011
255029931,1cyclictest0-21swapper/110:10:131
255029931,1cyclictest0-21swapper/110:00:141
255029931,1cyclictest0-21swapper/109:00:131
255029931,1cyclictest0-21swapper/108:20:131
255029931,1cyclictest0-21swapper/108:10:131
255029930,2cyclictest231322sleep108:06:121
255029930,2cyclictest207212sleep109:09:341
255029930,2cyclictest15470-21chrt09:39:171
255029930,2cyclictest0-21swapper/112:25:161
255029930,2cyclictest0-21swapper/110:35:131
255029930,2cyclictest0-21swapper/107:45:131
255029930,2cyclictest0-21swapper/107:15:151
255029930,1cyclictest0-21swapper/111:55:121
255029930,1cyclictest0-21swapper/109:50:131
255019932,1cyclictest6013-21cat12:00:160
255019931,2cyclictest12634-21fschecks_time12:20:140
255019930,3cyclictest477-21ssh11:25:150
255019930,3cyclictest17630-21apt11:50:120
255019930,3cyclictest13144-21/usr/sbin/munin10:10:150
255019930,3cyclictest0-21swapper/012:15:030
255019930,3cyclictest0-21swapper/012:05:070
255019930,3cyclictest0-21swapper/011:15:050
255019930,3cyclictest0-21swapper/011:10:050
255019930,3cyclictest0-21swapper/010:40:030
255019930,3cyclictest0-21swapper/010:00:110
255019930,3cyclictest0-21swapper/009:20:020
255019930,3cyclictest0-21swapper/008:50:120
255019930,3cyclictest0-21swapper/008:30:140
255019930,3cyclictest0-21swapper/008:25:070
255019930,3cyclictest0-21swapper/008:20:070
255019930,3cyclictest0-21swapper/008:05:150
255019930,3cyclictest0-21swapper/007:55:010
255019930,3cyclictest0-21swapper/007:10:110
16259230,0sleep10-21swapper/112:38:481
1564230,0sleep10-21swapper/109:31:521
255029921,1cyclictest72242chrt08:40:181
255029921,1cyclictest192132chrt09:25:031
255029921,1cyclictest0-21swapper/112:10:011
255029921,1cyclictest0-21swapper/111:05:011
255029921,1cyclictest0-21swapper/108:25:131
255029921,1cyclictest0-21swapper/108:00:151
255029921,0cyclictest109082sleep107:43:071
255029920,1cyclictest224342sleep111:20:021
255029920,1cyclictest13587-21chrt08:55:121
255029920,1cyclictest0-21swapper/112:30:131
255029920,1cyclictest0-21swapper/112:20:121
255029920,1cyclictest0-21swapper/111:45:161
255029920,1cyclictest0-21swapper/111:25:151
255029920,1cyclictest0-21swapper/111:00:131
255029920,1cyclictest0-21swapper/110:50:131
255029920,1cyclictest0-21swapper/110:30:151
255029920,1cyclictest0-21swapper/109:15:131
255029920,1cyclictest0-21swapper/108:50:121
255029920,1cyclictest0-21swapper/108:35:131
255029920,1cyclictest0-21swapper/108:30:141
255029920,1cyclictest0-21swapper/107:30:141
255029920,1cyclictest0-21swapper/107:25:131
255029920,1cyclictest0-21swapper/107:20:131
255029920,1cyclictest0-21swapper/107:10:101
255029910,1cyclictest0-21swapper/107:50:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional