You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-09 - 08:12
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot2.osadl.org (updated Wed Jul 09, 2025 00:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
298152204188,11sleep10-21swapper/119:09:521
295762198181,12sleep30-21swapper/319:06:443
296602190172,13sleep00-21swapper/019:07:500
276072173154,14sleep20-21swapper/219:05:072
134342520,0sleep30-21swapper/300:19:483
290672460,0sleep10-21swapper/100:27:101
38632440,0sleep10-21swapper/100:14:571
315432440,0sleep10-21swapper/121:12:101
236332410,0sleep30-21swapper/300:25:003
196022400,0sleep10-21swapper/122:10:501
100022380,0sleep10-21swapper/122:39:291
2998999113,4cyclictest974-21munin-run22:35:010
228112100,0sleep30-21swapper/300:07:343
299899991,4cyclictest1278-21munin-run21:30:000
14782280,0sleep00-21swapper/022:41:120
299899960,6cyclictest0-21swapper/000:31:440
299899960,5cyclictest0-21swapper/021:40:020
299899960,3cyclictest10995-21sendmail-msp20:39:590
299899960,1cyclictest19790-21ssh00:39:200
299989955,0cyclictest2466-21kworker/3:100:32:253
299989955,0cyclictest0-21swapper/322:30:003
299989954,1cyclictest0-21swapper/323:18:013
299989951,3cyclictest3952-21latency_hist00:15:003
299989950,5cyclictest0-21swapper/323:03:563
299989950,5cyclictest0-21swapper/322:10:053
299989950,5cyclictest0-21swapper/300:26:513
299989950,4cyclictest0-21swapper/323:45:003
299989950,0cyclictest26087-21diskmemload23:53:153
299949950,5cyclictest0-21swapper/223:08:482
299949950,5cyclictest0-21swapper/222:50:532
299949950,5cyclictest0-21swapper/221:00:012
299949950,4cyclictest18891-21apt-get20:55:172
299949950,0cyclictest0-21swapper/223:38:292
299909951,2cyclictest0-21swapper/123:00:121
299909951,2cyclictest0-21swapper/121:20:131
299899952,3cyclictest4193-21apt-get22:20:130
299899952,3cyclictest14582-21ssh21:35:490
299899952,3cyclictest0-21swapper/023:20:440
299899952,3cyclictest0-21swapper/021:58:570
299899952,3cyclictest0-21swapper/000:28:120
299899951,4cyclictest11461-21/usr/sbin/munin23:45:160
299899950,5cyclictest26646-21ssh23:36:320
299899950,5cyclictest17126-21ssh22:59:290
299899950,5cyclictest0-21swapper/023:50:170
299899950,5cyclictest0-21swapper/023:31:200
299899950,5cyclictest0-21swapper/022:17:210
299899950,5cyclictest0-21swapper/000:21:200
299899950,5cyclictest0-21swapper/000:16:130
299899950,5cyclictest0-21swapper/000:10:330
299899950,5cyclictest0-21swapper/000:02:560
299899950,1cyclictest4112-21ssh22:52:280
299899950,0cyclictest5473-21ssh23:25:370
299899950,0cyclictest4167-21ssh23:58:120
299899950,0cyclictest21594-21ssh23:01:010
299899950,0cyclictest16761-21ssh21:53:410
299899950,0cyclictest0-21swapper/023:40:240
299899950,0cyclictest0-21swapper/023:19:360
299899950,0cyclictest0-21swapper/023:12:330
299899950,0cyclictest0-21swapper/019:20:200
29886250,0sleep10-21swapper/123:38:381
299989944,0cyclictest0-21swapper/321:13:113
299989943,0cyclictest8584-21ssh22:55:113
299989942,2cyclictest2466-21kworker/3:123:36:273
299989942,1cyclictest25908-21find21:25:193
299989941,3cyclictest18688-21latency_hist20:55:003
299989941,2cyclictest0-21swapper/320:40:143
299989940,4cyclictest6814-21perl19:25:153
299989940,4cyclictest6166-21ssh22:37:023
299989940,4cyclictest0-21swapper/323:25:123
299989940,4cyclictest0-21swapper/323:10:323
299989940,4cyclictest0-21swapper/322:45:143
299989940,4cyclictest0-21swapper/322:43:033
299989940,4cyclictest0-21swapper/321:55:113
299989940,4cyclictest0-21swapper/321:50:203
299989940,4cyclictest0-21swapper/321:41:013
299989940,4cyclictest0-21swapper/321:32:053
299989940,4cyclictest0-21swapper/320:46:293
299989940,4cyclictest0-21swapper/319:55:123
299989940,4cyclictest0-21swapper/300:37:273
299989940,4cyclictest0-21swapper/300:00:113
299989940,3cyclictest909-21ntp_states21:45:193
299989940,3cyclictest5198-21ssh23:42:163
299989940,3cyclictest4999-21perl22:20:153
299989940,3cyclictest4245-21/usr/sbin/munin21:15:163
299989940,3cyclictest32567-21munin-node19:15:013
299989940,3cyclictest31198-21ssh22:00:213
299989940,3cyclictest29147-21ssh22:32:193
299989940,3cyclictest281062sleep323:05:023
299989940,3cyclictest27425-21grep22:15:173
299989940,3cyclictest25649-21/usr/sbin/munin23:20:183
299989940,3cyclictest20920-21munin-run19:55:013
299989940,3cyclictest16118-21gpgv19:45:003
299989940,3cyclictest13605-21sh22:08:213
299989940,3cyclictest13230-21ntp_states21:35:203
299989940,3cyclictest12579-21sh23:30:023
299989940,3cyclictest10222-21grep19:30:193
299989940,3cyclictest0-21swapper/320:30:143
299989940,0cyclictest756-21runrttasks21:08:583
299949944,0cyclictest0-21swapper/223:45:012
299949944,0cyclictest0-21swapper/223:31:552
299949944,0cyclictest0-21swapper/200:03:482
299949943,1cyclictest19328-21cut19:50:142
299949942,2cyclictest14543-21sed20:45:162
299949942,1cyclictest9827-21ntp_states19:30:172
299949942,1cyclictest57250irq/125-eth020:05:132
299949942,1cyclictest30997-21snmpd19:12:312
299949941,1cyclictest0-21swapper/223:10:152
299949940,4cyclictest8577-21apt-get22:55:112
299949940,4cyclictest7445-21ssh22:37:502
299949940,4cyclictest15489-21ssh21:36:232
299949940,4cyclictest0-21swapper/223:56:062
299949940,4cyclictest0-21swapper/223:15:162
299949940,4cyclictest0-21swapper/222:10:132
299949940,4cyclictest0-21swapper/222:05:402
299949940,4cyclictest0-21swapper/221:44:512
299949940,4cyclictest0-21swapper/221:30:132
299949940,4cyclictest0-21swapper/221:27:542
299949940,4cyclictest0-21swapper/221:20:062
299949940,4cyclictest0-21swapper/220:40:212
299949940,4cyclictest0-21swapper/220:35:122
299949940,4cyclictest0-21swapper/220:27:282
299949940,4cyclictest0-21swapper/220:10:122
299949940,4cyclictest0-21swapper/200:35:502
299949940,4cyclictest0-21swapper/200:06:482
299949940,3cyclictest5865-21/usr/sbin/munin21:15:212
299949940,3cyclictest4843-21/usr/sbin/munin00:15:142
299949940,3cyclictest26383-21ssh21:10:032
299949940,3cyclictest25398-21sh22:30:172
299949940,3cyclictest25301-21irqstats00:25:182
299949940,3cyclictest2501-21latency_hist23:25:012
299949940,3cyclictest21143-21apt19:55:112
299949940,3cyclictest11029-21chrt19:35:102
299949940,2cyclictest0-21swapper/221:45:122
299949940,1cyclictest751-21df23:40:152
299949940,1cyclictest3691-21users00:30:222
299949940,1cyclictest28137-21ntp_states22:15:212
299949940,1cyclictest27475-21cut00:10:142
299949940,1cyclictest23630-21/usr/sbin/munin20:00:132
299949940,1cyclictest19082-21df_inode23:00:142
299949940,1cyclictest14072-21/usr/sbin/munin22:25:152
299949940,0cyclictest0-21swapper/223:51:332
299909942,1cyclictest0-21swapper/123:15:161
299909942,1cyclictest0-21swapper/122:25:131
299909941,2cyclictest0-21swapper/122:45:141
299909941,2cyclictest0-21swapper/121:15:141
299909941,2cyclictest0-21swapper/120:50:151
299909941,2cyclictest0-21swapper/120:45:131
299909941,2cyclictest0-21swapper/120:25:131
299909941,2cyclictest0-21swapper/120:20:121
299909941,2cyclictest0-21swapper/120:15:131
299909941,2cyclictest0-21swapper/119:50:121
299909941,2cyclictest0-21swapper/119:30:101
299909941,2cyclictest0-21swapper/119:15:141
299909941,2cyclictest0-21swapper/100:35:141
299909941,2cyclictest0-21swapper/100:20:121
299909941,2cyclictest0-21swapper/100:15:131
299909940,4cyclictest0-21swapper/122:07:151
299909940,4cyclictest0-21swapper/121:28:321
299909940,2cyclictest0-21swapper/123:50:111
299909940,2cyclictest0-21swapper/121:50:141
299899944,0cyclictest0-21swapper/022:12:280
299899944,0cyclictest0-21swapper/020:18:320
299899943,0cyclictest32630-21ntp_states19:15:000
299899942,2cyclictest0-21swapper/019:30:120
299899941,3cyclictest0-21swapper/022:45:410
299899941,3cyclictest0-21swapper/022:35:460
299899941,3cyclictest0-21swapper/021:45:060
299899940,4cyclictest27168-21ssh21:10:140
299899940,4cyclictest22044-21perl21:00:140
299899940,4cyclictest16188-21apt-get19:45:010
299899940,4cyclictest15764-21turbostat.cron19:45:000
299899940,4cyclictest0-21swapper/023:06:280
299899940,4cyclictest0-21swapper/022:25:120
299899940,4cyclictest0-21swapper/022:05:180
299899940,4cyclictest0-21swapper/022:00:290
299899940,4cyclictest0-21swapper/021:32:210
299899940,4cyclictest0-21swapper/021:20:090
299899940,4cyclictest0-21swapper/021:15:150
299899940,4cyclictest0-21swapper/020:57:300
299899940,4cyclictest0-21swapper/020:47:280
299899940,4cyclictest0-21swapper/020:04:240
299899940,4cyclictest0-21swapper/000:05:130
299899940,3cyclictest0-21swapper/019:55:120
299899940,3cyclictest0-21swapper/019:50:130
26517240,0sleep10-21swapper/122:15:071
299989933,0cyclictest0-21swapper/320:25:103
299989932,1cyclictest30997-21snmpd20:15:043
299989931,1cyclictest10789-21cron19:35:003
299989931,1cyclictest0-21swapper/319:10:133
299989930,3cyclictest9994-21ntp_states20:35:183
299989930,3cyclictest3173-21apt-get19:20:013
299989930,3cyclictest31020-21ssh22:50:073
299989930,3cyclictest26420-21cpuspeed_turbos20:05:133
299989930,3cyclictest26087-21diskmemload23:55:023
299989930,3cyclictest21377-21apt-get21:00:123
299989930,3cyclictest0-21swapper/321:20:093
299989930,3cyclictest0-21swapper/320:55:153
299989930,3cyclictest0-21swapper/320:20:133
299989930,3cyclictest0-21swapper/320:10:013
299989930,3cyclictest0-21swapper/320:00:123
299989930,3cyclictest0-21swapper/319:40:013
299949933,0cyclictest0-21swapper/222:20:162
299949932,1cyclictest30997-21snmpd23:25:052
299949932,1cyclictest30997-21snmpd22:45:092
299949931,1cyclictest0-21swapper/221:05:132
299949931,1cyclictest0-21swapper/220:30:152
299949930,3cyclictest19332-21sh21:55:112
299949930,3cyclictest164052sleep220:50:072
299949930,3cyclictest0-21swapper/222:40:032
299949930,3cyclictest0-21swapper/222:00:122
299949930,3cyclictest0-21swapper/221:50:142
299949930,3cyclictest0-21swapper/220:20:122
299949930,3cyclictest0-21swapper/220:15:022
299949930,3cyclictest0-21swapper/219:45:012
299949930,3cyclictest0-21swapper/219:40:122
299949930,3cyclictest0-21swapper/219:25:112
299949930,3cyclictest0-21swapper/219:20:012
299949930,3cyclictest0-21swapper/219:15:132
299949930,3cyclictest0-21swapper/200:20:022
299909933,0cyclictest0-21swapper/123:23:561
299909933,0cyclictest0-21swapper/100:01:011
299909931,2cyclictest0-21swapper/119:55:121
299909931,2cyclictest0-21swapper/119:10:141
299909931,1cyclictest0-21swapper/123:55:001
299909931,1cyclictest0-21swapper/123:45:011
299909931,1cyclictest0-21swapper/123:40:151
299909931,1cyclictest0-21swapper/123:30:121
299909931,1cyclictest0-21swapper/123:25:121
299909931,1cyclictest0-21swapper/122:55:111
299909931,1cyclictest0-21swapper/122:50:131
299909931,1cyclictest0-21swapper/122:20:131
299909931,1cyclictest0-21swapper/122:00:131
299909931,1cyclictest0-21swapper/121:40:011
299909931,1cyclictest0-21swapper/121:30:131
299909931,1cyclictest0-21swapper/120:35:131
299909931,1cyclictest0-21swapper/119:45:151
299909931,1cyclictest0-21swapper/119:40:131
299909931,1cyclictest0-21swapper/119:20:011
299909931,1cyclictest0-21swapper/100:30:131
299909930,2cyclictest21939-21chrt00:07:011
299909930,2cyclictest0-21swapper/121:35:151
299909930,2cyclictest0-21swapper/120:05:121
299899931,2cyclictest0-21swapper/020:50:010
299899931,2cyclictest0-21swapper/020:40:140
299899931,1cyclictest1836-21df20:20:150
299899930,3cyclictest24541-21perl21:05:150
299899930,3cyclictest0-21swapper/020:30:010
299899930,3cyclictest0-21swapper/020:25:130
299899930,3cyclictest0-21swapper/020:10:120
299899930,3cyclictest0-21swapper/020:05:120
299899930,3cyclictest0-21swapper/019:35:130
299899930,3cyclictest0-21swapper/019:25:130
299899930,3cyclictest0-21swapper/019:10:130
19587230,0sleep10-21swapper/122:44:171
299909921,1cyclictest0-21swapper/121:55:111
299909921,1cyclictest0-21swapper/121:05:141
299909921,1cyclictest0-21swapper/120:40:151
299909920,1cyclictest0-21swapper/123:10:081
299909920,1cyclictest0-21swapper/123:05:121
299909920,1cyclictest0-21swapper/122:30:131
299909920,1cyclictest0-21swapper/121:45:121
299909920,1cyclictest0-21swapper/121:00:121
299909920,1cyclictest0-21swapper/120:55:161
299909920,1cyclictest0-21swapper/120:30:141
299909920,1cyclictest0-21swapper/120:10:021
299909920,1cyclictest0-21swapper/120:00:111
299909920,1cyclictest0-21swapper/119:35:131
299909920,1cyclictest0-21swapper/119:25:111
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional