You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-04 - 11:33
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot2.osadl.org (updated Mon Dec 04, 2023 00:44:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
236022205189,11sleep10-21swapper/119:08:441
214832197179,13sleep00-21swapper/019:05:140
233582194178,11sleep30-21swapper/319:05:373
23508210386,12sleep20-21swapper/219:07:322
288652430,0sleep10-21swapper/122:39:491
49152420,0sleep30-21swapper/300:28:493
267692410,0sleep30-21swapper/300:22:043
251892100,0sleep10-21swapper/121:34:081
238849992,3cyclictest14582-21perf19:55:000
238849991,4cyclictest1965-21perf20:35:000
238849980,8cyclictest6520-21ssh00:07:430
238849972,5cyclictest0-21swapper/022:11:160
238849970,7cyclictest0-21swapper/022:30:200
238849970,7cyclictest0-21swapper/022:22:200
238849970,1cyclictest0-21swapper/022:04:200
238949960,5cyclictest16292-21ssh00:15:153
238849962,4cyclictest11111-21ssh21:45:170
238849962,4cyclictest0-21swapper/022:51:040
238849962,4cyclictest0-21swapper/021:50:080
238849960,6cyclictest30611-21ssh23:00:440
238849960,6cyclictest0-21swapper/023:55:470
238849960,6cyclictest0-21swapper/023:49:430
238849960,6cyclictest0-21swapper/023:36:120
238849960,6cyclictest0-21swapper/023:30:530
238849960,6cyclictest0-21swapper/023:26:040
238849960,6cyclictest0-21swapper/023:20:480
238849960,6cyclictest0-21swapper/023:16:240
238849960,6cyclictest0-21swapper/023:10:440
238849960,6cyclictest0-21swapper/023:05:240
238849960,6cyclictest0-21swapper/022:55:400
238849960,6cyclictest0-21swapper/022:45:360
238849960,6cyclictest0-21swapper/022:40:040
238849960,6cyclictest0-21swapper/022:35:570
238849960,6cyclictest0-21swapper/022:19:290
238849960,6cyclictest0-21swapper/022:19:280
238849960,6cyclictest0-21swapper/022:07:240
238849960,6cyclictest0-21swapper/021:55:400
238849960,6cyclictest0-21swapper/021:41:120
238849960,6cyclictest0-21swapper/000:35:030
238849960,6cyclictest0-21swapper/000:25:190
238849960,6cyclictest0-21swapper/000:21:550
238849960,6cyclictest0-21swapper/000:16:270
238849960,6cyclictest0-21swapper/000:14:190
238849960,6cyclictest0-21swapper/000:02:040
238849960,1cyclictest17745-21perl21:05:200
238849960,1cyclictest0-21swapper/022:25:120
238849960,0cyclictest11083-21ssh23:50:070
238849960,0cyclictest0-21swapper/023:40:440
15095260,0sleep20-21swapper/219:55:202
26759250,0sleep10-21swapper/123:18:561
238949955,0cyclictest0-21swapper/321:30:483
238949955,0cyclictest0-21swapper/319:55:293
238949951,2cyclictest0-21swapper/323:35:153
238949950,5cyclictest19279-21diskmemload23:20:573
238949950,5cyclictest0-21swapper/322:58:243
238949950,5cyclictest0-21swapper/322:50:173
238949950,5cyclictest0-21swapper/322:08:033
238949950,5cyclictest0-21swapper/321:10:093
238949950,5cyclictest0-21swapper/320:55:173
238949950,5cyclictest0-21swapper/319:45:183
238949950,0cyclictest0-21swapper/321:24:423
238949950,0cyclictest0-21swapper/300:38:123
238909955,0cyclictest22442-21kworker/2:100:30:372
238909950,5cyclictest1138-21ntp_states22:00:212
238909950,5cyclictest0-21swapper/223:44:482
238909950,5cyclictest0-21swapper/221:43:092
238909950,2cyclictest0-21swapper/219:25:292
238909950,0cyclictest0-21swapper/223:26:362
238889951,2cyclictest0-21swapper/123:25:161
238889950,5cyclictest0-21swapper/122:46:171
238849952,3cyclictest0-21swapper/021:35:080
238849952,3cyclictest0-21swapper/021:16:120
238849950,5cyclictest0-21swapper/021:31:000
238849950,5cyclictest0-21swapper/021:25:520
238849950,5cyclictest0-21swapper/021:11:200
238849950,5cyclictest0-21swapper/020:55:270
238849950,5cyclictest0-21swapper/019:45:320
238849950,5cyclictest0-21swapper/019:40:210
238849950,5cyclictest0-21swapper/019:21:400
238849950,5cyclictest0-21swapper/019:10:280
238849950,5cyclictest0-21swapper/000:30:230
238849950,4cyclictest0-21swapper/019:25:180
238849950,0cyclictest0-21swapper/021:20:570
238949944,0cyclictest0-21swapper/323:11:473
238949944,0cyclictest0-21swapper/300:11:173
238949944,0cyclictest0-21swapper/300:08:493
238949943,1cyclictest28551-21tr23:00:193
238949942,1cyclictest4028-21ssh22:02:143
238949942,1cyclictest29357-21ssh22:40:023
238949941,2cyclictest0-21swapper/320:25:163
238949941,1cyclictest0-21swapper/320:30:223
238949940,4cyclictest4345-21ssh23:25:163
238949940,4cyclictest25542-21hddtemp_smartct20:15:213
238949940,4cyclictest22006-21munin-run20:10:003
238949940,4cyclictest21788-21users23:55:413
238949940,4cyclictest0-21swapper/323:30:173
238949940,4cyclictest0-21swapper/322:48:213
238949940,4cyclictest0-21swapper/322:35:593
238949940,4cyclictest0-21swapper/322:30:203
238949940,4cyclictest0-21swapper/321:54:023
238949940,4cyclictest0-21swapper/321:40:023
238949940,4cyclictest0-21swapper/321:35:023
238949940,4cyclictest0-21swapper/321:15:163
238949940,4cyclictest0-21swapper/300:03:063
238949940,3cyclictest6907-21latency_hist20:45:003
238949940,3cyclictest30833-21/usr/sbin/munin22:20:233
238949940,3cyclictest28086-21latency_hist23:20:003
238949940,3cyclictest18039-21perl20:00:233
238949940,3cyclictest17165-21cat21:50:013
238949940,3cyclictest0-21swapper/321:00:203
238949940,1cyclictest4460-21latency_hist19:35:003
238909944,0cyclictest13724-21kworker/2:023:57:012
238909944,0cyclictest13724-21kworker/2:022:49:502
238909944,0cyclictest0-21swapper/221:35:182
238909944,0cyclictest0-21swapper/200:36:282
238909943,1cyclictest24749-21gltestperf19:10:232
238909941,3cyclictest9068-21cat19:40:292
238909941,2cyclictest0-21swapper/221:10:212
238909941,2cyclictest0-21swapper/220:20:172
238909940,4cyclictest28687-21ssh21:57:392
238909940,4cyclictest0-21swapper/223:47:172
238909940,4cyclictest0-21swapper/223:36:042
238909940,4cyclictest0-21swapper/223:30:222
238909940,4cyclictest0-21swapper/223:14:482
238909940,4cyclictest0-21swapper/223:05:482
238909940,4cyclictest0-21swapper/222:51:352
238909940,4cyclictest0-21swapper/222:31:122
238909940,4cyclictest0-21swapper/222:25:172
238909940,4cyclictest0-21swapper/222:16:322
238909940,4cyclictest0-21swapper/222:16:322
238909940,4cyclictest0-21swapper/222:11:262
238909940,4cyclictest0-21swapper/221:50:582
238909940,4cyclictest0-21swapper/221:32:162
238909940,4cyclictest0-21swapper/221:20:452
238909940,4cyclictest0-21swapper/200:22:442
238909940,4cyclictest0-21swapper/200:13:172
238909940,3cyclictest32162-21latency_hist19:25:012
238909940,3cyclictest3070-21hddtemp_smartct19:30:212
238909940,3cyclictest26430-21munin-node19:15:012
238909940,3cyclictest22487-21wc23:15:402
238909940,3cyclictest14316-21dump-pmu-power21:00:002
238909940,3cyclictest12280-21ls19:50:002
238909940,3cyclictest12023-21ssh23:50:182
238909940,3cyclictest0-21swapper/220:05:182
238909940,3cyclictest0-21swapper/200:15:152
238909940,1cyclictest22233-21/usr/sbin/munin20:10:242
238909940,1cyclictest22049-21df22:35:242
238909940,1cyclictest20076-21/usr/sbin/munin22:55:252
238909940,1cyclictest1613-21/usr/sbin/munin00:05:232
238909940,1cyclictest12337-21/usr/sbin/munin19:50:292
238909940,0cyclictest0-21swapper/200:26:422
238889941,2cyclictest0-21swapper/123:55:221
238889941,2cyclictest0-21swapper/123:00:181
238889941,2cyclictest0-21swapper/121:50:151
238889941,2cyclictest0-21swapper/119:40:191
238889941,2cyclictest0-21swapper/119:15:191
238889941,2cyclictest0-21swapper/100:25:141
238889941,1cyclictest0-21swapper/100:05:141
238889940,4cyclictest0-21swapper/123:12:191
238889940,4cyclictest0-21swapper/122:18:031
238889940,4cyclictest0-21swapper/122:18:031
238889940,4cyclictest0-21swapper/122:05:071
238889940,4cyclictest0-21swapper/121:20:081
238889940,4cyclictest0-21swapper/119:36:401
238889940,4cyclictest0-21swapper/119:29:411
238889940,4cyclictest0-21swapper/100:34:141
238889940,4cyclictest0-21swapper/100:21:091
238889940,3cyclictest0-21swapper/120:05:181
238889940,0cyclictest0-21swapper/122:41:201
238889940,0cyclictest0-21swapper/122:00:181
238889940,0cyclictest0-21swapper/121:25:311
238849944,0cyclictest0-21swapper/020:25:280
238849941,3cyclictest0-21swapper/019:35:080
238849940,4cyclictest0-21swapper/021:03:390
238849940,4cyclictest0-21swapper/020:50:350
238849940,4cyclictest0-21swapper/020:45:310
238849940,4cyclictest0-21swapper/020:15:180
238849940,4cyclictest0-21swapper/019:30:320
238849940,3cyclictest29589-21sed20:25:000
238849940,3cyclictest26636-21awk19:15:010
238849940,3cyclictest19681-21sed20:05:000
31857230,0sleep10-21swapper/121:37:531
29037230,0sleep10-21swapper/122:20:111
238949930,3cyclictest7511-21expr20:45:203
238949930,3cyclictest6659-21sh00:30:023
238949930,3cyclictest19279-21diskmemload22:15:103
238949930,3cyclictest19279-21diskmemload22:15:093
238949930,3cyclictest19279-21diskmemload21:55:113
238949930,3cyclictest17790-21fw_packets21:05:203
238949930,3cyclictest0-21swapper/323:50:153
238949930,3cyclictest0-21swapper/323:45:323
238949930,3cyclictest0-21swapper/323:40:063
238949930,3cyclictest0-21swapper/323:05:033
238949930,3cyclictest0-21swapper/322:25:033
238949930,3cyclictest0-21swapper/322:10:033
238949930,3cyclictest0-21swapper/321:25:153
238949930,3cyclictest0-21swapper/320:50:173
238949930,3cyclictest0-21swapper/320:35:203
238949930,3cyclictest0-21swapper/320:20:173
238949930,3cyclictest0-21swapper/320:10:193
238949930,3cyclictest0-21swapper/319:50:213
238949930,3cyclictest0-21swapper/319:40:013
238949930,3cyclictest0-21swapper/319:35:203
238949930,3cyclictest0-21swapper/319:25:183
238949930,3cyclictest0-21swapper/319:20:173
238949930,3cyclictest0-21swapper/319:15:013
238949930,3cyclictest0-21swapper/319:10:193
238909932,1cyclictest28269-21/usr/sbin/munin23:00:202
238909931,1cyclictest0-21swapper/221:00:212
238909931,1cyclictest0-21swapper/220:45:182
238909931,1cyclictest0-21swapper/220:25:162
238909930,3cyclictest32147-21apt-get20:30:222
238909930,3cyclictest29736-21ssh22:40:202
238909930,3cyclictest28468-21ssh00:00:542
238909930,3cyclictest28404-21ssh23:20:042
238909930,3cyclictest28046-21apt-get21:15:162
238909930,3cyclictest24710-21apt-get20:15:192
238909930,3cyclictest10108-21ssh21:25:032
238909930,3cyclictest0-21swapper/222:20:152
238909930,3cyclictest0-21swapper/222:05:092
238909930,3cyclictest0-21swapper/221:45:252
238909930,3cyclictest0-21swapper/221:05:182
238909930,3cyclictest0-21swapper/220:50:172
238909930,3cyclictest0-21swapper/220:40:172
238909930,3cyclictest0-21swapper/220:35:202
238909930,3cyclictest0-21swapper/220:00:202
238909930,1cyclictest4700-21/usr/sbin/munin19:35:212
238889933,0cyclictest0-21swapper/123:42:181
238889933,0cyclictest0-21swapper/121:45:011
238889931,2cyclictest0-21swapper/119:20:171
238889931,1cyclictest177952sleep122:11:311
238889931,1cyclictest0-21swapper/123:35:141
238889931,1cyclictest0-21swapper/123:30:221
238889931,1cyclictest0-21swapper/123:05:211
238889931,1cyclictest0-21swapper/122:50:151
238889931,1cyclictest0-21swapper/122:30:201
238889931,1cyclictest0-21swapper/121:40:021
238889931,1cyclictest0-21swapper/121:15:161
238889931,1cyclictest0-21swapper/121:10:211
238889931,1cyclictest0-21swapper/121:05:181
238889931,1cyclictest0-21swapper/120:35:191
238889931,1cyclictest0-21swapper/120:10:191
238889931,1cyclictest0-21swapper/119:55:191
238889931,1cyclictest0-21swapper/119:45:171
238889931,1cyclictest0-21swapper/119:30:171
238889930,2cyclictest0-21swapper/123:50:161
238889930,2cyclictest0-21swapper/120:25:161
238889930,1cyclictest0-21swapper/100:20:001
238849931,1cyclictest0-21swapper/020:35:200
238849930,3cyclictest0-21swapper/020:40:030
238849930,3cyclictest0-21swapper/020:10:190
238849930,3cyclictest0-21swapper/020:05:050
238849930,1cyclictest0-21swapper/019:55:180
238889922,0cyclictest0-21swapper/120:18:351
238889921,1cyclictest0-21swapper/122:25:171
238889921,1cyclictest0-21swapper/120:40:171
238889921,1cyclictest0-21swapper/120:00:201
238889921,0cyclictest0-21swapper/120:50:171
238889920,2cyclictest25896-21chrt19:10:591
238889920,2cyclictest166422sleep121:03:041
238889920,1cyclictest95872sleep100:10:011
238889920,1cyclictest7130-21chrt20:45:071
238889920,1cyclictest5256-21chrt23:45:351
238889920,1cyclictest30560-21chrt00:02:361
238889920,1cyclictest28182-21taskset21:57:141
238889920,1cyclictest199672sleep122:55:151
238889920,1cyclictest0-21swapper/123:20:211
238889920,1cyclictest0-21swapper/120:55:171
238889920,1cyclictest0-21swapper/120:30:221
238889920,1cyclictest0-21swapper/120:20:171
238889920,1cyclictest0-21swapper/119:50:001
238889920,1cyclictest0-21swapper/100:35:421
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional