You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-10-03 - 06:05
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackaslot8.osadl.org (updated Tue Oct 03, 2023 00:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
319772295262,20sleep20-21swapper/219:08:162
319782276241,22sleep30-21swapper/319:08:173
317592269212,19sleep10-21swapper/119:05:431
317652264232,20sleep00-21swapper/019:05:460
322702670,1sleep00-21swapper/022:18:340
197122640,0sleep00-21swapper/019:50:300
125302620,0sleep10-21swapper/122:25:281
219172610,0sleep20-21swapper/223:50:462
215442600,0sleep20-21swapper/221:50:532
175832590,0sleep10-21swapper/123:29:111
133792590,0sleep00-21swapper/022:46:330
72382570,0sleep00-21swapper/023:42:030
28572190,0sleep20-21swapper/223:20:202
176902190,0sleep20-21swapper/221:49:092
277572180,4sleep012-21ksoftirqd/021:55:250
231452180,0sleep30-21swapper/322:52:423
168682180,0sleep20-21swapper/223:08:462
278222160,0sleep00-21swapper/021:13:380
199472150,0sleep20-21swapper/222:30:272
168792150,0sleep10-21swapper/122:09:351
20642140,0sleep30-21swapper/320:25:213
129322130,0sleep10-21swapper/123:05:451
54062100,0sleep00-21swapper/023:21:050
3224599105,3cyclictest19982-21systemd-journal22:17:393
3223699109,1cyclictest22-21ksoftirqd/121:50:001
322369995,2cyclictest19982-21systemd-journal23:52:211
322369993,3cyclictest11000-21munin-run21:45:001
322329994,3cyclictest19982-21systemd-journal19:32:270
9963280,0sleep00-21swapper/023:04:520
322459984,3cyclictest508-21dbus-daemon23:14:483
322459984,3cyclictest508-21dbus-daemon23:14:483
322459984,3cyclictest508-21dbus-daemon21:12:103
322459984,3cyclictest19982-21systemd-journal22:12:133
322459984,3cyclictest19982-21systemd-journal00:00:003
322459983,4cyclictest19982-21systemd-journal21:32:333
322459983,3cyclictest19982-21systemd-journal20:10:333
322459980,7cyclictest8811-21diskstats20:40:203
322419984,3cyclictest508-21dbus-daemon23:47:212
322419984,3cyclictest508-21dbus-daemon20:37:562
322419983,3cyclictest19982-21systemd-journal21:34:402
322419983,3cyclictest19982-21systemd-journal20:00:242
322419982,3cyclictest3201-21idleruntime-cro00:00:002
322419981,4cyclictest1468-21perf20:25:002
322369982,3cyclictest508-21dbus-daemon21:25:001
322329985,2cyclictest11736-21in:imuxsock19:39:080
322329984,2cyclictest19982-21systemd-journal22:45:000
322329984,1cyclictest12-21ksoftirqd/000:20:000
322329983,3cyclictest19982-21systemd-journal22:53:320
17789280,0sleep00-21swapper/000:10:100
14925280,0sleep10-21swapper/100:07:481
322459974,2cyclictest508-21dbus-daemon19:37:523
322459973,3cyclictest508-21dbus-daemon19:17:263
322459973,3cyclictest3758-21snmpd19:32:083
322459973,3cyclictest3758-21snmpd19:25:113
322459973,3cyclictest29503-21cat21:35:213
322459973,3cyclictest19982-21systemd-journal23:31:333
322459973,3cyclictest19982-21systemd-journal21:01:573
322459973,3cyclictest17486-21systemctl19:45:253
322459973,3cyclictest0-21swapper/322:30:553
322459973,3cyclictest0-21swapper/322:02:493
322459973,3cyclictest0-21swapper/321:28:433
322459973,2cyclictest19982-21systemd-journal22:22:143
322459973,1cyclictest0-21swapper/322:46:373
322459972,3cyclictest19982-21systemd-journal19:56:483
322459971,5cyclictest19982-21systemd-journal21:55:003
322459970,7cyclictest0-21swapper/321:55:463
322459970,4cyclictest131532cstates20:50:203
322419973,3cyclictest19982-21systemd-journal20:44:322
322419973,3cyclictest19982-21systemd-journal00:20:012
322419973,3cyclictest18107-21sed21:00:252
322419973,2cyclictest502-21systemd-logind00:33:402
322419973,2cyclictest19982-21systemd-journal19:50:002
322419973,1cyclictest0-21swapper/223:37:052
322419973,1cyclictest0-21swapper/222:59:412
322419973,1cyclictest0-21swapper/222:20:462
322419973,1cyclictest0-21swapper/222:03:062
322419972,4cyclictest0-21swapper/222:25:202
322419972,3cyclictest19982-21systemd-journal19:52:292
322419972,3cyclictest16969-21lldpd20:12:002
322419971,3cyclictest25789-21munin-run22:15:012
322419971,2cyclictest13038-21ssh21:25:042
322419970,5cyclictest19982-21systemd-journal22:10:002
322419970,3cyclictest19982-21systemd-journal22:50:322
322369974,2cyclictest963-21ntpq19:10:251
322369974,2cyclictest508-21dbus-daemon22:30:101
322369974,2cyclictest508-21dbus-daemon22:03:291
322369974,2cyclictest22-21ksoftirqd/123:00:011
322369973,3cyclictest3758-21snmpd19:41:361
322369973,3cyclictest3758-21snmpd19:34:471
322369973,3cyclictest3758-21snmpd19:28:231
322369973,3cyclictest3758-21snmpd00:22:251
322369973,3cyclictest24738-21sshd21:32:051
322369973,3cyclictest19982-21systemd-journal19:47:331
322369973,3cyclictest19982-21systemd-journal00:11:331
322369973,3cyclictest19116-21gdbus19:40:021
322369973,3cyclictest0-21swapper/123:56:441
322369973,2cyclictest19982-21systemd-journal23:05:001
322369973,1cyclictest0-21swapper/122:46:281
322369973,1cyclictest0-21swapper/122:20:171
322369973,1cyclictest0-21swapper/121:36:101
322369973,1cyclictest0-21swapper/121:18:591
322369971,4cyclictest19982-21systemd-journal20:31:411
322369970,5cyclictest16969-21lldpd20:04:581
322329976,1cyclictest12-21ksoftirqd/000:25:000
322329976,1cyclictest0-21swapper/022:03:170
322329975,1cyclictest12-21ksoftirqd/021:40:010
322329974,2cyclictest508-21dbus-daemon20:32:070
322329974,2cyclictest19982-21systemd-journal20:20:250
322329973,3cyclictest508-21dbus-daemon23:20:010
322329973,3cyclictest3758-21snmpd23:45:140
322329973,3cyclictest3758-21snmpd23:05:590
322329973,3cyclictest3758-21snmpd22:20:480
322329973,3cyclictest3758-21snmpd22:14:420
322329973,3cyclictest3758-21snmpd19:15:160
322329973,3cyclictest19982-21systemd-journal19:26:370
322329973,3cyclictest12081-21ntpq22:25:250
322329973,3cyclictest0-21swapper/023:57:260
322329973,3cyclictest0-21swapper/022:58:470
322329973,3cyclictest0-21swapper/022:38:540
322329973,3cyclictest0-21swapper/000:25:570
322329973,1cyclictest12-21ksoftirqd/000:40:000
322329973,1cyclictest0-21swapper/023:35:470
322329973,1cyclictest0-21swapper/023:11:360
322329973,1cyclictest0-21swapper/023:11:350
322329973,1cyclictest0-21swapper/021:32:030
322329972,3cyclictest19982-21systemd-journal21:20:000
322329972,3cyclictest19982-21systemd-journal20:25:000
322329970,4cyclictest29403-21munin-run20:15:000
29820270,0sleep20-21swapper/220:15:192
12046270,0sleep00-21swapper/022:05:370
322459966,0cyclictest0-21swapper/323:00:143
322459965,1cyclictest2808-21kworker/3:221:40:203
322459963,2cyclictest8455-21ls00:25:303
322459963,2cyclictest3758-21snmpd20:45:243
322459963,2cyclictest2796-21ssh00:21:273
322459963,2cyclictest23087-21ssh00:37:443
322459963,2cyclictest14402-21sh23:06:523
322459963,2cyclictest12081-21ssh23:25:233
322459962,3cyclictest5850-21cat19:20:263
322459962,3cyclictest508-21dbus-daemon19:51:383
322459962,3cyclictest4086-21expr00:00:203
322459962,3cyclictest3758-21snmpd23:15:043
322459962,3cyclictest3758-21snmpd22:59:463
322459962,3cyclictest3758-21snmpd22:35:193
322459962,3cyclictest3758-21snmpd21:20:413
322459962,3cyclictest3758-21snmpd21:05:403
322459962,3cyclictest3758-21snmpd20:20:073
322459962,3cyclictest3758-21snmpd20:19:223
322459962,3cyclictest3758-21snmpd00:06:273
322459962,3cyclictest3282-21awk22:40:283
322459962,3cyclictest29741-11brltty20:07:553
322459962,3cyclictest19982-21systemd-journal21:47:093
322459962,3cyclictest19116-21gdbus20:40:003
322459962,3cyclictest19116-21gdbus20:35:003
322459962,3cyclictest15382-21expr20:55:203
322459962,1cyclictest0-21swapper/323:46:503
322459962,1cyclictest0-21swapper/323:41:253
322459962,1cyclictest0-21swapper/300:30:253
322459962,1cyclictest0-21swapper/300:10:243
322459961,3cyclictest19982-21systemd-journal23:35:163
322459960,6cyclictest0-21swapper/320:00:023
322459960,2cyclictest3451-21fschecks_time23:20:223
322459960,2cyclictest32432-21/usr/sbin/munin19:10:253
322459960,2cyclictest25832-21cstates00:15:213
322459960,2cyclictest19982-21systemd-journal23:50:413
322459960,0cyclictest0-21swapper/322:06:573
322419966,0cyclictest0-21swapper/219:10:232
322419963,2cyclictest8055-21if_enp4s019:25:242
322419963,2cyclictest32682-21ssh21:15:592
322419963,2cyclictest21417-21cat19:55:242
322419963,2cyclictest19208-21awk00:10:262
322419963,2cyclictest11736-21in:imuxsock20:08:202
322419963,2cyclictest11736-21in:imuxsock00:20:442
322419962,3cyclictest508-21dbus-daemon21:10:532
322419962,3cyclictest3758-21snmpd23:25:362
322419962,3cyclictest3758-21snmpd20:57:462
322419962,3cyclictest3758-21snmpd20:31:062
322419962,3cyclictest3758-21snmpd20:27:212
322419962,3cyclictest3758-21snmpd00:00:392
322419962,3cyclictest28278-21expr22:35:282
322419962,3cyclictest28197-21cat23:15:252
322419962,3cyclictest19568-21grep00:35:192
322419962,3cyclictest16969-21lldpd23:00:042
322419962,3cyclictest13231-21expr20:50:202
322419962,3cyclictest0-21swapper/222:17:352
322419962,1cyclictest0-21swapper/222:40:082
322419961,3cyclictest19982-21systemd-journal23:41:332
322419961,3cyclictest19982-21systemd-journal21:20:042
322419961,2cyclictest18932-21ssh23:30:022
322419960,6cyclictest0-21swapper/200:06:172
322419960,4cyclictest19982-21systemd-journal23:12:182
322419960,4cyclictest19982-21systemd-journal23:12:182
322419960,3cyclictest7149-21expr00:25:212
322419960,3cyclictest5599-21/usr/sbin/munin21:40:312
322419960,3cyclictest29735-21needreboot21:35:232
322419960,0cyclictest0-21swapper/221:55:402
322369964,1cyclictest9022-21hddtemp_smartct20:40:221
322369963,2cyclictest668-21runrttasks20:06:001
322369963,2cyclictest6384-21swap19:20:301
322369963,2cyclictest30015-21sh22:16:521
322369963,2cyclictest28046-21hddtemp_smartct20:10:241
322369963,2cyclictest28033-21ssh23:15:251
322369963,2cyclictest27454-21sh23:35:131
322369963,2cyclictest24343-21ssh23:32:561
322369963,2cyclictest17838-21fw_packets21:00:221
322369963,2cyclictest11736-21in:imuxsock19:16:101
322369963,2cyclictest11493-21rm23:45:111
322369963,1cyclictest0-21swapper/123:40:491
322369962,3cyclictest794-21grep20:20:271
322369962,3cyclictest3758-21snmpd21:05:251
322369962,3cyclictest3758-21snmpd20:36:061
322369962,3cyclictest3758-21snmpd20:26:211
322369962,3cyclictest3758-21snmpd19:50:181
322369962,3cyclictest3178-21cat23:20:221
322369962,3cyclictest30056-21sh20:15:191
322369962,3cyclictest21493-21ls19:55:241
322369962,3cyclictest20131-21cat00:35:221
322369962,3cyclictest19690-21cat21:50:201
322369962,3cyclictest18844-21cat22:50:201
322369962,3cyclictest16702-21sshd20:56:041
322369962,3cyclictest13657-21fw_packets00:30:231
322369962,3cyclictest0-21swapper/123:10:181
322369962,3cyclictest0-21swapper/123:10:181
322369962,3cyclictest0-21swapper/121:10:231
322369962,3cyclictest0-21swapper/100:26:341
322369962,1cyclictest0-21swapper/122:10:221
322369962,1cyclictest0-21swapper/121:56:221
322369962,1cyclictest0-21swapper/100:02:231
322369961,3cyclictest19982-21systemd-journal22:36:251
322369960,6cyclictest0-21swapper/121:27:021
322369960,6cyclictest0-21swapper/120:51:561
322369960,5cyclictest25616-21expr00:15:221
322369960,2cyclictest2180-21apt-get22:40:201
322329963,2cyclictest16290-21sensors_temp20:55:270
322329963,2cyclictest1-21systemd21:45:330
322329962,3cyclictest6176-21head21:20:190
322329962,3cyclictest508-21dbus-daemon20:40:270
322329962,3cyclictest3758-21snmpd21:50:060
322329962,3cyclictest3758-21snmpd21:05:550
322329962,3cyclictest3758-21snmpd20:15:390
322329962,3cyclictest3758-21snmpd20:06:020
322329962,3cyclictest3758-21snmpd19:20:540
322329962,3cyclictest3758-21snmpd19:14:160
322329962,3cyclictest20019-21expr23:50:190
322329962,3cyclictest19982-21systemd-journal19:40:220
322329962,3cyclictest13122-21expr20:50:200
322329962,3cyclictest1-21systemd21:41:230
322329962,3cyclictest1-21systemd00:30:190
322329962,3cyclictest0-21swapper/000:05:140
322329962,2cyclictest19982-21systemd-journal21:02:020
322329962,2cyclictest16969-21lldpd20:36:450
322329962,1cyclictest12-21ksoftirqd/019:55:200
322329962,1cyclictest0-21swapper/023:31:010
322329962,1cyclictest0-21swapper/021:25:120
322329961,3cyclictest19982-21systemd-journal20:01:320
322329960,3cyclictest4653-21rm00:00:250
322329960,3cyclictest22006-21rm22:31:390
322329960,3cyclictest16484-21rm23:28:190
13424260,0sleep20-21swapper/222:46:352
322459952,2cyclictest29842-21apt-key21:15:013
322459952,2cyclictest14766-21cat19:40:263
322459951,3cyclictest508-21dbus-daemon22:25:113
322419952,2cyclictest9752-21expr19:30:202
322419952,2cyclictest5406-21expr19:20:232
322419952,2cyclictest19530-21/usr/sbin/munin21:05:182
322419952,2cyclictest11910-21cat19:35:172
322419952,2cyclictest10601-21taskset20:45:012
322419951,3cyclictest2513-21ntp_states19:15:012
322419951,3cyclictest14564-21tune2fs19:40:252
322369951,3cyclictest11370-21cut20:45:281
322329952,2cyclictest3758-21snmpd19:45:020
322329950,4cyclictest10624-21/usr/sbin/munin20:45:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional