You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-23 - 08:47
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8.osadl.org (updated Sat Sep 23, 2023 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
126142309276,21sleep20-21swapper/219:08:372
125452270236,22sleep00-21swapper/019:07:420
125532264228,23sleep30-21swapper/319:07:503
124122259225,22sleep10-21swapper/119:06:141
179362690,0sleep30-21swapper/322:07:573
284062580,0sleep10-21swapper/123:25:261
281852580,0sleep20-21swapper/222:39:102
18692570,0sleep00-21swapper/021:56:080
24762550,0sleep30-21swapper/323:30:263
189282190,1sleep20-21swapper/223:42:132
102662180,0sleep20-21swapper/200:25:232
13212160,0sleep20-21swapper/223:53:102
267282150,0sleep00-21swapper/000:38:370
242462140,1sleep30-21swapper/319:30:353
140742130,0sleep30-21swapper/321:42:133
253662120,0sleep00-21swapper/023:23:450
107712120,2sleep1221ksoftirqd/122:25:281
260972110,1sleep30-21swapper/300:12:133
1283499104,4cyclictest19982-21systemd-journal22:52:241
22246290,1sleep00-21swapper/000:09:540
16127290,0sleep10-21swapper/122:06:101
128439994,3cyclictest19982-21systemd-journal23:36:133
128389994,3cyclictest19982-21systemd-journal21:59:522
128299995,1cyclictest30968-21turbostat21:00:000
128299994,3cyclictest19982-21systemd-journal20:39:450
6510280,0sleep30-21swapper/323:10:163
128439983,3cyclictest19982-21systemd-journal20:53:553
128439983,3cyclictest19982-21systemd-journal00:26:313
128389984,3cyclictest508-21dbus-daemon23:03:162
128389984,3cyclictest19982-21systemd-journal22:32:182
128389983,3cyclictest19982-21systemd-journal19:15:542
128389981,2cyclictest29613-21apt-get22:40:202
128389980,2cyclictest15288-21taskset23:15:372
128349983,3cyclictest19982-21systemd-journal00:28:241
128299985,2cyclictest508-21dbus-daemon22:00:160
128299985,2cyclictest30558-21ssh22:16:380
128299983,3cyclictest19982-21systemd-journal19:37:100
128299982,5cyclictest0-21swapper/021:34:490
128439974,2cyclictest508-21dbus-daemon23:20:483
128439974,2cyclictest508-21dbus-daemon00:38:483
128439974,2cyclictest15009-21latency_hist19:15:013
128439974,2cyclictest11736-21in:imuxsock19:25:013
128439973,3cyclictest4798-21sed23:55:223
128439973,3cyclictest19982-21systemd-journal19:49:173
128439973,3cyclictest19576-21chrt23:19:493
128439973,3cyclictest11499-21ntpq00:00:253
128439973,1cyclictest0-21swapper/322:10:013
128439973,1cyclictest0-21swapper/321:18:373
128439972,5cyclictest25475-21kworker/3:020:15:233
128439971,4cyclictest19982-21systemd-journal21:31:033
128439970,1cyclictest19107-21diskmemload00:06:233
128389974,2cyclictest19982-21systemd-journal22:24:552
128389974,2cyclictest1664-21cron22:20:002
128389974,2cyclictest14066-21ntp_states20:20:252
128389973,3cyclictest3758-21snmpd23:28:202
128389973,3cyclictest3758-21snmpd21:01:402
128389973,3cyclictest3758-21snmpd20:55:072
128389973,3cyclictest3758-21snmpd20:49:212
128389973,3cyclictest3758-21snmpd20:39:562
128389973,3cyclictest22619-21cron19:30:012
128389973,3cyclictest19982-21systemd-journal21:47:132
128389973,3cyclictest19982-21systemd-journal20:52:412
128389973,3cyclictest0-21swapper/222:01:432
128389973,1cyclictest0-21swapper/221:42:042
128389973,1cyclictest0-21swapper/200:01:032
128389972,3cyclictest19982-21systemd-journal00:08:032
128389970,7cyclictest0-21swapper/222:45:262
128389970,6cyclictest14175-21fschecks_time22:05:232
128389970,1cyclictest3315-21diskmemload22:56:022
128349974,2cyclictest508-21dbus-daemon20:38:551
128349974,2cyclictest20200-21sshd19:22:321
128349974,2cyclictest18344-21rm23:41:381
128349973,3cyclictest508-21dbus-daemon22:48:401
128349973,3cyclictest508-21dbus-daemon21:38:591
128349973,3cyclictest31904-21cat21:00:201
128349973,3cyclictest31833-21apt-get23:05:171
128349973,3cyclictest31170-21cat22:40:301
128349973,3cyclictest1-21systemd19:32:111
128349973,3cyclictest1-21systemd19:16:441
128349973,3cyclictest0-21swapper/123:53:021
128349973,3cyclictest0-21swapper/123:36:131
128349973,3cyclictest0-21swapper/122:21:061
128349973,2cyclictest19982-21systemd-journal20:45:011
128349973,1cyclictest0-21swapper/122:04:371
128349972,3cyclictest19982-21systemd-journal22:36:591
128349971,5cyclictest3758-21snmpd22:59:551
128349971,2cyclictest25981-21ssh00:12:051
128349970,7cyclictest3315-21diskmemload21:21:581
128299974,2cyclictest508-21dbus-daemon23:32:030
128299974,2cyclictest3620-21idleruntime-cro20:00:010
128299974,2cyclictest23821-21grep00:10:210
128299974,2cyclictest11736-21in:imuxsock23:59:090
128299974,2cyclictest11736-21in:imuxsock23:15:210
128299974,2cyclictest11736-21in:imuxsock20:14:440
128299973,3cyclictest7475-21memory23:10:230
128299973,3cyclictest508-21dbus-daemon23:05:450
128299973,3cyclictest3758-21snmpd20:23:330
128299973,3cyclictest19982-21systemd-journal23:39:320
128299973,3cyclictest19982-21systemd-journal19:51:580
128299973,3cyclictest19116-21gdbus23:45:010
128299973,3cyclictest10115-21cat22:25:250
128299973,3cyclictest0-21swapper/023:25:530
128299973,3cyclictest0-21swapper/023:01:040
128299973,3cyclictest0-21swapper/000:02:070
128299973,1cyclictest0-21swapper/022:50:500
128299972,3cyclictest19982-21systemd-journal22:11:320
128299971,3cyclictest5699-21rm21:35:320
128299971,3cyclictest16425-21rm22:30:230
128299970,5cyclictest19982-21systemd-journal22:44:540
17802260,0chrt0-21swapper/021:20:240
17229260,0sleep20-21swapper/200:30:262
16250260,1sleep116270-21ssh21:44:221
128439964,1cyclictest17860-21apt-get22:55:193
128439963,3cyclictest25475-21kworker/3:023:00:233
128439963,3cyclictest25475-21kworker/3:022:45:113
128439963,2cyclictest9495-21ssh22:25:203
128439963,2cyclictest668-21runrttasks21:00:123
128439963,2cyclictest668-21runrttasks20:11:123
128439963,2cyclictest3959-21/usr/sbin/munin21:35:283
128439963,2cyclictest3567-21cron20:00:023
128439963,2cyclictest3245-21tr00:20:203
128439963,2cyclictest30717-21sh23:27:173
128439963,2cyclictest29741-11brltty22:20:173
128439963,2cyclictest22500-21ssh22:35:043
128439963,2cyclictest20544-21ssh23:43:463
128439962,4cyclictest25475-21kworker/3:019:15:313
128439962,3cyclictest885-21ntpd20:25:263
128439962,3cyclictest7494-21cat22:00:253
128439962,3cyclictest5394-21unixbench_singl20:00:283
128439962,3cyclictest3758-21snmpd23:50:353
128439962,3cyclictest3758-21snmpd22:50:013
128439962,3cyclictest3758-21snmpd22:15:163
128439962,3cyclictest3758-21snmpd20:55:193
128439962,3cyclictest3758-21snmpd20:45:113
128439962,3cyclictest3758-21snmpd20:30:293
128439962,3cyclictest3758-21snmpd19:50:173
128439962,3cyclictest32610-21cat21:55:223
128439962,3cyclictest32294-21grep23:05:203
128439962,3cyclictest31027-21cat22:40:293
128439962,3cyclictest28366-21pluginstate19:40:253
128439962,3cyclictest25176-21fschecks_count21:50:243
128439962,3cyclictest23000-21cat20:40:233
128439962,3cyclictest22738-21expr23:45:193
128439962,3cyclictest2022-21cat21:05:213
128439962,3cyclictest18261-21cat21:45:213
128439962,3cyclictest17004-21ntp_kernel_pll_22:30:263
128439962,3cyclictest14073-21awk20:20:253
128439962,1cyclictest0-21swapper/321:10:183
128439962,1cyclictest0-21swapper/300:15:263
128439961,3cyclictest19982-21systemd-journal21:26:023
128439961,3cyclictest19982-21systemd-journal21:20:373
128439961,3cyclictest19982-21systemd-journal20:05:083
128439960,3cyclictest18312-21sh00:31:153
128389966,0cyclictest0-21swapper/221:25:042
128389964,1cyclictest22921-21hddtemp_smartct20:40:222
128389963,2cyclictest4820-21sh00:20:312
128389963,2cyclictest3758-21snmpd23:45:112
128389963,2cyclictest3758-21snmpd20:19:232
128389963,2cyclictest17167-21expr21:20:182
128389962,3cyclictest885-21ntpd00:15:222
128389962,3cyclictest508-21dbus-daemon20:30:002
128389962,3cyclictest4539-21if_enp4s021:10:222
128389962,3cyclictest3758-21snmpd23:31:082
128389962,3cyclictest3758-21snmpd21:07:342
128389962,3cyclictest3758-21snmpd20:10:082
128389962,3cyclictest3758-21snmpd19:56:462
128389962,3cyclictest3758-21snmpd19:54:022
128389962,3cyclictest3758-21snmpd19:36:532
128389962,3cyclictest3758-21snmpd19:22:242
128389962,3cyclictest32268-21diskstats23:05:202
128389962,3cyclictest25867-21cut21:50:292
128389962,3cyclictest22681-21grep00:35:202
128389962,3cyclictest21629-21ntpq22:10:252
128389962,1cyclictest0-21swapper/223:57:522
128389962,1cyclictest0-21swapper/223:20:182
128389962,1cyclictest0-21swapper/222:50:052
128389961,3cyclictest19982-21systemd-journal20:26:562
128389961,2cyclictest9544-21/usr/sbin/munin23:35:252
128389960,6cyclictest0-21swapper/222:25:352
128389960,5cyclictest9923-21cron21:40:012
128389960,5cyclictest13994-21ntpq19:10:232
128389960,4cyclictest7808-21sshd20:07:002
128389960,4cyclictest19982-21systemd-journal21:15:122
128389960,0cyclictest0-21swapper/223:10:212
128389960,0cyclictest0-21swapper/200:10:072
128349966,0cyclictest0-21swapper/123:00:301
128349966,0cyclictest0-21swapper/100:15:551
128349964,1cyclictest25712-21gltestperf19:35:241
128349964,1cyclictest17978-21sort21:45:211
128349963,2cyclictest9260-21chrt20:10:251
128349963,2cyclictest668-21runrttasks21:07:021
128349963,2cyclictest4812-21ssh23:32:021
128349963,2cyclictest20240-21sh23:20:131
128349963,2cyclictest19982-21systemd-journal23:15:061
128349963,2cyclictest18501-21sh00:05:431
128349963,2cyclictest11736-21in:imuxsock23:45:231
128349963,2cyclictest11736-21in:imuxsock22:15:171
128349963,2cyclictest11511-21cat20:15:251
128349963,2cyclictest11438-21pmu-power21:15:281
128349962,3cyclictest508-21dbus-daemon20:20:081
128349962,3cyclictest508-21dbus-daemon19:28:261
128349962,3cyclictest3758-21snmpd23:56:151
128349962,3cyclictest3758-21snmpd23:10:401
128349962,3cyclictest3758-21snmpd22:30:041
128349962,3cyclictest3758-21snmpd21:53:171
128349962,3cyclictest3758-21snmpd21:11:271
128349962,3cyclictest3758-21snmpd20:25:261
128349962,3cyclictest3758-21snmpd20:05:341
128349962,3cyclictest3758-21snmpd20:02:491
128349962,3cyclictest3758-21snmpd00:00:061
128349962,3cyclictest31186-21cat21:30:251
128349962,3cyclictest30173-21mailstats20:55:251
128349962,3cyclictest27995-21idleruntime19:40:231
128349962,3cyclictest27983-21sensors_temp20:50:311
128349962,3cyclictest22574-21expr20:40:211
128349962,3cyclictest19982-21systemd-journal19:45:311
128349962,3cyclictest19982-21systemd-journal19:12:081
128349962,2cyclictest31981-21lldpd19:58:101
128349962,1cyclictest0-21swapper/121:55:391
128349961,4cyclictest0-21swapper/100:35:211
128349961,3cyclictest19982-21systemd-journal00:22:061
128349960,2cyclictest20594-21/usr/sbin/munin22:10:211
128299966,0cyclictest0-21swapper/000:32:260
128299963,2cyclictest508-21dbus-daemon20:25:340
128299963,2cyclictest4373-21sendmail_mailqu00:20:280
128299963,2cyclictest27670-21cut20:50:260
128299963,2cyclictest25802-21ntpq21:50:280
128299963,2cyclictest24929-21df_inode20:45:210
128299963,2cyclictest24808-21sh23:46:100
128299963,2cyclictest19639-21taskset20:32:550
128299963,2cyclictest16350-21ssh22:06:240
128299962,3cyclictest3952-21wc21:10:200
128299962,3cyclictest3758-21snmpd21:00:060
128299962,3cyclictest3758-21snmpd20:42:080
128299962,3cyclictest3758-21snmpd19:40:100
128299962,3cyclictest3758-21snmpd19:31:130
128299962,3cyclictest3758-21snmpd19:25:490
128299962,3cyclictest3758-21snmpd19:21:190
128299962,3cyclictest3758-21snmpd19:19:400
128299962,3cyclictest3758-21snmpd19:11:030
128299962,3cyclictest30266-21cat19:45:220
128299962,3cyclictest29937-21basename23:50:240
128299962,3cyclictest23470-21cat22:35:220
128299962,3cyclictest11804-21ntpq20:15:280
128299962,3cyclictest0-21swapper/022:24:170
128299962,2cyclictest19982-21systemd-journal20:00:010
128299962,1cyclictest0-21swapper/022:55:170
128299962,1cyclictest0-21swapper/022:45:060
128299962,1cyclictest0-21swapper/021:45:220
128299962,1cyclictest0-21swapper/000:29:100
128299962,1cyclictest0-21swapper/000:16:120
128299961,4cyclictest24204-21wc21:25:250
128299961,2cyclictest3315-21diskmemload21:40:180
128299960,5cyclictest3319-21perf21:10:000
128299960,3cyclictest13613-21rm21:17:090
128439952,2cyclictest3758-21snmpd19:25:093
128439952,2cyclictest25704-21sed19:35:233
128439952,2cyclictest20747-21cat20:35:243
128389953,1cyclictest27358-21acpi19:40:202
128389952,2cyclictest29983-21sh19:45:212
128389952,2cyclictest23187-21cat19:30:252
128389950,4cyclictest4256-21diskstats20:00:192
128389950,2cyclictest29762-21find21:30:162
128349952,2cyclictest508-21dbus-daemon20:30:011
128349952,2cyclictest32178-21expr19:50:191
128349951,3cyclictest3758-21snmpd00:30:031
128349951,3cyclictest0-21swapper/121:25:071
128299952,2cyclictest6331-21cat20:05:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional