You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 08:41
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackaslot8.osadl.org (updated Sun Apr 28, 2024 00:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
264182300266,22sleep10-21swapper/119:08:311
261962274244,19sleep00-21swapper/019:05:530
265282262225,22sleep20-21swapper/219:09:572
263532258227,19sleep30-21swapper/319:07:413
125502650,0sleep20-21swapper/222:10:242
35342250,0sleep20-21swapper/222:46:072
256312210,1sleep30-21swapper/300:03:453
8942200,4sleep228-21ksoftirqd/223:05:212
138222180,1sleep10-21swapper/122:53:151
291432170,0sleep10-21swapper/123:45:011
91042160,0sleep30-21swapper/321:48:023
124742160,0sleep00-21swapper/000:16:590
236762150,0sleep20-21swapper/200:25:142
151222150,0sleep30-21swapper/323:56:003
214442140,0sleep00-21swapper/022:37:000
26649991312,1cyclictest34-21ksoftirqd/319:15:013
275202120,0sleep10-21swapper/100:27:491
2664999129,2cyclictest34-21ksoftirqd/322:40:003
26649991210,1cyclictest34-21ksoftirqd/321:40:013
2664099127,4cyclictest22-21ksoftirqd/121:10:001
17362120,0sleep20-21swapper/223:26:362
2664999118,2cyclictest34-21ksoftirqd/323:25:003
2664999118,2cyclictest34-21ksoftirqd/321:25:003
2664999118,2cyclictest34-21ksoftirqd/320:20:013
2664999118,2cyclictest34-21ksoftirqd/319:55:013
2664999113,5cyclictest502-21systemd-logind21:53:213
26649991110,1cyclictest34-21ksoftirqd/320:40:003
144362110,4sleep334-21ksoftirqd/321:30:233
2664999109,1cyclictest34-21ksoftirqd/321:45:013
2664999109,1cyclictest34-21ksoftirqd/300:10:013
2664999109,1cyclictest34-21ksoftirqd/300:10:013
2664999107,2cyclictest34-21ksoftirqd/321:20:003
2664999107,2cyclictest34-21ksoftirqd/321:00:003
2664999107,2cyclictest34-21ksoftirqd/320:30:013
2664999107,2cyclictest34-21ksoftirqd/319:50:013
266499998,1cyclictest34-21ksoftirqd/323:00:003
266499998,1cyclictest34-21ksoftirqd/322:25:003
266499998,1cyclictest34-21ksoftirqd/321:30:003
266499998,1cyclictest34-21ksoftirqd/321:15:003
266499998,1cyclictest34-21ksoftirqd/319:40:013
266499997,2cyclictest34-21ksoftirqd/323:10:003
266499997,2cyclictest34-21ksoftirqd/322:10:013
266499997,2cyclictest34-21ksoftirqd/319:30:013
266499996,2cyclictest34-21ksoftirqd/323:50:013
266499996,2cyclictest34-21ksoftirqd/321:09:593
266499996,2cyclictest34-21ksoftirqd/320:25:013
266499996,2cyclictest34-21ksoftirqd/319:20:013
266499996,2cyclictest34-21ksoftirqd/300:35:003
266499996,2cyclictest34-21ksoftirqd/300:25:013
266449996,2cyclictest508-21dbus-daemon20:10:012
266389994,3cyclictest19982-21systemd-journal20:43:400
266499987,1cyclictest34-21ksoftirqd/320:15:013
266499987,1cyclictest34-21ksoftirqd/320:15:013
266499986,2cyclictest34-21ksoftirqd/322:00:013
266499986,2cyclictest34-21ksoftirqd/320:02:173
266499986,1cyclictest34-21ksoftirqd/319:35:013
266499985,2cyclictest34-21ksoftirqd/322:15:013
266499984,3cyclictest19982-21systemd-journal22:45:583
266499984,3cyclictest19982-21systemd-journal22:05:013
266499983,3cyclictest19982-21systemd-journal22:50:553
266499983,3cyclictest19982-21systemd-journal19:44:013
266499982,4cyclictest34-21ksoftirqd/323:04:483
266499982,4cyclictest34-21ksoftirqd/322:15:113
266499982,4cyclictest34-21ksoftirqd/320:50:453
266499982,4cyclictest34-21ksoftirqd/300:35:573
266499982,4cyclictest34-21ksoftirqd/300:26:523
266499981,5cyclictest34-21ksoftirqd/323:26:043
266499981,5cyclictest34-21ksoftirqd/323:15:213
266499981,5cyclictest34-21ksoftirqd/322:25:173
266499981,4cyclictest19982-21systemd-journal00:17:413
266499980,5cyclictest34-21ksoftirqd/322:31:593
266449984,3cyclictest508-21dbus-daemon20:26:592
266449984,3cyclictest19982-21systemd-journal19:46:542
266449984,2cyclictest19982-21systemd-journal23:57:402
266449983,3cyclictest19982-21systemd-journal22:17:082
266449983,3cyclictest19982-21systemd-journal20:50:452
266409985,2cyclictest508-21dbus-daemon20:16:581
266409985,2cyclictest13388-21snmpd22:02:121
266409984,3cyclictest19982-21systemd-journal19:11:581
266409983,3cyclictest19982-21systemd-journal19:20:581
266389985,2cyclictest12-21ksoftirqd/021:20:000
266389984,3cyclictest19982-21systemd-journal21:51:160
266389984,2cyclictest19982-21systemd-journal20:35:440
266389983,3cyclictest19982-21systemd-journal21:45:000
266389983,3cyclictest19982-21systemd-journal00:35:000
266389982,3cyclictest5833-21turbostat23:30:000
266389980,3cyclictest1338850snmpd00:05:310
266389980,3cyclictest1338850snmpd00:05:310
266389980,2cyclictest22140-21sh22:58:420
12628280,0sleep10-21swapper/100:17:081
26792270,0sleep00-21swapper/022:20:110
266499976,1cyclictest34-21ksoftirqd/323:42:393
266499973,3cyclictest508-21dbus-daemon20:44:303
266499973,3cyclictest12634-21cat19:55:113
266499973,1cyclictest0-21swapper/323:51:193
266499972,4cyclictest6305-21apt-get23:30:123
266499972,1cyclictest141rcu_preempt20:48:153
266499971,4cyclictest9386-21ls21:00:153
266499971,4cyclictest508-21dbus-daemon20:32:493
266499970,5cyclictest34-21ksoftirqd/323:13:263
266499970,5cyclictest34-21ksoftirqd/322:43:073
266499970,5cyclictest17159-21cut20:05:163
266499970,3cyclictest161262sh23:35:393
266449976,1cyclictest28-21ksoftirqd/200:35:002
266449974,2cyclictest11736-21in:imuxsock21:02:012
266449973,3cyclictest19982-21systemd-journal22:31:582
266449973,3cyclictest19982-21systemd-journal20:01:192
266449973,3cyclictest19116-21gdbus21:10:002
266449973,3cyclictest0-21swapper/223:39:592
266449973,1cyclictest0-21swapper/222:57:092
266449973,1cyclictest0-21swapper/221:55:212
266449973,1cyclictest0-21swapper/200:19:412
266449972,3cyclictest19982-21systemd-journal20:40:452
266449972,1cyclictest22902-21idleruntime-cro20:20:012
266409976,1cyclictest0-21swapper/122:09:271
266409975,1cyclictest8164-21cron19:40:001
266409974,2cyclictest11736-21in:imuxsock22:43:491
266409973,3cyclictest8041-21awk23:30:231
266409973,3cyclictest508-21dbus-daemon21:12:271
266409973,3cyclictest1934-21apt-get20:45:101
266409973,3cyclictest13388-21snmpd20:01:411
266409973,3cyclictest13388-21snmpd19:55:501
266409973,3cyclictest0-21swapper/122:21:481
266409973,1cyclictest0-21swapper/123:48:101
266409973,1cyclictest0-21swapper/122:48:491
266409972,3cyclictest3682-21idleruntime-cro21:45:011
266409970,7cyclictest0-21swapper/122:30:301
266389974,2cyclictest502-21systemd-logind22:13:220
266389973,3cyclictest21747-21memory21:35:190
266389973,3cyclictest19982-21systemd-journal22:46:450
266389973,3cyclictest19982-21systemd-journal22:00:010
266389973,3cyclictest19982-21systemd-journal21:27:570
266389973,3cyclictest19982-21systemd-journal20:13:130
266389973,3cyclictest19982-21systemd-journal20:13:130
266389973,3cyclictest16969-21lldpd23:36:210
266389973,3cyclictest13388-21snmpd20:31:210
266389973,3cyclictest0-21swapper/021:49:060
266389973,1cyclictest0-21swapper/022:08:550
266389972,3cyclictest19982-21systemd-journal22:17:330
266389972,3cyclictest19982-21systemd-journal21:12:570
266389970,2cyclictest24123-21ssh23:40:490
266499963,1cyclictest0-21swapper/300:10:413
266499962,3cyclictest13388-21snmpd19:21:463
266449966,0cyclictest0-21swapper/221:14:512
266449964,1cyclictest27895-21sshd23:01:592
266449964,1cyclictest11736-21in:imuxsock20:21:232
266449963,2cyclictest8762-21ssh23:10:242
266449963,2cyclictest4033-21sh21:45:082
266449963,2cyclictest23937-21sh23:20:182
266449963,2cyclictest21288-21rm21:15:042
266449963,2cyclictest20852-21ssh00:00:152
266449963,2cyclictest19982-21systemd-journal21:20:012
266449963,2cyclictest19982-21systemd-journal20:30:442
266449963,2cyclictest17772-21dpkg22:35:112
266449963,2cyclictest16030-21sh00:20:072
266449963,2cyclictest14024-21rm21:30:192
266449963,2cyclictest13388-21snmpd23:51:412
266449963,2cyclictest10113-21ssh22:09:352
266449963,2cyclictest0-21swapper/222:20:412
266449962,3cyclictest5964-21wc21:25:182
266449962,3cyclictest5901-21seq19:34:332
266449962,3cyclictest5397-21cat00:35:152
266449962,3cyclictest508-21dbus-daemon22:51:202
266449962,3cyclictest29651-21ls19:15:022
266449962,3cyclictest2215-21expr00:10:112
266449962,3cyclictest13388-21snmpd23:41:382
266449962,3cyclictest13388-21snmpd23:30:322
266449962,3cyclictest13388-21snmpd22:27:572
266449962,3cyclictest13388-21snmpd21:35:342
266449962,3cyclictest13388-21snmpd20:55:362
266449962,3cyclictest13388-21snmpd20:13:562
266449962,3cyclictest13388-21snmpd20:13:562
266449962,3cyclictest13388-21snmpd19:51:172
266449962,3cyclictest13388-21snmpd19:43:172
266449962,3cyclictest13388-21snmpd19:20:012
266449962,3cyclictest13388-21snmpd00:05:062
266449962,3cyclictest13388-21snmpd00:05:052
266449962,3cyclictest12245-21expr21:50:132
266449962,1cyclictest0-21swapper/223:15:542
266449961,3cyclictest19982-21systemd-journal19:58:122
266449961,2cyclictest26178-21grep22:40:182
266449960,5cyclictest30253-21ssh23:45:142
266449960,5cyclictest19982-21systemd-journal20:47:502
266449960,3cyclictest3840-21cat19:30:012
266449960,3cyclictest29634-21ssh22:00:292
266449960,2cyclictest29732-21ntp_states21:40:182
266449960,1cyclictest28405-21users19:10:182
266409964,2cyclictest22-21ksoftirqd/121:53:201
266409964,1cyclictest309312sleep123:25:111
266409964,1cyclictest17034-21cut22:55:141
266409964,1cyclictest14382-21switchtime21:30:231
266409963,2cyclictest9478-21ssh23:10:431
266409963,2cyclictest6294-21sh00:35:251
266409963,2cyclictest5867-21sshd20:52:031
266409963,2cyclictest31680-21sshd20:37:571
266409963,2cyclictest27648-21cat00:05:131
266409963,2cyclictest27648-21cat00:05:131
266409963,2cyclictest20178-21grep20:10:271
266409963,2cyclictest20178-21grep20:10:271
266409963,2cyclictest18477-21seq23:16:451
266409963,2cyclictest13418-21ssh22:10:511
266409963,2cyclictest11736-21in:imuxsock20:44:551
266409963,1cyclictest0-21swapper/121:46:451
266409963,1cyclictest0-21swapper/121:20:301
266409963,1cyclictest0-21swapper/100:20:401
266409962,3cyclictest32289-21cat23:05:121
266409962,3cyclictest32183-21ssh00:30:361
266409962,3cyclictest29741-11brltty22:35:041
266409962,3cyclictest20608-21grep22:15:231
266409962,3cyclictest13388-21snmpd21:55:031
266409962,3cyclictest13388-21snmpd21:15:311
266409962,3cyclictest13388-21snmpd20:28:001
266409962,3cyclictest13388-21snmpd19:33:401
266409962,3cyclictest0-21swapper/123:35:461
266409962,3cyclictest0-21swapper/122:26:581
266409962,3cyclictest0-21swapper/100:10:321
266409962,1cyclictest0-21swapper/123:55:271
266409962,1cyclictest0-21swapper/123:50:041
266409962,1cyclictest0-21swapper/121:37:241
266409961,3cyclictest19982-21systemd-journal23:00:311
266409961,3cyclictest19982-21systemd-journal20:57:011
266409961,2cyclictest5803-21ssh21:25:161
266409961,2cyclictest19982-21systemd-journal00:00:011
266409960,6cyclictest0-21swapper/123:20:101
266409960,6cyclictest0-21swapper/120:06:441
266409960,6cyclictest0-21swapper/119:16:501
266389964,1cyclictest6446-21users00:35:250
266389963,2cyclictest668-21runrttasks19:49:010
266389963,2cyclictest508-21dbus-daemon19:32:450
266389963,2cyclictest3507-21rt-features22:25:200
266389963,2cyclictest3447-21sshd19:26:500
266389963,2cyclictest31203-21tail19:15:220
266389963,2cyclictest29741-11brltty23:50:590
266389963,2cyclictest28577-21ssh22:41:260
266389963,2cyclictest27520-21sort20:30:010
266389963,2cyclictest22959-21ssh00:01:110
266389963,2cyclictest2026-21ssh23:48:020
266389963,2cyclictest1-21systemd19:20:140
266389962,3cyclictest7194-21load20:55:270
266389962,3cyclictest3145-21swap20:45:190
266389962,3cyclictest29741-11brltty21:07:020
266389962,3cyclictest27715-21date19:10:130
266389962,3cyclictest15773-21basename23:15:150
266389962,3cyclictest13388-21snmpd23:00:160
266389962,3cyclictest13388-21snmpd21:01:450
266389962,3cyclictest13388-21snmpd20:23:140
266389962,3cyclictest13388-21snmpd19:53:230
266389962,3cyclictest0-21swapper/023:55:300
266389962,3cyclictest0-21swapper/023:11:320
266389962,3cyclictest0-21swapper/021:30:030
266389962,3cyclictest0-21swapper/021:20:340
266389962,2cyclictest502-21systemd-logind20:02:170
266389962,1cyclictest0-21swapper/022:01:190
266389961,3cyclictest19982-21systemd-journal23:05:060
266389960,5cyclictest9990-21which22:50:220
266389960,5cyclictest6272-21/usr/sbin/munin23:30:220
266389960,5cyclictest22847-21/usr/sbin/munin23:20:190
266389960,5cyclictest10830-21sed22:30:170
266389960,3cyclictest25015-21rm00:25:260
266389960,0cyclictest12811-21diskmemload00:11:430
266389960,0cyclictest0-21swapper/000:20:300
266449952,2cyclictest13388-21snmpd19:35:132
266449950,4cyclictest29897-21apt-get20:35:142
266409952,2cyclictest8488-21sshd19:40:571
266409952,2cyclictest27861-21cat20:30:101
266409952,2cyclictest23932-21latency20:20:181
266409952,2cyclictest1889-21expr19:25:121
266409951,3cyclictest9213-21expr19:45:091
266409950,2cyclictest9396-21sed21:00:151
266409950,2cyclictest11703-21latency_hist19:55:011
266389953,1cyclictest4398-21expr20:50:180
266389952,2cyclictest16611-21cut20:05:110
266389952,2cyclictest13388-21snmpd19:55:080
266389951,3cyclictest6488-21expr19:35:130
266389951,3cyclictest21527-21/usr/sbin/munin20:15:160
266389950,3cyclictest19982-21systemd-journal19:41:570
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional