You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-12-10 - 05:44
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack0slot2.osadl.org (updated Tue Dec 10, 2024 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
325042226353,6sleep70-21swapper/719:06:217
371059624846,1sleep70-21swapper/722:48:137
357508424846,1sleep70-21swapper/721:38:437
354399324846,1sleep70-21swapper/721:23:217
336845424846,1sleep70-21swapper/720:01:177
333738424846,1sleep70-21swapper/719:45:307
328550024846,1sleep70-21swapper/719:18:147
327526824846,1sleep70-21swapper/719:15:427
348193724745,1sleep70-21swapper/720:53:167
2708189390,32rtkit-daemon0-21swapper/419:02:014
330641723432,1sleep70-21swapper/719:30:327
325039223321,8sleep60-21swapper/619:05:566
380359023129,1sleep70-21swapper/723:32:437
347178522927,1sleep70-21swapper/720:50:367
325014422918,8sleep50-21swapper/519:02:265
368987322523,1sleep70-21swapper/722:37:567
3250689992218,3cyclictest3451600-21latency_hist20:41:295
3250685992220,2cyclictest3763234-21kworker/u16:2+events_unbound23:31:424
3250689992119,2cyclictest3638812-21kworker/u16:1+flush-8:022:32:005
3250691992016,3cyclictest3451979-21kworker/u16:1+flush-8:021:26:296
3250685992017,2cyclictest3813952-21kworker/u16:1+events_unbound23:56:464
3250695991917,1cyclictest3679800-21turbostat22:36:287
325069199190,15cyclictest0-21swapper/620:55:446
3250689991916,3cyclictest3451979-21kworker/u16:1+flush-8:021:06:595
3250689991916,2cyclictest3307135-21kworker/u16:0+flush-8:019:56:555
3250695991817,0cyclictest0-21swapper/723:02:167
3250691991817,1cyclictest0-21swapper/623:02:156
325068599180,17cyclictest3473148-21fschecks_count20:51:444
3250685991715,2cyclictest3638812-21kworker/u16:1+events_unbound22:51:304
3250685991715,2cyclictest3503610-21kworker/u16:4+events_unbound21:11:424
3250685991715,2cyclictest3295975-21kworker/u16:4+events_unbound19:41:444
3250695991615,0cyclictest0-21swapper/723:58:327
325069599160,14cyclictest3430871-21turbostat20:31:297
325068599164,5cyclictest0-21swapper/419:11:474
3250685991614,2cyclictest3628073-21kworker/u16:0+flush-8:023:16:554
3250685991614,2cyclictest3623023-21kworker/u16:0+events_unbound19:18:474
3250695991513,1cyclictest0-21swapper/720:57:057
3250691991511,3cyclictest26754-21gmain20:56:556
325069199150,10cyclictest0-21swapper/623:31:436
3250685991513,2cyclictest3523362-21kworker/u16:0+events_unbound21:22:394
3250685991513,2cyclictest3472583-21kworker/u16:2+events_unbound20:59:334
3250695991413,0cyclictest0-21swapper/719:36:297
3250695991412,1cyclictest0-21swapper/723:21:087
3250695991412,1cyclictest0-21swapper/722:51:437
325069199146,8cyclictest0-21swapper/623:10:096
325069199145,8cyclictest0-21swapper/622:39:186
3250691991412,1cyclictest3259243-21kworker/u16:3+events_unbound19:11:316
3250691991411,2cyclictest3638812-21kworker/u16:1+flush-8:022:51:396
3250691991410,3cyclictest3430870-21latency_hist20:31:296
325069199140,1cyclictest0-21swapper/619:41:456
325068999146,7cyclictest0-21swapper/522:46:425
3250689991412,2cyclictest3523362-21kworker/u16:0+flush-8:021:21:595
3250689991412,2cyclictest3472583-21kworker/u16:2+events_unbound20:56:435
325068599140,1cyclictest0-21swapper/422:51:284
3250695991311,1cyclictest0-21swapper/723:07:577
3250695991311,1cyclictest0-21swapper/721:11:477
3250695991311,1cyclictest0-21swapper/721:06:577
3250695991311,1cyclictest0-21swapper/700:06:297
325069199135,8cyclictest0-21swapper/619:21:096
325069199130,6cyclictest0-21swapper/623:56:436
325068999138,4cyclictest3296206-21latency_hist19:26:295
325068999135,7cyclictest0-21swapper/523:35:255
3250685991311,2cyclictest3628073-21kworker/u16:0+flush-8:023:01:384
3250685991311,2cyclictest3259243-21kworker/u16:3+flush-8:019:56:494
3250695991211,0cyclictest0-21swapper/721:46:297
3250695991211,0cyclictest0-21swapper/720:46:297
3250695991210,1cyclictest0-21swapper/720:16:297
3250695991210,1cyclictest0-21swapper/720:06:297
3250691991211,1cyclictest0-21swapper/622:50:386
325069199120,12cyclictest0-21swapper/622:34:436
325068999129,3cyclictest3638812-21kworker/u16:1+flush-8:022:36:475
325068999127,4cyclictest3772847-21latency_hist23:21:285
325068999127,4cyclictest3752114-21latency_hist23:11:285
3250689991211,1cyclictest0-21swapper/500:00:195
325068999120,12cyclictest0-21swapper/520:48:355
325068999120,12cyclictest0-21swapper/519:27:145
325068599128,3cyclictest3451979-21kworker/u16:1+flush-8:020:46:404
3250685991211,1cyclictest3231716-21kworker/u16:1+events_unbound19:30:564
3250685991210,2cyclictest3659846-21kworker/u16:2+events_unbound22:33:084
3250685991210,2cyclictest3638812-21kworker/u16:1+events_unbound22:40:314
3250685991210,2cyclictest3574160-21kworker/u16:3+events_unbound21:39:374
3250685991210,2cyclictest3503610-21kworker/u16:4+events_unbound21:09:394
325069599119,1cyclictest0-21swapper/723:46:297
325069599119,1cyclictest0-21swapper/723:41:287
325069599119,1cyclictest0-21swapper/723:26:287
325069599119,1cyclictest0-21swapper/721:31:297
325069599119,1cyclictest0-21swapper/719:51:297
325069599119,1cyclictest0-21swapper/719:51:297
325069599119,1cyclictest0-21swapper/700:36:297
325069599119,1cyclictest0-21swapper/700:21:297
325069599119,1cyclictest0-21swapper/700:16:297
3250695991110,0cyclictest0-21swapper/723:56:297
3250695991110,0cyclictest0-21swapper/723:31:297
3250695991110,0cyclictest0-21swapper/723:31:297
3250695991110,0cyclictest0-21swapper/722:26:297
3250695991110,0cyclictest0-21swapper/722:21:297
3250695991110,0cyclictest0-21swapper/722:01:297
3250695991110,0cyclictest0-21swapper/700:11:297
325069199119,2cyclictest27119-21goa-identity-se21:12:136
325069199118,2cyclictest1166-21snmpd20:01:166
325069199117,3cyclictest3348170-21latency_hist19:51:296
325069199117,3cyclictest3348170-21latency_hist19:51:296
3250691991110,1cyclictest0-21swapper/623:17:556
325069199110,2cyclictest0-21swapper/619:26:486
325069199110,11cyclictest0-21swapper/620:49:146
325068999117,3cyclictest3886991-21latency_hist00:16:295
325068999110,1cyclictest0-21swapper/521:16:115
325068999110,11cyclictest0-21swapper/519:17:285
325068599119,2cyclictest3628073-21kworker/u16:0+events_unbound23:10:144
325068599118,2cyclictest3638890-21latency_hist22:16:294
325068599118,2cyclictest3623023-21kworker/u16:0+flush-8:019:11:294
325069599109,1cyclictest0-21swapper/719:11:297
325069599109,0cyclictest0-21swapper/722:31:297
325069599109,0cyclictest0-21swapper/722:11:297
325069599109,0cyclictest0-21swapper/720:41:297
325069599109,0cyclictest0-21swapper/719:41:297
325069599109,0cyclictest0-21swapper/719:41:297
325069599108,1cyclictest0-21swapper/723:16:287
325069599108,1cyclictest0-21swapper/722:16:297
325069599108,1cyclictest0-21swapper/721:46:297
325069199109,1cyclictest0-21swapper/621:37:386
325069199109,1cyclictest0-21swapper/621:08:286
325068999107,2cyclictest1166-21snmpd23:05:455
325068999101,8cyclictest0-21swapper/519:41:505
325068599108,1cyclictest3379162-21latency_hist20:06:294
32506959999,0cyclictest0-21swapper/721:06:297
32506959999,0cyclictest0-21swapper/719:26:297
32506959998,0cyclictest0-21swapper/723:01:287
32506959998,0cyclictest0-21swapper/721:56:287
32506959997,1cyclictest0-21swapper/723:51:287
32506959997,1cyclictest0-21swapper/722:46:287
32506959997,1cyclictest0-21swapper/721:51:297
32506959997,1cyclictest0-21swapper/721:21:297
32506959997,1cyclictest0-21swapper/719:56:297
32506959997,1cyclictest0-21swapper/700:31:297
32506919990,8cyclictest3876796-21latency_hist00:11:296
32506919990,8cyclictest0-21swapper/622:26:296
32506919990,1cyclictest0-21swapper/600:31:296
32506899994,4cyclictest923-21gdbus22:56:285
32506859997,2cyclictest3659846-21kworker/u16:2+flush-8:022:31:294
32506859997,1cyclictest3794207-21latency_hist23:31:294
32506859997,1cyclictest3794207-21latency_hist23:31:294
32506859997,1cyclictest0-21swapper/400:26:294
32506959987,0cyclictest0-21swapper/720:36:297
32506959986,1cyclictest0-21swapper/720:21:297
32506919986,1cyclictest0-21swapper/622:16:296
32506919986,1cyclictest0-21swapper/600:36:296
32506919986,1cyclictest0-21swapper/600:16:296
32506919985,3cyclictest3825124-21kworker/u16:0+flush-8:023:56:296
32506919984,3cyclictest86-21ksoftirqd/619:11:296
32506919984,3cyclictest3628572-21cat22:11:296
32506919983,4cyclictest639-21systemd-journal22:06:296
32506919980,7cyclictest0-21swapper/620:16:296
32506899983,4cyclictest3567519-21kthreadcore21:36:445
32506899981,3cyclictest3649159-21munin-run22:21:295
32506899980,4cyclictest3597438-21munin-run21:56:295
32506859986,1cyclictest0-21swapper/400:06:294
32506859983,4cyclictest639-21systemd-journal23:56:294
32506859980,4cyclictest3917985-21awk00:31:284
32506959977,0cyclictest0-21swapper/722:06:297
32506919976,1cyclictest0-21swapper/621:31:296
32506919975,2cyclictest0-21swapper/622:21:296
32506919975,1cyclictest0-21swapper/600:21:286
32506919974,2cyclictest3461893-21latency_hist20:46:296
32506919972,4cyclictest3835289-21cut23:51:286
32506899975,1cyclictest0-21swapper/520:21:295
32506899975,1cyclictest0-21swapper/520:06:295
32506899974,2cyclictest0-21swapper/521:46:295
32506899974,2cyclictest0-21swapper/521:46:295
32506899973,3cyclictest639-21systemd-journal19:41:295
32506899973,3cyclictest639-21systemd-journal19:41:295
32506899973,3cyclictest3907975-21sed00:26:295
32506899973,3cyclictest3876862-21sed00:11:295
32506899973,3cyclictest3473831-21kthreadcore20:51:485
32506899973,3cyclictest3389824-21ls20:11:295
32506899973,3cyclictest3264336-21grep19:11:315
32506899972,4cyclictest639-21systemd-journal20:46:305
32506859975,1cyclictest0-21swapper/423:51:294
32506859973,3cyclictest3659620-21cat22:26:294
32506859973,3cyclictest3649411-21latency_hist22:21:294
32506859973,3cyclictest3586046-21cat21:46:294
32506859973,3cyclictest3586046-21cat21:46:294
32506859973,3cyclictest3431228-21cut20:31:294
32506859973,3cyclictest3348205-21systemd-run19:51:294
32506859973,3cyclictest3348205-21systemd-run19:51:294
32506859973,3cyclictest3327651-21grep19:41:294
32506859973,3cyclictest3327651-21grep19:41:294
32506859972,4cyclictest864-21dbus-daemon21:31:294
32506859972,4cyclictest864-21dbus-daemon20:41:294
32506859970,7cyclictest0-21swapper/420:46:294
32506959966,0cyclictest0-21swapper/721:36:297
32506959966,0cyclictest0-21swapper/721:36:297
32506959964,1cyclictest0-21swapper/720:11:297
32506959964,1cyclictest0-21swapper/700:26:297
32506919965,1cyclictest3251005-21cyclictest23:31:296
32506919965,1cyclictest3251005-21cyclictest23:31:296
32506919965,1cyclictest3251005-21cyclictest19:36:296
32506919963,2cyclictest3564933-21latency_hist21:36:296
32506919963,2cyclictest3564933-21latency_hist21:36:296
32506919963,2cyclictest3503502-21sed21:06:296
32506919963,2cyclictest3358154-21latency_hist19:56:296
32506919963,2cyclictest1132-21abrt-dump-journ19:26:296
32506919962,3cyclictest3814708-21latency_hist23:41:286
32506919962,3cyclictest3701130-21sed22:46:286
32506919962,3cyclictest3586003-21basename21:46:296
32506919962,3cyclictest3586003-21basename21:46:296
32506919962,3cyclictest3451849-21cat20:41:296
32506919960,2cyclictest86-21ksoftirqd/622:01:296
32506919960,2cyclictest0-21swapper/621:56:296
32506899964,1cyclictest0-21swapper/522:01:295
32506899964,1cyclictest0-21swapper/521:21:295
32506899964,1cyclictest0-21swapper/520:31:295
32506899963,2cyclictest3700580-21latency_hist22:46:285
32506899962,3cyclictest3565494-21cat21:36:295
32506899962,3cyclictest3565494-21cat21:36:295
32506899962,3cyclictest3441457-21cat20:36:295
32506899962,3cyclictest3358754-21latency_hist19:56:295
32506899960,5cyclictest3628490-21munin-run22:11:295
32506899960,5cyclictest3261535-21munin-node19:11:295
32506899960,3cyclictest3835526-21cat23:51:295
32506899960,3cyclictest3554905-21sendmail21:31:285
32506899960,2cyclictest3731454-21latency_hist23:01:285
32506859964,1cyclictest0-21swapper/423:26:284
32506859964,1cyclictest0-21swapper/400:36:284
32506859963,2cyclictest3701047-21latency_hist22:46:284
32506859963,2cyclictest3607746-21latency_hist22:01:294
32506859963,2cyclictest3587345-21exim421:51:294
32506859962,3cyclictest3897333-21basename00:21:284
32506859962,3cyclictest3534632-21sed21:21:294
32506859962,3cyclictest3503546-21latency_hist21:06:294
32506859962,3cyclictest3441331-21date20:36:294
32506859962,3cyclictest3410408-21cat20:21:294
32506859962,3cyclictest3296781-21cat19:26:294
32506859960,5cyclictest639-21systemd-journal21:56:294
32506959954,1cyclictest0-21swapper/720:26:297
32506919953,1cyclictest0-21swapper/623:01:286
32506919953,1cyclictest0-21swapper/600:26:296
32506919951,3cyclictest3669889-21cat22:31:296
32506919950,4cyclictest3866493-21latency_hist00:06:296
32506919950,4cyclictest3825235-21sendmail23:46:296
32506919950,4cyclictest3783511-21latency_hist23:26:286
32506919950,4cyclictest3763117-21cut23:16:286
32506919950,4cyclictest3586717-21latency_hist21:51:296
32506919950,4cyclictest3534461-21sendmail21:21:296
32506919950,4cyclictest3441314-21sed20:36:296
32506919950,4cyclictest3410222-21systemd-run20:21:296
32506919950,4cyclictest3389772-21sendmail20:11:296
32506919950,4cyclictest3379394-21cat20:06:296
32506919950,4cyclictest3327689-21cut19:41:296
32506919950,4cyclictest3327689-21cut19:41:296
32506899953,1cyclictest3794187-21latency_hist23:31:285
32506899953,1cyclictest3794187-21latency_hist23:31:285
32506899953,1cyclictest3763234-21kworker/u16:2+flush-8:023:41:285
32506899953,1cyclictest0-21swapper/523:26:285
32506899950,4cyclictest969-21runrttasks00:31:285
32506899950,4cyclictest3928310-21crond00:36:285
32506899950,4cyclictest3897309-21munin-plugin-st00:21:285
32506899950,4cyclictest3866393-21awk00:06:285
32506899950,4cyclictest3845587-21bash23:56:285
32506899950,4cyclictest3825233-21cat23:46:295
32506899950,4cyclictest3763089-21cron23:16:285
32506899950,4cyclictest3670172-21sed22:31:295
32506899950,4cyclictest3659483-21basename22:26:295
32506899950,4cyclictest3618464-21latency_hist22:06:295
32506899950,4cyclictest3420220-21sh20:26:295
32506899950,4cyclictest3399872-21crond20:16:295
32506899950,4cyclictest3348173-21dump-pmu-power19:51:295
32506899950,4cyclictest3348173-21dump-pmu-power19:51:295
32506899950,4cyclictest3316805-21cron19:36:295
32506899950,4cyclictest1133-21abrt-dump-journ21:06:285
32506899950,2cyclictest3587341-21cat21:51:295
32506899950,2cyclictest0-21swapper/522:16:295
32506859953,1cyclictest3814603-21systemd-run23:41:284
32506859953,1cyclictest0-21swapper/419:56:294
32506859953,1cyclictest0-21swapper/400:11:294
32506859952,2cyclictest3825239-21latency_hist23:46:294
32506859950,4cyclictest3886979-21exim400:16:284
32506859950,4cyclictest3763092-21latency_hist23:16:284
32506859950,4cyclictest3731986-21latency_hist23:01:284
32506859950,4cyclictest3628480-21sh22:11:294
32506859950,4cyclictest3618452-21ls22:06:294
32506859950,4cyclictest3565502-21latency_hist21:36:294
32506859950,4cyclictest3565502-21latency_hist21:36:294
32506859950,4cyclictest3420869-21cat20:26:294
32506859950,4cyclictest3399884-21awk20:16:294
32506859950,4cyclictest3317161-21munin-plugin-st19:36:294
32506859950,4cyclictest1133-21abrt-dump-journ20:11:294
32506919940,3cyclictest3420867-21sendmail20:26:296
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional