You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 15:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8s.osadl.org (updated Sun Apr 28, 2024 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5970512780,1sleep3481rcuc/321:25:203
5766292580,0sleep0576628-21open_inodes21:10:270
7052042490,1sleep10-21swapper/122:45:231
4749492330,1chrt474945-21munin-run19:40:000
436353992825,2cyclictest806629-21systemd00:00:001
436356992725,1cyclictest468228-21lspci19:34:162
436351992620,5cyclictest799919-21kworker/u8:4-nfsiod00:00:000
6650802250,4sleep2391rcuc/222:15:172
436358992515,3cyclictest512-21rs:main3
436358992513,3cyclictest851325-21kthreadcore00:30:213
436356992522,2cyclictest505962-21sh20:15:022
436353992521,1cyclictest191rcu_preempt23:15:481
436358992414,2cyclictest533-21gdbus00:05:003
43635899239,3cyclictest0-21swapper/320:35:003
43635699233,2cyclictest518251-21cat20:25:002
436353992318,3cyclictest0-21swapper/119:15:011
436353992218,1cyclictest191rcu_preempt22:17:491
436353992218,1cyclictest191rcu_preempt21:22:271
436353992217,4cyclictest640546-21munin-run22:00:001
436353992216,1cyclictest191rcu_preempt23:11:141
436358992117,3cyclictest533-21gdbus20:30:003
436358992115,4cyclictest682579-21sed22:30:173
436358992111,2cyclictest0-21swapper/322:00:013
436356992118,2cyclictest473606-21tr19:35:192
436353992118,2cyclictest546-21ModemManager19:35:151
436353992117,1cyclictest191rcu_preempt22:05:251
436351992118,2cyclictest730518-21cron23:05:000
436356992016,3cyclictest620017-21idleruntime-cro21:45:002
43635399201,7cyclictest796771-21cut23:50:171
43635399201,7cyclictest796771-21cut23:50:171
436353992016,1cyclictest191rcu_preempt23:21:481
436353992016,1cyclictest191rcu_preempt00:30:491
269698200,3rtkit-daemon707-21ntpd21:15:192
436358991916,2cyclictest709812-21gpgconf22:50:003
436358991916,2cyclictest695847-21apt-key22:40:013
436358991916,2cyclictest498-21NetworkManager19:34:473
436358991915,2cyclictest774916-21kthreadcore23:35:153
436356991916,2cyclictest709693-21turbostat.cron22:50:002
43635399191,6cyclictest482770-21basename19:55:101
436353991915,3cyclictest2609-21cron23:30:001
436353991915,1cyclictest191rcu_preempt22:43:151
43635399191,13cyclictest725399-21grep23:00:171
436351991916,2cyclictest599339-21idleruntime-cro21:30:010
436351991916,2cyclictest594-21gdbus19:35:010
436351991916,2cyclictest2609-21cron23:45:000
436351991915,2cyclictest0-21swapper/019:30:000
436358991814,3cyclictest510-21in:imuxsock20:06:543
436358991814,2cyclictest442713-21irqcore19:15:013
436358991811,5cyclictest506-21systemd-logind20:19:233
436356991814,3cyclictest503-21polkitd20:19:232
436356991813,3cyclictest0-21swapper/200:00:012
43635699180,17cyclictest481490-21munin-run19:50:002
43635399189,4cyclictest0-21swapper/122:27:031
43635399183,4cyclictest782687-21cut23:40:201
43635399183,4cyclictest521142-21cut20:25:201
43635399182,5cyclictest825572-21mailstats00:10:261
43635399182,5cyclictest495650-21uname20:05:161
43635399182,5cyclictest494202-21latency_hist20:05:011
43635399182,4cyclictest856726-21grep00:35:201
43635399182,4cyclictest669718-21irqrtprio22:20:171
436353991815,2cyclictest59322-21packagekitd20:19:351
436353991812,5cyclictest476797-21cut19:45:151
43635399180,6cyclictest596066-21kthreadcore21:25:191
436351991812,4cyclictest585219-21cron21:20:000
43635199180,2cyclictest499963-21cron20:10:010
269698180,1rtkit-daemon0-21swapper/100:22:551
436358991714,2cyclictest0-21swapper/322:10:003
436358991714,2cyclictest0-21swapper/320:15:013
436358991713,3cyclictest718526-21kthreadcore22:55:163
436358991713,2cyclictest0-21swapper/323:25:173
436356991714,2cyclictest855032-21idleruntime-cro00:35:002
436356991714,2cyclictest696210-21gdbus22:40:012
436356991714,2cyclictest633640-21sh21:55:012
436356991712,3cyclictest797497-21kthreadcore23:50:172
436356991712,3cyclictest797497-21kthreadcore23:50:172
43635399172,6cyclictest777737-21runrttasks23:36:281
43635399172,5cyclictest789001-21cat23:45:171
43635399172,5cyclictest505530-21grep20:10:301
43635399172,4cyclictest769974-21awk23:30:241
43635399171,6cyclictest816970-21cut00:05:211
43635399171,4cyclictest737229-21runrttasks23:09:311
436353991714,2cyclictest436344-21cyclictest20:25:001
436351991713,3cyclictest487595-21unixbench_multi19:55:350
436351991713,3cyclictest0-21swapper/023:25:010
436351991713,2cyclictest0-21swapper/022:50:000
6800682160,2chrt481rcuc/322:25:203
436358991613,2cyclictest0-21swapper/320:45:003
43635699161,8cyclictest827188-21idleruntime-cro00:15:012
436356991612,3cyclictest731165-21gltestperf23:05:162
436356991612,3cyclictest561280-21kthreadcore21:00:162
436356991612,3cyclictest493032-21needreboot20:00:232
436356991610,5cyclictest533-21gdbus23:15:232
43635399167,1cyclictest0-21swapper/120:50:281
43635399163,3cyclictest690655-21cat22:35:201
43635399162,5cyclictest530325-21/usr/sbin/munin20:35:261
43635399162,3cyclictest632055-21sed21:50:241
43635399161,5cyclictest468623-21sshd19:34:571
43635399161,4cyclictest455783-21runrttasks19:21:561
436353991613,2cyclictest559625-21munin-run21:00:011
436353991612,3cyclictest832817-21unixbench_singl00:15:281
436353991612,1cyclictest191rcu_preempt21:41:001
436351991613,2cyclictest737358-21turbostat23:10:010
436351991613,2cyclictest533-21gdbus23:55:000
436351991613,2cyclictest533-21gdbus23:55:000
436351991613,2cyclictest518672-21fschecks_time20:25:150
436351991613,2cyclictest436344-21cyclictest22:55:250
436351991612,2cyclictest621710-21kthreadcore21:45:160
436351991612,2cyclictest0-21swapper/022:20:130
5182502150,1sleep3481rcuc/320:25:003
43635899157,5cyclictest495-21dbus-daemon00:00:003
436358991512,2cyclictest843117-21kthreadcore00:25:193
436358991511,3cyclictest661646-21cpuspeed_turbos22:15:113
436358991511,3cyclictest567198-21kthreadcore21:05:193
436358991511,2cyclictest535973-21sh20:40:003
436358991511,2cyclictest474589-21kworker/u8:0-rpciod21:10:213
436358991511,2cyclictest0-21swapper/319:25:193
436358991511,2cyclictest0-21swapper/300:15:263
436358991510,4cyclictest696619-21grep22:40:173
43635699159,4cyclictest820285-21cron00:10:002
436356991512,2cyclictest442264-21latency_hist19:15:012
436356991512,2cyclictest0-21swapper/221:40:002
436356991512,2cyclictest0-21swapper/221:00:002
436356991511,3cyclictest571453-21diskmemload00:35:212
436356991511,2cyclictest0-21swapper/222:30:292
436356991510,3cyclictest0-21swapper/222:05:002
43635399155,3cyclictest0-21swapper/122:30:141
43635399152,3cyclictest845209-21cut00:25:211
43635399152,2cyclictest577375-21ssh21:11:191
43635399151,4cyclictest541960-21latency_hist20:45:011
43635399151,4cyclictest461794-21runrttasks19:27:541
436353991512,1cyclictest191rcu_preempt22:50:161
43635399150,3cyclictest599631-21apt-get21:30:101
43635399150,3cyclictest475608-21munin-run19:45:001
43635199159,4cyclictest592221-21systemd-run21:25:000
436351991512,2cyclictest701098-21ntp_kernel_pll_22:40:250
436351991512,2cyclictest571453-21diskmemload22:15:270
436351991512,2cyclictest0-21swapper/000:05:170
436351991511,3cyclictest860650-21unixbench_multi00:35:300
436351991511,3cyclictest710334-21cpuspeed_turbos22:50:130
436351991511,3cyclictest565148-21unixbench_singl21:00:260
436351991511,2cyclictest524775-21gltestperf20:30:120
436351991510,3cyclictest0-21swapper/020:40:000
436358991410,3cyclictest730908-21cpuspeed_turbos23:05:133
436358991410,3cyclictest613534-21cpuspeed_turbos21:40:113
436358991410,2cyclictest0-21swapper/323:05:003
43635699149,4cyclictest835354-21sed00:20:152
43635699148,4cyclictest456024-21sshd19:24:582
43635699147,5cyclictest461972-21ls19:30:002
436356991411,2cyclictest827733-21fschecks_time00:15:132
436356991411,2cyclictest0-21swapper/223:40:002
436356991411,2cyclictest0-21swapper/221:45:162
436356991411,2cyclictest0-21swapper/221:10:302
436356991410,3cyclictest784274-21unixbench_multi23:40:282
436356991410,3cyclictest0-21swapper/223:35:002
436356991410,2cyclictest0-21swapper/223:00:262
436356991410,2cyclictest0-21swapper/220:06:542
43635699140,2cyclictest0-21swapper/222:10:182
43635399144,6cyclictest0-21swapper/120:31:261
43635399143,6cyclictest584290-21chrt21:16:321
43635399143,4cyclictest32-21ksoftirqd/121:08:411
43635399143,3cyclictest623832-21cut21:45:171
43635399143,3cyclictest446316-21cut19:15:171
43635399142,3cyclictest659521-21cat22:10:231
43635399142,3cyclictest0-21swapper/122:00:031
43635399141,4cyclictest565223-21runrttasks21:01:081
436353991412,1cyclictest739-21runrttasks21:39:001
436353991411,2cyclictest807855-21sed00:00:201
436353991411,2cyclictest542215-21cpuspeed_turbos20:45:111
43635199149,4cyclictest511784-21kworker/0:3+kacpi_notify21:57:320
43635199149,4cyclictest441231-21needreboot19:10:150
43635199149,4cyclictest353-21systemd-journal22:31:550
436351991411,2cyclictest772347-21gltestperf23:35:120
436351991411,2cyclictest512-21rs:main0
436351991411,2cyclictest0-21swapper/022:40:010
436351991411,2cyclictest0-21swapper/021:55:000
436351991411,2cyclictest0-21swapper/021:31:520
436351991410,3cyclictest456037-21idleruntime-cro19:25:000
436351991410,2cyclictest0-21swapper/020:20:000
43635199140,4cyclictest495-21dbus-daemon23:15:230
43635899139,3cyclictest633636-21cron21:55:013
43635899139,2cyclictest625016-21tail21:45:183
43635899139,2cyclictest0-21swapper/321:05:003
43635899138,4cyclictest797998-21ntpq23:50:213
43635899138,4cyclictest797998-21ntpq23:50:213
436358991310,2cyclictest855675-21grep00:35:163
436358991310,2cyclictest834487-21cpuspeed_turbos00:20:103
436358991310,2cyclictest768010-21tr23:30:223
436358991310,2cyclictest0-21swapper/320:55:283
436358991310,2cyclictest0-21swapper/319:45:253
43635699139,3cyclictest689168-21cpuspeed_turbos22:35:122
43635699139,3cyclictest634024-21cpuspeed_turbos21:55:142
43635699139,3cyclictest475625-21systemd-run19:45:012
43635699139,2cyclictest0-21swapper/223:25:142
43635699138,3cyclictest570500-21head21:05:222
43635699138,3cyclictest0-21swapper/220:30:012
43635699137,4cyclictest543006-21cut20:45:152
43635699137,4cyclictest1-21systemd22:55:002
43635699131,3cyclictest721185-21ssh22:55:182
436356991310,2cyclictest592828-21cpuspeed_turbos21:25:112
436356991310,2cyclictest491-21avahi-daemon22:30:012
436356991310,2cyclictest482849-21sed19:55:122
43635399133,6cyclictest0-21swapper/122:55:221
43635399131,3cyclictest482125-21latency_hist19:55:011
43635199139,3cyclictest846913-21unixbench_singl00:25:280
43635199139,3cyclictest61650irq/128-eno1:rx-020:55:220
43635199131,4cyclictest2695-21rtkit-daemon00:00:030
436351991311,1cyclictest511784-21kworker/0:3+kacpid20:46:530
436351991311,1cyclictest511784-21kworker/0:3+kacpid00:16:500
436351991311,1cyclictest401721-21kworker/0:2+kacpid20:15:010
436351991310,2cyclictest680612-21unixbench_multi22:25:270
436351991310,2cyclictest613171-21sendmail21:40:000
436351991310,2cyclictest548090-21cpuspeed_turbos20:50:180
436351991310,2cyclictest401721-21kworker/0:2+kacpid19:15:190
436351991310,2cyclictest0-21swapper/021:40:130
436351991310,2cyclictest0-21swapper/020:45:000
43635899129,2cyclictest670187-21kthreadcore22:20:163
43635899129,2cyclictest0-21swapper/320:50:323
43635899129,2cyclictest0-21swapper/320:05:013
43635899128,3cyclictest475592-21cron19:45:013
43635899128,3cyclictest0-21swapper/319:10:123
43635899127,4cyclictest742060-21tr23:10:243
43635899127,3cyclictest565544-21kworker/u8:2+nfsiod21:15:103
43635899126,4cyclictest825604-21python300:10:263
43635899126,4cyclictest644224-21tr22:00:203
43635899126,4cyclictest495-21dbus-daemon23:40:253
43635899120,5cyclictest745015-21ps23:15:163
43635699129,2cyclictest751703-21fschecks_time23:20:142
43635699129,2cyclictest571453-21diskmemload21:25:002
43635699129,2cyclictest536331-21cpuspeed_turbos20:40:132
43635699128,3cyclictest789032-21kthreadcore23:45:182
43635699127,4cyclictest1-21systemd20:55:002
43635199129,2cyclictest513353-21irqrtprio20:20:180
43635199129,2cyclictest511784-21kworker/0:3+kacpi_notify00:20:150
43635199129,2cyclictest0-21swapper/022:10:190
43635199128,3cyclictest488427-21cpuspeed_turbos20:00:150
43635199124,8cyclictest171ktimers/022:00:360
436351991211,1cyclictest401721-21kworker/0:2+kacpid19:47:360
43635199120,2cyclictest737678-21apt-get23:10:130
43635899119,1cyclictest0-21swapper/321:35:143
43635899117,2cyclictest0-21swapper/322:50:123
43635899117,2cyclictest0-21swapper/319:55:343
43635899115,4cyclictest353-21systemd-journal20:50:003
43635899114,6cyclictest592177-21cron21:25:013
43635699118,2cyclictest0-21swapper/223:10:152
43635699118,2cyclictest0-21swapper/200:25:132
43635699116,4cyclictest525600-21sed20:30:142
43635699115,5cyclictest812216-21head00:00:302
43635699115,5cyclictest353-21systemd-journal21:31:522
43635699115,4cyclictest651587-21tr22:05:202
43635699110,3cyclictest0-21swapper/222:23:422
43635199118,2cyclictest511784-21kworker/0:3+kacpi_notify23:45:200
43635199115,4cyclictest1-21systemd19:55:010
43635199110,3cyclictest825537-21mailstats00:10:250
43635899107,2cyclictest654800-21cpuspeed_turbos22:10:123
43635899107,2cyclictest0-21swapper/323:45:183
43635899106,3cyclictest751630-21df_inode23:20:123
43635899105,4cyclictest814650-21sed00:05:193
43635899105,4cyclictest601117-21tr21:30:173
43635899105,4cyclictest454903-21tr19:20:203
43635699106,3cyclictest442456-21interrupts19:15:012
43635199107,2cyclictest848754-21fschecks_time00:30:150
43635199107,2cyclictest0-21swapper/023:25:130
43635199106,3cyclictest648801-21cut22:05:170
43635199100,8cyclictest475813-21cat19:45:010
43635199100,10cyclictest0-21swapper/021:05:020
4353582101,5sleep0435482-21python319:05:190
4363589996,2cyclictest0-21swapper/319:52:583
4363569996,2cyclictest0-21swapper/220:35:162
26969890,5rtkit-daemon442303-21munin-node19:54:472
4363569981,5cyclictest436354-21latency_hist19:10:012
4363539982,4cyclictest436412-21wc19:10:011
4363589970,5cyclictest436375-21latency_hist19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional