You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-22 - 13:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Wed Oct 22, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
218830799747743,3cyclictest2163235-21kworker/2:0+events19:16:472
218830799743739,3cyclictest2415887-21kworker/2:1+events00:11:472
218830799743739,3cyclictest2163235-21kworker/2:0+events22:07:102
218830999742738,3cyclictest2194547-21kworker/3:3+events22:50:203
218830799742738,3cyclictest2415887-21kworker/2:1+events23:58:122
218830799742738,3cyclictest2415887-21kworker/2:1+events22:19:442
218830499742738,3cyclictest94083-21kworker/1:0+events21:22:311
218830499742737,4cyclictest2194449-21turbostat19:19:541
218830799741737,3cyclictest2415887-21kworker/2:1+events22:26:482
218830799741736,4cyclictest2163235-21kworker/2:0+events20:24:052
218830999740730,4cyclictest2194547-21kworker/3:3+events00:00:213
218830499740736,3cyclictest94083-21kworker/1:0+events21:45:201
218830999739734,4cyclictest2202773-21turbostat19:24:553
218830799739736,2cyclictest2415887-21kworker/2:1+events23:45:532
218830799739735,3cyclictest2415887-21kworker/2:1+events23:53:102
218830799739735,3cyclictest2415887-21kworker/2:1+events22:34:382
218830799739735,3cyclictest2163235-21kworker/2:0+events21:41:482
218830799739735,3cyclictest2163235-21kworker/2:0+events21:41:482
218830499739735,3cyclictest94083-21kworker/1:0+events22:01:041
218830999738733,4cyclictest2578940-21turbostat00:29:563
218830799738733,4cyclictest2415887-21kworker/2:1+events22:13:012
218830499738734,3cyclictest94083-21kworker/1:0+events21:25:191
218830799737733,3cyclictest2415887-21kworker/2:1+events23:28:552
218830499737733,3cyclictest94083-21kworker/1:0+events21:56:591
218830799736732,3cyclictest2415887-21kworker/2:1+events23:34:322
218830799736732,3cyclictest2163235-21kworker/2:0+events21:36:462
218830999735730,4cyclictest2194547-21kworker/3:3+events20:30:213
218830499735731,3cyclictest94083-21kworker/1:0+events21:33:321
218830999734731,2cyclictest2578622-21kworker/3:1+events00:39:283
218830999734730,3cyclictest2578622-21kworker/3:1+events00:34:413
218830499733729,3cyclictest2453368-21kworker/1:0+events22:46:381
218830999731727,3cyclictest2194547-21kworker/3:3+events19:35:423
218830999731727,3cyclictest2194547-21kworker/3:3+events19:35:413
218830999731726,4cyclictest2328245-21turbostat21:09:553
218830499731727,3cyclictest2403545-21kworker/1:1+events22:41:151
218830499731727,3cyclictest2403545-21kworker/1:1+events22:35:221
218830999730725,4cyclictest2378593-21turbostat21:49:553
218830499730726,3cyclictest94083-21kworker/1:0+events19:24:301
218830799729725,3cyclictest2415887-21kworker/2:1+events00:07:292
218830799728724,3cyclictest2415887-21kworker/2:1+events23:37:442
218830799728724,3cyclictest2415887-21kworker/2:1+events23:37:442
218830999724718,5cyclictest2194547-21kworker/3:3+events20:05:223
218830999723719,3cyclictest2194547-21kworker/3:3+events20:27:093
218830999723719,3cyclictest2194547-21kworker/3:3+events20:17:173
218830999720717,2cyclictest2194547-21kworker/3:3+events19:51:493
218830999720714,5cyclictest2194547-21kworker/3:3+events21:50:313
218830999719715,3cyclictest2194547-21kworker/3:3+events20:02:083
218830999719714,4cyclictest2277539-21turbostat20:24:563
218830999718715,2cyclictest2194547-21kworker/3:3+events23:11:383
218830999718714,3cyclictest2194547-21kworker/3:3+events22:21:483
218830999717713,3cyclictest2194547-21kworker/3:3+events19:33:373
218830999717712,4cyclictest3218111-21kworker/3:2+events19:11:583
218830999717711,5cyclictest2194547-21kworker/3:3+events00:20:223
218830999716712,3cyclictest2194547-21kworker/3:3+events23:43:143
218830999716711,4cyclictest2194547-21kworker/3:3+events20:57:073
218830999715712,2cyclictest2194547-21kworker/3:3+events22:55:413
218830999714711,2cyclictest2194547-21kworker/3:3+events21:16:383
218830999714711,2cyclictest2194547-21kworker/3:3+events20:41:393
218830799714709,4cyclictest2497277-21turbostat23:24:552
218830999713710,2cyclictest2194547-21kworker/3:3+events20:54:493
218830999712709,2cyclictest2194547-21kworker/3:3+events23:06:573
218830999712709,2cyclictest2194547-21kworker/3:3+events19:43:343
218830999712708,3cyclictest2194547-21kworker/3:3+events21:01:583
218830999712707,3cyclictest2194547-21kworker/3:3+events19:29:083
218830999711708,2cyclictest2194547-21kworker/3:3+events23:19:503
218830999710707,2cyclictest2194547-21kworker/3:3+events00:19:193
218830999709706,2cyclictest2194547-21kworker/3:3+events21:13:223
218830999708704,3cyclictest2194547-21kworker/3:3+events20:48:063
218830999708703,4cyclictest2194547-21kworker/3:3+events19:49:103
218830999704699,4cyclictest2194547-21kworker/3:3+events23:24:423
218830999702699,2cyclictest2194547-21kworker/3:3+events20:36:493
218830999701698,2cyclictest2194547-21kworker/3:3+events19:59:113
218830499701698,2cyclictest2384849-21turbostat21:54:551
218830999699695,3cyclictest2194547-21kworker/3:3+events20:10:213
218830799699695,3cyclictest2522208-21turbostat23:44:542
218830999697694,2cyclictest2194547-21kworker/3:3+events23:04:483
218830999691686,4cyclictest2453728-21turbostat22:49:553
218830999689685,3cyclictest2515973-21turbostat23:39:553
218830999689685,3cyclictest2515973-21turbostat23:39:553
218830999689684,4cyclictest2560255-21turbostat00:14:563
218830799680677,2cyclictest2271315-21turbostat20:19:552
218830799679674,4cyclictest2422502-21turbostat22:24:562
218830799657653,3cyclictest2188159-21turbostat19:14:552
218830799651647,3cyclictest2547453-21turbostat00:04:552
218830799648644,3cyclictest2359913-21turbostat21:34:552
218830499638634,3cyclictest2372356-21turbostat21:44:551
218830499638634,3cyclictest2372356-21turbostat21:44:551
218830999610608,1cyclictest2416249-21turbostat22:19:563
218830999599595,3cyclictest2541214-21turbostat23:59:553
218830799597595,1cyclictest2397353-21turbostat22:04:562
218830499597594,2cyclictest2435003-21turbostat22:34:561
218830499596594,1cyclictest2340703-21turbostat21:19:551
250327821090,2sleep10-21swapper/123:22:011
24979082890,3sleep02497914-21gltestperf23:20:160
2188304992723,3cyclictest506-21in:imuxsock19:55:001
218830499223,18cyclictest0-21swapper/122:20:201
2188304992218,3cyclictest0-21swapper/100:00:001
2188307992118,2cyclictest2390163-21head21:50:262
2188304992117,3cyclictest2309304-21cron20:50:011
2188300992118,2cyclictest0-21swapper/022:00:010
2188309992016,3cyclictest0-21swapper/319:20:003
2188304992015,3cyclictest506-21in:imuxsock22:10:011
2188300992010,9cyclictest0-21swapper/019:55:000
2188304991915,3cyclictest2474124-21kthreadcore23:00:181
2188300991916,2cyclictest0-21swapper/023:05:000
2188300991916,2cyclictest0-21swapper/020:55:000
2188304991815,2cyclictest0-21swapper/123:15:001
2188304991814,2cyclictest0-21swapper/123:30:001
2188309991712,3cyclictest0-21swapper/322:15:003
2188307991714,2cyclictest0-21swapper/220:55:002
2188307991711,4cyclictest2284280-21sort20:25:162
218830499171,14cyclictest728-21snmpd00:10:541
2188300991714,2cyclictest0-21swapper/021:20:000
2188300991713,3cyclictest171ktimers/000:10:140
218830999164,4cyclictest3709-21packagekitd21:30:143
2188309991613,2cyclictest2373592-21irqrtprio21:40:173
2188309991613,2cyclictest2373592-21irqrtprio21:40:173
2188307991614,1cyclictest0-21swapper/200:40:012
2188307991613,2cyclictest2873-21cron21:05:002
2188307991613,2cyclictest2334467-21idleruntime-cro21:10:002
2188307991612,2cyclictest0-21swapper/220:00:012
2188304991612,3cyclictest2337760-21kthreadcore21:10:181
2188304991612,2cyclictest0-21swapper/123:15:191
2188304991610,4cyclictest2463261-21tail22:50:231
2188300991613,2cyclictest2422959-21fschecks_count22:20:160
2188300991613,2cyclictest2353401-21kworker/0:0+kacpid22:07:030
2188300991613,2cyclictest2353401-21kworker/0:0+kacpid22:04:330
2188300991613,2cyclictest2283334-21unixbench_multi20:20:270
2188300991613,2cyclictest0-21swapper/023:45:000
218830999158,5cyclictest3763191-21systemd-journal21:30:003
2188309991511,3cyclictest2429321-21grep22:25:163
218830799152,2cyclictest25800232cut00:25:172
2188307991512,2cyclictest2234401-21gltestperf19:45:142
2188307991511,2cyclictest509-21rs:main2
2188307991510,4cyclictest2484824-21cron23:09:592
218830499159,4cyclictest2276651-21ntpq20:15:221
218830499158,5cyclictest2316055-21apt-get20:55:111
218830499157,5cyclictest493-21dbus-daemon23:09:591
2188304991511,3cyclictest2516560-21gltestperf23:35:151
2188304991511,3cyclictest2516560-21gltestperf23:35:151
2188304991511,3cyclictest2221948-21fschecks_time19:35:171
2188304991511,3cyclictest2221948-21fschecks_time19:35:161
2188304991511,3cyclictest0-21swapper/120:30:011
2188304991510,4cyclictest0-21swapper/100:40:001
2188304991510,3cyclictest0-21swapper/100:25:001
218830499150,13cyclictest2580320-21cut00:25:171
218830099159,4cyclictest3763191-21systemd-journal00:05:000
218830099157,2cyclictest295798rtkit-daemon22:10:210
2188300991513,1cyclictest2086027-21kworker/0:3+kacpid20:08:590
2188300991513,1cyclictest0-21swapper/020:05:010
2188300991512,2cyclictest2334477-21turbostat21:10:000
2188300991512,2cyclictest0-21swapper/021:25:010
2188300991511,3cyclictest2579769-21gltestperf00:25:160
295798140,2rtkit-daemon2308853-21grep20:45:222
218830799149,3cyclictest2222159-21ps19:35:192
218830799149,3cyclictest2222159-21ps19:35:182
2188307991411,2cyclictest2269537-21kthreadcore20:10:202
2188307991411,2cyclictest0-21swapper/222:55:162
2188307991411,2cyclictest0-21swapper/221:55:172
2188307991411,2cyclictest0-21swapper/219:20:162
2188307991410,3cyclictest2447819-21cpuspeed_turbos22:40:142
218830499149,3cyclictest0-21swapper/123:45:001
218830499148,4cyclictest3763191-21systemd-journal20:07:171
218830499148,4cyclictest2553624-21sshd00:03:281
218830499148,4cyclictest0-21swapper/122:20:011
218830499144,4cyclictest493-21dbus-daemon00:05:251
2188304991411,2cyclictest2569757-21kthreadcore00:15:181
2188304991411,2cyclictest2529463-21irqcore23:45:171
2188304991411,2cyclictest2282509-21kthreadcore20:20:211
2188304991411,2cyclictest2215560-21cpuspeed_turbos19:30:131
2188304991410,3cyclictest311ktimers/120:35:131
2188304991410,3cyclictest2415268-21needreboot22:10:231
218830099149,4cyclictest2197150-21gltestperf19:15:140
218830099148,4cyclictest2308452-21ntpq20:45:180
218830099142,5cyclictest191rcu_preempt00:10:010
2188300991412,1cyclictest2086027-21kworker/0:3+kacpid20:28:530
2188300991411,2cyclictest2504024-21fschecks_time23:25:140
2188300991411,2cyclictest2490098-21needreboot23:10:230
2188300991411,2cyclictest2422158-21kworker/0:3+kacpid00:40:000
2188300991411,2cyclictest2339975-21systemctl21:10:230
2188300991410,3cyclictest499-21polkitd19:35:000
218830099140,13cyclictest0-21swapper/022:55:230
218830999138,4cyclictest3763191-21systemd-journal22:35:013
218830999138,3cyclictest0-21swapper/321:39:593
2188309991310,2cyclictest0-21swapper/322:00:003
2188309991310,2cyclictest0-21swapper/321:25:003
218830799139,3cyclictest2568197-21kthreadcore00:15:172
218830799138,4cyclictest2490159-21ntpq23:10:232
218830799137,4cyclictest3763191-21systemd-journal21:30:002
2188307991311,1cyclictest2253792-21irqrtprio20:00:192
2188307991310,2cyclictest2585636-21/usr/sbin/munin00:30:202
2188307991310,2cyclictest2301471-21needreboot20:35:212
218830499139,3cyclictest2309786-21fschecks_count20:50:161
218830499139,3cyclictest2209574-21gltestperf19:25:151
218830499139,2cyclictest0-21swapper/120:05:011
218830499136,5cyclictest493-21dbus-daemon00:30:261
2188304991310,2cyclictest2249502-21tr19:55:191
2188304991310,2cyclictest2234283-21fschecks_count19:45:141
2188304991310,2cyclictest2232478-21kthreadcore19:40:201
2188304991310,2cyclictest0-21swapper/122:55:241
2188304991310,2cyclictest0-21swapper/121:35:191
218830099138,4cyclictest2227624-21cron19:40:000
218830099138,4cyclictest2227624-21cron19:40:000
218830099131,4cyclictest0-21swapper/000:20:010
2188300991310,2cyclictest2510192-21cpuspeed_turbos23:30:130
2188300991310,2cyclictest2378184-21unixbench_singl21:40:250
2188300991310,2cyclictest2378184-21unixbench_singl21:40:240
2188300991310,2cyclictest2359474-21unixbench_singl21:25:250
2188300991310,2cyclictest2353401-21kworker/0:0+kacpi_notify23:05:230
2188300991310,2cyclictest2291232-21irqrtprio20:30:180
2188300991310,2cyclictest2086027-21kworker/0:3+kacpid19:55:200
2188300991310,2cyclictest0-21swapper/022:15:170
218830099130,5cyclictest730-21ntpd22:45:150
218830999129,2cyclictest2536683-21kthreadcore23:50:183
2188309991210,1cyclictest0-21swapper/322:00:033
218830799128,3cyclictest2321576-21unixbench_singl20:55:212
218830799128,3cyclictest0-21swapper/222:40:012
218830799126,4cyclictest3763191-21systemd-journal19:55:002
218830799126,4cyclictest2491271-21sshd23:15:022
218830799125,5cyclictest3763191-21systemd-journal19:35:002
218830799124,2cyclictest191rcu_preempt23:05:002
218830799121,2cyclictest2232491-21kthreadcore19:40:202
218830499129,2cyclictest2535336-21cpuspeed_turbos23:50:141
218830499129,2cyclictest0-21swapper/121:05:181
218830499129,1cyclictest0-21swapper/122:25:161
2188304991210,1cyclictest0-21swapper/119:10:191
218830099129,2cyclictest2422158-21kworker/0:3+kacpi_notify00:20:260
218830099129,2cyclictest2316370-21gltestperf20:55:130
218830099129,1cyclictest0-21swapper/020:20:010
218830099128,3cyclictest2353401-21kworker/0:0+kacpi_notify22:55:000
218830099128,3cyclictest171ktimers/021:30:140
218830099128,2cyclictest2447453-21latency_hist22:40:010
218830099126,4cyclictest2516246-21apt-get23:35:130
218830099126,4cyclictest2516246-21apt-get23:35:120
218830099121,4cyclictest0-21swapper/023:45:180
2188300991210,1cyclictest2491772-21cut23:15:170
2188300991210,1cyclictest2422158-21kworker/0:3+kacpid00:31:040
2188300991210,1cyclictest0-21swapper/020:35:230
218830999116,4cyclictest2404620-21sed22:05:193
218830999115,4cyclictest2533848-21ntp_states23:45:233
218830799118,2cyclictest2454106-21cpuspeed_turbos22:45:162
218830799118,2cyclictest2347640-21cpuspeed_turbos21:20:172
218830799117,3cyclictest2384382-21grep21:45:242
218830799117,3cyclictest2290417-21cut20:30:152
218830799116,3cyclictest2578956-21basename00:25:002
218830499118,2cyclictest2303085-21latency_hist20:45:011
218830499117,3cyclictest0-21swapper/120:10:211
218830499110,3cyclictest0-21swapper/120:30:241
218830099119,1cyclictest2086027-21kworker/0:3+kacpid19:45:170
218830099119,1cyclictest0-21swapper/022:40:240
218830099119,1cyclictest0-21swapper/019:25:200
218830099118,2cyclictest2390205-21ntp_kernel_err21:50:270
218830099118,2cyclictest21750irq/159-ahci[0000:00:17.0]19:15:010
218830099118,2cyclictest0-21swapper/000:00:000
218830099117,3cyclictest0-21swapper/022:25:240
218830099116,4cyclictest2383852-21sed21:45:200
218830099110,9cyclictest2271379-21cat20:15:010
21880342117,2sleep30-21swapper/319:08:393
21822772111,3sleep10-21swapper/119:05:171
218830999108,1cyclictest0-21swapper/300:05:143
218830999106,2cyclictest0-21swapper/322:40:013
218830999105,4cyclictest2512967-21cut23:30:173
218830999105,4cyclictest2450779-21cut22:40:183
218830999105,3cyclictest2506345-21tr23:25:183
218830799106,3cyclictest2262276-21tr20:05:192
218830799104,1cyclictest2347206-21munin-run21:20:002
218830799103,2cyclictest295798rtkit-daemon21:14:172
218830499108,1cyclictest0-21swapper/123:30:171
218830499105,4cyclictest2323982-21tr21:00:211
218830099109,1cyclictest0-21swapper/020:44:520
218830099107,2cyclictest2366410-21cpu21:35:140
218830099107,2cyclictest0-21swapper/019:10:010
218830099106,3cyclictest2323679-21sed21:00:200
218830099106,3cyclictest2233804-21sshd19:44:460
218830099106,3cyclictest2208216-21mailstats19:20:220
218830099101,1cyclictest2435271-21apt-get22:30:130
218830099100,10cyclictest0-21swapper/023:54:360
2188018296,2sleep20-21swapper/219:08:262
21883079986,1cyclictest0-21swapper/219:25:122
21883079985,2cyclictest401ktimers/220:40:272
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional