You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 11:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8s.osadl.org (updated Sat Jul 12, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
210119599754748,5cyclictest2402238-21kworker/1:2+events23:59:131
210119599753747,5cyclictest2215492-21kworker/1:1+events21:10:201
210119599752746,5cyclictest2215492-21kworker/1:1+events21:28:421
210119599750747,2cyclictest2215492-21kworker/1:1+events21:44:171
210119599750746,3cyclictest2215492-21kworker/1:1+events21:18:491
210119599750745,4cyclictest2215492-21kworker/1:1+events21:45:441
210119599747741,5cyclictest2364636-21turbostat22:44:561
210119599747739,3cyclictest2215492-21kworker/1:1+events21:50:231
210119599747739,3cyclictest2215492-21kworker/1:1+events21:50:221
210119599745740,4cyclictest2402238-21kworker/1:2+events23:34:501
210119599745740,4cyclictest2402238-21kworker/1:2+events00:27:471
210119599745739,5cyclictest2402238-21kworker/1:2+events23:44:441
210119599744740,3cyclictest2402238-21kworker/1:2+events00:36:281
210119599744739,4cyclictest2402238-21kworker/1:2+events23:48:501
210119599743739,3cyclictest2215492-21kworker/1:1+events21:00:441
210119599743738,4cyclictest2215492-21kworker/1:1+events21:34:491
210120199742738,3cyclictest2088346-21kworker/3:0+events20:00:213
210119599742738,3cyclictest2402238-21kworker/1:2+events00:34:541
210119599742737,4cyclictest4186258-21kworker/1:1+events19:27:401
210119599742737,4cyclictest2184461-21kworker/1:2+events20:30:211
210119599742731,6cyclictest4186258-21kworker/1:1+events19:14:081
210119599741737,3cyclictest4186258-21kworker/1:1+events20:15:451
210119599741737,3cyclictest4186258-21kworker/1:1+events19:19:111
210119599741737,3cyclictest2215492-21kworker/1:1+events20:52:571
210119599741736,4cyclictest4186258-21kworker/1:1+events19:24:101
210119599740736,3cyclictest4186258-21kworker/1:1+events19:32:551
210119599740732,4cyclictest2402238-21kworker/1:2+events23:51:581
210119599739735,3cyclictest2402238-21kworker/1:2+events23:35:221
210119599739735,3cyclictest2402238-21kworker/1:2+events00:21:501
210119599739735,3cyclictest2402238-21kworker/1:2+events00:16:061
210119599739735,3cyclictest2184461-21kworker/1:2+events20:26:061
210119599739735,3cyclictest2184461-21kworker/1:2+events20:21:281
210119599739734,4cyclictest2215492-21kworker/1:1+events21:05:531
210119599739733,5cyclictest2215492-21kworker/1:1+events22:48:161
210120199738730,6cyclictest2088346-21kworker/3:0+events19:55:213
210119599738734,3cyclictest2215492-21kworker/1:1+events22:10:221
210119599738734,3cyclictest2215492-21kworker/1:1+events20:57:411
210120199737729,7cyclictest2171783-21kworker/3:1+events20:40:213
210120199736733,2cyclictest2265061-21kworker/3:0+events22:25:243
210119599735731,3cyclictest2215492-21kworker/1:1+events21:37:281
210120199731725,5cyclictest2265061-21kworker/3:0+events22:18:383
210119799730727,2cyclictest2314951-21kworker/2:2+events00:11:042
210120199729724,4cyclictest2265061-21kworker/3:0+events22:38:353
210119799729724,4cyclictest2147175-21kworker/2:0+events21:20:222
210120199726722,3cyclictest2265061-21kworker/3:0+events22:50:233
210120199724720,3cyclictest2265061-21kworker/3:0+events22:41:073
210120199724720,3cyclictest2265061-21kworker/3:0+events21:59:543
210120199723719,3cyclictest2265061-21kworker/3:0+events22:58:463
210120199721714,3cyclictest2265061-21kworker/3:0+events23:25:413
210120199720717,2cyclictest2265061-21kworker/3:0+events23:24:223
210120199720716,3cyclictest2171783-21kworker/3:1+events20:46:393
210120199720716,3cyclictest2088346-21kworker/3:0+events19:51:263
210120199720716,3cyclictest2088346-21kworker/3:0+events19:41:503
210120199720716,3cyclictest2088346-21kworker/3:0+events19:41:493
210120199720714,5cyclictest2265061-21kworker/3:0+events23:17:093
210120199719715,3cyclictest2088346-21kworker/3:0+events19:46:253
210119599719716,2cyclictest2265025-21turbostat21:24:551
210120199718714,3cyclictest2265061-21kworker/3:0+events22:05:223
210120199717714,2cyclictest2265061-21kworker/3:0+events22:21:463
210120199717713,3cyclictest2088346-21kworker/3:0+events19:39:273
210120199715711,3cyclictest2265061-21kworker/3:0+events23:02:293
210120199715711,3cyclictest2171783-21kworker/3:1+events20:39:463
210120199714709,3cyclictest2265061-21kworker/3:0+events23:11:583
210120199713710,2cyclictest2265061-21kworker/3:0+events23:07:003
210120199713710,2cyclictest2265061-21kworker/3:0+events22:31:263
210119799712709,2cyclictest2458153-21turbostat23:59:552
210119799711705,5cyclictest2314951-21kworker/2:2+events00:00:232
210120199710706,3cyclictest2265061-21kworker/3:0+events22:02:123
210120199709704,4cyclictest2508400-21turbostat00:39:553
210119799708701,5cyclictest2314951-21kworker/2:2+events00:05:222
210119199703699,3cyclictest2165793-21turbostat20:04:550
210119199701694,6cyclictest2165510-21kworker/0:1+events20:10:200
210119599700696,3cyclictest2420859-21turbostat23:29:541
210119599683678,4cyclictest2221669-21turbostat20:49:561
210120199677674,2cyclictest2128542-21turbostat19:34:563
210120199677673,3cyclictest2370853-21turbostat22:49:553
210119199674671,2cyclictest2165510-21kworker/0:1+events20:06:500
210120199659656,2cyclictest2327322-21turbostat22:14:563
210120199657655,1cyclictest2203054-21turbostat20:34:563
210119599655652,2cyclictest2178192-21turbostat20:14:551
210120199654651,2cyclictest2302512-21turbostat21:54:563
210120199654651,2cyclictest2302512-21turbostat21:54:553
210119599651648,2cyclictest2476853-21turbostat00:14:551
210119799648644,3cyclictest2258809-21turbostat21:19:542
210119599623618,4cyclictest2321116-21turbostat22:09:551
2101201992030,201cyclictest2246136-21taskset21:01:173
24697842740,0sleep02469787-21grep00:00:220
2101201992825,2cyclictest2240234-21sendmail21:00:003
2101191992420,3cyclictest745-21cups-browsed22:05:010
2101201992320,2cyclictest0-21swapper/300:00:003
210119799223,7cyclictest2489846-21grep00:20:112
210119799223,7cyclictest2346448-21cat22:25:142
210119799222,8cyclictest2389904-21ls23:00:002
210119799215,5cyclictest2456701-21grep23:50:202
210119799215,5cyclictest2172915-21cut20:05:192
210119799214,5cyclictest2153581-21latency_hist19:50:002
210119799212,7cyclictest2382792-21processes22:50:252
210119799212,7cyclictest2239071-21cut20:55:182
2101197992118,2cyclictest2497740-21kthreadcore00:25:182
2101201992015,3cyclictest2107658-21latency_hist19:15:003
210119799204,5cyclictest2445980-21cat23:45:112
210119799204,5cyclictest2315644-21irqcore22:00:192
210119799204,5cyclictest2111096-21cut19:15:192
210119799203,6cyclictest2293866-21grep21:40:192
210119799203,5cyclictest2425631-21tr23:25:172
210119799203,5cyclictest2370860-21latency_hist22:45:022
210119799202,7cyclictest2202512-21sed20:25:232
210119799202,7cyclictest2124210-21grep19:25:182
2101197992015,4cyclictest2254347-21kthreadcore21:10:182
2101197992011,1cyclictest2159563-21dump-pmu-power19:55:012
2101191992017,2cyclictest2265024-21apt21:20:010
210119799195,5cyclictest2248105-21sed21:05:162
210119799194,5cyclictest2514160-21awk00:35:252
210119799194,5cyclictest2412283-21grep23:15:192
210119799194,5cyclictest2355913-21cut22:30:182
210119799194,5cyclictest2340235-21fschecks_count22:20:142
210119799194,5cyclictest2233363-21mailstats20:50:232
210119799194,5cyclictest2146083-21cat19:40:242
210119799194,5cyclictest2146083-21cat19:40:232
210119799193,7cyclictest2244939-21kthreadcore21:00:212
210119799193,6cyclictest2441469-21cut23:40:152
210119799193,6cyclictest2118805-21kthreadcore19:20:162
210119799192,7cyclictest2209358-21cat20:35:022
210119799192,7cyclictest2165138-21cat19:55:262
210119799192,6cyclictest2360638-21cut22:35:182
210119799191,6cyclictest2483595-21apt00:15:132
2101197991915,3cyclictest2507422-21needreboot00:30:202
2101197991915,3cyclictest2377081-21idleruntime-cro22:50:002
2101197991915,2cyclictest2327321-21sh22:10:002
2101195991915,3cyclictest745-21cups-browsed23:04:591
210119199195,12cyclictest493-21dbus-daemon23:50:000
2101191991916,2cyclictest2101182-21cyclictest19:50:000
2101201991814,2cyclictest0-21swapper/321:25:003
210119799182,6cyclictest2226963-21awk20:45:222
210119799182,6cyclictest2181557-21grep20:10:182
210119799182,6cyclictest2102473-21irqrtprio19:10:152
210119799182,5cyclictest2215938-21fschecks_count20:40:162
210119799182,5cyclictest2196264-21wc20:20:272
2101197991814,3cyclictest2427531-21fschecks_count23:30:142
2101197991813,4cyclictest523-21gdbus23:05:002
2101195991814,2cyclictest0-21swapper/120:40:011
2101195991813,3cyclictest506-21in:imuxsock22:05:011
2101201991714,2cyclictest2178918-21irqcore20:10:163
210119799173,5cyclictest2308931-21/usr/sbin/munin21:55:262
2101197991712,3cyclictest2296926-21gltestperf21:45:142
2101195991713,3cyclictest0-21swapper/122:30:011
2101191991715,1cyclictest2308697-21idleruntime-cro21:55:010
2101191991715,1cyclictest2308697-21idleruntime-cro21:55:000
2101191991714,2cyclictest2154601-21cut19:50:170
210119799165,3cyclictest21346232sleep219:34:002
210119799164,3cyclictest2416952-21cut23:20:182
210119799164,3cyclictest2190802-21date20:20:012
210119799163,5cyclictest0-21swapper/223:05:172
210119799163,4cyclictest2214526-21netstat20:35:202
210119799162,5cyclictest2281561-21cut21:30:182
2101197991613,2cyclictest2308686-21sh21:55:012
2101197991613,2cyclictest2308686-21sh21:55:002
2101197991611,4cyclictest2333068-21grep22:10:252
210119599164,2cyclictest191rcu_preempt22:25:001
2101195991613,2cyclictest2358423-21cron22:35:001
2101195991612,3cyclictest2470220-21unixbench_singl00:00:261
2101195991612,3cyclictest2358892-21fschecks_count22:35:151
2101191991613,2cyclictest2190660-21sendmail20:20:000
2101191991610,4cyclictest3614154-21systemd-journal00:15:000
2101201991512,2cyclictest2502226-21latency_hist00:30:013
2101201991512,2cyclictest2484553-21cut00:15:173
2101201991512,2cyclictest2445947-21latency_hist23:45:013
2101201991512,2cyclictest0-21swapper/320:25:163
2101201991511,3cyclictest2264551-21unixbench_multi21:15:253
210119799158,5cyclictest2277474-21true21:30:012
210119799157,6cyclictest2101199-21latency_hist19:10:012
210119799156,8cyclictest493-21dbus-daemon20:05:002
210119799153,3cyclictest2284438-21perl21:35:162
210119799151,5cyclictest0-21swapper/222:15:222
2101197991512,2cyclictest2439496-21cron23:40:002
2101195991512,2cyclictest2476842-21cron00:10:001
2101195991512,2cyclictest2169531-21kthreadcore20:00:191
2101191991513,1cyclictest0-21swapper/022:25:170
2101191991512,2cyclictest2483694-21cpuspeed_turbos00:15:150
2101191991512,2cyclictest2290070-21idleruntime-cro21:40:000
2101191991512,2cyclictest2197304-21fschecks_count20:25:140
2101191991512,2cyclictest0-21swapper/023:29:590
2101191991511,2cyclictest2147140-21sh19:45:010
2101191991511,2cyclictest2147140-21sh19:45:000
2101201991410,2cyclictest0-21swapper/321:05:163
210120199140,3cyclictest2122474-21apt-key19:25:003
210119799147,5cyclictest2135126-21tr19:35:132
210119799146,5cyclictest2402513-21grep23:10:162
2101195991410,2cyclictest2309315-21gltestperf21:55:181
210119599140,4cyclictest2148871-21ps19:45:201
210119199143,2cyclictest295798rtkit-daemon00:26:450
2101191991412,1cyclictest2377083-21latency_hist22:50:000
2101191991412,1cyclictest0-21swapper/022:55:200
2101191991411,2cyclictest2458397-21/usr/sbin/munin23:55:190
2101191991411,2cyclictest2215473-21sendmail20:40:010
2101191991411,2cyclictest2184230-21kworker/0:0+kacpid21:25:210
2101191991411,2cyclictest0-21swapper/019:35:000
2101191991411,2cyclictest0-21swapper/000:40:000
2101191991410,2cyclictest0-21swapper/023:40:130
2101191991410,2cyclictest0-21swapper/023:04:590
210119199140,2cyclictest191rcu_preempt20:20:270
20949772142,6sleep32095010-21apt-get19:05:113
210120199139,3cyclictest2277432-21cron21:30:013
210120199138,4cyclictest2283452-21grep21:30:243
210120199137,4cyclictest493-21dbus-daemon23:55:003
210120199137,4cyclictest2115713-21sshd19:18:373
2101201991310,2cyclictest2439864-21cpuspeed_turbos23:40:113
2101201991310,2cyclictest2172721-21irqcore20:05:183
2101201991310,2cyclictest0-21swapper/321:40:143
2101195991310,2cyclictest491-21avahi-daemon20:00:001
2101195991310,2cyclictest2141580-21gltestperf19:40:191
2101195991310,2cyclictest2141580-21gltestperf19:40:181
210119599130,4cyclictest2335316-21cut22:15:211
210119599130,3cyclictest2135164-21fschecks_count19:35:141
210119599130,2cyclictest0-21swapper/120:40:361
210119599130,12cyclictest0-21swapper/123:21:321
210119199139,2cyclictest2439512-21turbostat23:40:010
210119199138,4cyclictest2282975-21awk21:30:200
210119199135,2cyclictest295798rtkit-daemon21:07:410
210119199134,8cyclictest211rcuc/021:45:160
210119199131,3cyclictest2107535-21munin-run19:15:000
2101191991311,1cyclictest2968852-21kworker/0:3+kacpid19:15:160
2101191991311,1cyclictest2416642-21kthreadcore23:20:180
2101191991311,1cyclictest0-21swapper/019:25:010
2101191991310,2cyclictest523-21gdbus20:45:000
2101191991310,2cyclictest2327738-21cpuspeed_turbos22:10:140
2101191991310,2cyclictest2184230-21kworker/0:0+kacpid20:55:160
2101191991310,2cyclictest2159867-21cpu19:55:170
210119199130,2cyclictest191rcu_preempt23:05:120
210120199127,4cyclictest2302225-21sshd21:46:363
210120199127,4cyclictest2257801-21sed21:10:203
210120199127,4cyclictest2228487-21gltestperf20:50:153
210120199126,4cyclictest2128282-21sshd19:26:593
2101201991210,1cyclictest0-21swapper/300:11:023
210120199120,1cyclictest0-21swapper/323:30:173
210120199120,1cyclictest0-21swapper/320:21:113
210119599129,2cyclictest2383970-21cpuspeed_turbos22:55:141
210119599127,4cyclictest2173286-21sed20:05:191
210119599122,3cyclictest2956-21rtkit-daemon23:18:081
2101195991210,1cyclictest0-21swapper/123:07:411
210119599120,3cyclictest0-21swapper/119:50:201
210119199129,2cyclictest2184230-21kworker/0:0+kacpid21:20:170
210119199129,2cyclictest2184230-21kworker/0:0+kacpid20:55:000
210119199129,2cyclictest0-21swapper/023:30:180
210119199128,3cyclictest3614154-21systemd-journal21:14:360
210119199128,3cyclictest3614154-21systemd-journal00:34:010
210119199128,3cyclictest3614154-21systemd-journal00:05:360
210119199127,4cyclictest2309170-21cut21:55:180
210119199127,4cyclictest0-21swapper/022:15:160
210119199124,7cyclictest211rcuc/022:30:170
2101191991210,1cyclictest2968852-21kworker/0:3+kacpid19:35:200
2101191991210,1cyclictest2490272-21grep00:20:140
2101191991210,1cyclictest2370783-21kworker/0:2+kacpid22:50:230
2101191991210,1cyclictest2370392-21unixbench_multi22:40:270
2101191991210,1cyclictest2165510-21kworker/0:1+kacpid22:05:170
2101191991210,1cyclictest211rcuc/023:15:160
2101191991210,1cyclictest2101067-21apt-key19:10:010
210119199120,4cyclictest2205675-21grep20:30:170
210119199120,3cyclictest21260401kthreadcore19:25:190
210119199120,2cyclictest2290620-21grep21:40:150
210119199120,2cyclictest191rcu_preempt23:50:160
210119199120,11cyclictest0-21swapper/020:45:130
210120199119,1cyclictest0-21swapper/320:15:113
210120199116,4cyclictest2466933-21cut00:00:193
210120199114,5cyclictest523-21gdbus00:25:003
210120199110,1cyclictest0-21swapper/300:06:213
210119199118,2cyclictest2358696-21apt-get22:35:120
210119199115,4cyclictest2408624-21sed23:15:010
210119199110,10cyclictest2340504-21irqcore22:20:160
210119199110,10cyclictest0-21swapper/021:05:000
21007672113,3sleep10-21swapper/119:08:071
210120199108,1cyclictest0-21swapper/321:38:223
210120199107,2cyclictest2505482-21tr00:30:193
210120199105,4cyclictest2434656-21sed23:35:213
210119599106,3cyclictest2407605-21ps23:10:251
210119599100,2cyclictest0-21swapper/122:50:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional