You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-31 - 11:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Wed Dec 31, 2025 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
212366923070,9sleep1251rcuc/119:10:001
212240122620,9sleep30-21swapper/319:05:153
197399840,5rtkit-daemon4497-21PK-Backend23:49:500
212344227739,32sleep00-21swapper/019:07:350
197399760,62rtkit-daemon0-21swapper/219:05:162
22253992740,7sleep24497-21PK-Backend23:55:222
22253992740,7sleep24497-21PK-Backend23:55:212
197399710,8rtkit-daemon1051-21thermald00:10:340
197399690,5rtkit-daemon4497-21PK-Backend23:43:240
197399690,4rtkit-daemon0-21swapper/000:15:020
197399680,4rtkit-daemon4497-21PK-Backend00:22:070
197399620,9rtkit-daemon1972-21rtkit-daemon23:43:051
197399590,9rtkit-daemon1972-21rtkit-daemon23:58:221
197399590,9rtkit-daemon1972-21rtkit-daemon23:58:211
2123781995825,17cyclictest42-21ksoftirqd/300:14:093
197399560,9rtkit-daemon1972-21rtkit-daemon23:48:401
212377199540,49cyclictest1972-21rtkit-daemon00:31:381
212377199540,49cyclictest1972-21rtkit-daemon00:31:371
197399540,9rtkit-daemon1972-21rtkit-daemon00:12:531
21317792520,11sleep3411rcuc/319:30:133
2123781995017,17cyclictest42-21ksoftirqd/319:50:013
22234022490,8chrt0-21swapper/223:50:142
2123771994938,8cyclictest26-21ksoftirqd/100:22:221
2123781994827,7cyclictest42-21ksoftirqd/321:12:223
2123781994823,7cyclictest42-21ksoftirqd/319:24:143
2123781994814,18cyclictest42-21ksoftirqd/300:40:023
2123781994813,6cyclictest21513872/usr/sbin/munin20:25:223
212378199476,10cyclictest2222961-21cron23:49:593
212378199474,10cyclictest937-21polkitd00:22:223
212378199463,13cyclictest2135441-21cstates19:40:183
2123781994625,7cyclictest42-21ksoftirqd/319:54:043
2123781994616,16cyclictest42-21ksoftirqd/321:05:023
2123781994616,16cyclictest42-21ksoftirqd/321:05:023
2123781994614,17cyclictest42-21ksoftirqd/320:40:023
212378199461,13cyclictest2150669-21sed20:25:013
2123776994621,20cyclictest2234432-21irqstats00:20:232
2123768994615,12cyclictest438-20systemd-journal00:30:000
2123781994525,7cyclictest42-21ksoftirqd/319:11:463
2123781994517,15cyclictest42-21ksoftirqd/322:13:163
2123781994517,15cyclictest42-21ksoftirqd/322:13:163
197399450,31rtkit-daemon0-21swapper/223:30:312
21987382440,5sleep2331rcuc/222:39:592
212378199445,18cyclictest22155002sed23:25:303
2123781994424,7cyclictest42-21ksoftirqd/323:38:023
2123771994428,12cyclictest26-21ksoftirqd/123:55:001
2123771994424,5cyclictest26-21ksoftirqd/100:15:231
2123771994415,6cyclictest21341752ntp_states19:35:261
2123781994321,7cyclictest42-21ksoftirqd/321:16:093
2123771994325,6cyclictest26-21ksoftirqd/123:30:261
2123771994324,7cyclictest26-21ksoftirqd/122:05:121
2123771994323,12cyclictest26-21ksoftirqd/123:30:011
197399430,32rtkit-daemon0-21swapper/222:58:532
2123781994223,7cyclictest42-21ksoftirqd/320:18:153
2123781994223,7cyclictest42-21ksoftirqd/320:18:143
2123781994210,12cyclictest42-21ksoftirqd/321:50:013
2123776994225,13cyclictest2148758-21cron20:20:002
2123776994225,13cyclictest2148758-21cron20:20:002
2123771994226,5cyclictest26-21ksoftirqd/122:52:551
2123771994224,7cyclictest26-21ksoftirqd/119:50:191
2123771994224,6cyclictest26-21ksoftirqd/122:00:021
2123771994224,6cyclictest26-21ksoftirqd/121:50:081
2123771994224,6cyclictest26-21ksoftirqd/120:05:291
2123771994223,14cyclictest2211487-21df23:15:151
197399420,10rtkit-daemon2195915-21df22:30:142
21744222410,8sleep00-21swapper/021:30:150
212378199418,18cyclictest42-21ksoftirqd/300:05:143
212378199417,17cyclictest42-21ksoftirqd/323:40:293
2123781994112,11cyclictest42-21ksoftirqd/322:57:013
2123781994111,11cyclictest42-21ksoftirqd/321:59:593
2123771994124,7cyclictest26-21ksoftirqd/121:15:111
2123771994123,5cyclictest26-21ksoftirqd/100:05:021
212377199412,12cyclictest26-21ksoftirqd/119:49:251
197399410,30rtkit-daemon0-21swapper/200:33:372
197399410,30rtkit-daemon0-21swapper/200:33:372
197399410,28rtkit-daemon0-21swapper/322:35:113
212378199409,12cyclictest42-21ksoftirqd/320:41:013
212378199409,12cyclictest42-21ksoftirqd/320:41:003
212378199406,17cyclictest42-21ksoftirqd/323:57:093
212378199406,17cyclictest42-21ksoftirqd/323:57:093
212378199405,17cyclictest2173195-21ntp_states21:25:263
212378199405,17cyclictest2173195-21ntp_states21:25:253
2123781994022,7cyclictest42-21ksoftirqd/300:31:283
2123781994022,7cyclictest42-21ksoftirqd/300:31:273
2123781994022,14cyclictest2228057-21chrt00:03:353
2123781994012,15cyclictest42-21ksoftirqd/319:19:053
212378199401,14cyclictest2172104-21taskset21:24:133
2123776994023,12cyclictest438-20systemd-journal23:29:592
2123771994024,6cyclictest26-21ksoftirqd/119:30:191
2123771994024,5cyclictest26-21ksoftirqd/100:25:131
2123771994021,6cyclictest26-21ksoftirqd/122:25:291
2123771994019,17cyclictest2173898-21cron21:29:591
2123771994019,17cyclictest2173898-21cron21:29:591
2123768994024,12cyclictest2206122-21tr23:00:150
2123768994017,19cyclictest2160784-21ntp_states20:50:270
197399400,25rtkit-daemon0-21swapper/323:15:023
212378199398,18cyclictest42-21ksoftirqd/321:40:063
212378199398,11cyclictest42-21ksoftirqd/320:54:233
212378199397,16cyclictest42-21ksoftirqd/323:15:093
212378199397,16cyclictest42-21ksoftirqd/300:25:023
212378199397,15cyclictest42-21ksoftirqd/320:31:593
212378199397,10cyclictest42-21ksoftirqd/320:05:293
212378199395,18cyclictest42-21ksoftirqd/321:31:463
2123781993930,6cyclictest411rcuc/320:59:463
2123781993928,7cyclictest2134891-21idleruntime-cro19:39:593
2123781993927,7cyclictest438-20systemd-journal22:54:553
2123781993922,12cyclictest438-20systemd-journal21:54:553
2123781993922,12cyclictest438-20systemd-journal00:20:013
2123781993920,7cyclictest42-21ksoftirqd/321:36:173
2123781993915,8cyclictest42-21ksoftirqd/320:02:563
212378199391,4cyclictest4497-21PK-Backend22:05:433
212378199391,22cyclictest42-21ksoftirqd/323:23:593
212378199391,22cyclictest42-21ksoftirqd/323:23:593
2123776993923,12cyclictest2125440-21/usr/sbin/munin19:15:012
2123776993923,12cyclictest1-21systemd19:55:022
2123771993929,6cyclictest26-21ksoftirqd/119:45:011
2123771993923,5cyclictest26-21ksoftirqd/121:38:051
2123771993921,13cyclictest2164357-21sensors_temp21:00:271
2123771993921,13cyclictest2164357-21sensors_temp21:00:271
212376899391,6cyclictest141rcu_preempt21:20:020
212376899391,10cyclictest4497-21PK-Backend20:25:130
212376899390,35cyclictest1972-21rtkit-daemon00:02:360
197399390,27rtkit-daemon4497-21PK-Backend22:29:093
197399390,27rtkit-daemon4497-21PK-Backend21:05:143
197399390,26rtkit-daemon4497-21PK-Backend22:05:082
197399390,26rtkit-daemon0-21swapper/222:02:422
21478932380,6chrt0-21swapper/120:15:231
21478932380,6chrt0-21swapper/120:15:231
212378199386,17cyclictest42-21ksoftirqd/323:09:013
212378199384,18cyclictest42-21ksoftirqd/322:45:033
212378199382,15cyclictest2199499-21irqstats22:40:213
212378199381,21cyclictest2186069-21taskset22:01:513
212378199381,17cyclictest2191505-21taskset22:16:083
212378199381,14cyclictest438-20systemd-journal22:32:073
2123771993821,13cyclictest2146442-21sensors_temp20:10:271
2123771993815,19cyclictest2124180-21cstates19:10:151
212376899382,31cyclictest2217787-21turbostat23:35:000
2123768993822,12cyclictest2195270-21cron22:30:000
212376899381,5cyclictest4497-21PK-Backend23:52:590
197399380,27rtkit-daemon4497-21PK-Backend23:24:112
197399380,27rtkit-daemon4497-21PK-Backend23:24:112
197399380,27rtkit-daemon4497-21PK-Backend19:23:202
197399380,27rtkit-daemon34-21ksoftirqd/220:12:262
197399380,24rtkit-daemon0-21swapper/220:44:032
197399380,24rtkit-daemon0-21swapper/220:44:032
197399380,24rtkit-daemon0-21swapper/200:38:132
22062552370,5sleep14497-21PK-Backend23:00:151
212378199375,9cyclictest42-21ksoftirqd/322:20:133
212378199374,13cyclictest42-21ksoftirqd/323:31:103
212378199374,12cyclictest42-21ksoftirqd/323:50:113
2123781993719,13cyclictest438-20systemd-journal19:59:593
212377699375,27cyclictest21893042sleep222:10:262
212377699375,27cyclictest21893042sleep222:10:252
212377699372,7cyclictest2220262-21irqstats23:40:192
2123776993723,10cyclictest2154543-21apt-config20:35:022
2123776993718,14cyclictest2166765-40apt-check21:07:572
212377699371,12cyclictest924-21NetworkManager21:49:322
2123771993721,12cyclictest2214307-21cron23:25:011
2123771993721,12cyclictest2214307-21cron23:25:001
2123771993710,5cyclictest141rcu_preempt20:36:361
212376899372,31cyclictest2198777-21munin-run22:40:000
212376899372,30cyclictest2182107-21ntp_states21:50:230
2123768993720,12cyclictest2124702-21ntpq19:10:250
197399370,26rtkit-daemon4497-21PK-Backend22:27:232
197399370,26rtkit-daemon4497-21PK-Backend21:01:162
197399370,26rtkit-daemon4497-21PK-Backend21:01:152
197399370,25rtkit-daemon0-21swapper/221:21:212
197399370,24rtkit-daemon0-21swapper/223:09:182
197399370,24rtkit-daemon0-21swapper/222:51:072
212378199363,11cyclictest42-21ksoftirqd/319:25:123
2123781993627,6cyclictest1137-21runrttasks20:13:093
212378199361,20cyclictest2207069-21taskset23:01:003
212378199361,20cyclictest2159727-21runrttasks20:49:443
2123776993619,6cyclictest34-21ksoftirqd/222:45:022
2123776993619,13cyclictest2236368-21cat00:25:202
2123776993611,6cyclictest141rcu_preempt21:40:302
2123771993625,7cyclictest2069-21gnome-shell19:57:581
2123771993621,11cyclictest2131413-21cron19:30:001
2123771993620,12cyclictest2159172-21sensors_temp20:45:301
212377199361,8cyclictest0-21swapper/100:06:021
2123771993618,13cyclictest26-21ksoftirqd/122:00:221
2123771993618,13cyclictest26-21ksoftirqd/120:35:001
2123771993615,6cyclictest141rcu_preempt22:39:291
2123771993614,6cyclictest141rcu_preempt22:45:001
2123771993613,5cyclictest141rcu_preempt19:20:151
212377199361,31cyclictest2157747-21chrt20:41:311
212377199361,31cyclictest2157747-21chrt20:41:311
212376899362,8cyclictest13-21ksoftirqd/019:20:200
2123768993620,12cyclictest2193583-21pluginstate22:25:010
2123768993619,13cyclictest2190504-21tr22:15:140
212376899361,13cyclictest4497-21PK-Backend21:47:100
212376899360,33cyclictest1972-21rtkit-daemon00:30:100
212376899360,33cyclictest1972-21rtkit-daemon00:30:090
197399360,25rtkit-daemon0-21swapper/222:23:372
197399360,25rtkit-daemon0-21swapper/221:31:462
197399360,24rtkit-daemon0-21swapper/223:19:482
2123776993520,11cyclictest2168128-21python321:10:292
2123776993519,12cyclictest2176169-21ls21:35:142
2123776993518,13cyclictest34-21ksoftirqd/219:35:292
2123776993518,12cyclictest2150763-21awk20:25:112
2123776993518,12cyclictest2142857-21ntpq20:00:252
2123776993517,13cyclictest34-21ksoftirqd/219:49:262
2123776993514,18cyclictest2202212-21turbostat22:50:012
2123776993511,6cyclictest141rcu_preempt00:05:312
212377699351,11cyclictest141rcu_preempt20:45:272
2123776993510,10cyclictest141rcu_preempt21:50:192
2123771993520,11cyclictest2143565-21cron20:05:011
2123771993519,12cyclictest2210405-21sed23:10:291
2123771993518,12cyclictest26-21ksoftirqd/121:05:221
2123771993515,17cyclictest2190122-21unin-run22:15:011
2123771993515,17cyclictest2190122-21unin-run22:15:011
2123768993520,11cyclictest2164820-21sshd21:03:300
2123768993520,11cyclictest2164820-21sshd21:03:290
2123768993520,11cyclictest2127652-21apt-get19:20:010
2123768993519,12cyclictest923-21dbus-daemon19:49:260
2123768993515,7cyclictest141rcu_preempt20:04:070
2123768993515,16cyclictest2147040-21cron20:15:010
212376899351,30cyclictest4497-21PK-Backend23:35:210
212376899351,30cyclictest2138661-21apt19:50:120
212376899351,13cyclictest2166789-40lsb_release21:08:010
197399350,25rtkit-daemon4497-21PK-Backend21:16:322
197399350,24rtkit-daemon0-21swapper/223:35:382
197399350,24rtkit-daemon0-21swapper/219:34:502
2123776993420,10cyclictest2233664-21cron00:20:002
2123776993420,10cyclictest2173401-21threads21:25:302
2123776993420,10cyclictest2173401-21threads21:25:302
2123776993419,11cyclictest2162013-21cat20:55:152
2123776993418,12cyclictest2226825-21sort00:00:152
2123776993418,12cyclictest2191523-21seq22:16:242
2123776993418,12cyclictest2156183-21seq20:38:402
2123776993418,12cyclictest2141746-21seq19:58:532
2123776993418,12cyclictest2136205-21wc19:40:322
2123776993417,13cyclictest2193-21gdbus23:49:562
2123776993416,4cyclictest141rcu_preempt20:50:242
2123776993410,10cyclictest141rcu_preempt20:05:252
2123771993425,5cyclictest26-21ksoftirqd/100:37:051
2123771993424,6cyclictest26-21ksoftirqd/120:50:281
2123771993423,7cyclictest2126313-21tr19:15:121
2123771993423,6cyclictest26-21ksoftirqd/123:07:111
2123771993422,7cyclictest26-21ksoftirqd/121:15:021
2123771993420,10cyclictest2163229-21awk21:00:001
212377199341,9cyclictest4497-21PK-Backend21:30:121
2123768993423,8cyclictest2147055-21perf20:19:550
2123768993423,8cyclictest2147055-21perf20:19:550
212376899342,13cyclictest2133143-21cron19:34:590
2123768993419,11cyclictest2228658-21cat00:05:160
2123768993419,11cyclictest2209253-21sed23:10:010
2123768993418,12cyclictest2240714-21seq00:37:160
2123768993418,11cyclictest438-20systemd-journal22:34:300
2123768993417,13cyclictest2212492-21seq23:18:570
197399340,23rtkit-daemon0-21swapper/219:17:572
2123776993324,6cyclictest2130097-21cpuspeed_turbos19:25:142
2123776993321,8cyclictest2149226-21tr20:20:142
2123776993311,5cyclictest141rcu_preempt21:57:512
2123771993324,6cyclictest26-21ksoftirqd/120:22:361
2123771993323,6cyclictest26-21ksoftirqd/123:39:341
2123771993323,6cyclictest26-21ksoftirqd/122:46:301
2123771993323,6cyclictest26-21ksoftirqd/122:20:141
2123771993323,6cyclictest26-21ksoftirqd/121:40:291
2123771993323,5cyclictest26-21ksoftirqd/122:55:081
2123771993322,6cyclictest26-21ksoftirqd/122:15:251
2123771993317,12cyclictest976-21rs:main1
212377199331,29cyclictest0-21swapper/121:46:331
212376899335,13cyclictest141rcu_preempt20:05:210
212376899332,8cyclictest2204611-21smartctl22:55:200
212376899331,8cyclictest2159773-21turbostat20:49:590
2123768993318,11cyclictest2199515-21sed22:40:210
2123768993317,12cyclictest2178529-21swap21:40:290
2123768993317,12cyclictest2141008-21wc19:55:240
2123768993316,12cyclictest438-20systemd-journal20:44:590
2123768993316,12cyclictest438-20systemd-journal20:44:590
212376899331,29cyclictest4497-21PK-Backend19:27:030
212376899331,28cyclictest4497-21PK-Backend20:30:140
2123768993311,7cyclictest141rcu_preempt22:05:000
212376899331,15cyclictest2188813-21tr22:10:170
212376899331,15cyclictest2188813-21tr22:10:160
197399330,25rtkit-daemon1972-21rtkit-daemon23:56:390
197399330,25rtkit-daemon1972-21rtkit-daemon23:56:390
212377699321,28cyclictest0-21swapper/223:01:142
2123776993210,9cyclictest141rcu_preempt00:10:222
2123771993223,5cyclictest26-21ksoftirqd/121:21:391
2123771993222,6cyclictest2152231-21sh20:30:011
212376899321,4cyclictest4497-21PK-Backend21:22:090
212376899321,29cyclictest0-21swapper/023:27:140
212376899321,29cyclictest0-21swapper/020:57:270
212376899321,28cyclictest4497-21PK-Backend19:42:240
212376899321,27cyclictest2209737-21expr23:10:130
212376899321,27cyclictest2156302-21turbostat20:40:010
2123768993211,6cyclictest21876612grep22:05:280
212377699313,11cyclictest141rcu_preempt23:10:272
212376899315,7cyclictest0-21swapper/020:20:130
2123768993117,10cyclictest2202196-21cron22:50:000
2123768993116,11cyclictest2184625-21systemd-run22:00:000
212376899311,4cyclictest4497-21PK-Backend22:52:120
2123768993019,7cyclictest2168712-21grep21:14:590
2123768993016,10cyclictest2172970-21wc21:25:220
2123768993016,10cyclictest2172970-21wc21:25:210
2123768992918,8cyclictest4497-21PK-Backend19:38:570
2123768992913,12cyclictest2177312-21seq21:39:260
2123768992910,5cyclictest141rcu_preempt23:22:280
2123768992910,5cyclictest141rcu_preempt23:22:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional