You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-09 - 05:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Jan 09, 2026 00:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
626129785,8sleep20-21swapper/219:07:552
622029177,9sleep00-21swapper/019:07:240
633928464,16sleep30-21swapper/319:08:563
617428164,13sleep10-21swapper/119:06:491
283952710,2sleep1241rcuc/122:10:011
6503993024,2cyclictest32-21ksoftirqd/219:20:002
6503992218,3cyclictest9248-21irqstats22:35:182
649799211,10cyclictest0-21swapper/100:05:131
6494992119,1cyclictest0-21swapper/020:30:140
650399200,2cyclictest0-21swapper/219:45:132
6507991913,4cyclictest0-21swapper/323:25:133
649799190,9cyclictest0-21swapper/123:20:111
649499191,7cyclictest225-21systemd-journal23:00:000
649499183,6cyclictest0-21swapper/022:40:120
6497991710,5cyclictest0-21swapper/122:40:121
649799163,6cyclictest225-21systemd-journal21:55:001
649499164,6cyclictest273-21systemd-udevd20:51:040
6507991511,2cyclictest15917-21grep21:40:243
6503991511,3cyclictest29453-21sort22:10:172
6497991510,3cyclictest0-21swapper/100:25:141
649799150,14cyclictest22965-21wc00:10:201
6494991510,3cyclictest18586-21sh19:35:000
649499150,1cyclictest0-21swapper/020:55:120
6507991411,2cyclictest31470-21grep23:20:233
6507991410,3cyclictest17355-21cut21:45:173
649799149,3cyclictest28561-21wc19:55:001
6497991410,3cyclictest9163-21kworker/1:119:35:131
6497991410,2cyclictest9466-21grep20:20:181
6497991410,2cyclictest3606-21tr22:25:141
6497991410,2cyclictest29797-21basename21:04:591
6497991410,2cyclictest12280-21cut19:20:151
649499142,6cyclictest30749-21munin-run22:15:000
649499141,6cyclictest13628-21apt-get19:25:000
6494991411,2cyclictest29122-21sed23:15:270
6494991410,3cyclictest29031-21wc00:25:000
650799139,3cyclictest7932-21cat23:40:193
650799139,3cyclictest20548-21grep20:45:153
650799139,3cyclictest14552-21basename23:55:133
650799134,4cyclictest225-21systemd-journal21:25:003
6507991310,2cyclictest32687-21grep22:15:273
6507991310,2cyclictest18579-21cron19:35:003
6507991310,2cyclictest17597-21perf22:54:593
6507991310,2cyclictest12697-21head21:35:183
6507991310,2cyclictest10729-21grep19:15:263
650799130,1cyclictest0-21swapper/321:10:123
650799130,12cyclictest11638-21cron21:35:003
650399139,3cyclictest32-21ksoftirqd/200:40:002
650399139,3cyclictest32149-21cron21:10:002
650399138,4cyclictest32-21ksoftirqd/220:50:012
650399137,3cyclictest18341-21df_abs22:55:132
6503991310,2cyclictest27478-21grep00:20:152
6503991310,2cyclictest19094-21basename00:05:002
650399130,12cyclictest14728-21cut19:25:172
649799139,3cyclictest8564-21tr22:35:141
649799139,3cyclictest22758-21cut20:50:131
649799139,3cyclictest21971-21grep20:45:291
649799139,3cyclictest15227-21ls20:35:001
649799139,2cyclictest7000-21wc22:30:221
649799137,5cyclictest16244-21sed23:55:261
649799136,3cyclictest225-21systemd-journal20:05:001
6497991310,2cyclictest9815-21tr23:45:141
6497991310,2cyclictest7409-21tr21:25:151
6497991310,2cyclictest29743-21cat23:20:001
6497991310,2cyclictest25577-21grep19:45:271
6497991310,2cyclictest24654-21grep22:00:161
6497991310,2cyclictest18593-21apt19:35:001
6497991310,2cyclictest13142-21head21:35:221
649499139,3cyclictest20118-21wc20:45:000
649499134,8cyclictest28391-21munin-run19:55:000
6494991311,1cyclictest6316-21dnf22:30:160
6494991310,2cyclictest31668-21grep21:05:260
6494991310,2cyclictest29746-21grep00:25:160
6494991310,2cyclictest25814-21head23:10:160
6494991310,2cyclictest2181-21cat21:15:000
6494991310,2cyclictest20554-21grep20:45:150
6494991310,2cyclictest15220-21sleep22:50:000
650799129,2cyclictest9833-21tr23:45:143
650799129,2cyclictest9109-21grep20:20:153
650799129,2cyclictest7493-21sed19:10:163
650799129,2cyclictest5345-21sed23:35:173
650799129,2cyclictest4973-21grep22:25:263
650799129,2cyclictest4044-21sed21:15:263
650799129,2cyclictest28438-21grep00:20:263
650799129,2cyclictest27569-21tr23:15:143
650799129,2cyclictest27209-21munin-plugin-st23:15:003
650799129,2cyclictest22160-21grep19:40:153
650799129,2cyclictest18628-21grep20:40:163
650799129,2cyclictest17595-21sort00:00:163
650799129,2cyclictest13701-21grep19:25:003
650799129,2cyclictest11941-21ls22:40:203
650799128,3cyclictest7066-21grep20:15:203
650799128,3cyclictest25446-21cat22:00:253
650799127,4cyclictest13070-21/usr/sbin/munin22:45:143
650399129,2cyclictest9836-21taskset21:30:142
650399129,2cyclictest7669-21cut23:40:172
650399129,2cyclictest6266-21grep23:35:262
650399129,2cyclictest4289-21grep20:10:162
650399129,2cyclictest32-21ksoftirqd/221:45:002
650399129,2cyclictest29773-21sort00:25:162
650399129,2cyclictest17250-21grep19:30:152
650399129,2cyclictest12832-21sh22:45:002
650399128,3cyclictest7939-21tail21:25:162
650399128,2cyclictest1855-21cut20:05:162
650399128,2cyclictest1078-21grep22:20:132
650399126,4cyclictest32-21ksoftirqd/221:00:002
650399123,5cyclictest0-21swapper/220:20:122
650399121,8cyclictest14220-21strings20:30:242
6503991211,1cyclictest0-21swapper/222:50:132
649799129,2cyclictest5338-21sed23:35:171
649799129,2cyclictest25698-21sort23:10:151
649799129,2cyclictest22713-21tr23:05:131
649799129,2cyclictest21956-21tail23:00:251
649799129,2cyclictest17881-21cut22:55:121
649799128,3cyclictest32291-21grep00:30:171
649799128,3cyclictest16200-21tail20:35:161
649799128,2cyclictest24623-21grep00:15:161
649799127,3cyclictest26020-21wc20:55:181
6497991211,1cyclictest38150irq/131-snd_hda23:30:181
6497991210,1cyclictest0-21swapper/123:50:141
649499129,2cyclictest9089-21head20:20:160
649499129,2cyclictest20796-21grep21:50:270
649499129,2cyclictest19897-21grep19:35:160
649499129,2cyclictest14816-21grep19:25:160
649499128,3cyclictest16023-21sed21:40:250
649499128,2cyclictest6968-21tr19:10:130
649499128,2cyclictest4007-21grep22:25:160
649499123,5cyclictest225-21systemd-journal23:55:000
649499123,4cyclictest225-21systemd-journal23:30:000
650799118,2cyclictest8468-21tr22:35:143
650799118,2cyclictest7162-21apache_volume21:25:123
650799118,2cyclictest5762-21cut22:30:133
650799118,2cyclictest4114-21chrt23:32:443
650799118,2cyclictest29783-21apt21:04:593
650799118,2cyclictest29181-21sed22:10:153
650799118,2cyclictest27984-21who22:05:263
650799118,2cyclictest24117-21sleep00:14:593
650799118,2cyclictest22899-21tr23:05:143
650799118,2cyclictest21744-21tr21:55:143
650799118,2cyclictest19890-21chrt22:58:343
650799118,2cyclictest14103-21seq23:53:093
650799118,2cyclictest12863-21perf20:30:003
650799117,3cyclictest17098-21grep20:35:253
650799114,6cyclictest27169-21latency_hist21:00:013
650799113,6cyclictest30763-21munin-run20:00:003
650399118,2cyclictest6072-21basename20:15:142
650399118,2cyclictest5544-21head21:20:172
650399118,2cyclictest31754-21expr20:00:142
650399118,2cyclictest31749-21grep22:15:172
650399118,2cyclictest29587-21run-parts23:20:002
650399118,2cyclictest24810-21turbostat20:55:002
650399118,2cyclictest2461-21cut23:30:142
650399118,2cyclictest23386-21head23:05:172
650399118,2cyclictest23156-21sed19:40:242
650399118,2cyclictest22232-21grep00:10:162
650399118,2cyclictest21324-21taskset00:08:242
650399118,2cyclictest19858-21grep19:35:162
650399118,2cyclictest17280-21sort21:45:172
650399118,2cyclictest16785-21uname20:35:212
650399118,2cyclictest13171-21grep22:45:142
650399118,2cyclictest12172-21cut21:35:152
650399118,2cyclictest11442-21grep20:25:152
650399111,6cyclictest25327-21cpuspeed_turbos23:10:142
650399110,10cyclictest23195-21sed21:55:252
649799118,2cyclictest6511-21sed20:15:161
649799118,2cyclictest5118-21expr21:20:151
649799118,2cyclictest4531-21sendmail-msp21:20:001
649799118,2cyclictest32214-21cut22:15:221
649799118,2cyclictest31816-21taskset21:05:481
649799118,2cyclictest27260-21df_abs00:20:141
649799118,2cyclictest22613-21cat19:40:191
649799118,2cyclictest22008-21sed21:55:151
649799118,2cyclictest1952-21expr00:35:141
649799118,2cyclictest17691-21wc22:55:001
649799118,2cyclictest16817-21grep21:45:141
649799118,2cyclictest13700-21cut22:45:181
649799118,2cyclictest11070-21taskset19:18:051
649499118,2cyclictest8819-21sed22:35:150
649499118,2cyclictest8075-21sh20:20:000
649499118,2cyclictest653-21taskset22:19:460
649499118,2cyclictest643-21taskset20:04:140
649499118,2cyclictest5640-21cat23:35:190
649499118,2cyclictest5377-21grep21:20:160
649499118,2cyclictest3907-21cat21:15:250
649499118,2cyclictest26479-21cron00:20:000
649499118,2cyclictest25697-21taskset19:46:150
649499118,2cyclictest2566-21cut00:35:190
649499118,2cyclictest21694-21egrep00:09:520
649499118,2cyclictest18014-21cat00:00:190
649499118,2cyclictest16228-21sed23:55:260
649499118,2cyclictest15965-21grep22:50:150
649499118,2cyclictest12709-21smartctl21:35:170
650799108,1cyclictest392-21taskset20:01:243
650799108,1cyclictest31480-21chrt00:30:023
650799108,1cyclictest30403-21expr21:05:143
650799108,1cyclictest23825-21ntpq20:50:203
650799108,1cyclictest21185-21taskset00:06:383
650799108,1cyclictest20675-21cat23:00:133
650799108,1cyclictest14025-21cat19:25:003
650799107,2cyclictest5026-21cat20:10:243
650799107,2cyclictest30459-21grep00:25:223
650799107,2cyclictest24620-21head19:45:173
650799107,2cyclictest24374-21apache_accesses00:15:143
650799107,2cyclictest2404-21expr22:20:253
650799107,2cyclictest2212-21cat20:05:183
650799107,2cyclictest19352-21tr19:35:143
650799107,2cyclictest1911-21expr00:35:143
650799107,2cyclictest13867-21head20:30:193
650399108,1cyclictest8306-21cat19:10:242
650399108,1cyclictest5703-21taskset22:30:122
650399108,1cyclictest32534-21wc23:25:152
650399108,1cyclictest324242sleep221:10:122
650399108,1cyclictest30439-21chrt19:56:102
650399108,1cyclictest30298-21expr23:20:122
650399108,1cyclictest26232-21taskset19:50:082
650399108,1cyclictest24670-21cstates00:15:162
650399108,1cyclictest22145-21taskset23:01:472
650399108,1cyclictest19084-21cat21:50:132
650399108,1cyclictest18501-21fschecks_count20:40:162
650399107,2cyclictest27655-21basename21:00:102
650399107,2cyclictest24026-21egrep22:00:132
650399107,2cyclictest15380-21if_enp0s31f623:55:172
650399107,2cyclictest13141-21cat23:50:192
650399107,2cyclictest12285-21grep19:20:152
650399104,3cyclictest225-21systemd-journal21:20:002
650399103,3cyclictest225-21systemd-journal00:35:012
650399100,1cyclictest0-21swapper/222:05:132
649799108,1cyclictest28472sleep122:21:581
649799108,1cyclictest16887-21taskset00:00:101
649799108,1cyclictest158482sleep119:25:291
649799108,1cyclictest15206-21cat21:40:161
649799107,2cyclictest8402-21grep19:10:251
649799107,2cyclictest363-21ls21:10:131
649799107,2cyclictest32507-21grep23:25:161
649799107,2cyclictest29208-21taskset22:10:151
649799107,2cyclictest28827-21grep19:55:131
649799107,2cyclictest18976-21grep20:40:191
649799107,2cyclictest11514-21cut20:25:151
649799107,2cyclictest10636-21grep21:30:201
649499108,1cyclictest7165-21chrt23:40:150
649499108,1cyclictest40372sleep023:31:450
649499108,1cyclictest3672-21aten2_r7power_p20:10:130
649499108,1cyclictest26836-21dnf22:05:150
649499108,1cyclictest24575-21perl22:00:160
649499108,1cyclictest22126-21cut00:10:150
649499108,1cyclictest20509-21grep23:00:140
649499108,1cyclictest16653-21basename21:45:110
649499108,1cyclictest11039-21cat20:25:130
649499108,1cyclictest10396-21basename23:45:170
649499108,1cyclictest1009-21apt-config20:05:000
649499107,3cyclictest24277-21ls23:05:270
649499107,2cyclictest8844-21grep21:25:260
649499107,2cyclictest30914-21wc23:20:170
649499107,2cyclictest30551-21chrt19:57:370
649499107,2cyclictest27922-21tr21:00:140
649499107,2cyclictest1387-21expr22:20:150
649499107,2cyclictest11025-21taskset19:17:300
649499106,2cyclictest16445-21wc20:35:180
649499103,4cyclictest225-21systemd-journal21:35:000
649499102,4cyclictest225-21systemd-journal19:45:010
65079997,1cyclictest262292sleep319:50:073
65079997,1cyclictest19595-21sh21:50:163
65039997,1cyclictest9686-21chrt23:45:132
65039990,1cyclictest0-21swapper/222:25:132
64979997,1cyclictest4525-21cut20:10:171
64979997,1cyclictest1393-21egrep20:05:131
64979992,2cyclictest225-21systemd-journal23:40:431
64949996,2cyclictest32271-21sed00:30:170
64949991,4cyclictest225-21systemd-journal21:55:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional