You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-20 - 17:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu Aug 20, 2026 12:45:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1409217497,24sleep20-21swapper/207:06:382
13142160104,29sleep30-21swapper/307:05:273
15842159107,28sleep10-21swapper/107:08:531
15382159101,28sleep00-21swapper/007:08:170
254922590,2sleep00-21swapper/012:23:570
221572590,2sleep00-21swapper/011:33:400
192782580,1sleep10-21swapper/110:43:541
117072570,3sleep2271rcuc/212:13:402
149702560,2sleep30-21swapper/309:52:313
88872550,1sleep10-21swapper/110:35:261
196952530,3sleep3331rcuc/312:19:553
313542520,2sleep012-21ksoftirqd/010:28:480
300242520,2sleep10-21swapper/111:39:481
58592510,2sleep30-21swapper/311:45:153
167625131,14sleep00-21swapper/007:10:000
291252480,1sleep10-21swapper/112:02:061
76962470,1chrt7697-21/usr/sbin/munin07:20:251
266942460,1sleep00-21swapper/009:37:150
149382460,1sleep00-21swapper/011:51:390
251302420,4sleep325128-21ssh10:00:143
78662400,2sleep20-21swapper/211:46:192
309282400,2sleep30-21swapper/310:28:233
89172370,1sleep10-21swapper/109:24:561
78002330,2sleep30-21swapper/309:23:493
203762320,2sleep00-21swapper/009:32:410
1949993217,9cyclictest52950irq/141-enp3s0-09:45:112
265552310,2sleep20-21swapper/210:49:252
1950993017,7cyclictest21148-21diskmemload11:25:043
1948993021,7cyclictest25856-21/usr/sbin/munin10:25:001
99502290,2sleep30-21swapper/307:25:233
56982290,3sleep30-21swapper/310:57:153
195099282,17cyclictest30739-21ssh11:40:043
1950992710,9cyclictest29923-21diskmemload09:40:123
194899271,24cyclictest0-21swapper/107:16:571
193352270,2chrt0-21swapper/007:50:000
195099261,2cyclictest0-21swapper/307:16:393
194999261,23cyclictest0-21swapper/207:18:572
1947992623,1cyclictest0-21swapper/007:16:170
8182250,2sleep20-21swapper/211:17:502
242062250,3sleep2271rcuc/211:10:312
195099252,17cyclictest23926-21ssh11:35:123
194999258,11cyclictest52950irq/141-enp3s0-10:20:122
194799257,11cyclictest52750irq/139-enp3s0-10:40:100
194799252,15cyclictest4269-21apt-get09:45:020
123042250,1sleep20-21swapper/210:38:412
303842240,11sleep012-21ksoftirqd/012:27:060
195099249,5cyclictest4525-21tr09:45:113
195099242,15cyclictest28324-21sh09:15:113
1949992422,1cyclictest0-21swapper/207:20:512
194999241,11cyclictest194882expr09:05:132
194799240,1cyclictest0-21swapper/007:20:100
1950992311,7cyclictest20400-21grep12:20:143
1948992314,6cyclictest32180-21fw_packets10:05:171
1948992314,6cyclictest11304-21ls11:25:201
1948992312,8cyclictest17059-21cron07:45:001
1948992312,7cyclictest53050irq/142-enp3s0-10:00:131
1948992312,6cyclictest53050irq/142-enp3s0-09:45:031
1947992315,5cyclictest21407-21rm11:56:210
1947992310,10cyclictest75-21kswapd011:10:100
33972220,1sleep00-21swapper/009:44:370
315032220,3sleep00-21swapper/010:52:230
311202220,1sleep00-21swapper/012:04:070
220352221,1sleep10-21swapper/112:20:301
1949992214,6cyclictest25096-21basename11:59:592
1949992213,5cyclictest52950irq/141-enp3s0-12:30:122
1949992212,7cyclictest4549-21cat08:30:152
1949992211,9cyclictest13950-21cut12:15:132
1949992211,8cyclictest28358-21df_inode08:10:132
194999220,9cyclictest245842needreboot08:00:202
194899223,14cyclictest17962-21ssh09:55:041
194899223,14cyclictest17962-21ssh09:55:031
1948992214,6cyclictest23243-21memory11:10:191
1948992213,7cyclictest27676-21seq09:14:471
1948992211,8cyclictest12824-21apt-get08:50:141
194799222,14cyclictest2148-21ssh10:55:110
1947992214,6cyclictest10048-21ssh12:35:270
1947992210,8cyclictest31646-21apt-get10:05:120
75102210,2sleep20-21swapper/211:22:432
75102210,2sleep20-21swapper/211:22:422
52442211,6sleep20-21swapper/209:21:172
270982210,1sleep10-21swapper/109:37:391
245032210,3sleep31945-21cyclictest11:59:283
233512210,1sleep30-21swapper/310:22:323
229112210,1sleep30-21swapper/309:58:293
229112210,1sleep30-21swapper/309:58:283
219102210,2sleep30-21swapper/307:55:143
195099219,7cyclictest52850irq/140-enp3s0-12:00:113
195099212,8cyclictest28699-21munin-run11:14:593
1950992116,3cyclictest52850irq/140-enp3s0-11:19:083
1950992114,4cyclictest11964-21/usr/sbin/munin11:50:173
1950992113,6cyclictest5859-21seq10:10:063
1949992113,6cyclictest560-21dbus-daemon11:25:002
1949992113,6cyclictest25417-21ls11:35:242
1949992113,5cyclictest18184-21ntpq11:30:192
1949992112,6cyclictest29345-21ntpq09:15:192
1949992111,6cyclictest52950irq/141-enp3s0-12:00:232
194899218,10cyclictest22193-21basename07:55:171
194899215,4cyclictest66050lldpd11:48:581
1948992113,6cyclictest29905-21cron08:15:001
1948992113,6cyclictest27608-21sed12:25:171
1948992113,6cyclictest20298-21seq11:55:241
1948992112,6cyclictest16017-21cut09:30:001
194899211,10cyclictest26619-21df_inode10:25:141
1947992113,6cyclictest278-21systemd-journal08:40:000
1947992112,7cyclictest9706-21python308:40:240
1947992111,8cyclictest9921-21perl09:25:160
194799211,13cyclictest10271-21sh11:25:040
102082210,1sleep00-21swapper/012:12:080
277282200,2sleep20-21swapper/209:38:172
195099209,7cyclictest15014-21cat10:40:193
1950992014,4cyclictest31629-21acpi10:05:113
1950992012,6cyclictest7307-21ntp_states08:35:223
1950992011,7cyclictest14688-21wc12:40:003
1950992010,7cyclictest7413-21cat12:10:163
1950992010,7cyclictest17920-21sed09:00:193
1950992010,2cyclictest34-21ksoftirqd/312:25:393
194999209,6cyclictest52950irq/141-enp3s0-09:55:132
194999209,6cyclictest52950irq/141-enp3s0-09:55:122
1949992014,4cyclictest25550-21idleruntime10:00:162
1949992013,5cyclictest12315-21basename10:15:012
1949992012,6cyclictest7270-21ntp_kernel_err08:35:202
1949992010,7cyclictest15563-21cat08:55:172
1948992013,4cyclictest32424-21/usr/sbin/munin12:05:241
1948992012,6cyclictest6077-21seq08:34:261
1948992012,6cyclictest28230-21seq09:15:081
1948992012,5cyclictest12082-21ls09:50:181
1948992011,7cyclictest14472-21seq11:51:101
1948992011,6cyclictest278-21systemd-journal12:35:001
1948992011,6cyclictest13232-21cat12:15:011
194899201,13cyclictest580-21ssh10:30:121
194799209,7cyclictest1422-21wc12:05:250
194799205,12cyclictest13428-21cat10:15:160
194799201,5cyclictest0-21swapper/009:54:380
1947992014,4cyclictest15704-21sh12:15:550
1947992012,6cyclictest15575-21cut07:40:150
194799201,12cyclictest636-21runrttasks11:00:120
1947992010,7cyclictest17447-21df_inode09:00:140
300972190,2sleep10-21swapper/109:40:131
241382191,3sleep024140-21ssh09:59:430
241382191,3sleep024140-21ssh09:59:420
195099198,8cyclictest0-21swapper/312:05:173
195099193,13cyclictest15504-21cut08:55:163
1950991917,1cyclictest0-21swapper/307:20:033
1950991913,4cyclictest34-21ksoftirqd/309:30:013
1950991912,5cyclictest14047-21which07:35:223
1950991912,5cyclictest14047-21which07:35:213
1950991912,4cyclictest13173-21sh10:15:153
1950991911,6cyclictest2912-21ntpq07:10:203
1950991911,6cyclictest19770-21expr07:50:103
1950991910,6cyclictest28721-21grep08:10:183
194999197,9cyclictest26239-21apt-get10:25:102
194999197,3cyclictest2738-21gltestperf10:55:152
194999193,8cyclictest15640-21apt-get11:05:112
194999191,9cyclictest31713-21ssh11:40:172
1949991914,3cyclictest45392sleep212:08:152
1949991913,4cyclictest17166-21sh10:18:072
1949991912,5cyclictest8788-21perl12:35:152
1949991912,5cyclictest26232-21diskstats08:05:172
1949991912,5cyclictest11661-21perl09:50:162
1949991912,4cyclictest3011-21processes07:10:202
1949991912,4cyclictest1696-21sh09:42:542
1949991911,6cyclictest9855-21expr09:25:152
1949991911,6cyclictest5716-21date10:10:012
1949991910,7cyclictest12409-21seq08:48:072
1949991910,6cyclictest30501-21df08:15:142
1949991910,6cyclictest17721-21cat09:00:152
194899199,7cyclictest10085-21cut11:00:221
1948991912,5cyclictest11303-21if_err_enp3s008:45:141
1948991911,6cyclictest5021-21date11:20:231
1948991911,6cyclictest5021-21date11:20:221
1948991911,6cyclictest31109-21seq10:51:591
1948991911,6cyclictest3054-21cut07:10:211
1948991911,6cyclictest2552-21grep08:25:151
1948991911,6cyclictest18818-21seq09:02:481
194899191,11cyclictest0-21swapper/111:09:271
1948991910,7cyclictest10131-21awk07:25:251
1948991910,6cyclictest7392-21which08:35:231
1948991910,6cyclictest3452-21sensors_temp10:55:221
194799198,3cyclictest255742rm10:48:250
194799192,11cyclictest5235-21apt-get11:45:120
1947991913,4cyclictest13995-21rt-features07:35:210
1947991913,4cyclictest13995-21rt-features07:35:210
1947991913,3cyclictest3851-21sh11:44:030
1947991912,5cyclictest15911-21awk08:55:220
1947991912,4cyclictest4857-21sh10:32:570
1947991912,4cyclictest2117-21sh11:19:060
1947991910,7cyclictest4744-21seq09:20:450
1947991910,6cyclictest28292-21cat09:15:100
1947991910,6cyclictest0-21swapper/008:10:000
195099189,7cyclictest560-21dbus-daemon08:05:003
195099189,6cyclictest18097-21sed07:45:183
195099189,5cyclictest0-21swapper/309:35:133
195099188,7cyclictest7359-21ssh10:35:133
195099182,11cyclictest30167-21apt-get08:15:113
195099182,10cyclictest15670-21ssh11:05:133
1950991811,5cyclictest560-21dbus-daemon09:35:013
1950991811,5cyclictest4541-21cat08:30:143
1950991811,5cyclictest27908-21cat10:50:133
1950991811,5cyclictest26114-21expr08:05:163
1950991811,4cyclictest21148-21diskmemload11:00:243
1950991810,6cyclictest2261-21grep12:30:193
1950991810,5cyclictest21878-21cat09:10:133
1950991810,5cyclictest0-21swapper/311:34:593
194999189,7cyclictest8820-21ls07:25:132
194999182,12cyclictest20266-21ssh12:20:122
194999182,11cyclictest32328-21apt-get08:20:102
194999182,11cyclictest27813-21apt-get10:50:122
1949991812,4cyclictest26942-21sh12:25:112
1949991811,5cyclictest9461-21expr11:00:162
1949991811,5cyclictest2692-21cat08:25:152
1949991811,5cyclictest21913-21expr07:55:132
1949991811,4cyclictest23852-21sh09:10:582
1949991810,6cyclictest793-21cat10:30:142
1949991810,5cyclictest15386-21run-parts10:40:222
194899189,7cyclictest21329-21wc09:10:011
194899183,4cyclictest21148-21diskmemload11:32:191
1948991813,3cyclictest12534-21sh10:15:021
1948991812,4cyclictest9142-21sh10:11:501
1948991812,4cyclictest53050irq/142-enp3s0-10:45:151
1948991811,5cyclictest1730-21seq08:24:161
1948991811,5cyclictest14466-21ntp_kernel_err12:15:171
194899181,12cyclictest14961-21apt-get08:55:121
194899181,11cyclictest16246-21apt-get09:30:041
1948991810,6cyclictest7963-21sed12:35:021
194799189,7cyclictest19588-21df_inode09:05:140
194799189,6cyclictest3462-21unixbench_multi08:25:230
194799189,6cyclictest1996-21sed08:25:010
194799182,5cyclictest24883-21apt-get10:00:120
194799182,12cyclictest7026-21ntp_states10:10:210
1947991813,3cyclictest636-21runrttasks08:30:240
1947991813,3cyclictest13738-21ntp_states08:50:220
1947991812,3cyclictest3706-21sh11:20:120
1947991812,3cyclictest3706-21sh11:20:110
1947991811,5cyclictest21536-21date07:55:010
1947991811,4cyclictest22167-21sed09:10:160
1947991811,4cyclictest21148-21diskmemload12:33:470
1947991810,6cyclictest31538-21cat08:15:250
1947991810,6cyclictest21041-21sed10:20:260
1947991810,5cyclictest30029-21cut08:14:590
194799180,10cyclictest970050kworker/u8:111:05:110
195099179,6cyclictest12131-21unixbench_singl08:45:233
195099179,5cyclictest20592-21sed09:05:233
195099178,6cyclictest320-21systemd-udevd08:40:213
195099172,9cyclictest208502sleep310:45:123
195099171,8cyclictest2032-21/usr/sbin/munin08:25:183
1950991712,3cyclictest21148-21diskmemload10:30:233
1950991711,3cyclictest4202-21sh11:20:163
1950991711,3cyclictest4202-21sh11:20:153
1950991710,5cyclictest15379-21expr07:40:133
1949991712,3cyclictest8524-21acpi08:40:132
1949991712,3cyclictest15160-21/usr/sbin/munin07:40:112
1949991712,3cyclictest12799-21/usr/sbin/munin08:50:172
194999171,14cyclictest16758-21df09:30:132
1949991711,4cyclictest12365-21sh11:50:152
1949991710,5cyclictest20089-21cat07:50:142
194899179,6cyclictest30524-21cat11:15:231
1948991712,3cyclictest32699-21sh11:40:301
1948991711,4cyclictest29478-21munin-node08:00:121
194899171,12cyclictest10781-21apt-get07:30:111
1948991710,5cyclictest26965-21ps08:05:231
1948991710,5cyclictest20669-21cat07:50:201
1948991710,5cyclictest17593-21expr07:45:131
1948991710,5cyclictest13042-21grep07:35:131
1948991710,5cyclictest13042-21grep07:35:121
194799179,6cyclictest9449-21cat07:25:180
194799179,5cyclictest22494-21ntp_kernel_pll_07:55:200
1947991711,3cyclictest21148-21diskmemload10:37:380
1947991710,5cyclictest12479-21seq07:32:480
1947991710,4cyclictest0-21swapper/011:35:210
195099168,6cyclictest32417-21expr08:20:123
1950991610,4cyclictest13985-21swap08:50:253
1950991610,4cyclictest11011-21expr07:30:123
194999169,5cyclictest13097-21expr07:35:132
194999169,5cyclictest13097-21expr07:35:122
1949991611,3cyclictest17295-21/usr/sbin/munin07:45:102
194899169,5cyclictest9618-21cut08:40:241
1948991611,3cyclictest30550-21fschecks_count08:15:141
194799169,5cyclictest24029-21expr08:00:130
194799168,6cyclictest11895-21cat08:45:220
1949991510,3cyclictest11183-21fschecks_time07:30:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional