You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-22 - 00:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu May 21, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
255062187166,14sleep20-21swapper/207:09:042
253182165110,28sleep30-21swapper/307:06:363
253832162104,28sleep10-21swapper/107:07:261
252532161107,27sleep00-21swapper/007:05:440
79422740,2sleep00-21swapper/011:05:210
16772730,9sleep122-21ksoftirqd/111:49:041
230812710,1sleep10-21swapper/112:05:011
89082690,6sleep18912-21cpuspeed_turbos07:45:141
237032620,2sleep10-21swapper/111:16:531
153902620,3sleep215376-21ssh11:10:242
268822600,2sleep20-21swapper/212:07:142
80872580,1sleep30-21swapper/309:30:003
169742580,1sleep016978-21cut12:00:140
23082560,2sleep10-21swapper/112:36:431
213862560,1sleep00-21swapper/012:26:560
213862560,1sleep00-21swapper/012:26:550
275062530,2sleep30-21swapper/312:31:153
259222520,2sleep00-21swapper/011:42:370
278302510,1sleep10-21swapper/110:55:421
267622510,1sleep10-21swapper/109:43:351
188312500,1sleep10-21swapper/111:37:161
132432480,1sleep10-21swapper/109:33:311
50782460,2sleep00-21swapper/011:03:400
280302460,1sleep20-21swapper/210:55:552
93342420,2chrt0-21swapper/110:42:221
303432410,3sleep00-21swapper/012:10:130
83202400,2chrt0-21swapper/211:53:592
2606099383,14cyclictest7782-21ssh11:05:183
46642360,2sleep30-21swapper/312:15:143
103522360,3sleep110351-21ssh09:55:141
2606099341,9cyclictest1421-21mailstats11:00:223
2605699348,7cyclictest32598-21missed_timers12:35:190
26056993412,6cyclictest25423-21sed10:55:000
26056993411,6cyclictest27812-21seq08:26:570
2606099332,10cyclictest32143-21tune2fs12:35:153
2605699339,6cyclictest560-21dbus-daemon07:20:000
26056993310,5cyclictest27977-21expr09:20:160
26056993310,5cyclictest27977-21expr09:20:160
135682330,2chrt0-21swapper/311:57:293
323202320,2chrt0-21swapper/309:23:393
323202320,2chrt0-21swapper/309:23:383
2606099322,11cyclictest25324-21cat08:20:233
2605699329,7cyclictest560-21dbus-daemon12:20:010
2605699329,5cyclictest6617-21cut07:40:000
26056993213,4cyclictest8274-21/usr/sbin/munin11:30:130
26056993210,5cyclictest2725-21sed08:45:000
2606099312,10cyclictest13297-21ssh10:21:003
2605699311,7cyclictest4620-21sed11:50:240
26056993113,3cyclictest6835-21latency_hist08:55:010
2605699310,29cyclictest0-21swapper/012:32:230
2606099304,11cyclictest23073-21sh10:52:423
2606099303,17cyclictest15828-21ssh09:10:363
2606099303,10cyclictest16359-21sed10:00:013
2606099302,13cyclictest29283-21grep11:45:173
2605899301,27cyclictest26946-21chrt12:30:432
2605699309,5cyclictest9385-21needreboot09:30:210
2605699307,6cyclictest13662-21sed07:55:170
26056993012,3cyclictest443-21sh11:24:180
26056993012,3cyclictest30131-21sh11:45:260
26056993012,3cyclictest18552-21sh11:13:270
2605699301,11cyclictest22418-21grep09:40:140
26056993010,4cyclictest13358-21sh10:21:050
26056993010,3cyclictest21563-21sendmail11:40:000
26060992923,4cyclictest24367-21ssh12:29:533
26060992923,4cyclictest24367-21ssh12:29:523
26057992920,7cyclictest12155-21hddtemp_smartct09:05:171
2605699298,5cyclictest10208-21fschecks_count11:55:150
26056992913,3cyclictest10961-21uptime09:00:260
26056992911,3cyclictest27262-21sh10:31:130
2605699291,10cyclictest4812-21fschecks_time10:15:160
26056992910,5cyclictest12-21ksoftirqd/012:20:210
2606099284,10cyclictest13488-21diskmemload09:40:353
2606099281,10cyclictest1706-21rm10:12:493
2606099280,9cyclictest640-21snmpd09:53:043
2605799280,26cyclictest27597-21ssh12:31:211
2605699281,13cyclictest43092wc09:50:230
26056992811,2cyclictest8000-21latency_hist09:30:000
26056992810,3cyclictest13488-21diskmemload10:39:320
2606099274,9cyclictest7738-21sh10:40:463
2606099274,9cyclictest2816-21sh10:37:313
2606099274,10cyclictest27606-21sh10:08:023
2606099274,10cyclictest27606-21sh10:08:013
2606099271,9cyclictest5777-21ssh11:51:253
2606099271,11cyclictest4865-21ssh11:26:583
2606099270,9cyclictest23283-21apt-get12:05:123
26057992720,5cyclictest22-21ksoftirqd/108:19:591
2605799271,25cyclictest0-21swapper/112:29:031
2605799271,25cyclictest0-21swapper/112:29:021
26056992711,5cyclictest12-21ksoftirqd/008:45:190
2605699271,13cyclictest20368-21ssh10:26:140
2605699271,12cyclictest12354-21apt-get10:45:110
26056992710,3cyclictest20715-21sh09:39:180
2606099265,4cyclictest0-21swapper/311:35:133
2606099264,8cyclictest13488-21diskmemload09:15:303
2606099262,8cyclictest24113-21ssh11:17:173
2606099261,9cyclictest30550-21expr07:20:123
2606099261,8cyclictest18091-21ssh11:12:593
2606099261,8cyclictest0-21swapper/310:25:123
2606099261,23cyclictest0-21swapper/312:22:573
2605899261,2cyclictest13488-21diskmemload12:29:592
2605899261,2cyclictest13488-21diskmemload12:29:582
26056992611,3cyclictest16420-21sh10:00:030
57942250,9sleep228-21ksoftirqd/211:27:552
2606099253,9cyclictest2648-21rm12:13:403
2606099253,9cyclictest23180-21ps11:40:213
2606099252,8cyclictest27285-21apt11:20:123
2606099252,6cyclictest27634-21ssh10:55:313
2606099251,9cyclictest20319-21ssh09:38:543
2606099251,7cyclictest8356-21ssh09:30:083
2605899250,24cyclictest0-21swapper/211:47:412
26056992512,3cyclictest18300-21ntp_states08:05:210
158252250,10sleep228-21ksoftirqd/211:59:442
314532240,1sleep10-21swapper/107:20:211
2606099242,8cyclictest5392-21ntp_states10:15:223
2606099242,8cyclictest29974-21python309:45:243
2605899243,14cyclictest16555-21ssh12:00:122
2605799244,12cyclictest32534-21ssh11:00:111
26057992411,3cyclictest1320-21latency_hist11:25:011
2606099238,12cyclictest31790-21/usr/sbin/munin08:35:193
2606099236,10cyclictest34-21ksoftirqd/311:31:353
2606099232,6cyclictest0-21swapper/310:00:173
26060992320,2cyclictest331rcuc/310:45:493
26058992310,9cyclictest32039-21df12:35:122
26057992315,6cyclictest27283-21ssh09:20:051
26057992315,6cyclictest27283-21ssh09:20:051
253222230,1sleep10-21swapper/111:42:011
227802230,2sleep122-21ksoftirqd/110:28:421
50482220,2sleep20-21swapper/210:15:182
326252220,1sleep00-21swapper/009:47:430
313772220,2sleep10-21swapper/109:46:291
2606099228,11cyclictest438-21/usr/sbin/munin07:25:153
2606099225,8cyclictest34-21ksoftirqd/307:45:223
2606099224,8cyclictest34-21ksoftirqd/312:01:083
2606099223,6cyclictest23578-21taskset08:17:523
2606099223,16cyclictest11501-21latency_hist09:05:003
26060992213,6cyclictest25623-21ls10:30:193
26060992212,7cyclictest13099-21awk09:05:253
2605899229,4cyclictest1354-21cron08:00:002
26058992215,5cyclictest8910-21grep07:45:132
26058992213,6cyclictest6737-21pluginstate10:40:202
2605799228,9cyclictest13488-21diskmemload11:30:101
2605799223,7cyclictest27519-21ssh10:07:571
2605799223,7cyclictest27519-21ssh10:07:561
26057992214,6cyclictest6847-21ls08:55:001
26057992213,7cyclictest16895-21seq08:02:091
2605799221,15cyclictest12381-21ssh10:45:111
2605699221,15cyclictest2277-21chrt07:30:110
172262220,2sleep20-21swapper/210:00:172
2606099213,6cyclictest6530-21taskset08:51:123
2606099213,6cyclictest4513-21taskset08:47:473
2606099212,8cyclictest12755-21runrttasks07:54:103
2605899213,15cyclictest29600-21grep10:10:002
2605899213,15cyclictest29600-21grep10:10:002
2605899212,13cyclictest14643-21apt-get11:35:032
26058992115,4cyclictest28-21ksoftirqd/209:00:002
26058992114,4cyclictest13488-21diskmemload10:25:232
26058992113,5cyclictest20527-21timerandwakeup10:50:242
26058992112,7cyclictest25806-21ntp_states10:30:232
2605799213,6cyclictest14651-21ssh10:22:221
26057992116,3cyclictest16115-21munin-plugin-st12:00:011
26057992112,7cyclictest31208-21pluginstate12:10:221
26057992112,6cyclictest7996-21ntpq08:55:191
26057992111,7cyclictest14931-21seq07:59:401
2605699212,15cyclictest24234-21tr12:05:180
26056992113,6cyclictest31748-21users10:10:270
2606099209,8cyclictest26359-21cat08:25:133
2606099208,10cyclictest26504-21tune2fs07:10:143
2606099202,6cyclictest30070-21runrttasks08:30:463
2606099202,5cyclictest0-21swapper/307:55:263
26060992011,7cyclictest30194-21cat07:20:003
26060992010,7cyclictest16236-21awk08:00:213
26058992014,4cyclictest6259-21sh12:15:322
26058992012,6cyclictest12970-21sed12:20:252
2605899201,17cyclictest31113-21ntp_kernel_pll_12:10:202
26058992011,6cyclictest22030-21cpu11:40:122
26058992011,6cyclictest20272-21ls08:10:182
2605799203,5cyclictest9827-21ssh11:06:401
2605799201,2cyclictest0-21swapper/110:39:591
26057992012,6cyclictest12704-21seq07:53:311
26057992011,6cyclictest4530-21cat12:15:141
26057992011,6cyclictest3988-21awk08:45:241
26056992014,4cyclictest31237-21sh10:59:100
26056992014,3cyclictest657-21sshd10:05:180
26056992014,3cyclictest657-21sshd10:05:170
26056992012,6cyclictest12825-21cron07:55:000
2606099199,7cyclictest3234-21cut07:30:223
2606099192,6cyclictest7054-21runrttasks08:55:053
26060991913,3cyclictest7946-21timerandwakeup07:40:243
26060991911,6cyclictest18572-21cat08:05:233
2605899199,7cyclictest3118-21apt-get09:50:142
2605899192,12cyclictest670-21apt-get08:40:112
2605899191,4cyclictest0-21swapper/210:24:022
26058991914,3cyclictest636-21runrttasks08:45:192
26058991912,5cyclictest24224-21expr08:20:122
26058991912,5cyclictest22154-21expr08:15:142
26058991911,6cyclictest7510-21cat11:05:152
26058991911,6cyclictest22705-21cut09:40:172
26058991911,6cyclictest17834-21cut08:05:152
26058991911,6cyclictest10314-21perl09:55:142
26058991910,7cyclictest11407-21tune2fs07:50:152
26058991910,6cyclictest6630-21sed07:40:002
2605799197,10cyclictest22-21ksoftirqd/109:30:001
26057991913,4cyclictest18609-21sh11:13:311
26057991912,5cyclictest30619-21seq08:34:231
26057991912,5cyclictest21715-21ntp_kernel_pll_09:15:211
26057991911,6cyclictest560-21dbus-daemon11:55:001
26057991911,6cyclictest4006-21memory09:50:201
26057991911,6cyclictest28590-21diskstats07:15:151
26057991911,6cyclictest21407-21seq10:03:321
26057991911,6cyclictest19755-21expr08:10:131
26057991911,6cyclictest13953-21ssh09:10:131
26057991911,5cyclictest25379-21timerwakeupswit08:20:241
26057991911,5cyclictest10354-21ntp_kernel_pll_09:00:191
26057991910,7cyclictest7832-21python307:40:231
26057991910,6cyclictest24963-21expr10:30:131
2605699199,7cyclictest15217-21seq08:00:090
26056991914,3cyclictest8114-21processes08:55:200
26056991914,3cyclictest5626-21apt10:40:090
26056991913,4cyclictest24152-21sh11:17:190
26056991912,5cyclictest25837-21seq08:24:080
26056991911,6cyclictest794-21sed07:25:170
26056991911,6cyclictest10410-21seq07:46:280
26056991910,7cyclictest20859-21cat08:10:240
47102180,3chrt271rcuc/211:03:182
2606099187,7cyclictest4407-21apt-get07:35:103
26060991810,6cyclictest20099-21cat08:10:163
2606099180,6cyclictest2528-21runrttasks08:44:473
2605899182,11cyclictest4953-21apt-get08:50:122
2605899181,8cyclictest13943-21ssh09:10:122
26058991813,3cyclictest30183-21cut10:10:122
26058991813,3cyclictest13488-21diskmemload11:21:182
26058991813,3cyclictest11331-21cron09:04:592
26058991812,4cyclictest14652-21perf11:35:002
26058991811,5cyclictest7025-21forks07:40:142
26058991811,5cyclictest52950irq/141-enp3s0-10:46:082
26058991811,5cyclictest4899-21seq09:26:522
26058991811,5cyclictest32294-21sed10:35:182
26058991811,5cyclictest20508-21basename11:15:122
26058991811,5cyclictest11913-21expr09:05:142
2605899181,15cyclictest0-21swapper/209:32:232
26058991811,4cyclictest13488-21diskmemload09:20:162
26058991811,4cyclictest13488-21diskmemload09:20:152
26058991810,6cyclictest31606-21sed07:20:232
26058991810,6cyclictest30699-21basename08:35:002
26058991810,6cyclictest15899-21uname08:00:172
2605799181,5cyclictest0-21swapper/109:35:221
26057991813,3cyclictest6443-21sh10:15:511
26057991813,3cyclictest28055-21sh08:30:001
2605799181,2cyclictest141rcu_preempt08:35:141
26057991812,4cyclictest31687-21sh10:10:251
26057991812,4cyclictest27550-21sh12:07:551
26057991811,5cyclictest2495-21sed11:25:191
26057991810,6cyclictest23911-21seq10:53:321
26057991810,6cyclictest12745-21swap12:20:241
26057991810,6cyclictest1238-21sed08:40:151
26056991813,3cyclictest12499-21sh09:56:220
26056991811,5cyclictest14676-21awk09:10:170
26056991810,6cyclictest30922-21cat08:35:100
26056991810,6cyclictest2552-21needreboot11:25:190
26056991810,6cyclictest13448-21seq09:09:470
2605899172,6cyclictest28-21ksoftirqd/207:15:122
2605899171,3cyclictest0-21swapper/209:15:132
26058991712,3cyclictest31048-21sh09:46:082
26058991711,4cyclictest26991-21sed08:25:192
26058991710,5cyclictest32675-21expr07:25:142
26058991710,5cyclictest26232-21grep07:10:122
26058991710,5cyclictest15473-21cat09:35:132
2605799174,11cyclictest4148-21cron07:34:591
26057991712,3cyclictest26866-21missed_timers07:10:171
26057991711,4cyclictest2013-21sh07:29:591
26057991710,5cyclictest6563-21grep07:40:001
2605699172,10cyclictest28069-21apt-get08:30:030
26056991712,3cyclictest27560-21unixbench_singl07:10:240
26056991712,3cyclictest23491-21sh09:16:310
26056991712,3cyclictest21847-21/usr/sbin/munin08:15:260
2605899169,5cyclictest2594-21expr07:30:132
2605899168,6cyclictest395-21basename08:40:002
26057991611,3cyclictest18478-21sendmail_mailst08:05:221
2605699169,5cyclictest7811-21tr07:40:230
2605699158,5cyclictest31563-21mailstats07:20:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional