You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-27 - 03:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed May 27, 2026 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
38572222102,96sleep00-21swapper/019:08:360
3941221798,94sleep20-21swapper/219:09:422
37862142106,25sleep10-21swapper/119:07:401
37562141103,27sleep30-21swapper/319:07:183
716821030,1sleep00-21swapper/022:57:270
277152670,2chrt0-21swapper/200:00:302
28152580,3sleep122-21ksoftirqd/122:05:231
120732550,2sleep00-21swapper/022:36:450
96012510,1sleep30-21swapper/321:23:053
78532510,1sleep20-21swapper/222:10:102
159632500,1sleep00-21swapper/023:28:340
138162490,2sleep228-21ksoftirqd/223:26:232
188112480,2sleep00-21swapper/019:40:250
301232470,1sleep30-21swapper/322:50:213
87072460,2chrt0-21swapper/323:22:593
91012440,1sleep10-21swapper/123:47:171
241312420,1sleep00-21swapper/022:21:320
80522400,2sleep00-21swapper/022:34:280
444399372,33cyclictest75-21kswapd023:15:003
4442993710,5cyclictest0-21swapper/222:25:122
444299353,13cyclictest8228-21ssh00:35:132
4439993513,19cyclictest23091-21cron23:10:000
444399342,13cyclictest5694-21sh21:44:423
4439993411,7cyclictest14893-21seq19:34:180
4439993410,6cyclictest5989-21ls23:20:240
444399336,9cyclictest23416-21rm23:34:203
444399336,9cyclictest23416-21rm23:34:203
444399332,17cyclictest26951-21apt-get00:25:103
444299330,30cyclictest0-21swapper/223:14:552
443999338,5cyclictest7840-21expr21:45:230
4439993312,18cyclictest24806-21seq22:46:040
4439993310,5cyclictest14904-21expr22:15:130
443999330,8cyclictest31425-21irqstats21:15:170
4439993213,16cyclictest18377-21sh23:30:160
4439993213,16cyclictest18377-21sh23:30:150
4439993211,6cyclictest3899-21seq20:23:580
4439993211,5cyclictest1659-21seq22:29:470
21642320,3sleep20-21swapper/223:42:052
444399314,8cyclictest23512-21ssh21:33:313
444399314,16cyclictest186091ls19:40:233
444399312,11cyclictest4893-21sh22:31:173
444399311,6cyclictest0-21swapper/323:15:103
444399311,12cyclictest1015-21sh23:40:553
443999319,6cyclictest25840-21smartctl23:35:230
443999319,5cyclictest23460-21seq23:58:000
443999316,6cyclictest14863-21cat20:50:000
443999311,7cyclictest3982-21cron23:20:000
4439993113,15cyclictest31220-21sh00:04:010
4439993112,16cyclictest14535-21sh21:26:160
444399305,10cyclictest14589-21rm22:39:173
444399304,9cyclictest26715-21ssh22:00:023
444399302,10cyclictest3414-21apt-get22:55:113
443999301,6cyclictest75-21kswapd022:10:100
443999301,27cyclictest0-21swapper/023:13:000
4439993012,15cyclictest14825-21sh23:03:270
4439993011,16cyclictest636-21runrttasks21:41:040
444399295,11cyclictest9207-21rm23:47:243
444399294,15cyclictest14636-21sh23:27:133
444399293,9cyclictest20467-21mailstats23:55:223
444399293,12cyclictest9634-21systemctl20:35:233
444399292,12cyclictest14231-21date22:15:003
444399292,10cyclictest30221-21apt-get20:10:113
444399291,7cyclictest24282-21ssh21:58:023
444399291,10cyclictest14234-21sh23:02:513
443999299,5cyclictest20509-21fschecks_time21:55:150
443999298,6cyclictest15281-21cat19:35:120
443999298,5cyclictest5107-21cat19:10:160
443999298,5cyclictest19419-21grep21:00:130
443999296,5cyclictest7265-21df00:10:150
443999296,5cyclictest20131-21if_err_enp3s019:45:140
443999293,22cyclictest0-21swapper/022:05:170
443999293,10cyclictest21809-21cut21:05:150
443999292,8cyclictest13346-21ssh21:50:120
443999291,6cyclictest26740-21sh22:00:030
4439992912,14cyclictest9992-21sh21:23:290
444399282,9cyclictest32592-21sh00:05:113
4442992818,1cyclictest0-21swapper/223:07:222
4440992823,3cyclictest25979-21sh23:11:101
4440992823,3cyclictest23080-21ssh23:10:011
444099281,25cyclictest0-21swapper/123:30:121
444099281,25cyclictest0-21swapper/123:30:111
443999289,3cyclictest24168-21sh21:10:150
443999285,5cyclictest11886-21switchtime20:40:230
443999281,8cyclictest32233-21ntp_kernel_pll_23:40:190
443999281,8cyclictest18663-21cat23:55:000
443999281,5cyclictest16120-21df22:40:130
4439992811,3cyclictest32370-21rm22:51:580
443999281,10cyclictest9345-21pmu-power00:35:220
443999281,10cyclictest28064-21pmu-power00:25:190
443999281,10cyclictest15348-21cut20:50:150
138542280,2sleep10-21swapper/120:45:221
61232270,2chrt0-21swapper/300:33:353
444399274,8cyclictest27118-21sh21:12:033
444399274,8cyclictest13813-21sh00:39:203
444399273,9cyclictest23343-21diskmemload00:10:353
444399272,8cyclictest26259-21ssh23:35:283
4443992721,4cyclictest22760-21ssh23:09:423
444399271,13cyclictest18314-21sh22:17:263
444399271,12cyclictest1871-21ssh22:29:593
4442992717,5cyclictest21474-21cpuspeed_turbos22:20:142
444299271,2cyclictest23343-21diskmemload23:04:012
4442992712,8cyclictest25442-21ssh21:35:132
444099271,25cyclictest0-21swapper/123:19:591
443999276,6cyclictest11023-21tr19:25:120
443999276,4cyclictest344-21tr00:05:150
443999274,6cyclictest17896-21cat20:55:190
443999271,8cyclictest560-21dbus-daemon20:10:000
443999271,7cyclictest22316-21ssh00:20:430
443999271,13cyclictest18733-21ssh21:30:140
443999271,12cyclictest7111-21tail19:15:160
4439992711,14cyclictest6366-21/usr/sbin/munin20:30:200
443999270,8cyclictest1392-21cut00:30:130
93232260,2sleep228-21ksoftirqd/223:47:302
444399264,11cyclictest14543-21sh23:51:003
444399263,8cyclictest17941-21sh00:18:053
444399263,8cyclictest17941-21sh00:18:053
444399261,9cyclictest24996-21sh22:46:153
444299261,24cyclictest0-21swapper/223:19:482
444299261,23cyclictest0-21swapper/223:20:582
4440992624,1cyclictest0-21swapper/123:20:561
444099261,24cyclictest0-21swapper/123:26:461
444099260,24cyclictest0-21swapper/123:03:431
443999269,3cyclictest1110-21switchtime20:15:230
443999268,2cyclictest5715-21cstates23:45:130
443999266,5cyclictest10076-21timerandwakeup19:20:240
443999261,8cyclictest25391-21/usr/sbin/munin21:35:110
29512260,1sleep30-21swapper/321:18:223
444399253,11cyclictest1104950ssh21:48:213
444399251,8cyclictest23561-21sh22:20:573
444399251,8cyclictest15528-21sh21:50:543
4442992522,2cyclictest0-21swapper/219:13:532
4440992523,1cyclictest0-21swapper/119:14:181
443999256,6cyclictest12-21ksoftirqd/020:00:120
443999251,9cyclictest5170-21ntp_states20:25:220
443999250,3cyclictest141rcu_preempt00:15:280
443999250,3cyclictest141rcu_preempt00:15:270
444399242,8cyclictest12053-21mailstats19:25:213
444399241,8cyclictest25410-21apt-get21:35:123
444399240,8cyclictest28521-21sh00:01:193
444399240,23cyclictest0-21swapper/319:13:563
444299247,13cyclictest17815-21ssh23:30:122
444299247,13cyclictest17815-21ssh23:30:112
443999241,9cyclictest21985-21diskstats19:50:160
180672240,2sleep20-21swapper/221:53:302
16222240,2sleep30-21swapper/322:05:143
70342231,19sleep20-21swapper/221:20:312
444399233,7cyclictest15514-21diskmemload21:27:143
4443992313,7cyclictest1092-21sensors_temp20:15:233
444299239,7cyclictest18723-21ssh21:30:132
444299237,3cyclictest160452uname20:50:222
444299232,7cyclictest28-21ksoftirqd/200:14:182
444299232,14cyclictest23343-21diskmemload00:05:152
4442992313,7cyclictest10408-21cron20:40:002
4442992310,10cyclictest25490-40mandb00:00:012
444099239,12cyclictest31667-21ssh22:51:141
444099231,16cyclictest14598-21ssh22:15:111
240052230,2sleep10-21swapper/121:10:131
149552230,3sleep114953-21sh19:34:591
444399226,13cyclictest21341-21grep00:20:213
444399222,6cyclictest20941-21sh22:43:563
444299222,14cyclictest24524-21ssh23:35:122
444299221,16cyclictest30539-21ssh21:15:032
4440992214,6cyclictest29242-21python320:05:241
4440992213,7cyclictest17915-21memory20:55:191
4440992212,7cyclictest2106-21apt20:20:121
443999221,9cyclictest31381-21python320:10:240
444399217,11cyclictest23525-21cron19:55:003
444399213,14cyclictest0-21swapper/319:55:093
4443992112,7cyclictest26367-21tune2fs20:00:153
444299212,12cyclictest15759-21sh22:40:112
4442992113,6cyclictest2682-21sed20:20:182
4442992112,7cyclictest19773-21tune2fs21:00:162
4442992112,6cyclictest20936-21grep19:45:222
4440992115,4cyclictest1-21systemd00:00:001
4440992113,6cyclictest9444-21rt-features00:35:231
4440992113,6cyclictest32521-21seq22:04:221
4440992113,6cyclictest29436-21cat22:25:241
4440992112,7cyclictest11590-21cut21:25:001
4440992112,6cyclictest13568-21python323:50:241
4440992111,7cyclictest25091-21date19:55:231
4440992111,6cyclictest52950irq/141-enp3s0-22:20:141
213892210,3chrt0-21swapper/121:05:001
208262210,4chrt20819-21ssh00:20:161
444399203,6cyclictest12440-21taskset20:43:233
4443992012,6cyclictest21792-21entropy21:05:153
444399200,17cyclictest2838-21cat20:20:193
444299202,8cyclictest1300-21ssh00:30:132
444299202,16cyclictest10673-21ssh21:47:582
4442992012,5cyclictest14803-21cat22:15:122
4442992011,5cyclictest52750irq/139-enp3s0-23:50:122
4442992011,5cyclictest0-21swapper/222:45:132
4440992014,4cyclictest17904-21sh21:29:401
4440992013,5cyclictest26521-21seq22:47:481
4440992012,6cyclictest17699-21df19:40:131
4440992012,6cyclictest14935-21seq20:50:071
4440992012,6cyclictest11128-21seq20:40:161
4440992011,6cyclictest17316-21seq21:52:421
444099201,14cyclictest2283-21ssh22:30:111
443999202,5cyclictest8533-21apt-get20:35:140
311192200,9sleep122-21ksoftirqd/123:40:061
229332200,1sleep10-21swapper/119:50:241
444399194,12cyclictest15490-21tail20:50:173
4443991914,3cyclictest20459-21open_inodes19:45:183
4443991912,5cyclictest7075-21uniq19:15:143
4443991912,5cyclictest28161-21ls20:05:133
444299199,6cyclictest52750irq/139-enp3s0-22:55:122
444299197,10cyclictest30022-21sh22:01:512
444299192,12cyclictest26980-21runrttasks00:25:112
4442991913,4cyclictest22692-21sh00:21:062
4442991913,4cyclictest0-21swapper/222:35:112
4442991912,5cyclictest12893-21sed19:30:012
444299191,13cyclictest8795-21apt-get19:20:122
4442991910,7cyclictest32203-21sed20:15:012
4442991910,7cyclictest18227-21cut20:55:242
4442991910,5cyclictest28-21ksoftirqd/220:30:132
444099191,5cyclictest0-21swapper/121:32:111
4440991913,4cyclictest30584-21sh00:27:181
4440991913,3cyclictest30491-21sh21:15:001
4440991912,5cyclictest636-21runrttasks00:11:491
4440991912,5cyclictest1485-21seq20:15:431
4440991912,5cyclictest10110-21seq20:36:141
4440991912,4cyclictest636-21runrttasks21:56:171
4440991912,4cyclictest18398-21sh00:18:341
4440991912,4cyclictest18398-21sh00:18:331
444099191,16cyclictest9581-21ssh22:35:141
4440991911,6cyclictest27807-21seq20:04:521
4440991910,6cyclictest30789-21cat20:10:171
250082190,1sleep20-21swapper/221:58:462
444399189,6cyclictest19652-21diskstats21:00:163
4443991811,5cyclictest13148-21cat20:45:133
4443991810,6cyclictest13141-21cat19:30:133
4443991810,3cyclictest15211-21apt-get19:35:113
444299189,5cyclictest23343-21diskmemload21:42:342
444299181,8cyclictest12493-21ssh21:25:152
444299181,7cyclictest24896-21/usr/sbin/munin21:10:222
4442991812,4cyclictest25920-21/usr/sbin/munin20:00:152
4442991811,5cyclictest7425-21wc19:15:182
4442991811,5cyclictest29253-21wc22:50:142
4442991811,5cyclictest28-21ksoftirqd/220:05:252
4442991810,6cyclictest2586-21ls22:05:222
4442991810,6cyclictest21864-21cat19:50:142
4442991810,6cyclictest13194-21cut20:45:162
4442991810,5cyclictest23343-21diskmemload00:19:402
4442991810,5cyclictest23343-21diskmemload00:19:392
444099189,7cyclictest5864-21seq20:26:391
444099182,11cyclictest7864-21apt-get22:10:131
444099182,11cyclictest1274-21apt-get00:30:141
444099181,8cyclictest6412-21ssh21:45:121
444099181,7cyclictest3649-21cat22:55:131
444099181,7cyclictest25921-21apt-get00:00:121
4440991812,4cyclictest23343-21diskmemload21:40:521
4440991812,4cyclictest18485-21sh22:41:271
4440991811,5cyclictest6709-21expr20:30:141
4440991811,5cyclictest19712-21expr19:45:111
4440991810,6cyclictest6699-21cat19:15:121
4440991810,6cyclictest25744-21cat21:35:151
4440991810,6cyclictest15808-21cat19:35:161
443999188,8cyclictest25660-21cron20:00:000
444399179,6cyclictest4469-21expr20:25:143
444399176,8cyclictest6397-21/usr/sbin/munin20:30:133
444399170,6cyclictest34-21ksoftirqd/320:55:153
444399170,6cyclictest34-21ksoftirqd/319:20:223
444299177,7cyclictest13451-21fschecks_count19:30:142
444299172,10cyclictest23785-21apt-get19:55:112
4442991711,4cyclictest2387-21sh22:30:112
4442991710,5cyclictest11823-21systemctl20:40:232
4442991710,4cyclictest17940-21hwlatdetect19:40:162
444299170,3cyclictest22533-21grep21:05:232
4440991711,4cyclictest24469-21/usr/sbin/munin23:35:161
4440991710,5cyclictest899-21basename00:05:191
4440991710,5cyclictest21661-21expr21:05:141
444299168,6cyclictest15556-21forks19:35:142
444299162,9cyclictest4248-21apt-get20:25:112
4442991611,3cyclictest636-21runrttasks20:18:042
444099169,5cyclictest12850-21ls19:30:001
444099158,5cyclictest9157-21cut19:20:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional