You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-13 - 04:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Mon Jul 13, 2026 00:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5306216297,54sleep10-21swapper/119:05:281
53182141104,26sleep30-21swapper/319:05:383
54192139102,25sleep00-21swapper/019:06:550
53072139104,24sleep20-21swapper/219:05:292
306312580,2sleep20-21swapper/222:01:282
186072580,2sleep30-21swapper/322:16:503
217392550,2sleep122-21ksoftirqd/122:44:011
9712530,3sleep20-21swapper/223:16:042
116432520,1sleep00-21swapper/021:47:370
116432520,1sleep00-21swapper/021:47:370
248722490,1sleep10-21swapper/121:33:341
68452480,1chrt0-21swapper/022:56:280
119962480,1sleep00-21swapper/023:25:130
325262470,2sleep00-21swapper/000:28:420
327192450,2chrt0-21swapper/021:15:150
47392440,3chrt141rcu_preempt23:19:543
187842440,2sleep00-21swapper/019:40:010
145982440,2sleep30-21swapper/321:50:133
305522430,1sleep30-21swapper/300:02:483
593099421,4cyclictest11049-21sh00:36:140
106232420,1sleep00-21swapper/023:24:050
230202410,3sleep228-21ksoftirqd/223:09:182
200972400,4sleep00-21swapper/019:40:250
5933993718,5cyclictest3061-21ssh21:40:443
593099360,6cyclictest10954-21cpuspeed_turbos23:00:140
240472360,3chrt0-21swapper/222:20:342
593099353,3cyclictest0-21swapper/023:05:120
593099352,2cyclictest0-21swapper/021:55:130
593099350,6cyclictest18165-21grep22:40:260
593099350,5cyclictest8185-21tr20:30:140
593099341,11cyclictest15472-21grep21:50:210
593099340,7cyclictest13551-21chrt00:14:570
593099340,6cyclictest14280-21taskset19:29:530
593099340,5cyclictest2901-21ssh00:30:210
593099338,22cyclictest14316-21perf19:34:550
593099338,21cyclictest19838-21latency_hist21:30:010
593099331,7cyclictest8968-21cpuspeed_turbos22:10:140
5930993311,18cyclictest760-21cpu00:05:130
593099330,7cyclictest12967-21idleruntime20:40:180
593099330,6cyclictest22366-21rm21:31:020
593099330,5cyclictest21167-21cstates00:20:130
593099330,5cyclictest18110-21unixbench_singl20:50:260
593099330,5cyclictest11913-21ssh22:35:500
593199327,7cyclictest22-21ksoftirqd/123:24:161
593099328,20cyclictest18644-21apt-get23:30:130
5931993111,9cyclictest22-21ksoftirqd/100:18:381
593099318,20cyclictest2690-21tr22:05:160
593099315,5cyclictest32307-21ssh22:03:090
593099315,5cyclictest23115-21ssh22:45:100
593099312,3cyclictest0-21swapper/023:15:120
593099311,5cyclictest30655-21missed_timers22:50:190
593099311,4cyclictest22676-21tune2fs22:20:160
593099311,26cyclictest13818-21ssh23:50:160
5930993110,19cyclictest29609-21rm23:14:010
5930993110,19cyclictest29609-21rm23:14:010
593099309,18cyclictest624-21gdbus22:15:210
593099309,18cyclictest29825-21netstat22:25:180
593099307,6cyclictest27315-21wc20:00:000
593099307,5cyclictest19500-21cat20:55:200
593099302,7cyclictest6634-21ssh21:20:110
593099302,3cyclictest0-21swapper/020:05:130
593099302,2cyclictest0-21swapper/020:25:110
593099301,8cyclictest32661-21sed23:40:170
593099301,6cyclictest20452-21ssh23:55:170
593099301,5cyclictest0-21swapper/000:00:130
593099301,4cyclictest26746-21ssh21:35:120
5930993010,17cyclictest25198-21diskmemload21:10:260
593199296,11cyclictest23720-21apt-get23:10:031
593199296,11cyclictest23720-21apt-get23:10:031
593099299,17cyclictest23811-21gltestperf21:05:170
593099298,19cyclictest24746-21seq19:51:400
593099297,5cyclictest2333-21sed20:15:180
593099295,7cyclictest7372-21grep19:10:270
593099292,8cyclictest10024-21perf19:20:000
593099291,9cyclictest3236-21grep22:30:140
593099291,8cyclictest27090-21vmstat23:35:280
593099291,7cyclictest683-21unixbench-2d20:10:270
593099291,7cyclictest683-21unixbench-2d20:10:270
593099291,26cyclictest7530-21grep23:45:230
593099291,25cyclictest3478-21ssh21:41:090
5930992910,17cyclictest27834-21fschecks_time20:00:140
593099282,9cyclictest13613-21apt-get00:15:040
593099281,8cyclictest4173-21fschecks_count20:20:160
593099281,8cyclictest16396-21cron20:50:000
593099281,7cyclictest22462-21grep19:45:250
593099281,7cyclictest10371-21cpuspeed_turbos20:35:130
593099281,6cyclictest11694-21tail19:20:260
5931992723,3cyclictest211rcuc/122:05:401
5931992723,3cyclictest211rcuc/100:26:271
593099271,13cyclictest21868-21ntp_states21:00:180
593099271,13cyclictest21868-21ntp_states21:00:180
37972270,5sleep228-21ksoftirqd/220:20:112
593199267,4cyclictest141rcu_preempt00:30:031
593199265,10cyclictest22-21ksoftirqd/123:54:551
593199262,5cyclictest22-21ksoftirqd/122:15:501
5931992622,3cyclictest211rcuc/121:18:561
593199260,24cyclictest0-21swapper/119:12:461
593199260,14cyclictest22-21ksoftirqd/123:40:211
326552260,2chrt0-21swapper/300:28:493
247182260,2chrt0-21swapper/223:34:502
5933992523,1cyclictest0-21swapper/319:12:283
5933992515,7cyclictest5640-21df23:20:143
593299251,1cyclictest0-21swapper/219:12:252
593199258,10cyclictest141rcu_preempt00:01:301
593199257,9cyclictest22-21ksoftirqd/123:48:571
593199256,9cyclictest22-21ksoftirqd/122:59:381
5931992517,5cyclictest10566-21diskmemload00:11:551
5931992512,7cyclictest6403-21kworker/u8:123:01:071
50462250,4sleep15049-21latency_hist23:20:011
184272250,3sleep20-21swapper/223:30:012
593299244,17cyclictest28328-21sed21:35:252
593199249,9cyclictest22-21ksoftirqd/122:33:221
593199248,8cyclictest22-21ksoftirqd/121:35:361
593199246,5cyclictest22-21ksoftirqd/123:56:171
593199245,2cyclictest141rcu_preempt21:10:281
593199244,9cyclictest22-21ksoftirqd/122:38:431
593199244,7cyclictest22-21ksoftirqd/121:25:261
593199244,11cyclictest22-21ksoftirqd/122:27:091
593199243,3cyclictest22-21ksoftirqd/100:35:171
593199242,6cyclictest22-21ksoftirqd/122:45:201
593199241,7cyclictest22-21ksoftirqd/123:31:461
593199241,6cyclictest211rcuc/120:15:001
593199241,6cyclictest211rcuc/120:15:001
5931992415,6cyclictest278-21systemd-journal20:35:001
5931992410,4cyclictest22-21ksoftirqd/121:22:361
593199240,7cyclictest22-21ksoftirqd/123:08:301
593199240,7cyclictest22-21ksoftirqd/122:00:111
593199240,11cyclictest64762ssh00:09:311
593399232,14cyclictest28838-21ssh22:25:103
5933992314,6cyclictest6410-21fw_packets20:25:143
593299233,14cyclictest27601-21ssh00:25:122
593299232,14cyclictest13158-21ssh23:50:122
5932992312,8cyclictest278-21systemd-journal22:44:552
593199238,2cyclictest141rcu_preempt23:25:481
593199236,7cyclictest22-21ksoftirqd/122:14:261
593199235,8cyclictest141rcu_preempt22:23:111
593199233,9cyclictest22-21ksoftirqd/100:20:561
593199233,4cyclictest22-21ksoftirqd/121:45:581
593199233,4cyclictest22-21ksoftirqd/121:45:581
593199231,9cyclictest22-21ksoftirqd/122:54:311
5931992310,2cyclictest141rcu_preempt19:15:171
593199230,7cyclictest22-21ksoftirqd/121:41:171
268172230,3sleep2271rcuc/221:10:232
593399222,14cyclictest32349-21ssh21:15:113
5933992214,6cyclictest31193-21ls22:50:243
5933992214,6cyclictest17411-21ssh00:16:543
5933992214,5cyclictest15669-21sh23:03:373
5933992211,8cyclictest24749-21df_inode23:10:163
5933992211,8cyclictest24749-21df_inode23:10:163
593299229,6cyclictest13076-21cut21:25:012
5932992214,5cyclictest25198-21diskmemload23:35:212
5932992213,7cyclictest15504-21ntp_states21:50:222
593199223,9cyclictest22-21ksoftirqd/123:35:041
593199221,8cyclictest22-21ksoftirqd/120:55:151
593199221,7cyclictest22-21ksoftirqd/120:35:181
593199221,6cyclictest211rcuc/120:45:001
593199220,7cyclictest22-21ksoftirqd/119:55:231
246392220,3chrt34-21ksoftirqd/300:22:273
142562220,2sleep30-21swapper/322:38:133
63832210,2chrt0-21swapper/200:33:172
593399212,13cyclictest2160-21ssh22:05:123
5933992113,6cyclictest29912-21forks20:05:163
5933992112,7cyclictest17590-21awk20:50:213
5933992112,6cyclictest25248-21seq23:35:083
593299212,14cyclictest5305-21runrttasks23:20:132
5932992114,4cyclictest636-21runrttasks22:14:542
5932992113,6cyclictest13302-21ntp_states20:40:222
5932992112,7cyclictest560-21dbus-daemon23:45:002
593199219,2cyclictest141rcu_preempt20:08:341
593199218,4cyclictest22-21ksoftirqd/121:56:281
593199218,2cyclictest141rcu_preempt20:51:191
593199213,6cyclictest22-21ksoftirqd/120:20:111
593199212,4cyclictest22-21ksoftirqd/120:45:201
593199211,8cyclictest22-21ksoftirqd/120:15:261
5931992112,6cyclictest14802-21cat19:30:131
593199210,6cyclictest15305-21memory21:50:191
46152210,4sleep24613-21ls22:30:252
181572210,2chrt0-21swapper/323:29:583
111182210,2sleep30-21swapper/321:23:123
82752200,3chrt0-21swapper/321:45:163
82752200,3chrt0-21swapper/321:45:163
6682200,3chrt667-21ssh23:40:213
5933992014,4cyclictest11784-21users19:20:273
5933992013,4cyclictest21031-21sh21:30:203
5933992013,4cyclictest20902-21sh23:55:223
5933992012,6cyclictest29868-21seq21:12:583
5933992012,6cyclictest27929-21seq22:48:333
5933992012,6cyclictest19548-21ls20:55:203
5933992012,5cyclictest20951-21ssh21:55:013
5933992012,5cyclictest17589-21ntp_kernel_pll_19:35:203
5933992011,7cyclictest2922-21cat20:15:253
5933992011,7cyclictest21765-21ls19:45:183
5933992011,7cyclictest13174-21seq20:40:203
5933992011,6cyclictest24949-21seq22:21:283
593299203,14cyclictest9496-21strings00:35:202
593299203,10cyclictest13362-21ssh21:25:132
593299201,8cyclictest30678-21taskset00:02:562
5932992014,4cyclictest22461-21sh21:31:082
5932992012,6cyclictest6979-21basename00:10:002
5932992012,6cyclictest11562-21sed20:35:252
593299201,15cyclictest9799-21ssh22:35:122
5932992010,6cyclictest25185-21grep23:10:192
5932992010,6cyclictest25185-21grep23:10:192
5931992012,2cyclictest141rcu_preempt19:52:521
5931992010,2cyclictest141rcu_preempt19:36:361
593199200,6cyclictest211rcuc/120:05:001
593199200,5cyclictest0-21swapper/119:40:131
126542200,2sleep10-21swapper/119:25:131
593399199,7cyclictest8829-21cat20:30:193
593399197,10cyclictest27135-21cat21:35:153
5933991914,3cyclictest9209-21sendmail_mailst19:15:233
5933991913,4cyclictest5507-21sh00:08:323
5933991913,4cyclictest4210-21sh00:31:053
5933991913,4cyclictest18822-21swap23:05:233
5933991913,4cyclictest14518-21switchtime23:50:243
5933991913,3cyclictest14002-21sh22:13:553
5933991911,6cyclictest4004-21sed22:30:203
5933991911,6cyclictest27172-21readlink20:00:003
5933991911,6cyclictest15483-21ntpq19:30:213
5933991911,6cyclictest14673-21expr20:45:143
5933991911,5cyclictest27691-21sh22:00:023
5933991910,7cyclictest32526-21date20:10:213
5933991910,7cyclictest32526-21date20:10:213
5933991910,7cyclictest27542-21grep20:00:113
5933991910,7cyclictest21904-21pluginstate21:00:193
5933991910,7cyclictest21904-21pluginstate21:00:193
5933991910,6cyclictest25198-21diskmemload00:11:373
5933991910,6cyclictest25031-1kworker/u9:222:55:113
5933991910,5cyclictest53050irq/142-enp3s0-21:25:133
593299199,8cyclictest32331-21apt-get21:15:122
593299198,9cyclictest20019-21cstates23:55:142
593299193,7cyclictest28826-21apt-get22:25:102
593299192,12cyclictest29795-21ssh22:50:112
5932991913,4cyclictest9877-21sh22:59:332
5932991913,4cyclictest25185-21sh21:10:002
5932991912,5cyclictest8849-21sed19:15:182
5932991911,6cyclictest1965-21sed21:40:192
5932991910,7cyclictest21556-21cat21:00:142
5932991910,7cyclictest21556-21cat21:00:142
593199198,3cyclictest141rcu_preempt19:46:231
593199198,2cyclictest141rcu_preempt21:00:241
593199198,2cyclictest141rcu_preempt21:00:241
593199197,3cyclictest22-21ksoftirqd/120:25:471
74022180,3sleep20-21swapper/223:45:222
593399189,6cyclictest19177-21cat19:40:143
593399186,3cyclictest141rcu_preempt23:45:153
5933991811,4cyclictest8447-21sh00:35:093
5933991810,6cyclictest17143-21basename22:40:173
5933991810,6cyclictest13400-21grep19:25:213
593299182,11cyclictest23143-21apt-get22:45:122
593299182,11cyclictest12531-21users23:00:272
5932991813,3cyclictest22173-21sh21:55:212
5932991811,5cyclictest7630-21expr00:10:152
5932991811,5cyclictest24585-21seq00:22:242
5932991811,5cyclictest19058-21expr20:55:152
5932991811,4cyclictest25198-21diskmemload21:49:012
5932991811,4cyclictest25198-21diskmemload21:49:012
5932991811,4cyclictest25198-21diskmemload00:15:552
5932991810,6cyclictest2468-21ls22:05:132
5932991810,6cyclictest15558-21cat22:15:112
5932991810,5cyclictest23488-21cut19:50:152
5932991810,5cyclictest14293-21cron19:30:002
593199180,6cyclictest22-21ksoftirqd/121:05:171
593199180,6cyclictest22-21ksoftirqd/119:20:191
593399179,6cyclictest624-21gdbus23:30:243
593399172,11cyclictest10226-21apt-get20:35:133
593399171,11cyclictest23133-21apt-get19:50:123
5933991710,5cyclictest3980-21expr20:20:143
5933991710,5cyclictest23608-21expr21:05:153
593299179,5cyclictest26027-21load19:55:202
593299178,6cyclictest278-21systemd-journal20:20:002
5932991712,3cyclictest16636-21/usr/sbin/munin20:50:212
5932991711,4cyclictest9454-21sed20:30:262
593299171,13cyclictest31453-21munin-run20:10:002
5932991710,5cyclictest27642-21cat20:00:112
5932991710,5cyclictest19084-21expr19:40:142
5932991710,5cyclictest14821-21expr20:45:142
593299169,5cyclictest6210-21expr20:25:132
593299169,4cyclictest31681-21acpi20:10:122
593299169,4cyclictest31681-21acpi20:10:122
593299168,5cyclictest21977-21head19:45:212
593299167,7cyclictest12217-21cat19:25:002
593299162,11cyclictest17615-21ntp_states19:35:212
593299151,11cyclictest15472-21ntp_states19:30:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional