You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-19 - 07:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri Jun 19, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
46672177105,29sleep00-21swapper/019:07:450
46832176121,47sleep10-21swapper/119:07:571
45132165114,28sleep20-21swapper/219:05:452
46502158106,28sleep30-21swapper/319:07:323
283352740,7sleep012-21ksoftirqd/022:24:420
204502630,1sleep20-21swapper/221:30:222
117452630,3sleep20-21swapper/222:11:392
169612600,2sleep20-21swapper/221:51:292
271492570,1sleep10-21swapper/120:00:131
180502560,2sleep30-21swapper/319:40:013
180502560,2sleep30-21swapper/319:40:003
117092550,2sleep30-21swapper/320:40:123
200742540,2sleep30-21swapper/300:18:263
217172530,1sleep10-21swapper/122:43:281
226152520,3sleep30-21swapper/319:50:113
100142470,2sleep30-21swapper/300:10:233
65342440,1chrt0-21swapper/019:10:230
217952440,4sleep321793-21memory21:55:193
128572440,2sleep30-21swapper/322:12:453
101872440,1sleep10-21swapper/100:35:141
248172420,2chrt0-21swapper/123:57:381
154252420,1sleep20-21swapper/222:38:562
511799351,2cyclictest141rcu_preempt23:44:243
511599356,27cyclictest27457-21ssh00:00:011
5117993428,4cyclictest1880-21kworker/u8:022:53:583
511599341,30cyclictest29780-21ssh23:37:151
5114993410,12cyclictest12-21ksoftirqd/020:43:220
5114993316,6cyclictest12-21ksoftirqd/022:55:010
5114993314,7cyclictest12-21ksoftirqd/000:00:000
5116993218,5cyclictest28-21ksoftirqd/200:00:012
511599320,29cyclictest21973-21ssh00:20:061
511599320,29cyclictest21973-21ssh00:20:051
511499323,6cyclictest12-21ksoftirqd/022:00:170
511499323,6cyclictest12-21ksoftirqd/022:00:170
236952320,2chrt0-21swapper/023:32:560
511799311,27cyclictest20435-21ssh23:54:563
511699312,27cyclictest20510-21kworker/u8:122:53:592
5116993122,6cyclictest17148-21ssh23:51:412
5115993121,7cyclictest4192-21ssh22:54:541
511599311,28cyclictest0-21swapper/123:17:191
511499311,27cyclictest2996-21ssh23:17:280
511499310,6cyclictest9446-21sh22:58:270
511799309,11cyclictest34-21ksoftirqd/323:05:003
5117993027,2cyclictest0-21swapper/322:47:513
5117993023,5cyclictest14802-21diskmemload00:38:373
511799301,27cyclictest12388-21ssh23:48:363
5114993026,3cyclictest24969-1kworker/u9:123:36:040
511799298,4cyclictest34-21ksoftirqd/300:20:163
511799298,4cyclictest34-21ksoftirqd/300:20:153
511799290,27cyclictest8783-21ssh23:21:283
511799290,26cyclictest23668-21ssh23:56:263
511699299,11cyclictest28-21ksoftirqd/223:49:152
5116992922,5cyclictest1181-21ssh23:15:382
5114992923,4cyclictest12-21ksoftirqd/021:40:000
511499290,5cyclictest16898-21ssh23:51:250
511499290,27cyclictest12473-21ssh00:36:170
5117992820,6cyclictest24244-21diskmemload23:09:463
511799281,3cyclictest5589-21/usr/sbin/munin22:55:193
5117992811,8cyclictest141rcu_preempt22:25:243
5116992822,4cyclictest640-21snmpd00:34:352
511699281,20cyclictest0-21swapper/200:17:442
5116992810,15cyclictest210211cpuspeed_turbos21:55:122
5116992810,10cyclictest28-21ksoftirqd/223:40:262
5115992819,6cyclictest3578-21fschecks_time00:30:141
511599281,3cyclictest25133-21ssh23:34:231
511599281,25cyclictest10091-21ssh22:59:051
511599281,1cyclictest0-21swapper/123:09:491
511499283,7cyclictest12-21ksoftirqd/022:35:220
511499283,2cyclictest141rcu_preempt23:25:230
511499282,10cyclictest12-21ksoftirqd/023:42:490
511499281,7cyclictest12-21ksoftirqd/000:15:230
511499281,6cyclictest32390-21apt-get21:40:020
511499281,2cyclictest0-21swapper/023:23:500
511499281,25cyclictest0-21swapper/000:34:450
511799277,11cyclictest34-21ksoftirqd/323:35:183
511699278,11cyclictest28-21ksoftirqd/200:20:022
511699278,11cyclictest28-21ksoftirqd/200:20:012
5116992722,3cyclictest10242-21ssh22:59:132
511699271,24cyclictest0-21swapper/219:17:302
5116992711,4cyclictest28-21ksoftirqd/221:15:002
5116992711,2cyclictest141rcu_preempt22:16:082
511699270,25cyclictest10066-21cstates00:35:142
511599272,17cyclictest26901-21kworker/u8:121:40:031
511599271,2cyclictest0-21swapper/123:22:111
511599271,24cyclictest0-21swapper/123:45:421
511599271,24cyclictest0-21swapper/123:11:201
511499276,9cyclictest12-21ksoftirqd/021:11:300
511499274,9cyclictest12-21ksoftirqd/000:12:040
5114992723,2cyclictest0-21swapper/023:11:590
511499271,8cyclictest12-21ksoftirqd/021:45:120
511499271,24cyclictest0-21swapper/023:05:050
511499270,6cyclictest15841-21apt-get20:50:140
315772270,3sleep00-21swapper/020:10:140
511799268,7cyclictest141rcu_preempt21:51:113
5117992621,3cyclictest8282-21rm00:33:503
511799261,23cyclictest0-21swapper/323:25:043
511799261,23cyclictest0-21swapper/323:17:103
511799261,1cyclictest0-21swapper/323:34:243
511799261,18cyclictest0-21swapper/319:18:073
511799261,11cyclictest409-21apt-get21:40:043
511699268,8cyclictest28-21ksoftirqd/221:40:212
511699267,8cyclictest141rcu_preempt00:05:182
511699266,8cyclictest141rcu_preempt00:27:512
5116992623,2cyclictest0-21swapper/223:21:472
511699261,2cyclictest0-21swapper/222:47:372
511699261,24cyclictest0-21swapper/223:35:552
511699261,24cyclictest0-21swapper/223:05:052
511699261,23cyclictest0-21swapper/200:00:022
511699260,14cyclictest23230-21ssh23:32:282
5115992624,1cyclictest0-21swapper/123:04:311
5115992623,2cyclictest0-21swapper/123:50:281
511599261,24cyclictest0-21swapper/100:16:461
511599261,1cyclictest0-21swapper/123:42:501
511499264,11cyclictest12-21ksoftirqd/021:57:290
5114992624,1cyclictest0-21swapper/023:04:280
5114992623,2cyclictest0-21swapper/000:20:060
5114992623,2cyclictest0-21swapper/000:20:050
5114992623,2cyclictest0-21swapper/000:05:090
5114992623,1cyclictest0-21swapper/019:17:520
511499261,24cyclictest0-21swapper/023:47:470
511499261,1cyclictest0-21swapper/022:47:350
511499260,24cyclictest0-21swapper/000:00:240
511799259,9cyclictest34-21ksoftirqd/322:40:013
5117992518,5cyclictest16270-21ssh22:39:483
511799251,23cyclictest0-21swapper/323:11:403
511799251,23cyclictest0-21swapper/300:01:003
511699257,3cyclictest141rcu_preempt23:10:402
511699256,4cyclictest28-21ksoftirqd/221:15:202
5116992522,2cyclictest0-21swapper/223:04:452
511699251,9cyclictest0-21swapper/221:23:142
511699251,23cyclictest0-21swapper/223:29:102
5116992511,4cyclictest141rcu_preempt22:06:192
511599258,5cyclictest15627-21tr22:15:131
5115992523,1cyclictest0-21swapper/122:47:171
511599251,23cyclictest0-21swapper/123:26:531
511499257,8cyclictest12-21ksoftirqd/021:16:040
511499256,8cyclictest12-21ksoftirqd/022:31:190
511499256,7cyclictest12-21ksoftirqd/021:22:010
511499255,11cyclictest12-21ksoftirqd/020:38:510
511499253,6cyclictest12-21ksoftirqd/022:27:270
511499251,8cyclictest13441-21ls21:25:200
5114992518,4cyclictest12-21ksoftirqd/021:10:000
511499250,7cyclictest12-21ksoftirqd/021:30:220
511799249,8cyclictest28802-21ssh21:36:273
511799249,7cyclictest141rcu_preempt22:00:153
511799249,7cyclictest141rcu_preempt22:00:143
511799248,2cyclictest141rcu_preempt21:20:243
511799248,14cyclictest141rcu_preempt00:09:243
511799247,9cyclictest141rcu_preempt21:26:113
511799247,7cyclictest141rcu_preempt22:08:033
511699246,8cyclictest141rcu_preempt22:31:422
511699241,12cyclictest0-21swapper/222:40:442
511499247,7cyclictest12-21ksoftirqd/022:43:380
511499245,7cyclictest12-21ksoftirqd/000:26:340
511499243,3cyclictest12-21ksoftirqd/020:00:160
5114992419,4cyclictest131rcuc/022:14:510
511499241,8cyclictest12-21ksoftirqd/022:05:180
511499241,8cyclictest12-21ksoftirqd/020:55:180
5114992410,5cyclictest12-21ksoftirqd/021:50:130
511499240,6cyclictest21889-21sendmail22:20:000
194112240,2sleep10-21swapper/119:40:221
511799239,2cyclictest141rcu_preempt22:32:183
511799239,2cyclictest141rcu_preempt21:32:403
511799235,9cyclictest34-21ksoftirqd/321:17:323
511799235,2cyclictest141rcu_preempt21:14:023
511699239,5cyclictest28-21ksoftirqd/200:10:582
511699236,7cyclictest141rcu_preempt22:25:112
511699231,8cyclictest0-21swapper/222:20:332
511699231,12cyclictest24168-21diskmemload22:01:432
511699231,12cyclictest24168-21diskmemload22:01:422
511699231,11cyclictest24168-21diskmemload21:38:562
511499237,8cyclictest12-21ksoftirqd/020:21:090
511499236,9cyclictest12-21ksoftirqd/020:48:250
511499232,4cyclictest12-21ksoftirqd/019:25:130
511499231,6cyclictest809-21apt-get20:15:110
511499230,7cyclictest624-21gdbus21:04:590
511499230,6cyclictest19558-21ls19:40:230
511799225,7cyclictest141rcu_preempt22:21:223
5117992210,9cyclictest8034-21smartctl21:45:163
511699229,10cyclictest0-21swapper/221:25:242
511699221,8cyclictest0-21swapper/221:45:542
511699221,6cyclictest0-21swapper/220:45:192
511699221,5cyclictest0-21swapper/220:00:172
511699220,5cyclictest0-21swapper/219:30:232
511599229,10cyclictest20414-21sed21:30:221
511599222,14cyclictest8799-21ssh22:10:131
5115992214,6cyclictest17466-21ls19:35:261
5115992214,6cyclictest17466-21ls19:35:251
5114992216,4cyclictest12-21ksoftirqd/020:00:010
5114992212,7cyclictest29877-21ntpq20:05:210
511499220,12cyclictest12-21ksoftirqd/019:50:120
511799219,10cyclictest24195-21ls21:10:003
511799218,11cyclictest141rcu_preempt00:25:043
5117992113,6cyclictest6307-21sensors_temp19:10:213
5117992113,6cyclictest12975-21ls19:25:243
511699216,3cyclictest141rcu_preempt20:24:592
5116992113,6cyclictest8631-21sed20:30:242
511599214,6cyclictest636-21runrttasks22:20:011
5115992119,1cyclictest0-21swapper/119:18:141
511599211,4cyclictest0-21swapper/122:38:551
5115992114,5cyclictest12974-21fschecks_time21:25:161
5115992113,6cyclictest32010-21ntpq20:10:201
5115992113,6cyclictest29089-21ls22:00:231
5115992113,6cyclictest29089-21ls22:00:221
5115992112,6cyclictest3527-21cat22:05:231
5115992112,6cyclictest13713-21cron21:50:001
511499211,8cyclictest12-21ksoftirqd/019:45:160
511499211,6cyclictest12-21ksoftirqd/019:30:280
5114992115,4cyclictest12-21ksoftirqd/019:40:010
5114992115,4cyclictest12-21ksoftirqd/019:40:000
511499211,2cyclictest141rcu_preempt20:30:170
182632210,2sleep122-21ksoftirqd/120:55:141
89812200,3sleep18975-21ssh00:10:131
511799208,2cyclictest141rcu_preempt20:50:153
5117992012,6cyclictest15726-21ssh22:15:123
5117992011,6cyclictest1268-21tune2fs20:15:143
511699201,5cyclictest0-21swapper/220:05:152
5116992012,6cyclictest25172-21sed19:55:142
5115992014,4cyclictest17722-21sh20:55:001
5115992012,6cyclictest6716-21seq19:11:011
5115992012,6cyclictest3844-21seq21:18:361
5115992012,6cyclictest26950-21sed21:35:181
5115992012,6cyclictest24095-21seq19:51:081
5115992012,6cyclictest23172-21ntpq21:05:201
5115992012,6cyclictest20560-21sed21:55:001
5115992011,6cyclictest21673-21cat19:45:221
511499207,5cyclictest636-21runrttasks20:26:270
511499201,6cyclictest12-21ksoftirqd/019:20:130
511799199,2cyclictest34-21ksoftirqd/320:00:093
511799198,2cyclictest34-21ksoftirqd/319:40:143
511799194,10cyclictest331rcuc/320:25:013
5117991914,3cyclictest8341-21rt-features20:30:223
5117991911,6cyclictest28954-21apt20:05:123
5117991911,6cyclictest25417-21memory19:55:183
5117991911,6cyclictest21694-21ls19:45:233
5117991911,6cyclictest10345-21open_inodes20:35:213
511799190,11cyclictest331rcuc/320:55:143
511699196,4cyclictest141rcu_preempt20:15:212
511699192,4cyclictest141rcu_preempt20:50:252
511699191,4cyclictest0-21swapper/219:40:242
5116991914,3cyclictest7004-21cron20:30:002
511699191,3cyclictest141rcu_preempt19:50:232
5116991912,5cyclictest23457-21ls21:05:232
5115991913,4cyclictest11375-21sh21:24:201
5115991913,4cyclictest10612-21sensors19:20:211
5115991912,5cyclictest13211-21seq19:26:371
5115991912,4cyclictest24168-21diskmemload00:27:301
5115991911,6cyclictest9466-21sort20:35:141
5115991911,6cyclictest15012-21date20:45:261
511799181,3cyclictest31644-21hddtemp_smartct20:10:143
5117991813,3cyclictest6537-21unixbench_multi20:25:243
5117991810,5cyclictest13713-21acpi19:30:093
511699184,3cyclictest141rcu_preempt20:55:252
511699181,5cyclictest0-21swapper/219:12:552
511699181,4cyclictest141rcu_preempt19:35:122
511699181,4cyclictest141rcu_preempt19:35:112
511699181,2cyclictest141rcu_preempt20:10:102
5116991811,5cyclictest28-21ksoftirqd/220:40:002
5116991811,5cyclictest12047-21cut19:25:152
511599184,12cyclictest1161-21kworker/1:220:19:591
511599182,12cyclictest7265-21apt-get20:30:121
5115991812,4cyclictest27558-21sh21:11:391
5115991811,5cyclictest4617-21seq00:06:231
5115991811,5cyclictest3879-21ntp_kernel_pll_20:20:191
5115991811,5cyclictest25462-21sed19:55:171
5115991810,6cyclictest9710-21seq22:34:571
5115991810,6cyclictest5593-21tr20:25:151
5115991810,6cyclictest29055-21expr20:05:131
5115991810,6cyclictest21269-21run-parts21:00:221
511799179,5cyclictest10711-21timerandwakeup19:20:223
511799173,12cyclictest14187-21grep20:45:173
5117991712,3cyclictest20186-21cpu21:00:123
511699174,11cyclictest22461-21latency_hist19:50:002
511699172,2cyclictest141rcu_preempt20:40:212
511599176,9cyclictest0-21swapper/122:25:221
5115991710,5cyclictest20877-21basename21:55:111
511699161,2cyclictest0-21swapper/221:04:052
5116991611,3cyclictest636-21runrttasks19:20:202
511599169,5cyclictest15364-21seq19:31:471
511599168,5cyclictest11705-21cpuspeed_turbos20:40:111
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional