You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 13:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri May 08, 2026 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
276832177121,48sleep30-21swapper/319:05:293
278582143106,26sleep20-21swapper/219:07:452
278322142106,25sleep10-21swapper/119:07:241
278142139103,25sleep00-21swapper/019:07:110
138532940,2sleep10-21swapper/123:55:191
88152650,2sleep20-21swapper/223:04:032
76232620,2sleep30-21swapper/323:02:523
79562600,1sleep20-21swapper/222:39:352
222832590,2sleep20-21swapper/223:13:592
274102580,2sleep10-21swapper/122:05:231
274102580,2sleep10-21swapper/122:05:231
125132570,11sleep012-21ksoftirqd/000:18:260
266022560,2sleep10-21swapper/100:29:021
266022560,2sleep10-21swapper/100:29:021
71802530,1sleep10-21swapper/121:27:231
278422520,2sleep10-21swapper/123:17:471
71682480,2sleep20-21swapper/219:35:132
133512480,2sleep30-21swapper/321:05:143
117892470,2sleep10-21swapper/100:17:421
299632450,2sleep30-21swapper/320:27:063
198882420,1sleep00-21swapper/000:24:040
17252420,1sleep31724-21ssh21:23:443
17252420,1sleep31724-21ssh21:23:443
302002410,1sleep230201-21ssh23:20:012
176722400,1sleep30-21swapper/300:21:503
2803323727,4sleep10-21swapper/119:10:011
67452360,1sleep00-21swapper/000:14:240
295262310,1sleep30-21swapper/322:07:103
295262310,1sleep30-21swapper/322:07:103
139652310,2sleep20-21swapper/222:20:102
2830499309,11cyclictest12-21ksoftirqd/021:10:240
92562290,3chrt0-21swapper/122:40:151
2830599298,12cyclictest22-21ksoftirqd/100:02:131
2830599296,13cyclictest22-21ksoftirqd/123:23:151
2830599285,13cyclictest22-21ksoftirqd/123:47:141
2830499288,4cyclictest12-21ksoftirqd/023:03:380
2830499286,10cyclictest12-21ksoftirqd/022:31:520
28305992711,4cyclictest22-21ksoftirqd/123:42:031
28305992711,2cyclictest141rcu_preempt21:35:121
2830499276,6cyclictest12-21ksoftirqd/022:05:130
2830499276,6cyclictest12-21ksoftirqd/022:05:120
28304992711,8cyclictest141rcu_preempt22:10:520
28304992711,8cyclictest141rcu_preempt21:21:130
28304992711,8cyclictest141rcu_preempt21:21:130
2830599266,8cyclictest22-21ksoftirqd/100:13:541
2830599266,13cyclictest22-21ksoftirqd/122:57:311
28305992619,5cyclictest22-21ksoftirqd/100:40:001
2830499268,3cyclictest141rcu_preempt22:56:010
2830499265,4cyclictest12-21ksoftirqd/023:20:300
28304992611,2cyclictest141rcu_preempt23:32:410
28304992610,4cyclictest12-21ksoftirqd/022:53:330
154922260,1sleep00-21swapper/023:56:060
28307992523,1cyclictest0-21swapper/319:13:303
2830699251,1cyclictest0-21swapper/219:13:012
2830699251,12cyclictest1-21systemd21:40:062
2830599259,8cyclictest141rcu_preempt21:46:191
2830599259,5cyclictest22-21ksoftirqd/123:33:281
2830599258,4cyclictest22-21ksoftirqd/122:17:541
2830599257,6cyclictest22-21ksoftirqd/121:15:111
2830599255,4cyclictest22-21ksoftirqd/100:21:271
2830599254,5cyclictest22-21ksoftirqd/121:11:551
28305992519,4cyclictest22-21ksoftirqd/122:40:001
28305992518,5cyclictest22-21ksoftirqd/123:40:001
2830599251,12cyclictest22-21ksoftirqd/121:55:161
2830499257,7cyclictest141rcu_preempt22:36:480
2830499254,2cyclictest141rcu_preempt21:35:160
28304992523,1cyclictest0-21swapper/019:14:010
28304992519,4cyclictest12-21ksoftirqd/023:40:000
28304992510,2cyclictest141rcu_preempt23:53:420
28307992415,6cyclictest278-21systemd-journal23:40:013
28307992412,8cyclictest19340-21sort22:00:123
2830599249,4cyclictest22-21ksoftirqd/122:30:441
2830599249,2cyclictest141rcu_preempt23:05:431
2830599248,4cyclictest22-21ksoftirqd/122:29:261
2830499248,4cyclictest12-21ksoftirqd/023:29:140
2830499247,5cyclictest12-21ksoftirqd/023:10:330
2830499243,4cyclictest12-21ksoftirqd/023:08:450
28304992418,4cyclictest12-21ksoftirqd/022:00:000
28304992412,2cyclictest141rcu_preempt00:08:430
2830799235,7cyclictest34-21ksoftirqd/321:45:373
2830799233,14cyclictest26761-21ssh00:05:113
2830799231,9cyclictest0-21swapper/322:57:023
2830799231,8cyclictest26154-21grep23:40:193
2830799231,8cyclictest11761-21ssh22:18:113
2830799231,10cyclictest14017-21apt-get22:20:123
2830799230,11cyclictest0-21swapper/321:19:013
2830599234,4cyclictest22-21ksoftirqd/121:50:371
28305992317,4cyclictest22-21ksoftirqd/123:15:001
28305992311,2cyclictest141rcu_preempt21:05:191
28305992310,2cyclictest141rcu_preempt21:33:181
28305992310,2cyclictest141rcu_preempt00:05:201
2830499238,7cyclictest141rcu_preempt21:29:320
2830499233,4cyclictest12-21ksoftirqd/022:46:440
2830499232,10cyclictest12-21ksoftirqd/023:17:340
28304992311,2cyclictest141rcu_preempt21:45:140
2830499231,11cyclictest12-21ksoftirqd/000:37:070
28304992310,2cyclictest141rcu_preempt22:22:400
28304992310,2cyclictest141rcu_preempt00:27:370
28304992310,2cyclictest141rcu_preempt00:27:370
42602220,2sleep334-21ksoftirqd/323:24:473
2830799223,6cyclictest0-21swapper/323:15:163
2830799223,4cyclictest0-21swapper/323:48:583
28307992213,7cyclictest11341-21ntp_states21:30:223
28307992212,5cyclictest0-21swapper/300:15:123
28307992210,7cyclictest27779-21tr22:30:153
2830699224,9cyclictest19868-21ssh00:00:122
2830699221,6cyclictest3034-21munin-run21:25:002
2830699221,6cyclictest3034-21munin-run21:25:002
28306992214,6cyclictest29459-21diskstats22:55:142
28306992213,6cyclictest7865-21sed22:15:182
28306992213,6cyclictest6172-21sendmail_mailqu23:25:232
28306992212,7cyclictest8165-21diskstats00:15:142
28306992212,7cyclictest23230-21cat22:50:192
2830599228,4cyclictest22-21ksoftirqd/123:28:511
2830599228,3cyclictest141rcu_preempt22:00:111
2830599226,4cyclictest22-21ksoftirqd/123:01:531
2830599226,2cyclictest141rcu_preempt21:20:121
2830599226,2cyclictest141rcu_preempt21:20:121
2830599225,3cyclictest141rcu_preempt21:40:521
2830599225,2cyclictest141rcu_preempt22:47:561
2830599224,3cyclictest141rcu_preempt19:35:221
2830599223,7cyclictest141rcu_preempt00:30:291
2830599221,18cyclictest14947-21diskmemload22:54:081
2830499228,7cyclictest141rcu_preempt00:00:510
2830499228,2cyclictest141rcu_preempt22:18:140
2830499225,3cyclictest12-21ksoftirqd/000:30:230
2830499224,4cyclictest12-21ksoftirqd/023:40:220
56532210,1sleep20-21swapper/200:13:182
28307992112,7cyclictest10260-21ls19:40:233
28307992112,6cyclictest4240-21basename20:45:003
28307992112,6cyclictest30998-21sed20:30:163
28307992112,6cyclictest21207-21grep20:05:243
28307992112,5cyclictest636-21runrttasks23:31:183
2830699219,4cyclictest141rcu_preempt22:00:132
2830699216,5cyclictest7445-21pmu-power23:50:212
2830699212,6cyclictest28275-21tail22:30:182
28306992113,6cyclictest815-21diskstats22:10:162
28306992113,6cyclictest27764-21users22:05:252
28306992113,6cyclictest27764-21users22:05:252
28306992113,6cyclictest11236-21df21:00:132
28306992110,8cyclictest26741-21apt-get00:05:112
2830599219,3cyclictest141rcu_preempt22:14:591
2830599216,3cyclictest141rcu_preempt20:10:001
2830599213,6cyclictest22-21ksoftirqd/123:50:161
2830599213,3cyclictest141rcu_preempt20:45:221
2830599213,3cyclictest141rcu_preempt20:25:221
28305992113,6cyclictest10362-21sed19:40:241
2830499219,2cyclictest141rcu_preempt21:40:230
2830499218,2cyclictest141rcu_preempt22:43:020
2830499215,4cyclictest12-21ksoftirqd/021:15:020
28304992114,5cyclictest12-21ksoftirqd/020:30:000
28304992111,7cyclictest6407-21ntp_kernel_err21:50:200
190782210,2sleep30-21swapper/323:59:423
101882210,11sleep334-21ksoftirqd/323:52:323
314082200,2chrt0-21swapper/120:30:211
28307992016,2cyclictest0-21swapper/320:55:173
28307992015,3cyclictest22327-21sh00:25:223
28307992015,3cyclictest22327-21sh00:25:213
28307992013,4cyclictest22303-21sh00:01:103
28307992013,4cyclictest20878-21sh23:12:353
28307992012,4cyclictest2776-21sh00:10:263
2830799200,9cyclictest16196-21sed22:45:163
2830699208,9cyclictest32016-21cpu23:45:132
2830699204,5cyclictest141rcu_preempt21:35:262
2830699202,13cyclictest30991-21apt-get21:45:132
2830699201,4cyclictest0-21swapper/222:28:042
28306992013,4cyclictest26240-21ntp_offset23:40:202
28306992013,4cyclictest1313-21sh23:21:512
28306992012,6cyclictest5303-21needreboot20:45:192
28306992012,6cyclictest20634-21ntp_kernel_pll_20:05:192
28306992012,6cyclictest1625-21grep19:20:262
28306992011,6cyclictest10900-21cat23:05:182
28306992010,3cyclictest141rcu_preempt20:10:212
2830599206,6cyclictest22-21ksoftirqd/119:15:251
2830599206,4cyclictest22-21ksoftirqd/120:50:181
2830599204,4cyclictest22-21ksoftirqd/119:49:071
2830599203,3cyclictest141rcu_preempt19:50:141
28305992015,3cyclictest22-21ksoftirqd/119:35:001
28305992014,4cyclictest22-21ksoftirqd/120:00:001
28305992010,2cyclictest141rcu_preempt20:00:191
2830499205,3cyclictest141rcu_preempt19:20:000
28304992012,6cyclictest922-21ls20:35:170
28304992012,6cyclictest3187-21cat19:25:190
28304992012,6cyclictest21927-21awk20:10:120
28304992011,7cyclictest21039-21cat20:05:230
2830499201,10cyclictest23667-21rm22:26:300
2830499201,10cyclictest12-21ksoftirqd/023:47:130
28304992010,8cyclictest23350-21ssh22:02:460
2830799199,7cyclictest22448-21sed20:10:163
2830799198,3cyclictest141rcu_preempt22:35:233
2830799193,11cyclictest28006-21ssh00:30:113
2830799192,12cyclictest4907-21apt-get23:25:133
2830799191,6cyclictest0-21swapper/322:50:203
28307991913,4cyclictest34-21ksoftirqd/319:35:003
28307991913,3cyclictest1739-21uptime19:20:263
2830799191,2cyclictest141rcu_preempt21:25:243
2830799191,2cyclictest0-21swapper/321:10:113
28307991912,5cyclictest19611-21sed20:05:013
28307991912,4cyclictest5992-21vmstat20:45:253
28307991911,6cyclictest12211-21sh21:55:123
28307991910,6cyclictest52950irq/141-enp3s0-21:36:063
2830799190,5cyclictest5377-21apt-get21:50:133
2830699199,6cyclictest53050irq/142-enp3s0-21:30:142
2830699199,6cyclictest53050irq/142-enp3s0-00:30:102
2830699199,4cyclictest28-21ksoftirqd/221:20:002
2830699192,12cyclictest8755-21apt-get20:55:122
2830699191,8cyclictest10155-21python322:40:232
28306991914,3cyclictest3669-21sendmail_mailtr00:35:252
28306991914,3cyclictest17761-21sh23:34:462
28306991913,4cyclictest9656-21sh21:52:562
28306991912,5cyclictest14424-21run-parts23:55:252
28306991912,4cyclictest3649-21timerwakeupswit20:40:232
28306991911,6cyclictest31254-21memory19:15:212
28306991911,6cyclictest16309-21needreboot21:10:172
28306991911,6cyclictest13355-21cat21:05:142
2830699191,15cyclictest18457-21ssh23:35:122
28306991910,7cyclictest978-21cat20:35:182
28306991910,7cyclictest16127-21tune2fs22:45:152
28306991910,7cyclictest12384-21cat19:45:252
28306991910,6cyclictest2805-21tune2fs19:25:162
28306991910,6cyclictest24647-21cat20:15:182
2830599196,4cyclictest14947-21diskmemload22:20:161
2830599194,4cyclictest22-21ksoftirqd/121:02:061
28305991912,5cyclictest687-21sed20:35:151
2830499194,8cyclictest141rcu_preempt20:55:280
2830499192,3cyclictest141rcu_preempt19:20:130
2830499192,12cyclictest6615-21apt-get20:50:130
28304991911,6cyclictest10519-21expr21:30:140
2830499191,13cyclictest30457-21apt-get20:30:120
28304991910,6cyclictest2682-21cat20:40:140
2830799189,7cyclictest8498-21ssh00:39:533
2830799185,3cyclictest141rcu_preempt19:30:003
2830799181,5cyclictest0-21swapper/323:08:543
2830799181,4cyclictest0-21swapper/322:10:273
2830799181,3cyclictest0-21swapper/322:29:273
28307991813,3cyclictest14750-21sh22:44:353
28307991811,5cyclictest30474-21cat19:15:133
28307991811,5cyclictest25208-21sed20:15:243
2830699189,6cyclictest13691-21sed21:55:242
2830699189,6cyclictest13619-21cat19:50:132
2830699182,11cyclictest17598-21apt-get20:00:112
28306991813,4cyclictest28-21ksoftirqd/220:25:002
28306991813,3cyclictest16466-21processes19:55:222
28306991812,4cyclictest4446-21sh21:25:202
28306991812,4cyclictest23875-21sh00:26:192
28306991812,4cyclictest23875-21sh00:26:192
28306991812,4cyclictest17097-21sh00:21:152
28306991811,5cyclictest9241-21cat19:40:132
28306991811,5cyclictest32425-21sed20:35:002
28306991811,5cyclictest28488-21cat20:25:132
2830599189,2cyclictest22-21ksoftirqd/120:18:211
28305991812,4cyclictest22-21ksoftirqd/120:25:001
28305991812,4cyclictest22-21ksoftirqd/119:20:261
28305991810,6cyclictest3154-21cat19:25:181
28305991810,5cyclictest2461-21tr20:40:121
2830499185,2cyclictest141rcu_preempt19:50:140
2830499182,11cyclictest15441-21apt-get19:55:130
2830499181,3cyclictest141rcu_preempt19:40:110
2830499181,2cyclictest141rcu_preempt19:45:240
28304991811,5cyclictest8436-21seq19:36:070
28304991811,5cyclictest18751-21sensors20:00:210
28304991811,5cyclictest12-21ksoftirqd/020:50:000
28304991810,6cyclictest6019-21users19:30:260
28304991810,6cyclictest26794-21if_err_enp3s020:20:160
2830799178,6cyclictest11637-21sort21:00:173
2830799171,5cyclictest0-21swapper/320:35:203
28307991711,4cyclictest24280-21tr21:40:143
28307991710,5cyclictest7018-21grep20:50:153
28307991710,4cyclictest7334-21grep19:35:153
2830699179,6cyclictest6681-21cat20:50:132
2830699176,3cyclictest141rcu_preempt19:35:002
2830599179,6cyclictest22796-21ntp_states20:10:201
28305991711,4cyclictest22-21ksoftirqd/121:00:001
28304991711,4cyclictest13805-21load21:05:190
28304991710,5cyclictest11195-21cat21:00:130
2830799168,6cyclictest26299-21grep20:20:123
28307991611,3cyclictest15194-21latency_hist19:55:003
2830499163,8cyclictest24015-21apt-get20:15:130
2830799158,5cyclictest12654-21awk19:45:263
28307991510,3cyclictest636-21runrttasks19:59:573
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional