You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 09:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu Apr 16, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10756222698,50sleep10-21swapper/119:08:011
883621660,1sleep20-21swapper/223:44:042
3159121640,3sleep231593-21fschecks_time00:25:152
2250921570,2sleep10-21swapper/121:51:191
106012152101,24sleep00-21swapper/019:05:590
107282138102,25sleep30-21swapper/319:07:393
10640213496,26sleep20-21swapper/219:06:282
216412720,3sleep10-21swapper/122:40:111
124762650,7sleep0141rcu_preempt00:35:160
174942640,2sleep00-21swapper/019:24:590
6882630,1sleep30-21swapper/321:11:093
255702630,1sleep10-21swapper/100:20:201
110642620,2sleep20-21swapper/222:07:142
62502560,2sleep30-21swapper/322:28:223
173162560,1sleep1211rcuc/120:40:011
34922550,1sleep30-21swapper/323:40:093
140372550,1sleep00-21swapper/020:30:210
251312540,3sleep2271rcuc/221:30:022
48132510,2sleep30-21swapper/322:51:133
316722510,1sleep00-21swapper/022:22:570
322262480,1sleep20-21swapper/223:36:352
43802470,1sleep20-21swapper/221:38:362
217642470,1sleep10-21swapper/121:26:481
11184994714,18cyclictest16187-21ntp_states20:35:200
11184994610,22cyclictest22997-21date23:55:010
258932440,1sleep30-21swapper/321:00:013
1118499447,23cyclictest636-21runrttasks19:40:080
1118499441,25cyclictest22130-21ssh23:05:130
11184994412,19cyclictest30244-21diskmemload21:11:100
214412430,1sleep30-21swapper/300:17:173
112602430,2sleep311247-21apt-get00:10:113
1118499435,23cyclictest18062-21sh21:48:340
1118499432,27cyclictest9956-21latency_hist23:45:010
1118499431,23cyclictest0-21swapper/022:00:110
1118499431,23cyclictest0-21swapper/022:00:110
1118499430,6cyclictest21402-21sh23:29:020
239392410,1sleep00-21swapper/021:52:470
1118499415,26cyclictest30244-21diskmemload23:35:280
1118499414,21cyclictest27495-21sh21:55:200
1118499414,21cyclictest27495-21sh21:55:200
11184994111,16cyclictest636-21runrttasks19:58:070
1118499403,3cyclictest8778-21sh21:41:050
1118499401,22cyclictest636-21runrttasks19:15:240
130282390,1sleep00-21swapper/000:10:280
1118499396,19cyclictest30244-21diskmemload22:32:400
1118499395,19cyclictest7356-21sh00:06:400
1118499391,6cyclictest24668-21sh21:29:470
1118499384,20cyclictest12255-21sh23:45:480
1118499384,20cyclictest0-21swapper/023:03:410
260252360,2sleep20-21swapper/219:45:002
1118499369,15cyclictest30244-21diskmemload22:27:340
1118499358,16cyclictest30244-21diskmemload23:14:010
1118499352,28cyclictest5130-21ssh21:15:110
1118499344,17cyclictest30244-21diskmemload00:20:400
1118499342,26cyclictest0-21swapper/000:25:110
1118499340,6cyclictest18480-21runrttasks22:37:220
1118499340,3cyclictest6734-21wc00:30:250
1118499339,21cyclictest28641-21expr19:50:110
1118499333,25cyclictest26214-21apt-get21:00:120
1118499333,25cyclictest0-21swapper/023:55:110
1118499333,23cyclictest21951-21apt-get19:35:120
1118499332,25cyclictest0-21swapper/021:30:120
11184993310,20cyclictest11955-21rm23:21:080
1118499330,7cyclictest0-21swapper/020:55:250
1118499330,5cyclictest22444-21sed22:15:260
1118499330,2cyclictest0-21swapper/022:45:170
1118499327,22cyclictest2780-21tune2fs20:05:150
1118499322,25cyclictest11379-21apt-get21:20:010
1118499321,5cyclictest30244-21diskmemload21:39:180
1118499321,3cyclictest0-21swapper/000:00:090
1118499321,10cyclictest21506-21taskset22:40:110
1118499321,10cyclictest14382-21ssh22:59:140
1118499318,21cyclictest17859-21expr19:25:140
1118499317,21cyclictest27014-21missed_timers19:45:190
11184993111,18cyclictest636-21runrttasks00:15:010
1118499311,10cyclictest23901-21ssh23:30:210
1118499311,10cyclictest23901-21ssh23:30:200
1118499310,6cyclictest20140-21taskset19:30:130
1118499310,5cyclictest5247-21sed20:10:180
1118499310,3cyclictest8798-21grep22:05:220
11185993012,5cyclictest22-21ksoftirqd/121:10:201
1118499306,21cyclictest12416-21sed19:10:230
1118499301,9cyclictest3948-21ntp_states23:15:200
1118499300,4cyclictest52850irq/140-enp3s0-22:10:140
1118499300,3cyclictest7637-21pmu-power20:15:220
1118599299,5cyclictest22-21ksoftirqd/123:38:571
11185992913,4cyclictest22-21ksoftirqd/123:08:341
1118499296,21cyclictest560-21dbus-daemon21:09:590
1118499291,8cyclictest1700-21grep20:00:230
1118499290,7cyclictest6226-21ssh22:52:400
1118499289,16cyclictest23163-21unixbench_singl20:50:250
1118499289,16cyclictest20840-21timerandwakeup20:45:220
1118499281,8cyclictest18557-21mailstats20:40:220
95072270,8sleep228-21ksoftirqd/221:41:502
42642270,1sleep10-21swapper/100:05:021
20192270,1sleep30-21swapper/323:38:533
11185992711,2cyclictest141rcu_preempt23:17:331
1118499271,6cyclictest9553-21sed20:20:180
1118499270,8cyclictest11190-21expr20:25:120
232572260,2sleep30-21swapper/323:05:223
1118699262,16cyclictest28-21ksoftirqd/223:12:052
11185992612,3cyclictest141rcu_preempt00:17:361
11185992610,8cyclictest22-21ksoftirqd/122:26:221
11185992610,4cyclictest22-21ksoftirqd/122:19:371
11187992514,7cyclictest11547-21df19:10:123
1118699257,11cyclictest28-21ksoftirqd/223:03:382
1118699257,10cyclictest28-21ksoftirqd/223:05:172
1118699256,3cyclictest141rcu_preempt21:21:072
1118699255,10cyclictest28-21ksoftirqd/222:54:232
11186992516,6cyclictest17806-21sh21:48:172
1118599259,4cyclictest22-21ksoftirqd/100:11:271
1118599257,5cyclictest22-21ksoftirqd/122:23:201
1118599253,15cyclictest30244-21diskmemload23:45:091
11185992518,5cyclictest22-21ksoftirqd/121:35:001
11185992514,7cyclictest23361-21apt-get23:55:111
11185992510,4cyclictest22-21ksoftirqd/122:45:141
11185992510,4cyclictest22-21ksoftirqd/121:55:161
11185992510,4cyclictest22-21ksoftirqd/121:55:151
11185992510,4cyclictest22-21ksoftirqd/100:25:181
11187992413,9cyclictest21316-21sh23:04:383
1118699246,9cyclictest28-21ksoftirqd/200:21:052
1118699243,14cyclictest141rcu_preempt22:56:532
1118699241,1cyclictest0-21swapper/219:10:092
1118599249,4cyclictest22-21ksoftirqd/121:40:151
1118599248,5cyclictest22-21ksoftirqd/100:00:131
1118599248,4cyclictest22-21ksoftirqd/123:43:341
1118599248,2cyclictest141rcu_preempt20:05:011
1118599242,11cyclictest22-21ksoftirqd/123:54:451
11185992419,3cyclictest22-21ksoftirqd/100:40:011
11185992419,3cyclictest16532-21grep23:00:191
11185992418,4cyclictest22-21ksoftirqd/121:20:001
11185992411,4cyclictest22-21ksoftirqd/122:33:011
11185992410,2cyclictest141rcu_preempt23:21:331
297012230,2sleep20-21swapper/221:05:262
185242230,10sleep228-21ksoftirqd/222:13:042
11187992314,6cyclictest30275-21sed21:10:003
11187992312,8cyclictest19897-21df_inode20:45:123
1118699239,4cyclictest28-21ksoftirqd/221:53:542
1118699239,11cyclictest53050irq/142-enp3s0-00:10:032
1118699236,8cyclictest141rcu_preempt22:41:212
1118699236,7cyclictest28-21ksoftirqd/200:19:022
1118699235,10cyclictest28-21ksoftirqd/200:35:292
1118699233,14cyclictest14848-21kworker/u8:122:25:092
11186992314,5cyclictest53050irq/142-enp3s0-23:25:112
1118699231,3cyclictest141rcu_preempt22:45:122
11186992312,8cyclictest17319-21grep20:40:012
1118599239,2cyclictest141rcu_preempt21:45:001
1118599237,4cyclictest22-21ksoftirqd/100:30:361
1118599237,2cyclictest141rcu_preempt19:26:171
1118599232,2cyclictest141rcu_preempt22:00:251
1118599232,2cyclictest141rcu_preempt22:00:241
1118599231,9cyclictest141rcu_preempt22:35:121
11185992310,2cyclictest141rcu_preempt21:09:371
60912220,1sleep10-21swapper/122:52:311
47422220,3sleep228-21ksoftirqd/200:05:122
11187992215,5cyclictest25248-21tr19:40:223
11187992213,7cyclictest15265-21forks22:35:153
1118699228,12cyclictest28-21ksoftirqd/223:47:202
1118699227,6cyclictest141rcu_preempt21:55:532
1118699227,6cyclictest141rcu_preempt21:55:522
1118699227,3cyclictest141rcu_preempt00:00:132
11186992215,5cyclictest28-21ksoftirqd/219:29:592
11186992212,2cyclictest14041sed22:25:012
11186992211,2cyclictest141rcu_preempt23:57:072
1118699221,11cyclictest28-21ksoftirqd/223:15:472
11186992210,3cyclictest141rcu_preempt22:30:112
11186992210,2cyclictest141rcu_preempt21:28:462
1118599229,4cyclictest22-21ksoftirqd/123:34:351
1118599229,4cyclictest22-21ksoftirqd/123:34:341
1118599229,2cyclictest141rcu_preempt23:12:451
1118599228,2cyclictest141rcu_preempt20:23:191
1118599227,4cyclictest22-21ksoftirqd/123:26:211
1118599226,3cyclictest141rcu_preempt22:10:241
1118599226,3cyclictest141rcu_preempt19:35:121
1118599223,10cyclictest22-21ksoftirqd/122:58:071
1118599221,6cyclictest22-21ksoftirqd/122:05:251
11185992216,4cyclictest22-21ksoftirqd/121:40:011
11185992213,7cyclictest27499-21cat21:00:241
11185992210,3cyclictest22-21ksoftirqd/121:20:451
11185992210,2cyclictest141rcu_preempt19:43:181
1118599220,5cyclictest0-21swapper/119:45:131
11187992113,6cyclictest9586-21python322:30:233
11187992113,6cyclictest29607-21sed23:10:183
11187992113,6cyclictest28386-21apt-get19:50:103
11187992112,7cyclictest16839-21cat23:25:163
11187992112,6cyclictest21094-21ls19:30:233
11187992111,6cyclictest52750irq/139-enp3s0-21:40:113
1118699217,7cyclictest141rcu_preempt22:38:342
1118699214,5cyclictest28-21ksoftirqd/222:15:252
1118699214,15cyclictest171141ssh23:50:132
11186992110,2cyclictest28-21ksoftirqd/222:00:122
11186992110,2cyclictest28-21ksoftirqd/222:00:112
1118699210,7cyclictest4950-21fschecks_time20:10:142
1118599219,4cyclictest22-21ksoftirqd/119:18:261
1118599219,3cyclictest22-21ksoftirqd/120:25:181
1118599212,3cyclictest141rcu_preempt19:32:011
11185992112,6cyclictest278-21systemd-journal19:15:001
69822200,2sleep30-21swapper/322:04:503
69822200,2sleep30-21swapper/322:04:503
11187992015,3cyclictest34-21ksoftirqd/320:00:013
11187992014,4cyclictest34-21ksoftirqd/321:35:013
11187992014,4cyclictest34-21ksoftirqd/321:15:013
11187992014,4cyclictest26668-21sh23:57:033
11187992013,5cyclictest9542-21sh00:32:573
11187992012,6cyclictest9181-21diskstats20:20:143
11187992012,6cyclictest5716-21readlink00:05:213
11187992012,6cyclictest26622-21cat21:55:133
11187992012,6cyclictest26622-21cat21:55:133
11187992012,6cyclictest1170-21awk21:35:243
11187992011,6cyclictest12401-21diskstats21:20:153
11187992011,5cyclictest52750irq/139-enp3s0-00:00:163
1118699209,2cyclictest28-21ksoftirqd/223:30:122
1118699209,2cyclictest28-21ksoftirqd/223:30:122
1118699204,9cyclictest141rcu_preempt19:35:132
1118699204,5cyclictest0-21swapper/223:21:402
1118699202,9cyclictest141rcu_preempt20:40:112
1118699202,4cyclictest141rcu_preempt19:50:102
11186992012,6cyclictest20943-21systemctl19:30:232
11186992012,6cyclictest13595-21fw_packets20:30:162
11186992011,6cyclictest11686-21cat20:25:182
1118699201,12cyclictest141rcu_preempt00:33:082
1118699201,10cyclictest0-21swapper/221:10:072
1118599206,4cyclictest211rcuc/120:15:131
1118599204,8cyclictest22-21ksoftirqd/119:59:421
1118599203,2cyclictest141rcu_preempt20:10:221
11185992014,4cyclictest22-21ksoftirqd/120:10:001
11185992012,6cyclictest20889-21ls20:45:241
84432190,2sleep30-21swapper/322:05:183
1118799192,12cyclictest26176-21apt-get21:00:043
11187991914,3cyclictest11794-21sh23:20:583
11187991913,4cyclictest5377-21sh23:16:133
11187991913,4cyclictest34-21ksoftirqd/321:55:003
11187991913,4cyclictest29391-21sh22:20:363
11187991913,4cyclictest22634-21sh23:54:423
11187991913,4cyclictest17748-21sh22:12:173
11187991913,4cyclictest15256-21sh21:45:423
11187991913,4cyclictest14943-21sh00:36:463
11187991913,4cyclictest13357-21sh22:58:113
11187991913,4cyclictest13134-21sh23:46:423
11187991912,5cyclictest21380-21fw_packets22:15:163
11187991912,4cyclictest30244-21diskmemload00:27:593
11187991911,6cyclictest535-21expr20:00:133
11187991911,6cyclictest11148-21seq20:25:123
1118699192,3cyclictest141rcu_preempt20:45:142
1118699191,5cyclictest0-21swapper/221:00:252
1118699191,4cyclictest0-21swapper/221:15:162
11186991914,3cyclictest8820-21acpi20:20:102
11186991913,4cyclictest636-21runrttasks19:15:112
11186991912,5cyclictest24235-21cut20:55:162
11186991910,7cyclictest31525-21awk19:55:212
1118699190,2cyclictest141rcu_preempt20:20:002
11185991911,5cyclictest13597-21gltestperf20:30:161
11185991910,6cyclictest15529-21apt19:20:101
1118799189,7cyclictest624-21gdbus20:39:593
11187991813,3cyclictest23650-21sh22:40:513
11187991813,3cyclictest18669-21threads20:40:233
11187991812,4cyclictest26568-21sh00:20:453
11187991812,4cyclictest21926-21/usr/sbin/munin19:35:153
11187991812,4cyclictest21853-21sh21:26:543
11187991811,5cyclictest18232-21smartctl19:25:173
11187991811,5cyclictest16518-21awk19:20:203
1118799181,13cyclictest28151-21ssh22:45:123
11187991810,6cyclictest23887-21cut23:30:213
11187991810,6cyclictest23887-21cut23:30:213
11187991810,5cyclictest2915-21gltestperf20:05:163
1118699181,5cyclictest0-21swapper/219:45:142
11186991810,6cyclictest3326-21ntpq20:05:202
11186991810,6cyclictest21701-21awk20:50:112
11186991810,6cyclictest16289-21load19:20:172
1118599189,6cyclictest17757-21df_inode20:40:131
11185991811,5cyclictest22-21ksoftirqd/120:50:151
11185991810,6cyclictest24446-21cat20:55:171
1118799178,6cyclictest4543-21apt-get20:10:123
1118799171,11cyclictest21704-21apt-get20:50:123
1118799171,11cyclictest13370-21apt-get19:15:123
11187991710,5cyclictest26445-21expr19:45:133
11187991710,5cyclictest13202-21cat20:30:133
11186991710,5cyclictest503-21expr20:00:122
11185991710,5cyclictest28662-21expr19:50:121
1118799169,5cyclictest6814-21expr20:15:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional