You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-06 - 11:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun Sep 06, 2026 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
290922180106,29sleep00-21swapper/019:08:200
289242169114,28sleep30-21swapper/319:06:103
290012166110,28sleep10-21swapper/119:07:091
289062158108,28sleep20-21swapper/219:05:572
273192710,3sleep227315-21awk20:20:012
199672690,2sleep00-21swapper/022:48:170
236462650,2sleep20-21swapper/221:15:092
123052630,1sleep20-21swapper/222:18:382
279342580,3sleep3331rcuc/322:54:343
214012570,8sleep122-21ksoftirqd/123:13:381
70592510,2sleep10-21swapper/120:45:231
37742490,2sleep00-21swapper/021:48:060
10122480,2sleep21003-21ssh22:10:162
271482470,2sleep20-21swapper/221:40:452
173452470,3sleep00-21swapper/023:58:490
11682460,4chrt1144-21apt-get00:35:163
205792430,2chrt0-21swapper/021:12:300
240462420,3chrt0-21swapper/300:27:413
285832410,1sleep20-21swapper/223:42:412
111752410,2sleep00-21swapper/022:41:110
39952400,8sleep122-21ksoftirqd/121:48:201
29497994023,11cyclictest12283-21apt-get23:55:123
2949799398,16cyclictest13406-21rm23:30:583
2949799388,13cyclictest4178-21sed23:25:003
2949799379,11cyclictest20146-21ssh21:12:043
2949799363,16cyclictest0-21swapper/321:25:153
29497993510,16cyclictest14148-21ssh22:44:113
2949799349,15cyclictest16343-21diskmemload00:04:203
2949799348,16cyclictest6530-21sh23:02:093
2949799348,10cyclictest16343-21diskmemload22:22:543
2949799347,10cyclictest19038-21grep21:35:173
2949799345,13cyclictest6938-21rm23:50:233
29497993410,9cyclictest4355-21rm00:13:063
2949799338,9cyclictest11578-21rm23:05:303
2949799338,16cyclictest8161-21sh22:39:563
2949799337,10cyclictest29054-21cstates22:55:133
2949799337,10cyclictest16343-21diskmemload22:48:443
2949799336,10cyclictest26968-21taskset22:29:433
2949799329,14cyclictest23501-21sh22:02:313
2949799329,14cyclictest23501-21sh22:02:303
2949799328,9cyclictest16343-21diskmemload00:07:053
2949799328,9cyclictest16343-21diskmemload00:07:053
2949799328,8cyclictest19893-21sh23:35:433
2949799326,9cyclictest8444-21ssh21:51:053
2949799325,11cyclictest13507-21if_err_enp3s021:55:183
2949799325,10cyclictest16343-21diskmemload22:30:373
29497993217,9cyclictest26393-21chrt22:05:103
2949799318,9cyclictest26835-21ssh23:40:553
2949799318,8cyclictest23332-21chrt23:15:103
2949799317,9cyclictest2424-21grep20:35:193
2949799316,9cyclictest13098-21rm22:19:263
2949799316,7cyclictest11499-21sh00:18:353
2949799315,9cyclictest11521-21ssh21:30:053
2949799309,11cyclictest29969-21sh00:31:523
2949799305,14cyclictest0-21swapper/321:15:433
2949799305,11cyclictest32202-21cat21:20:263
29497993010,17cyclictest12278-21grep19:45:013
29497993010,17cyclictest12278-21grep19:45:003
2949799296,9cyclictest16343-21diskmemload23:28:083
2949799296,9cyclictest16343-21diskmemload23:28:073
2949799296,14cyclictest20117-21ssh23:12:213
2949799296,10cyclictest4016-21sh23:48:513
2949799293,6cyclictest26072-21ls21:40:213
2949799292,16cyclictest0-21swapper/322:10:143
287492290,9sleep012-21ksoftirqd/023:42:510
29496992818,7cyclictest19022-21apt-get00:00:142
2949499281,25cyclictest0-21swapper/019:28:350
186272280,1sleep10-21swapper/122:23:021
2949799273,9cyclictest0-21swapper/320:50:273
2949799273,14cyclictest0-21swapper/300:20:123
2949799271,8cyclictest16343-21diskmemload21:46:213
2949799270,16cyclictest0-21swapper/319:30:123
29495992725,1cyclictest0-21swapper/119:21:151
2949599271,24cyclictest0-21swapper/119:27:511
241542270,14sleep224149-21cron23:40:002
29497992622,3cyclictest34-21ksoftirqd/319:19:353
2949799261,3cyclictest34-21ksoftirqd/319:36:403
2949799261,15cyclictest0-21swapper/319:20:243
29497992611,13cyclictest32259-21fschecks_time20:30:153
29497992611,13cyclictest32259-21fschecks_time20:30:153
2949599260,24cyclictest0-21swapper/119:19:091
29494992615,6cyclictest16343-21diskmemload22:50:170
2949499261,24cyclictest0-21swapper/019:50:050
2949499261,24cyclictest0-21swapper/019:35:570
2949499261,24cyclictest0-21swapper/019:30:070
2949499261,24cyclictest0-21swapper/019:20:290
2949499261,1cyclictest0-21swapper/019:19:540
2949499260,24cyclictest0-21swapper/020:49:220
252242260,3chrt0-21swapper/200:28:522
2949799253,9cyclictest4458-21cut19:25:183
2949799252,14cyclictest0-21swapper/320:10:243
29496992523,1cyclictest0-21swapper/219:35:072
29496992523,1cyclictest0-21swapper/219:28:092
2949699251,1cyclictest0-21swapper/219:20:022
2949599259,11cyclictest15446-21ssh22:45:141
29495992523,1cyclictest0-21swapper/119:30:081
2949599251,1cyclictest0-21swapper/119:35:581
8842240,1sleep00-21swapper/000:35:000
2949799243,8cyclictest0-21swapper/320:05:153
29497992416,3cyclictest10154-21apt-get20:55:123
29497992413,9cyclictest3841-21tr20:40:143
29497992412,8cyclictest0-21swapper/319:50:123
29496992422,1cyclictest0-21swapper/219:19:492
29495992414,7cyclictest13157-21df_inode21:55:161
292192240,2sleep00-21swapper/022:06:320
169402240,1sleep00-21swapper/021:57:530
29497992312,8cyclictest28133-21/usr/sbin/munin20:20:163
29497992310,10cyclictest12698-21/usr/sbin/munin19:45:163
29496992321,1cyclictest0-21swapper/219:30:452
2949599232,19cyclictest4543-21cron21:25:001
2949599232,17cyclictest12288-21gltestperf21:30:201
29495992316,5cyclictest9699-21ssh23:52:491
29495992312,9cyclictest663-21/usr/sbin/munin22:10:151
6102220,9sleep228-21ksoftirqd/222:57:582
2949799229,11cyclictest30879-21/usr/sbin/munin20:25:253
2949799227,12cyclictest20539-21seq20:00:483
29497992214,5cyclictest5982-21tr20:45:123
29497992213,7cyclictest12496-21expr21:00:133
29497992210,9cyclictest18809-21cat20:00:013
29496992214,6cyclictest7479-21sh22:39:162
2949699220,5cyclictest0-21swapper/221:59:482
29495992212,7cyclictest278-21systemd-journal21:05:001
2949499222,14cyclictest18385-21sh21:35:120
2949499221,9cyclictest209072ssh23:13:080
29494992214,6cyclictest31714-21ssh22:32:460
269762220,3sleep226995-21ssh22:53:382
29497992110,9cyclictest26334-21ntpq20:15:213
29496992114,5cyclictest6679-21open_files23:50:212
29495992115,4cyclictest11094-21acpi23:30:111
29495992113,6cyclictest670-21ls20:30:251
29495992113,6cyclictest670-21ls20:30:241
29495992113,6cyclictest14792-21diskstats21:05:171
29495992113,6cyclictest13207-21seq22:19:331
29495992112,7cyclictest560-21dbus-daemon00:30:001
2949499214,6cyclictest0-21swapper/022:56:500
29494992116,3cyclictest27302-21dump-pmu-power20:20:000
29494992113,6cyclictest24594-21seq22:27:180
29494992112,7cyclictest12920-21cron00:20:010
29494992110,9cyclictest10254-21sh23:29:340
29494992110,9cyclictest10254-21sh23:29:340
2949499210,6cyclictest11222-21cron21:30:000
294722210,2sleep20-21swapper/222:30:292
12882210,1sleep00-21swapper/023:46:050
29497992011,7cyclictest29601-21cat19:10:133
29497992010,7cyclictest15922-21unixbench_singl21:05:283
29496992012,6cyclictest2743-21unixbench-2d00:35:272
29496992012,6cyclictest11932-21seq19:43:032
29496992012,6cyclictest11932-21seq19:43:022
2949699201,16cyclictest20794-21ssh22:25:012
29496992011,6cyclictest12449-21grep19:45:142
2949699201,14cyclictest19547-21ssh22:00:042
2949699201,14cyclictest19547-21ssh22:00:032
29496992010,7cyclictest560-21dbus-daemon23:19:592
29496992010,6cyclictest52750irq/139-enp3s0-22:05:122
29496992010,6cyclictest52750irq/139-enp3s0-21:20:152
2949599207,3cyclictest141rcu_preempt22:50:181
2949599202,13cyclictest636-21runrttasks00:00:121
2949599202,13cyclictest19757-21ssh22:00:141
2949599202,13cyclictest19757-21ssh22:00:141
29495992014,4cyclictest8949-21sh00:15:581
29495992012,6cyclictest560-21dbus-daemon21:15:001
29495992012,6cyclictest31081-21seq19:10:481
29495992012,6cyclictest27846-21sed22:05:231
29495992012,6cyclictest27387-21sed22:30:021
29495992012,6cyclictest25498-21seq21:40:151
29495992011,7cyclictest17789-21swap19:55:241
2949599201,17cyclictest16139-21sh00:21:281
2949499206,8cyclictest141rcu_preempt21:34:040
2949499206,12cyclictest141rcu_preempt22:11:310
29494992015,3cyclictest29464-21awk20:25:000
29494992014,4cyclictest15307-21sh23:32:520
29494992012,6cyclictest31495-21ls21:20:200
29494992012,6cyclictest25868-21seq00:29:310
29494992011,7cyclictest30076-21expr19:10:160
29494992011,7cyclictest2952-21cat20:35:250
29494992011,7cyclictest10862-21ntp_states23:05:230
2949499200,4cyclictest0-21swapper/000:23:330
116662201,1sleep20-21swapper/221:30:162
2949699198,7cyclictest13443-21cpu00:20:142
2949699192,15cyclictest28538-21grep00:30:302
2949699192,12cyclictest12311-21sh23:55:122
2949699191,9cyclictest17524-21irqstats21:10:172
29496991912,5cyclictest560-21dbus-daemon00:20:002
29496991912,5cyclictest32740-21cat00:10:142
29496991912,5cyclictest31275-21seq19:13:182
29496991912,5cyclictest13193-21awk21:00:192
29496991912,4cyclictest16343-21diskmemload23:21:362
29496991911,6cyclictest560-21dbus-daemon20:05:002
29496991911,6cyclictest4490-21ls20:40:182
29496991911,6cyclictest23355-21sort20:10:142
29496991911,6cyclictest2319-21sed20:35:182
2949699191,13cyclictest4976-21ssh21:25:122
29496991910,7cyclictest28873-21cat20:20:242
2949599197,9cyclictest4428-21apt-get23:25:111
2949599197,9cyclictest4428-21apt-get23:25:111
2949599192,2cyclictest141rcu_preempt21:26:411
29495991913,4cyclictest788-21sh22:58:081
29495991913,4cyclictest53050irq/142-enp3s0-00:05:221
29495991913,4cyclictest53050irq/142-enp3s0-00:05:211
29495991912,5cyclictest4387-21seq23:49:131
29495991912,5cyclictest15094-21sed22:45:001
29495991912,4cyclictest16343-21diskmemload21:38:051
29495991911,6cyclictest17155-21seq23:58:361
29495991911,6cyclictest12423-21cat19:45:151
2949599191,13cyclictest26860-21apt-get00:30:171
2949499199,8cyclictest3266-21ssh00:35:510
2949499191,4cyclictest0-21swapper/023:54:360
29494991914,3cyclictest26466-21kernelversion00:05:180
29494991914,3cyclictest26466-21kernelversion00:05:180
29494991912,5cyclictest2224-21expr22:35:150
29494991911,6cyclictest3860-21cut23:00:180
29494991911,6cyclictest344-21pluginstate20:30:230
29494991911,6cyclictest344-21pluginstate20:30:220
29494991911,6cyclictest20544-21seq20:00:530
2949499191,15cyclictest25271-21cron00:05:000
29494991910,6cyclictest25181-21timerandwakeup21:15:240
2949699188,7cyclictest31953-21tr21:45:132
2949699187,8cyclictest4730-21grep23:00:262
2949699182,12cyclictest20996-21apt-get22:25:122
2949699182,11cyclictest8697-21apt-get22:40:132
2949699182,11cyclictest11124-21apt-get23:30:112
2949699181,8cyclictest32343-21/usr/sbin/munin23:45:182
2949699181,8cyclictest17626-21ssh22:45:572
29496991811,5cyclictest11282-21run-parts20:55:222
29496991811,5cyclictest10872-21ntpq23:05:222
29496991810,6cyclictest7611-21sed21:50:272
29496991810,6cyclictest30628-21cat20:25:232
29496991810,6cyclictest29572-21seq00:07:362
29496991810,6cyclictest29572-21seq00:07:352
29496991810,6cyclictest15202-21expr21:05:202
2949599189,7cyclictest31701-21cat23:20:251
2949599189,6cyclictest53050irq/142-enp3s0-22:30:151
2949599188,7cyclictest10575-21tune2fs20:55:141
2949599182,11cyclictest18993-21apt-get20:00:101
2949599182,10cyclictest1393-21gpm20:50:131
29495991812,4cyclictest5436-21sh22:37:111
29495991812,4cyclictest23208-21readlink21:15:001
29495991812,4cyclictest18227-21sh23:35:131
29495991811,5cyclictest6957-21cat21:50:211
29495991811,5cyclictest3394-21expr23:00:151
29495991811,5cyclictest24858-21expr23:40:131
29495991811,4cyclictest4728-21sh00:37:211
29495991810,6cyclictest5028-21cat20:40:241
29495991810,6cyclictest25717-21expr20:15:141
29495991810,6cyclictest24989-21grep23:15:251
29495991810,6cyclictest10773-21smartctl19:40:161
29495991810,6cyclictest10773-21smartctl19:40:161
2949499188,8cyclictest3435-21ssh00:12:100
2949499182,12cyclictest19067-21apt-get22:00:040
2949499182,12cyclictest19067-21apt-get22:00:030
2949499182,11cyclictest12353-21apt-get19:45:140
2949499181,5cyclictest0-21swapper/021:40:140
29494991812,4cyclictest30025-21sh23:20:060
29494991812,4cyclictest25509-21sh23:15:490
29494991812,4cyclictest15396-21sh22:20:220
29494991812,4cyclictest12092-21sh19:45:000
29494991812,4cyclictest12092-21sh19:45:000
29494991811,5cyclictest6528-21awk21:50:190
29494991811,5cyclictest11300-21mailstats20:55:210
29494991811,4cyclictest7415-21sh22:15:110
2949499181,11cyclictest8020-21apt-get20:50:150
29494991810,6cyclictest18749-21cat23:35:190
2949699178,6cyclictest8363-21cat20:50:172
2949699171,6cyclictest4609-21cpuspeed_turbos23:25:142
2949699171,6cyclictest4609-21cpuspeed_turbos23:25:132
29496991712,3cyclictest16361-21sh23:10:042
29496991711,4cyclictest18221-21sh21:35:042
29496991710,5cyclictest22619-21sed20:05:242
29496991710,5cyclictest16877-21expr19:55:142
2949599179,6cyclictest640-21snmpd00:10:401
2949599179,6cyclictest23177-21cut20:10:011
2949599179,6cyclictest14862-21tune2fs19:50:161
2949599171,7cyclictest27546-21/usr/sbin/munin20:20:141
29495991712,3cyclictest13495-21sh23:07:251
29495991710,5cyclictest24511-21find20:10:241
29495991710,4cyclictest2091-21gltestperf20:35:161
2949499179,6cyclictest21289-21cstates20:05:120
2949499176,3cyclictest141rcu_preempt20:10:200
2949499175,3cyclictest0-21swapper/020:40:120
29494991712,3cyclictest12288-21/usr/sbin/munin21:00:220
15072170,1sleep20-21swapper/220:35:002
15072170,1sleep20-21swapper/220:35:002
2949699161,11cyclictest5879-21apt-get20:45:132
2949699161,11cyclictest14492-21apt-get19:50:132
2949599169,5cyclictest29878-21ls20:25:151
2949499169,5cyclictest16907-21expr19:55:140
2949499168,6cyclictest14431-21/usr/sbin/munin21:05:180
29494991610,4cyclictest30091-21fschecks_count20:25:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional