You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 02:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed Mar 25, 2026 12:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
166702156101,29sleep00-21swapper/007:06:050
166512155104,26sleep10-21swapper/107:05:491
166482154103,25sleep30-21swapper/307:05:483
167042149110,27sleep20-21swapper/207:06:312
2177321200,2sleep00-21swapper/009:22:230
2177321200,2sleep00-21swapper/009:22:220
1697528757,23sleep30-21swapper/307:10:013
255922790,17sleep122-21ksoftirqd/112:15:011
9292720,2sleep20-21swapper/209:55:282
310602620,12sleep012-21ksoftirqd/008:55:190
111402620,1sleep30-21swapper/310:50:513
111402620,1sleep30-21swapper/310:50:503
125092600,9sleep122-21ksoftirqd/110:52:141
125092600,9sleep122-21ksoftirqd/110:52:131
275452590,3sleep00-21swapper/010:15:200
191452590,2sleep10-21swapper/110:33:231
206782580,8sleep334-21ksoftirqd/311:46:173
10692580,3sleep11067-21ssh10:20:141
287902560,4sleep0131rcuc/011:05:000
104502560,1sleep20-21swapper/209:38:422
115662550,2sleep10-21swapper/110:04:211
320502540,2sleep317245-21cyclictest10:42:483
125972530,2sleep2141rcu_preempt10:28:262
33192520,2sleep3331rcuc/311:10:003
324702490,1sleep10-21swapper/110:19:381
318152480,1sleep10-21swapper/110:42:341
226512470,2sleep30-21swapper/312:12:063
94842460,1chrt0-21swapper/310:25:353
224232410,1sleep00-21swapper/007:20:200
128352400,1sleep30-21swapper/311:40:223
1724799363,6cyclictest12-21ksoftirqd/009:20:000
109552360,2sleep00-21swapper/010:27:040
17247993315,5cyclictest12-21ksoftirqd/011:45:010
1724799312,4cyclictest12-21ksoftirqd/012:05:000
1724799310,7cyclictest12-21ksoftirqd/010:30:120
1724799303,3cyclictest12-21ksoftirqd/009:10:000
17247993026,3cyclictest131rcuc/012:23:460
1724799302,5cyclictest12-21ksoftirqd/010:20:010
1724799302,4cyclictest12-21ksoftirqd/012:20:000
1724799301,7cyclictest12-21ksoftirqd/012:00:010
17247993014,6cyclictest12-21ksoftirqd/010:00:000
79992290,2sleep30-21swapper/311:36:573
56392290,10sleep122-21ksoftirqd/109:10:201
306842290,10sleep012-21ksoftirqd/011:30:130
306842290,10sleep012-21ksoftirqd/011:30:120
17249992920,3cyclictest29255-21ssh10:16:262
1724799294,5cyclictest12-21ksoftirqd/010:15:000
1724799293,5cyclictest12-21ksoftirqd/011:15:160
17247992925,3cyclictest131rcuc/011:36:360
17247992925,3cyclictest131rcuc/010:46:280
1725099286,16cyclictest34-21ksoftirqd/309:15:133
1724799285,6cyclictest12-21ksoftirqd/011:50:220
1724799283,4cyclictest12-21ksoftirqd/008:40:000
1724799282,5cyclictest12-21ksoftirqd/011:30:000
1724799281,8cyclictest12-21ksoftirqd/012:35:000
1725099277,17cyclictest34-21ksoftirqd/311:50:123
17248992720,5cyclictest22-21ksoftirqd/112:00:001
1724799273,10cyclictest5909-21ssh10:00:120
1724799272,4cyclictest12-21ksoftirqd/008:00:000
1724799270,5cyclictest0-21swapper/012:40:010
1725099269,9cyclictest34-21ksoftirqd/310:05:203
1725099269,9cyclictest34-21ksoftirqd/310:05:203
1725099269,9cyclictest34-21ksoftirqd/309:40:143
1725099267,16cyclictest34-21ksoftirqd/312:35:023
1725099266,17cyclictest34-21ksoftirqd/310:10:033
1724799267,7cyclictest12-21ksoftirqd/012:10:170
1724799264,7cyclictest12-21ksoftirqd/011:10:100
17247992620,4cyclictest12-21ksoftirqd/010:10:010
17247992620,4cyclictest12-21ksoftirqd/010:10:000
1724799261,7cyclictest12-21ksoftirqd/011:20:230
1724799261,4cyclictest12-21ksoftirqd/011:45:220
1725099255,15cyclictest34-21ksoftirqd/311:55:223
17250992516,7cyclictest4188-21diskmemload11:24:043
17248992513,7cyclictest52950irq/141-enp3s0-11:05:111
1724799253,3cyclictest12-21ksoftirqd/009:30:000
1724799252,5cyclictest12-21ksoftirqd/007:45:010
1724799252,3cyclictest12-21ksoftirqd/008:50:000
17247992522,2cyclictest131rcuc/010:51:270
17247992522,2cyclictest131rcuc/010:51:270
17247992520,4cyclictest131rcuc/008:01:160
1724799251,8cyclictest12-21ksoftirqd/011:05:320
1724799251,7cyclictest12-21ksoftirqd/008:10:010
1724799251,6cyclictest12-21ksoftirqd/012:25:010
1724799251,6cyclictest12-21ksoftirqd/009:12:020
1724799251,4cyclictest12-21ksoftirqd/012:10:010
1724799251,4cyclictest12-21ksoftirqd/007:20:020
1724799251,10cyclictest12-21ksoftirqd/008:10:240
1724799250,6cyclictest30211-21munin-run07:40:000
162752250,2sleep20-21swapper/212:07:272
1725099241,5cyclictest0-21swapper/310:02:223
17249992414,6cyclictest52850irq/140-enp3s0-09:20:142
17249992414,6cyclictest52850irq/140-enp3s0-09:20:132
17249992413,6cyclictest52850irq/140-enp3s0-11:55:122
17249992411,11cyclictest31787-21forks09:30:172
17248992413,7cyclictest31268-21grep11:55:001
17248992411,11cyclictest17339-21/usr/sbin/munin08:25:221
1724799243,7cyclictest12-21ksoftirqd/010:40:240
1724799243,6cyclictest12-21ksoftirqd/009:35:120
1724799242,3cyclictest12-21ksoftirqd/009:30:250
17247992420,3cyclictest131rcuc/010:56:190
1724799240,8cyclictest12-21ksoftirqd/009:50:480
1724799240,8cyclictest12-21ksoftirqd/009:40:310
1724799240,7cyclictest12-21ksoftirqd/008:50:250
141872240,4sleep114185-21ssh12:30:131
1725099238,4cyclictest141rcu_preempt08:40:003
1725099237,14cyclictest34-21ksoftirqd/309:21:523
1725099237,14cyclictest34-21ksoftirqd/309:21:523
1725099236,3cyclictest141rcu_preempt11:20:013
1725099236,3cyclictest141rcu_preempt09:10:113
1725099236,15cyclictest34-21ksoftirqd/309:26:113
17250992313,6cyclictest53050irq/142-enp3s0-09:55:233
1724999232,8cyclictest141rcu_preempt10:25:002
17249992310,3cyclictest4188-21diskmemload11:45:172
17248992316,5cyclictest22-21ksoftirqd/112:20:001
17248992313,7cyclictest19680-21smartctl09:45:231
1724799232,3cyclictest12-21ksoftirqd/010:35:170
1724799231,7cyclictest12-21ksoftirqd/009:49:290
17247992315,6cyclictest13380-21df_inode08:15:130
1725099228,12cyclictest53050irq/142-enp3s0-10:55:123
1725099227,11cyclictest34-21ksoftirqd/312:18:583
1725099223,3cyclictest162762ssh12:07:263
1725099223,13cyclictest28078-21apt-get08:50:133
1725099221,7cyclictest27853-21grep08:50:013
1725099221,5cyclictest0-21swapper/311:25:243
17250992215,4cyclictest4188-21diskmemload09:50:233
17250992212,7cyclictest7759-21sendmail_mailtr08:00:223
17250992210,3cyclictest26512-21/usr/sbin/munin10:15:263
17249992212,8cyclictest31263-21cat09:55:002
17249992211,7cyclictest52850irq/140-enp3s0-09:10:122
17249992211,7cyclictest27926-21apt-get10:40:112
17249992211,7cyclictest2341-21apt-get07:50:132
17249992211,7cyclictest15218-21apt-get08:20:132
17249992210,3cyclictest4188-21diskmemload11:50:162
1724999220,15cyclictest21010-21sshd10:10:232
1724899225,4cyclictest141rcu_preempt09:30:191
17248992215,5cyclictest22-21ksoftirqd/112:40:001
17248992215,4cyclictest3492-21/usr/sbin/munin11:10:151
17248992215,4cyclictest11900-21unixbench_multi11:15:251
17248992213,7cyclictest4188-21diskmemload12:24:091
17248992213,7cyclictest19658-21ls10:10:121
17247992219,2cyclictest131rcuc/007:12:100
1724799221,8cyclictest12-21ksoftirqd/008:30:160
17247992212,7cyclictest278-21systemd-journal07:29:590
1724799220,8cyclictest16802-21taskset08:21:190
22612210,2sleep30-21swapper/307:50:003
1725099218,10cyclictest7159-21grep12:00:223
1725099212,4cyclictest34-21ksoftirqd/309:35:223
1725099212,3cyclictest0-21swapper/310:35:163
17250992116,3cyclictest2624-21ssh09:32:263
1724999219,4cyclictest22929-21ssh11:24:372
1724999218,4cyclictest4188-21diskmemload11:40:182
17249992113,5cyclictest15052-21pluginstate12:30:222
17249992112,7cyclictest4704-21packagekitd11:35:002
17249992112,7cyclictest28-21ksoftirqd/209:45:002
17249992112,7cyclictest17509-21expr07:10:132
17249992112,6cyclictest52850irq/140-enp3s0-11:25:102
17249992111,6cyclictest52850irq/140-enp3s0-10:35:242
17249992111,2cyclictest4188-21diskmemload12:04:482
1724999211,11cyclictest4188-21diskmemload12:39:462
1724899217,10cyclictest52950irq/141-enp3s0-10:07:211
1724899217,10cyclictest52950irq/141-enp3s0-10:07:201
1724899215,9cyclictest0-21swapper/112:05:001
1724899214,3cyclictest4188-21diskmemload11:21:021
17248992113,6cyclictest710-21sed09:55:221
17248992113,6cyclictest5427-21pluginstate07:55:201
17248992113,6cyclictest32559-21find09:00:111
17248992113,6cyclictest29757-21seq08:52:341
17248992113,6cyclictest19394-21seq07:15:021
17248992112,7cyclictest24743-21apt-get09:25:121
17248992112,7cyclictest2041-21seq07:49:341
17248992111,7cyclictest6458-21sed09:35:181
17248992111,6cyclictest52950irq/141-enp3s0-09:40:101
1724799216,6cyclictest12-21ksoftirqd/007:46:110
17247992118,2cyclictest131rcuc/008:29:560
17247992112,6cyclictest25017-21sed08:40:230
1724799210,8cyclictest1949-21taskset09:03:510
249262200,2sleep20-21swapper/211:01:092
1725099208,2cyclictest34-21ksoftirqd/310:30:073
1725099208,10cyclictest53050irq/142-enp3s0-12:31:513
1725099207,2cyclictest34-21ksoftirqd/312:23:143
1725099207,11cyclictest53050irq/142-enp3s0-11:30:263
1725099207,11cyclictest53050irq/142-enp3s0-11:30:253
1725099204,2cyclictest141rcu_preempt09:49:383
1725099203,2cyclictest141rcu_preempt10:22:493
1725099202,2cyclictest141rcu_preempt07:25:013
17250992012,6cyclictest30387-21seq08:55:123
17250992012,5cyclictest31878-21unixbench-2d07:40:243
1725099201,10cyclictest0-21swapper/312:25:423
17250992010,7cyclictest25092-21cat08:40:233
1725099200,7cyclictest3275-21/usr/sbin/munin10:45:193
1724999209,7cyclictest52850irq/140-enp3s0-10:55:102
1724999209,7cyclictest31603-21pmu-power11:30:212
1724999209,7cyclictest31603-21pmu-power11:30:202
1724999205,3cyclictest25338-21gltestperf09:25:152
17249992014,4cyclictest32098-21idleruntime-cro09:00:012
1724999201,2cyclictest141rcu_preempt08:40:112
17249992012,6cyclictest30780-21cat07:40:132
17249992012,6cyclictest29592-21forks11:05:152
17249992012,6cyclictest28565-21forks07:35:152
17249992012,6cyclictest26273-21df08:45:142
17249992012,6cyclictest22897-21cat07:20:252
17249992012,6cyclictest20536-21which12:10:222
17249992011,7cyclictest25994-21cut07:30:012
1724999201,10cyclictest4188-21diskmemload12:17:262
1724999201,10cyclictest4188-21diskmemload10:06:472
1724999201,10cyclictest4188-21diskmemload10:06:462
1724999201,10cyclictest4188-21diskmemload09:48:492
1724899209,8cyclictest30825-21df11:30:141
1724899209,8cyclictest30825-21df11:30:131
1724899207,9cyclictest52950irq/141-enp3s0-11:27:541
1724899207,9cyclictest52950irq/141-enp3s0-10:57:411
1724899207,9cyclictest52950irq/141-enp3s0-10:45:201
1724899207,9cyclictest52950irq/141-enp3s0-09:51:141
1724899206,10cyclictest52950irq/141-enp3s0-11:46:261
1724899204,3cyclictest4188-21diskmemload12:00:391
1724899202,13cyclictest7087-21apt-get12:25:041
1724899201,6cyclictest27890-21latency_hist08:50:001
17248992013,4cyclictest4224-21turbostat.cron07:55:001
17248992013,4cyclictest17172-21sh09:19:301
17248992013,4cyclictest11055-21sh10:27:101
17248992012,6cyclictest8360-21seq08:03:051
17248992012,6cyclictest32239-21seq07:43:311
17248992012,6cyclictest18850-21seq07:11:011
17248992011,6cyclictest278-21systemd-journal09:09:591
17248992011,6cyclictest21944-21ls08:35:141
1724799205,6cyclictest12-21ksoftirqd/007:51:250
1724799204,6cyclictest12-21ksoftirqd/007:33:190
1725099197,8cyclictest19500-21apt-get08:30:123
1725099192,12cyclictest23938-21apt-get07:25:103
1725099191,6cyclictest0-21swapper/311:00:463
1725099191,4cyclictest0-21swapper/311:10:243
17250991911,5cyclictest16657-21grep08:20:273
1724999195,5cyclictest0-21swapper/212:25:112
1724999192,15cyclictest10651-21cron08:10:002
1724999191,9cyclictest4188-21diskmemload10:48:112
1724999191,9cyclictest4188-21diskmemload10:33:442
17249991914,3cyclictest19476-21acpi08:30:112
17249991913,4cyclictest10699-21sh10:50:262
17249991913,4cyclictest10699-21sh10:50:252
17249991912,5cyclictest659-21fw_packets07:45:152
17249991912,5cyclictest18514-21sensors_temp08:25:242
17249991911,6cyclictest5868-21cat10:00:102
17249991911,6cyclictest17757-21date09:20:002
17249991910,6cyclictest4782-21cat07:55:132
17249991910,6cyclictest22103-21fw_packets08:35:152
17249991910,5cyclictest52850irq/140-enp3s0-12:20:132
1724899193,3cyclictest4188-21diskmemload10:36:061
1724899193,3cyclictest141rcu_preempt07:25:121
1724899193,14cyclictest18966-21cat09:20:181
1724899193,14cyclictest18966-21cat09:20:181
1724899192,14cyclictest19510-21grep08:30:121
17248991911,6cyclictest28887-21sed07:35:171
17248991911,6cyclictest22652-21cat11:00:131
17248991911,6cyclictest21666-21cat07:20:141
17248991911,6cyclictest14790-21seq08:18:141
17248991911,6cyclictest12071-21basename11:40:151
17248991910,6cyclictest26270-21expr07:30:121
1725099188,8cyclictest560-21dbus-daemon08:00:013
1725099182,11cyclictest28221-21apt-get07:35:123
1725099182,11cyclictest13053-21apt-get08:15:123
17250991813,3cyclictest12292-21unixbench-2d08:10:243
1725099181,2cyclictest141rcu_preempt07:50:133
17250991811,5cyclictest19336-21sed08:30:003
17250991811,5cyclictest10268-21awk08:05:263
1724999181,6cyclictest4188-21diskmemload11:15:302
17249991812,4cyclictest4460-21sh11:10:182
17249991810,6cyclictest7392-21memory08:00:182
17249991810,5cyclictest26670-21idleruntime07:30:162
1724899189,7cyclictest16479-21cat08:20:251
1724899189,6cyclictest30411-21expr08:55:121
1724899187,4cyclictest18950irq/143-ahci[0008:40:101
1724899182,11cyclictest4828-21apt-get11:35:111
17248991811,5cyclictest9772-21awk08:05:221
1725099179,6cyclictest907-21ls09:00:193
17250991711,3cyclictest34-21ksoftirqd/307:30:203
17250991710,5cyclictest19795-21cut07:15:163
1724999171,9cyclictest11065-21tr08:10:132
17249991713,2cyclictest13027-21/usr/sbin/munin08:15:172
17249991712,3cyclictest32556-21/usr/sbin/munin09:00:222
17249991712,3cyclictest28637-21sh08:50:172
17249991710,5cyclictest2492-21cat09:05:132
17249991710,5cyclictest19655-21expr07:15:142
1724899179,6cyclictest11028-21grep08:10:141
1725099161,3cyclictest141rcu_preempt09:05:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional