You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-05 - 08:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jan 05, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
124872204168,24sleep20-21swapper/219:06:282
127162201165,24sleep10-21swapper/119:09:241
125142201164,25sleep00-21swapper/019:06:450
124882201165,24sleep30-21swapper/319:06:293
929721450,3sleep01284999cyclictest21:25:180
174942480,2sleep011-21rcuc/019:15:290
1285999402,21cyclictest775-21dbus-daemon19:10:272
1285999392,26cyclictest23662-21sed19:30:242
1285999391,28cyclictest30988-21/usr/sbin/munin23:30:122
1285999381,20cyclictest19140-21sh21:50:012
12849993812,13cyclictest13182-21munin-run22:50:020
1285999361,20cyclictest16727-21pmu-power22:55:202
1285999351,5cyclictest22568-21cut21:55:212
1285999351,23cyclictest16944-21latency_hist21:45:012
1285999351,19cyclictest31442-21ntp_states22:15:222
1284999359,13cyclictest2193-21munin-run22:25:010
12849993510,12cyclictest16346-21basename20:30:010
1285999341,3cyclictest12050irq/123-ahci[0022:10:012
1285999341,3cyclictest12050irq/123-ahci[0022:10:012
1284999346,13cyclictest18225-21munin-run00:15:000
1285999331,6cyclictest23986-21awk00:25:242
1285999331,5cyclictest747-21/usr/sbin/munin23:35:242
1285999331,4cyclictest0-21swapper/223:15:132
1285999322,4cyclictest23468-21smart_sda23:10:292
1285999321,6cyclictest32660-21idleruntime21:05:222
1285999321,17cyclictest14039-21/usr/sbin/munin00:05:132
1284999329,11cyclictest24188-21awk23:15:010
1285999319,19cyclictest31749-21sort21:05:012
1285999315,12cyclictest88650irq/133-enp2s0-23:20:172
1285999312,5cyclictest15342-21diskstats19:15:032
1285999312,4cyclictest20014-21smart_sda20:35:232
1285999312,4cyclictest18859-21ntp_states23:00:222
1285999312,25cyclictest20632-21smart_sda21:50:262
1285999311,7cyclictest14653-21cstates20:25:122
1285999311,4cyclictest8876-21sed23:50:272
1285999311,4cyclictest13399-21/usr/sbin/munin22:50:122
1285999311,4cyclictest0-21swapper/222:10:162
1285999311,20cyclictest801-21snmpd22:35:142
1285999311,16cyclictest8154-21latency_hist21:25:022
1285599318,2cyclictest21384-21cat21:55:011
1285999308,19cyclictest6555-21tr20:05:202
1285999305,22cyclictest5870-21grep23:45:162
1285999305,13cyclictest12050irq/123-ahci[0023:45:022
1285999302,6cyclictest27543-21/usr/sbin/munin20:55:242
1285999302,6cyclictest27543-21/usr/sbin/munin20:55:242
1285999302,3cyclictest20004-21/usr/sbin/munin23:05:142
1285999301,4cyclictest0-21swapper/222:40:232
1285999301,3cyclictest29510-21basename23:25:192
1285999301,2cyclictest0-21swapper/222:05:022
1285999301,26cyclictest24143-21cut20:45:182
12859993011,16cyclictest9048-21grep20:10:272
1285999296,20cyclictest1305-21cpuspeed_turbos19:55:142
1285999296,19cyclictest24824-21cpuspeed_turbos19:35:132
1285999295,22cyclictest13433-21vmstat00:00:312
1285999295,20cyclictest3004-21grep22:25:172
1285999291,6cyclictest3361-21/usr/sbin/munin20:00:152
1285999291,5cyclictest3991-21/usr/sbin/munin21:15:142
1285999291,4cyclictest8729-21diskstats21:25:142
1285999291,4cyclictest22308-21/usr/sbin/munin19:30:012
1285999291,3cyclictest31326-21/usr/sbin/munin19:50:342
1285999291,2cyclictest11643-21ntp_states21:30:222
12859992912,15cyclictest25036-21/usr/sbin/munin00:30:212
12859992912,15cyclictest25036-21/usr/sbin/munin00:30:212
1285999291,15cyclictest20499-21sed00:20:012
12859992911,15cyclictest26201-21sort20:50:182
1285999290,4cyclictest0-21swapper/220:40:132
1285999290,2cyclictest0-21swapper/220:19:412
1285999290,13cyclictest0-21swapper/220:20:142
165862280,3sleep10-21swapper/122:55:181
1285999286,19cyclictest14298-21grep21:35:282
1285999282,4cyclictest27462-21uniq19:40:182
1285999282,3cyclictest457-21diskstats22:20:152
1285999281,4cyclictest904-21sssd_nss22:30:232
1285999281,4cyclictest17854-21sshd00:10:302
1285999281,4cyclictest10531-21memory23:55:202
1285999281,3cyclictest1876-21/usr/sbin/munin21:10:132
1284999280,13cyclictest22933-21awk20:45:010
272992270,2sleep1221ktimersoftd/100:35:141
1285999278,16cyclictest29874-21if_enp2s019:45:152
1285999278,16cyclictest29874-21if_enp2s019:45:152
1285999271,4cyclictest17372-21chrt20:30:182
1285999271,4cyclictest11259-21/usr/sbin/munin22:45:122
1285999271,3cyclictest904-21sssd_nss00:35:172
1285999271,3cyclictest21318-21hddtemp_smartct00:20:182
1285999271,3cyclictest18952-21irqstats19:20:212
1286699269,6cyclictest15682-21cpuspeed_turbos22:55:113
12849992611,4cyclictest0-21swapper/023:24:300
1286699258,3cyclictest27352-21grep22:05:253
1286699258,3cyclictest27352-21grep22:05:253
1285599259,13cyclictest22189-21unixbench_singl00:20:281
1285599251,17cyclictest0-21swapper/119:41:451
1284999250,12cyclictest12286-21kworker/0:221:35:010
228022230,3sleep31286699cyclictest23:10:203
12866992317,3cyclictest3879-21grep20:00:173
1285599239,4cyclictest324742grep19:50:271
1285599236,3cyclictest13206-21fschecks_count21:35:161
1284999234,16cyclictest25752-21munin-run22:05:000
1286699228,3cyclictest493-21jbd2/dm-0-822:39:363
1286699226,4cyclictest0-21swapper/321:25:233
1286699225,14cyclictest0-21swapper/320:55:203
1286699225,14cyclictest0-21swapper/320:55:193
1286699221,19cyclictest0-21swapper/300:30:273
1286699221,19cyclictest0-21swapper/300:30:273
1286699221,13cyclictest0-21swapper/323:35:253
1285599228,12cyclictest30312-21processes19:45:211
1285599228,12cyclictest30312-21processes19:45:211
1285599227,3cyclictest7484-21cpuspeed_turbos23:50:121
1285599221,3cyclictest31955-21/usr/sbin/munin21:05:231
1285599221,3cyclictest1724-21memory23:35:201
1285599221,18cyclictest0-21swapper/122:21:581
1285599221,12cyclictest6534-21diskstats21:20:161
98252210,4sleep121-21rcuc/123:55:131
286932210,4sleep339-21rcuc/323:25:013
1286699212,11cyclictest6186-21/usr/sbin/munin21:20:123
1286699212,11cyclictest1790-21/usr/sbin/munin21:10:173
1286699211,6cyclictest0-21swapper/321:53:283
1286699211,13cyclictest0-21swapper/322:33:583
1286699211,11cyclictest29741-21/usr/sbin/munin21:00:193
1286699211,11cyclictest19729-21users19:20:293
1285599219,4cyclictest0-21swapper/119:15:211
1285599217,12cyclictest23-21ksoftirqd/121:46:351
1285599216,3cyclictest15154-21cut00:05:221
1285599214,11cyclictest9373-40dnf20:10:321
1285599211,8cyclictest4592-21gltestperf21:15:161
1285599211,6cyclictest0-21swapper/123:11:171
1285599211,18cyclictest4616-21sensors_temp20:00:241
1285599211,18cyclictest0-21swapper/123:40:181
1285599211,12cyclictest20685-21diskstats19:25:171
12855992110,3cyclictest24685-21memory22:00:191
12855992110,3cyclictest10914-21df21:30:141
1284999216,6cyclictest0-21swapper/021:13:480
1284999213,16cyclictest6579-21munin-run22:35:020
1286699206,11cyclictest20950-21/usr/sbin/munin20:40:173
1286699205,9cyclictest0-21swapper/320:20:183
1286699201,9cyclictest0-21swapper/319:33:053
1286699201,8cyclictest2180-21sh22:25:013
1286699201,6cyclictest0-21swapper/323:58:333
1286699201,6cyclictest0-21swapper/322:11:113
12866992015,3cyclictest21990-21cut21:55:163
1286699201,12cyclictest0-21swapper/300:00:293
1286699201,11cyclictest29123-21/usr/sbin/munin19:45:223
1286699201,11cyclictest29123-21/usr/sbin/munin19:45:223
1286699201,11cyclictest14977-21/usr/sbin/munin21:40:183
1286699201,11cyclictest13809-21if_enp2s019:10:173
1285599209,3cyclictest16047-21ntp_states21:40:221
1285599206,8cyclictest87150irq/128-enp1s0-00:25:191
1285599205,4cyclictest0-21swapper/123:04:081
1285599203,9cyclictest87150irq/128-enp1s0-21:55:181
1285599202,10cyclictest12579-21if_enp2s000:00:191
12855992016,3cyclictest0-21swapper/120:21:561
1285599201,18cyclictest0-21swapper/122:39:111
1285599201,17cyclictest0-21swapper/123:25:071
1285599201,17cyclictest0-21swapper/120:50:201
1285599201,14cyclictest0-21swapper/121:28:461
1285599201,14cyclictest0-21swapper/119:10:571
1285599201,11cyclictest31335-21diskstats23:30:131
1285599201,11cyclictest24463-21egrep23:15:121
1285599201,11cyclictest23492-21diskstats20:45:131
1285599201,11cyclictest18523-21fschecks_count19:20:161
1285599201,11cyclictest13746-21diskstats22:50:131
1285599201,11cyclictest0-21swapper/123:05:251
1285599201,10cyclictest9979-21acpi20:15:141
1284999209,3cyclictest22531-21/usr/sbin/munin19:30:150
1284999208,3cyclictest0-21swapper/022:08:410
1284999208,3cyclictest0-21swapper/022:08:400
1284999205,6cyclictest0-21swapper/000:17:510
1284999201,4cyclictest0-21swapper/023:05:080
1284999201,11cyclictest21454-21irqstats00:20:190
1284999201,11cyclictest0-21swapper/021:45:190
43212191,3sleep30-21swapper/322:29:243
15862191,3sleep00-21swapper/023:35:180
1286699198,3cyclictest23146-21/usr/sbin/munin20:45:213
1286699198,3cyclictest17176-21/usr/sbin/munin21:45:203
1286699195,4cyclictest13399-21/usr/sbin/munin22:50:163
1286699194,13cyclictest39-21rcuc/300:25:203
1286699191,9cyclictest0-21swapper/322:15:173
1286699191,6cyclictest3379-21diskstats23:40:163
1286699191,6cyclictest0-21swapper/322:00:203
1286699191,3cyclictest11558-21users20:15:313
12866991912,4cyclictest5254-21smartctl21:15:243
1286699191,15cyclictest10592-21users22:40:303
1286699191,10cyclictest9337-21sshd23:54:313
1286699191,10cyclictest25542-21irqstats19:35:193
1286699191,10cyclictest16279-21sshd20:29:313
1286699190,6cyclictest39-21rcuc/300:15:273
1285599198,3cyclictest2762-21diskstats22:25:151
1285599194,11cyclictest87050irq/127-enp1s0-20:30:181
1285599192,9cyclictest87250irq/129-enp1s0-20:06:301
1285599191,8cyclictest0-21swapper/122:42:071
1285599191,7cyclictest0-21swapper/123:21:401
12855991916,2cyclictest0-21swapper/119:59:161
12855991915,3cyclictest0-21swapper/120:28:261
1285599191,2cyclictest0-21swapper/119:35:291
1285599191,17cyclictest0-21swapper/122:46:241
1285599191,17cyclictest0-21swapper/122:10:171
1285599191,15cyclictest21993-21ntpq20:40:211
1285599191,14cyclictest0-21swapper/123:49:271
1285599191,14cyclictest0-21swapper/100:12:311
1285599191,13cyclictest0-21swapper/122:05:061
1285599191,13cyclictest0-21swapper/122:05:061
1285599191,10cyclictest5997-21cat22:30:251
1285599191,10cyclictest30373-21/usr/sbin/munin22:15:171
1285599191,10cyclictest27002-21latency_hist00:35:011
1285599191,10cyclictest27002-21latency_hist00:35:011
1285599190,18cyclictest0-21swapper/121:11:141
1285599190,14cyclictest0-21swapper/119:30:271
1285599190,14cyclictest0-21swapper/100:18:291
1284999198,3cyclictest7688-21ls22:35:200
1284999197,7cyclictest86950irq/126-enp1s0-21:50:320
1284999192,9cyclictest31901-21memory23:30:170
1284999192,14cyclictest5455-21grep21:15:260
1284999192,10cyclictest31955-21/usr/sbin/munin21:05:290
1284999191,4cyclictest27956-21/usr/sbin/munin00:35:200
1284999191,16cyclictest16020-21ls00:10:000
1284999191,12cyclictest0-21swapper/000:27:080
1284999191,11cyclictest0-21swapper/023:51:140
1284999191,10cyclictest18754-21/usr/sbin/munin20:35:190
1284999191,10cyclictest16848-21sendmail_mailtr22:55:220
1284999191,10cyclictest10593-21users22:40:300
1284999191,10cyclictest0-21swapper/021:23:320
1284999190,6cyclictest0-21swapper/023:45:280
196162181,2sleep30-21swapper/323:03:063
1286699184,5cyclictest5568-21/usr/sbin/munin20:05:183
1286699181,6cyclictest30960-21cat23:30:013
1286699181,4cyclictest0-21swapper/300:13:503
1286699181,11cyclictest0-21swapper/323:32:153
1286699181,10cyclictest11216-21cpu22:45:133
1285599186,3cyclictest28613-21ntp_states20:55:231
1285599186,3cyclictest28613-21ntp_states20:55:231
1285599186,2cyclictest0-21swapper/120:35:411
1284999187,3cyclictest801-21snmpd19:49:350
1284999187,3cyclictest801-21snmpd19:49:350
12849991813,3cyclictest5568-21/usr/sbin/munin20:05:170
1284999181,2cyclictest0-21swapper/023:40:160
12849991812,4cyclictest14390-21needreboot22:50:190
1284999181,11cyclictest0-21swapper/023:15:330
1284999180,6cyclictest0-21swapper/023:26:480
1284999180,6cyclictest0-21swapper/022:25:200
287702171,2sleep00-21swapper/019:43:300
1286699176,3cyclictest86950irq/126-enp1s0-00:05:173
1286699171,9cyclictest0-21swapper/320:50:173
1286699171,6cyclictest0-21swapper/323:45:123
1286699171,6cyclictest0-21swapper/320:31:143
1286699171,6cyclictest0-21swapper/319:27:353
1286699171,4cyclictest0-21swapper/323:17:153
1286699171,3cyclictest0-21swapper/323:05:213
1286699171,2cyclictest462-21/usr/sbin/munin19:50:333
12866991712,3cyclictest0-21swapper/320:10:113
1286699171,15cyclictest0-21swapper/300:39:123
12866991711,3cyclictest28186-21sensors_temp19:40:263
1286699170,14cyclictest12923-21grep21:35:143
12855991712,3cyclictest31287-21grep21:00:301
1284999175,4cyclictest0-21swapper/000:02:040
1284999173,6cyclictest0-21swapper/021:38:310
1284999173,11cyclictest18301-21cpuspeed_turbos19:20:140
1284999171,7cyclictest27543-21/usr/sbin/munin20:55:270
1284999171,7cyclictest27543-21/usr/sbin/munin20:55:270
1284999171,7cyclictest0-21swapper/019:25:200
1284999171,6cyclictest10361-21fschecks_count20:15:180
1284999171,6cyclictest0-21swapper/023:04:150
1284999171,3cyclictest96112sleep023:55:080
1284999171,10cyclictest0-21swapper/021:56:380
1284999170,5cyclictest0-21swapper/022:10:460
1284999170,4cyclictest0-21swapper/022:18:050
30752160,2sleep30-21swapper/319:59:273
1286699165,4cyclictest0-21swapper/321:05:183
1286699165,3cyclictest87250irq/129-enp1s0-19:15:133
1286699161,6cyclictest10579-21/usr/sbin/munin21:30:233
1286699161,6cyclictest0-21swapper/300:23:573
1284999161,9cyclictest0-21swapper/020:00:220
1284999161,9cyclictest0-21swapper/019:50:180
1284999161,9cyclictest0-21swapper/000:30:500
1284999161,9cyclictest0-21swapper/000:30:500
1284999161,6cyclictest16655-21grep20:30:140
12849991612,3cyclictest0-21swapper/021:00:090
1284999161,14cyclictest0-21swapper/019:56:510
12849991611,3cyclictest25379-21cut19:35:160
12849991611,3cyclictest13256-21expr19:10:120
1284999161,13cyclictest0-21swapper/020:45:120
1284999161,10cyclictest0-21swapper/021:42:020
12849991610,3cyclictest12386-21ls20:20:150
1286699151,5cyclictest0-21swapper/320:35:113
1284999151,5cyclictest25422-21grep20:50:130
1284999150,6cyclictest0-21swapper/020:10:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional