You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-27 - 10:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Apr 27, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18102239166,25sleep20-21swapper/219:06:062
18552204153,38sleep30-21swapper/319:06:383
18632202165,25sleep00-21swapper/019:06:430
18532200163,24sleep10-21swapper/119:06:361
2274721800,5sleep0219299cyclictest22:20:180
944321710,4sleep2220699cyclictest21:50:192
2444721680,3sleep1220099cyclictest22:25:131
1680521590,4sleep3221399cyclictest23:20:213
4472600,2sleep2220699cyclictest20:15:232
122262520,4sleep230-21rcuc/223:10:172
213832510,3sleep230-21rcuc/221:05:122
112592440,2sleep121-21rcuc/119:25:271
2213994118,20cyclictest18023-21grep20:55:243
220699411,31cyclictest26870-21processes21:15:232
221399382,14cyclictest3715-21users21:35:273
2192993813,12cyclictest21069-21munin-run21:05:010
220699372,25cyclictest32058-21munin-run21:30:022
220699361,23cyclictest7925-21cpuspeed_turbos00:15:142
2213993514,3cyclictest9729-21/usr/sbin/munin21:50:233
220699351,25cyclictest12991-21irqstats00:25:192
220699351,22cyclictest0-21swapper/222:55:002
219299357,12cyclictest919-21awk00:00:000
220699344,23cyclictest87350irq/126-enp1s0-00:20:142
220699344,23cyclictest87350irq/126-enp1s0-00:20:142
220699342,28cyclictest3897-40dnf21:37:362
220699342,27cyclictest15766-40dnf19:36:352
220699341,21cyclictest4103-21tune2fs20:25:162
221399330,10cyclictest0-21swapper/300:25:213
220699332,26cyclictest30333-40dnf22:38:232
220699332,18cyclictest0-21swapper/222:45:222
220699331,8cyclictest22728-21cat19:55:022
220699331,29cyclictest13015-21pmu-power19:30:232
220699331,29cyclictest10076-21turbostat.cron20:40:002
220699331,22cyclictest16573-21/usr/sbin/munin00:35:262
221399321,5cyclictest24729-21/usr/sbin/munin23:40:133
220699321,19cyclictest6524-21/usr/sbin/munin21:45:142
220699321,17cyclictest21768-21cut22:20:012
219299324,13cyclictest13529-21awk23:15:010
2213993112,4cyclictest19280-21irqstats19:45:163
221399311,10cyclictest0-21swapper/323:46:583
220699313,24cyclictest19729-21smart_sda19:45:232
220699312,17cyclictest29473-21/usr/sbin/munin20:10:172
220699311,6cyclictest30701-21/usr/sbin/munin22:40:242
220699311,20cyclictest1550-21chrt21:30:572
2206993111,17cyclictest14693-21/usr/sbin/munin20:50:272
2206993111,17cyclictest14693-21/usr/sbin/munin20:50:272
219299312,14cyclictest15097-21munin-run22:05:010
219299312,14cyclictest15097-21munin-run22:05:010
221399301,14cyclictest0-21swapper/322:53:323
220699303,3cyclictest87650irq/129-enp1s0-22:30:192
220699303,14cyclictest87650irq/129-enp1s0-20:05:202
220699301,4cyclictest25545-21/usr/sbin/munin23:40:192
220699301,3cyclictest18841-21/usr/sbin/munin23:25:192
220699301,18cyclictest3615-21expr00:05:122
220699301,14cyclictest0-21swapper/222:05:012
220699301,14cyclictest0-21swapper/222:05:012
2206993010,17cyclictest16610-21gltestperf23:20:202
221399297,15cyclictest0-21swapper/300:19:053
221399291,14cyclictest0-21swapper/322:39:243
220699298,18cyclictest3487-21open_files19:10:212
220699297,19cyclictest827-21runrttasks23:50:562
220699295,9cyclictest87350irq/126-enp1s0-20:20:272
220699294,16cyclictest755-21rngd20:55:202
220699292,3cyclictest12050irq/123-ahci[0022:00:012
220699291,5cyclictest0-21swapper/222:25:242
220699291,4cyclictest446-21/usr/sbin/munin23:55:262
220699291,3cyclictest24376-40dnf23:38:352
220699291,3cyclictest21108-21/usr/sbin/munin23:30:172
220699291,3cyclictest19088-21/usr/sbin/munin21:00:172
2206992913,13cyclictest7799-21sed23:00:152
2206992910,16cyclictest6913-21systemd23:00:002
219299292,12cyclictest32682-21awk22:45:010
2213992817,4cyclictest0-21swapper/320:03:253
221399281,13cyclictest0-21swapper/320:19:213
221399281,13cyclictest0-21swapper/319:43:083
2213992810,10cyclictest0-21swapper/322:40:203
220699289,16cyclictest14582-21expr00:30:132
220699287,18cyclictest24677-21processes21:10:212
220699282,4cyclictest13755-21smart_sda20:45:232
220699282,2cyclictest87350irq/126-enp1s0-22:05:212
220699281,3cyclictest7836-21/usr/sbin/munin19:20:142
220699281,3cyclictest6384-21/usr/sbin/munin00:10:162
220699281,3cyclictest1922-21/usr/sbin/munin00:00:182
220699281,3cyclictest10889-21/usr/sbin/munin23:05:262
2206992813,13cyclictest13743-21/usr/sbin/munin23:15:182
220699281,2cyclictest0-21swapper/222:25:012
220699281,2cyclictest0-21swapper/222:10:192
220699281,15cyclictest27495-21fschecks_time23:45:162
2206992810,15cyclictest827-21runrttasks19:17:232
220099281,18cyclictest4713-21seq20:25:211
219299289,16cyclictest6920-21munin-run23:00:020
301182270,2sleep00-21swapper/023:50:200
2213992711,3cyclictest15607-21memory20:50:203
2213992711,3cyclictest15607-21memory20:50:203
221399271,12cyclictest0-21swapper/323:03:243
221399271,12cyclictest0-21swapper/322:01:573
221399271,12cyclictest0-21swapper/322:01:573
221399271,11cyclictest0-21swapper/323:55:053
221399271,11cyclictest0-21swapper/321:25:323
221399271,11cyclictest0-21swapper/321:13:363
221399271,11cyclictest0-21swapper/320:30:263
2213992710,9cyclictest28067-21kworker/u8:121:00:043
2213992710,4cyclictest15291-21memory00:30:203
2213992710,3cyclictest10364-21cpuspeed_turbos20:40:123
220699277,17cyclictest10364-21cpuspeed_turbos20:40:122
220699277,16cyclictest25169-21tr20:00:112
220699272,4cyclictest16276-21/usr/sbin/munin19:40:292
220699272,3cyclictest23541-21hddtemp_smartct19:55:172
220699271,3cyclictest6497-21grep20:30:152
220699271,3cyclictest29658-21sshd21:22:342
220699271,2cyclictest4326-21/usr/sbin/munin21:40:222
220099271,17cyclictest0-21swapper/121:59:591
219299271,14cyclictest29260-21basename20:10:010
221399269,3cyclictest0-21swapper/323:29:163
221399269,3cyclictest0-21swapper/322:10:253
221399269,3cyclictest0-21swapper/321:07:403
221399269,3cyclictest0-21swapper/320:37:363
221399269,3cyclictest0-21swapper/320:25:043
221399269,3cyclictest0-21swapper/320:23:413
221399269,3cyclictest0-21swapper/320:12:533
221399268,3cyclictest27456-21cstates20:05:133
221399268,3cyclictest1799-21sensors_temp22:45:243
221399268,3cyclictest0-21swapper/319:28:443
221399268,3cyclictest0-21swapper/300:37:043
2213992616,4cyclictest0-21swapper/320:47:203
221399261,12cyclictest0-21swapper/322:16:213
221399261,11cyclictest0-21swapper/322:20:043
221399261,11cyclictest0-21swapper/321:47:243
2213992610,3cyclictest0-21swapper/319:23:523
220699261,3cyclictest9771-21cpuspeed_turbos19:25:132
220099261,17cyclictest9540-21/usr/sbin/munin21:50:201
220099261,17cyclictest0-21swapper/123:33:451
219299263,11cyclictest4377-21munin-run19:15:000
219299261,7cyclictest0-21swapper/021:37:350
221399259,3cyclictest0-21swapper/323:15:043
221399259,3cyclictest0-21swapper/323:05:523
221399259,3cyclictest0-21swapper/319:16:083
221399258,3cyclictest0-21swapper/319:50:283
221399257,3cyclictest0-21swapper/319:14:093
221399256,3cyclictest0-21swapper/321:31:283
221399251,13cyclictest0-21swapper/300:05:093
221399251,11cyclictest0-21swapper/323:11:443
221399251,11cyclictest0-21swapper/319:55:053
221399251,10cyclictest0-21swapper/321:59:413
2213992510,3cyclictest0-21swapper/321:22:283
220099251,16cyclictest11600-21taskset00:20:381
220099251,16cyclictest11600-21taskset00:20:381
220099251,12cyclictest0-21swapper/120:52:381
220099251,12cyclictest0-21swapper/120:52:381
219299257,16cyclictest0-21swapper/022:40:020
219299251,12cyclictest7879-21munin-run20:35:010
221399249,3cyclictest0-21swapper/322:27:003
221399249,3cyclictest0-21swapper/322:06:093
221399248,3cyclictest0-21swapper/319:37:163
221399248,3cyclictest0-21swapper/300:10:053
221399245,3cyclictest0-21swapper/300:20:253
221399245,3cyclictest0-21swapper/300:20:253
221399241,11cyclictest0-21swapper/321:17:443
221399241,10cyclictest0-21swapper/323:50:203
220099249,2cyclictest15230-21wc00:30:191
220099241,16cyclictest0-21swapper/123:35:571
221399239,3cyclictest0-21swapper/323:36:043
221399239,3cyclictest0-21swapper/321:41:133
221399238,3cyclictest0-21swapper/322:56:003
221399237,3cyclictest0-21swapper/322:30:083
221399231,12cyclictest0-21swapper/323:30:363
221399230,10cyclictest0-21swapper/319:30:043
2200992317,3cyclictest23267-21diskmemload21:21:101
220099231,16cyclictest0-21swapper/123:48:281
220099231,16cyclictest0-21swapper/123:17:061
220099231,16cyclictest0-21swapper/123:07:071
220099231,12cyclictest0-21swapper/122:22:201
220099231,12cyclictest0-21swapper/100:17:031
220099231,10cyclictest0-21swapper/121:40:341
315832220,2sleep10-21swapper/121:25:251
221399227,3cyclictest0-21swapper/300:00:333
220099226,10cyclictest19263-21grep23:25:241
2200992216,3cyclictest2743-21df_inode19:10:131
220099221,16cyclictest0-21swapper/123:58:121
220099221,16cyclictest0-21swapper/123:53:191
220099221,16cyclictest0-21swapper/122:08:451
220099221,16cyclictest0-21swapper/119:48:031
220099221,11cyclictest0-21swapper/122:58:271
220099221,11cyclictest0-21swapper/122:40:401
220099221,11cyclictest0-21swapper/121:31:471
220099221,10cyclictest0-21swapper/100:36:371
219299226,14cyclictest0-21swapper/023:40:010
219299221,7cyclictest0-21swapper/000:20:260
219299221,7cyclictest0-21swapper/000:20:260
51702210,4sleep0219299cyclictest21:40:170
220099219,5cyclictest0-21swapper/121:18:021
220099218,4cyclictest28278-21munin-run22:35:011
220099216,9cyclictest0-21swapper/122:11:051
220099213,11cyclictest20434-21if_enp1s022:15:161
220099211,18cyclictest0-21swapper/119:50:131
220099211,15cyclictest0-21swapper/120:02:281
220099211,15cyclictest0-21swapper/119:55:461
220099211,14cyclictest0-21swapper/120:20:161
220099211,14cyclictest0-21swapper/120:15:121
220099211,11cyclictest8554-21ntp_states19:20:211
220099211,11cyclictest0-21swapper/123:23:411
220099211,10cyclictest29157-21sed22:35:151
220099211,10cyclictest0-21swapper/100:14:341
220099211,10cyclictest0-21swapper/100:07:481
2200992110,3cyclictest14425-21diskstats19:35:141
220099210,15cyclictest0-21swapper/123:10:071
2192992114,4cyclictest19689-21grep23:25:270
219299211,3cyclictest28910-21idleruntime-cro23:50:010
219299211,11cyclictest5264-21unixbench_singl20:25:270
250332200,7sleep1221ktimersoftd/121:10:251
220099209,3cyclictest6171-21cstates20:30:121
220099209,3cyclictest3714-21users21:35:261
2200992013,2cyclictest0-21swapper/120:39:081
2200992012,3cyclictest0-21swapper/120:44:591
220099201,15cyclictest0-21swapper/123:43:141
220099201,14cyclictest0-21swapper/122:50:061
220099201,14cyclictest0-21swapper/121:47:061
220099201,14cyclictest0-21swapper/120:45:351
220099201,14cyclictest0-21swapper/120:09:051
220099201,11cyclictest13458-21df_inode22:00:141
220099201,11cyclictest13458-21df_inode22:00:141
220099201,10cyclictest12228-21df_inode19:30:151
220099200,15cyclictest0-21swapper/123:01:281
220099200,14cyclictest0-21swapper/121:03:341
219299207,3cyclictest808-21snmpd21:19:450
219299202,10cyclictest14693-21/usr/sbin/munin20:50:230
219299202,10cyclictest14693-21/usr/sbin/munin20:50:230
219299201,6cyclictest0-21swapper/022:50:360
219299201,11cyclictest27890-21/usr/sbin/munin21:20:150
219299201,11cyclictest11115-21sshd23:07:330
220099194,4cyclictest0-21swapper/100:01:511
2200991912,3cyclictest0-21swapper/119:43:321
220099191,12cyclictest0-21swapper/122:46:591
220099191,10cyclictest12515-21df00:25:141
220099190,14cyclictest0-21swapper/121:05:281
220099190,14cyclictest0-21swapper/120:12:111
219299191,4cyclictest0-21swapper/021:13:260
219299191,11cyclictest19713-21/usr/sbin/munin22:15:220
219299191,10cyclictest16892-21/usr/sbin/munin20:55:270
219299191,10cyclictest11885-21/usr/sbin/munin19:30:200
219299191,10cyclictest10335-21sshd21:50:330
180712190,2sleep00-21swapper/022:10:150
26232180,4sleep011-21rcuc/020:20:230
2200991814,3cyclictest0-21swapper/119:18:541
220099180,14cyclictest0-21swapper/120:56:061
219299189,2cyclictest31451-21awk20:15:010
219299187,3cyclictest9686-21/usr/sbin/munin19:25:120
219299187,3cyclictest808-21snmpd00:36:160
219299181,9cyclictest14377-21/usr/sbin/munin00:30:240
219299181,4cyclictest24729-21/usr/sbin/munin23:40:160
219299181,3cyclictest9127-21basename23:05:010
219299181,10cyclictest0-21swapper/000:13:430
219299180,4cyclictest0-21swapper/021:46:520
177042181,2sleep00-21swapper/023:22:320
219299171,9cyclictest21293-21/usr/sbin/munin21:05:130
219299171,8cyclictest808-21snmpd21:29:360
219299171,7cyclictest13743-21/usr/sbin/munin23:15:230
219299171,6cyclictest0-21swapper/022:07:190
219299171,6cyclictest0-21swapper/021:56:200
219299171,6cyclictest0-21swapper/020:35:250
219299171,6cyclictest0-21swapper/000:04:380
219299171,5cyclictest0-21swapper/022:27:430
219299171,5cyclictest0-21swapper/000:05:180
219299171,4cyclictest0-21swapper/022:32:320
2192991712,3cyclictest16854-21perl19:40:160
2192991711,3cyclictest9919-21cut00:20:010
2192991711,3cyclictest1168-21smartctl21:30:250
2192991710,4cyclictest26305-21wc20:00:220
143362170,2sleep011-21rcuc/019:35:120
219299168,1cyclictest0-21swapper/000:25:190
219299161,9cyclictest0-21swapper/019:45:330
219299161,7cyclictest23267-21diskmemload23:30:170
219299161,6cyclictest0-21swapper/019:50:250
2192991612,2cyclictest454-21cpu22:45:130
219299161,14cyclictest0-21swapper/019:20:480
2192991611,3cyclictest31670-21/usr/sbin/munin20:15:200
2192991611,3cyclictest23123-21expr19:55:130
2192991610,4cyclictest6103-21irqstats19:15:180
219299159,3cyclictest10923-21gltestperf20:40:160
219299151,11cyclictest0-21swapper/020:48:330
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional