You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 19:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Apr 28, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
36262241165,24sleep10-21swapper/107:06:541
36872239164,23sleep20-21swapper/207:07:432
35252205169,24sleep00-21swapper/007:05:370
35332204166,26sleep30-21swapper/307:05:433
2081121390,4sleep3396899cyclictest09:00:023
1971421390,4sleep1395899cyclictest12:35:261
3963991090,107cyclictest0-21swapper/211:58:412
39639910299,2cyclictest0-21swapper/207:41:302
146502510,2sleep30-21swapper/311:10:243
249312490,1sleep20-21swapper/210:20:232
257652460,2sleep20-21swapper/211:35:282
396399433,34cyclictest12293-40dnf09:53:242
396399392,23cyclictest773-21NetworkManager10:31:422
396399392,23cyclictest773-21NetworkManager10:31:422
396399391,25cyclictest3052-21if_err_enp1s010:45:182
396399381,27cyclictest27190-21/usr/sbin/munin10:25:242
396399371,26cyclictest3254-21/usr/sbin/munin12:00:152
396399371,25cyclictest0-21swapper/208:30:172
396399371,23cyclictest10943-21timerwakeupswit12:15:232
396399371,22cyclictest18077-21perf12:35:012
396399371,20cyclictest22770-21/usr/sbin/munin11:30:172
396399361,25cyclictest9002-21/usr/sbin/munin09:45:162
396399361,25cyclictest6983-21memory09:40:192
396399361,21cyclictest23520-21fschecks_time09:05:162
396399353,27cyclictest30284-21smart_sda08:05:252
396399351,25cyclictest8325-21switchtime07:15:232
396399351,24cyclictest22390-21cat07:50:012
396399351,20cyclictest30729-21/usr/sbin/munin09:20:212
396399341,30cyclictest13986-21sort11:10:162
396399341,24cyclictest2111-21latency_hist10:45:022
396399341,21cyclictest26291-21ntpq09:10:192
396399341,21cyclictest10906-21taskset07:21:162
396399341,19cyclictest18724-21ntp_kernel_pll_11:20:222
396399341,18cyclictest18579-21cat08:55:022
396399341,16cyclictest29413-21munin-run09:20:002
396399341,16cyclictest29413-21munin-run09:20:002
395199342,21cyclictest25500-21kworker/0:010:36:090
49352330,5sleep3396899cyclictest09:35:203
396399332,27cyclictest6349-40dnf10:53:502
396399332,26cyclictest22529-21ntp_states10:15:212
396399331,23cyclictest8880-21/usr/sbin/munin11:00:222
396399331,23cyclictest31217-21/usr/sbin/munin08:10:252
396399331,23cyclictest31217-21/usr/sbin/munin08:10:252
396399331,23cyclictest22814-21latency_hist09:05:012
396399327,16cyclictest87650irq/129-enp1s0-09:39:072
396399321,28cyclictest11439-21/usr/sbin/munin07:25:152
396399321,23cyclictest30866-21/usr/sbin/munin11:50:212
396399321,21cyclictest17579-21cat11:20:012
396399321,19cyclictest0-21swapper/210:00:192
396399321,18cyclictest0-21swapper/210:35:132
396399321,16cyclictest0-21swapper/211:40:262
396399321,16cyclictest0-21swapper/211:40:252
396899310,10cyclictest0-21swapper/308:29:003
396399313,20cyclictest3044-21awk08:20:012
396399313,16cyclictest22305-1kworker/u9:209:55:202
396399312,25cyclictest20330-21ntp_states10:10:232
396399311,27cyclictest13306-21smart_sda08:40:232
396399311,21cyclictest7312-21/usr/sbin/munin12:10:142
396399311,21cyclictest12972-21kworker/u8:211:09:052
396399311,17cyclictest14713-21interrupts12:25:182
396399311,17cyclictest0-21swapper/211:47:342
396399304,20cyclictest87350irq/126-enp1s0-07:55:202
396399302,25cyclictest6717-21smart_sda08:25:252
396399302,25cyclictest3254-21/usr/sbin/munin08:20:272
396399301,26cyclictest2012-21/usr/sbin/munin09:30:142
396399301,26cyclictest18330-21smart_sda10:05:252
396399301,18cyclictest15832-21/usr/sbin/munin07:35:222
396399301,17cyclictest827-21runrttasks12:23:362
396399301,15cyclictest12050irq/123-ahci[0007:15:012
396399291,25cyclictest7896-21/usr/sbin/munin10:55:232
396399291,25cyclictest5118-21/usr/sbin/munin12:05:242
396399291,25cyclictest31832-21/usr/sbin/munin09:25:142
396399291,25cyclictest18300-21/usr/sbin/munin12:35:222
396399291,25cyclictest14243-21/usr/sbin/munin08:45:262
396399291,25cyclictest13636-21/usr/sbin/munin07:30:222
396399290,26cyclictest26816-21/usr/sbin/munin08:00:212
396399281,24cyclictest9850-21/usr/sbin/munin08:35:212
396399281,23cyclictest18976-21df08:55:142
3963992810,15cyclictest19878-21/usr/sbin/munin11:25:202
396399280,25cyclictest22423-21/usr/sbin/munin07:50:262
3951992818,3cyclictest0-21swapper/010:32:460
3951992818,3cyclictest0-21swapper/010:32:460
18122280,4sleep123-21ksoftirqd/108:15:171
3951992713,8cyclictest0-21swapper/011:09:260
396899269,8cyclictest3368-21NetworkManager09:31:413
396899261,22cyclictest24619-21/usr/sbin/munin07:55:233
396899260,8cyclictest0-21swapper/310:59:083
395899264,16cyclictest12293-40dnf09:53:261
3958992612,11cyclictest18353-21expr07:40:131
3951992614,3cyclictest0-21swapper/012:25:460
395199261,18cyclictest0-21swapper/011:26:250
395199261,17cyclictest0-21swapper/011:54:260
395199261,17cyclictest0-21swapper/011:16:170
395199261,17cyclictest0-21swapper/009:47:510
395199261,17cyclictest0-21swapper/009:35:060
395199261,12cyclictest4266-21munin-run10:50:010
396899251,21cyclictest27442-21/usr/sbin/munin09:15:133
396899251,21cyclictest27442-21/usr/sbin/munin09:15:133
396899251,21cyclictest15832-21/usr/sbin/munin07:35:233
396899251,21cyclictest14860-21/usr/sbin/munin10:00:203
395899251,22cyclictest0-21swapper/107:38:521
395199252,11cyclictest1133-21kworker/0:110:44:570
395199251,8cyclictest23249-21kworker/0:110:21:330
395199251,18cyclictest0-21swapper/010:10:210
395199251,17cyclictest0-21swapper/012:23:290
395199251,16cyclictest29105-21kworker/u8:009:21:350
395199250,17cyclictest0-21swapper/011:59:220
396899242,11cyclictest11707-21/usr/sbin/munin12:20:163
396899241,20cyclictest1032-21/usr/sbin/munin08:15:223
396899241,11cyclictest28668-21/usr/sbin/munin11:45:153
395899244,5cyclictest773-21NetworkManager09:11:411
395899242,11cyclictest6162-21munin-node07:15:021
3958992410,4cyclictest23436-21munin-run10:20:011
3958992410,12cyclictest26292-21kworker/u8:209:57:201
3951992413,3cyclictest0-21swapper/012:07:000
3951992412,7cyclictest0-21swapper/010:56:500
3951992412,3cyclictest0-21swapper/011:01:560
3951992412,2cyclictest6349-40dnf10:53:480
395199241,16cyclictest0-21swapper/009:54:040
395199241,15cyclictest0-21swapper/012:15:540
395199241,15cyclictest0-21swapper/010:06:040
395199241,11cyclictest6068-21/usr/sbin/munin09:40:230
395199240,16cyclictest0-21swapper/009:10:310
396899234,6cyclictest32229-21idleruntime-cro10:40:003
396899231,3cyclictest0-21swapper/307:25:273
396899231,19cyclictest28050-21/usr/sbin/munin10:30:163
396899231,19cyclictest28050-21/usr/sbin/munin10:30:163
396899231,11cyclictest23655-21/usr/sbin/munin10:20:183
396899231,11cyclictest0-21swapper/311:35:523
396899231,10cyclictest8880-21/usr/sbin/munin11:00:153
396899231,10cyclictest19266-21/usr/sbin/munin10:10:223
395899231,16cyclictest0-21swapper/107:31:581
395899231,11cyclictest0-21swapper/108:55:071
395199232,7cyclictest1066-21kworker/0:209:33:320
3951992312,3cyclictest0-21swapper/010:15:170
3951992312,3cyclictest0-21swapper/009:15:300
3951992312,3cyclictest0-21swapper/009:15:300
395199231,10cyclictest12695-21kworker/0:211:11:250
395199230,14cyclictest0-21swapper/011:20:280
396899228,3cyclictest22423-21/usr/sbin/munin07:50:253
396899226,14cyclictest87350irq/126-enp1s0-11:30:163
396899222,11cyclictest839-21lldpd09:06:543
396899221,9cyclictest0-21swapper/312:01:353
396899221,7cyclictest6127-21perf07:15:003
396899221,10cyclictest18139-21NetworkManager08:51:413
396899220,10cyclictest0-21swapper/312:36:553
395899223,6cyclictest21-21rcuc/109:00:261
395899222,11cyclictest18934-21/usr/sbin/munin11:20:241
395899221,9cyclictest0-21swapper/109:34:461
395899221,18cyclictest9508-21/usr/sbin/munin12:15:191
395899221,18cyclictest909-21sssd_nss12:10:151
395899221,16cyclictest0-21swapper/109:07:101
395899221,16cyclictest0-21swapper/107:49:021
395899221,15cyclictest0-21swapper/109:27:111
395199222,13cyclictest25015-21diskmemload12:31:550
3951992215,4cyclictest14229-21grep07:30:170
3951992213,4cyclictest0-21swapper/011:32:440
3951992212,3cyclictest0-21swapper/011:45:170
395199221,14cyclictest0-21swapper/012:37:380
395199221,14cyclictest0-21swapper/009:29:420
395199220,11cyclictest7-21ksoftirqd/007:20:010
3968992113,5cyclictest7204-21tr07:15:113
396899211,13cyclictest0-21swapper/309:40:173
396899211,10cyclictest839-21lldpd07:41:223
396899211,10cyclictest714-21sshd08:14:483
396899211,10cyclictest714-21sshd08:14:483
396899210,9cyclictest0-21swapper/309:26:403
395899217,12cyclictest23-21ksoftirqd/109:15:081
395899217,12cyclictest23-21ksoftirqd/109:15:081
395899211,9cyclictest0-21swapper/110:56:111
395899211,5cyclictest28439-21latency_hist11:45:011
395899211,5cyclictest28439-21latency_hist11:45:011
3958992113,3cyclictest0-21swapper/108:49:431
395899211,18cyclictest0-21swapper/110:25:321
395899211,15cyclictest0-21swapper/108:51:151
395899211,15cyclictest0-21swapper/108:11:281
395899211,15cyclictest0-21swapper/108:11:281
395899211,14cyclictest0-21swapper/108:36:481
395899211,14cyclictest0-21swapper/108:00:171
395899211,13cyclictest0-21swapper/110:44:211
395899211,13cyclictest0-21swapper/110:10:571
395899211,11cyclictest29012-21df_inode11:45:131
395899211,11cyclictest15775-21memory10:00:201
395899211,11cyclictest13404-21latency_hist07:30:011
395199218,6cyclictest0-21swapper/010:28:280
3951992116,3cyclictest13495-21cut09:55:170
3951992115,4cyclictest8416-21/usr/sbin/munin12:10:210
395199211,14cyclictest0-21swapper/011:43:390
395199211,14cyclictest0-21swapper/011:43:390
395199211,13cyclictest87350irq/126-enp1s0-08:05:260
395199210,13cyclictest30415-21kworker/u8:012:04:420
156542211,3sleep015759-21sed10:00:190
396899206,3cyclictest12050irq/123-ahci[0010:50:023
396899204,3cyclictest30220-21gltestperf09:20:163
396899202,15cyclictest20223-21/usr/sbin/munin07:45:233
3968992013,4cyclictest22301-21grep09:00:263
396899201,10cyclictest0-21swapper/312:12:073
396899200,1cyclictest0-21swapper/309:55:183
395899201,8cyclictest0-21swapper/111:11:441
395899201,8cyclictest0-21swapper/109:40:291
395899201,8cyclictest0-21swapper/109:21:361
3958992013,2cyclictest0-21swapper/108:08:121
395899201,2cyclictest0-21swapper/107:59:091
395899201,18cyclictest0-21swapper/112:34:071
395899201,18cyclictest0-21swapper/111:36:551
395899201,18cyclictest0-21swapper/111:28:261
395899201,17cyclictest0-21swapper/107:21:031
395899201,14cyclictest0-21swapper/108:40:491
395899201,14cyclictest0-21swapper/108:20:451
395899201,14cyclictest0-21swapper/107:53:181
395899201,13cyclictest0-21swapper/111:32:331
395899201,11cyclictest5973-21cat09:40:021
395899201,11cyclictest16549-21ntp_states11:15:221
395899201,11cyclictest0-21swapper/110:24:491
395899201,10cyclictest8326-21egrep09:45:121
395899201,10cyclictest4889-21latency_hist12:05:021
395899201,10cyclictest13055-21latency_hist11:10:011
395199205,5cyclictest20835-21/usr/sbin/munin09:00:140
395199205,3cyclictest25621-21timerandwakeup11:35:260
395199202,9cyclictest17360-21kworker/0:008:51:470
3951992016,2cyclictest0-21swapper/007:59:480
396899195,3cyclictest87350irq/126-enp1s0-11:55:293
396899195,3cyclictest87350irq/126-enp1s0-10:15:203
396899191,6cyclictest3311-21cpu08:20:133
396899191,6cyclictest17448-21latency_hist11:20:023
396899191,6cyclictest0-21swapper/310:42:043
396899191,4cyclictest7652-21/usr/sbin/munin08:30:173
395899199,3cyclictest87650irq/129-enp1s0-11:50:181
395899196,11cyclictest27570-21kworker/u8:010:35:351
395899191,8cyclictest0-21swapper/112:07:081
395899191,16cyclictest0-21swapper/111:55:091
395899191,16cyclictest0-21swapper/110:45:191
395899191,16cyclictest0-21swapper/110:07:551
395899191,10cyclictest7424-21latency_hist08:30:011
395899191,10cyclictest4826-21df_inode10:50:131
395899191,10cyclictest28234-21cat10:30:131
395899191,10cyclictest28234-21cat10:30:131
395899190,18cyclictest0-21swapper/112:25:461
395199192,9cyclictest21959-21kworker/0:007:53:380
395199192,5cyclictest5457-21/usr/sbin/munin08:25:190
395199192,4cyclictest909-21sssd_nss07:24:470
145282190,4sleep011-21rcuc/008:45:130
89712180,2sleep30-21swapper/309:45:163
396899189,7cyclictest26801-21unixbench_singl09:10:253
396899182,4cyclictest11815-21latency_hist08:40:023
396899181,9cyclictest0-21swapper/309:50:153
396899181,5cyclictest29022-21/usr/sbin/munin08:05:173
396899181,5cyclictest18067-21latency_hist12:35:003
396899181,5cyclictest17070-21/usr/sbin/munin10:05:143
396899181,4cyclictest27544-21ntp_states11:40:243
396899181,4cyclictest27544-21ntp_states11:40:243
3968991812,3cyclictest6694-21head12:05:253
3968991812,3cyclictest18673-21needreboot11:20:213
396899181,14cyclictest12177-21tr08:40:123
3958991815,2cyclictest0-21swapper/111:02:061
3958991814,3cyclictest0-21swapper/112:21:101
395899181,16cyclictest0-21swapper/107:17:241
3958991811,2cyclictest0-21swapper/108:30:521
395199188,1cyclictest0-21swapper/008:55:140
395199182,4cyclictest20223-21/usr/sbin/munin07:45:220
395199181,9cyclictest10764-21memory08:35:180
395199181,5cyclictest7652-21/usr/sbin/munin08:30:190
3951991814,3cyclictest0-21swapper/007:35:160
3951991813,3cyclictest18025-21/usr/sbin/munin07:40:250
395199181,15cyclictest4968-21sshd08:21:470
396899171,9cyclictest0-21swapper/310:25:573
396899171,6cyclictest14592-21diskstats08:45:153
396899171,6cyclictest0-21swapper/311:05:273
396899171,4cyclictest19878-21/usr/sbin/munin11:25:213
396899171,4cyclictest10782-21/usr/sbin/munin12:15:213
3968991712,3cyclictest10303-21ntp_states07:20:213
3968991711,3cyclictest15077-21proc_pri12:25:223
396899171,11cyclictest0-21swapper/308:00:143
396899171,10cyclictest0-21swapper/310:50:043
395199173,3cyclictest87550irq/128-enp1s0-08:40:100
395199171,5cyclictest0-21swapper/007:28:110
3951991712,3cyclictest1795-21if_enp1s008:15:160
3951991710,4cyclictest28791-21latency_hist08:05:010
395199170,9cyclictest25027-21idleruntime-cro09:10:010
396899161,4cyclictest32567-21taskset11:51:423
3968991610,3cyclictest15601-21latency_hist07:35:023
395199163,3cyclictest87550irq/128-enp1s0-07:11:420
395199161,9cyclictest0-21swapper/008:10:280
395199161,9cyclictest0-21swapper/008:10:280
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional