You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-17 - 15:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Mar 17, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
172052229156,21sleep10-21swapper/107:08:331
172872223192,21sleep00-21swapper/007:09:380
172362205169,25sleep20-21swapper/207:08:592
172272202166,24sleep30-21swapper/307:08:533
17418991081,105cyclictest0-21swapper/312:16:273
17418999086,3cyclictest0-21swapper/308:22:183
17418998278,3cyclictest0-21swapper/311:17:183
17418997672,3cyclictest0-21swapper/307:36:253
17418997269,2cyclictest0-21swapper/310:55:363
17418996562,2cyclictest0-21swapper/312:04:363
17418996562,2cyclictest0-21swapper/307:17:193
111972590,3sleep230-21rcuc/208:05:202
248712570,2sleep081ktimersoftd/007:25:010
18672540,3sleep01870-21unixbench_multi10:10:240
1741899391,7cyclictest15970-21cstates10:45:113
1741399396,21cyclictest5965-21grep11:35:152
1741399381,27cyclictest12796-21cat09:25:022
1741399371,26cyclictest20886-21latency_hist08:30:012
1741399371,25cyclictest833-21runrttasks10:58:072
1741399371,25cyclictest4942-21cstates10:20:102
1741399371,25cyclictest31485-21/usr/sbin/munin07:40:132
1741899362,22cyclictest5717-21/usr/sbin/munin07:55:163
1741899361,24cyclictest31678-21proc_pri10:05:233
1741899361,22cyclictest29909-21diskstats12:30:153
1741899361,11cyclictest27093-21/usr/sbin/munin07:30:133
1741899357,2cyclictest0-21swapper/310:38:153
17418993512,20cyclictest16039-21grep08:15:263
1741399351,24cyclictest7388-21awk09:10:202
1741399351,23cyclictest0-21swapper/211:45:022
1741399351,23cyclictest0-21swapper/211:45:022
1741399351,21cyclictest28308-21sed11:10:242
1741399351,20cyclictest782-21in:imjournal10:05:012
1741899345,3cyclictest833-21runrttasks10:53:003
1741399341,22cyclictest17326-21/usr/sbin/munin09:35:202
1740199344,14cyclictest6125-21munin-run09:10:000
1741899338,21cyclictest0-21swapper/311:00:143
1741899337,3cyclictest19157-21chrt09:38:213
1741899331,29cyclictest422-21cpuspeed_turbos10:10:113
17418993312,18cyclictest25506-21sh12:20:123
1741399331,28cyclictest21134-21awk09:40:272
1741399331,23cyclictest4099-21unixbench_singl10:15:252
1741399331,23cyclictest24293-21fschecks_count09:50:152
1741399331,23cyclictest17733-21timerandwakeup12:00:252
17413993311,19cyclictest22694-21sshd12:14:352
1741899327,3cyclictest0-21swapper/312:13:043
1741899325,3cyclictest86550irq/126-enp1s0-10:00:213
1741899325,2cyclictest86550irq/126-enp1s0-10:15:153
1741899323,3cyclictest18573-21/usr/sbin/munin12:05:153
1741899322,26cyclictest7115-21cut10:25:133
1741899321,7cyclictest21116-21/usr/sbin/munin08:30:203
1741899321,2cyclictest0-21swapper/311:35:143
1741399322,18cyclictest27249-21/usr/sbin/munin09:55:242
1741399321,23cyclictest833-21runrttasks10:14:492
1741399321,23cyclictest2290-21wc11:25:262
17413993212,17cyclictest14545-21uname10:40:182
1740199327,12cyclictest15526-21munin-run10:45:020
1741899319,20cyclictest23293-21grep07:20:163
1741899319,19cyclictest11968-21/usr/sbin/munin11:50:253
1741899317,3cyclictest19114-21tail07:10:253
1741899316,4cyclictest86550irq/126-enp1s0-09:40:263
17418993127,3cyclictest0-21swapper/311:29:113
1741899311,5cyclictest31484-21/usr/sbin/munin11:20:123
1741899311,11cyclictest10511-21turbostat.cron09:20:013
1741899311,11cyclictest10511-21turbostat.cron09:20:013
17418993111,17cyclictest29898-21/usr/sbin/munin08:50:113
17418993110,18cyclictest12318-21/usr/sbin/munin08:10:193
1741399314,18cyclictest771-21rngd11:08:322
1741399311,26cyclictest31072-21proc_pri08:50:202
17413993110,18cyclictest24237-21missed_timers08:35:192
1741899301,2cyclictest86550irq/126-enp1s0-09:20:163
17418993012,16cyclictest5495-21sort09:05:243
17418993011,16cyclictest14425-21wc10:40:183
1741899300,9cyclictest0-21swapper/309:46:593
1741399301,26cyclictest979-21smart_sda08:55:262
1741399301,26cyclictest5188-21awk09:05:212
1741399301,21cyclictest28971-21sshd12:25:342
1741399301,20cyclictest14418-21expr11:55:142
1741399301,20cyclictest10226-21ntp_states10:30:202
1741399301,20cyclictest10226-21ntp_states10:30:202
1741399301,19cyclictest12933-21/usr/sbin/munin09:25:162
1741399301,17cyclictest22286-21perl09:45:172
1740199301,13cyclictest0-21swapper/008:40:350
1741899299,17cyclictest833-21runrttasks11:55:463
1741899298,4cyclictest25303-21sort09:50:253
1741899298,4cyclictest25242-21/usr/sbin/munin07:25:163
1741899298,3cyclictest25364-21cat08:40:013
1741899298,1cyclictest0-21swapper/309:30:173
1741899297,4cyclictest26647-21tr08:40:223
1741899297,4cyclictest10241-21tr08:05:123
1741899297,3cyclictest9770-21munin-node11:45:113
1741899295,1cyclictest0-21swapper/308:55:183
1741899292,7cyclictest0-21swapper/311:10:223
1741899291,5cyclictest0-21swapper/311:43:593
1741899291,5cyclictest0-21swapper/311:43:593
1741899291,25cyclictest9767-21sshd08:03:323
17418992911,15cyclictest26856-21awk09:55:183
17418992910,16cyclictest833-21runrttasks07:48:413
1741899290,10cyclictest0-21swapper/310:20:113
1741399297,19cyclictest833-21runrttasks12:15:442
1741399295,21cyclictest26268-21/usr/sbin/munin12:20:212
1741399291,25cyclictest32094-21/usr/sbin/munin12:35:132
1741399291,25cyclictest30478-21memory12:30:212
1741399291,25cyclictest15132-21/usr/sbin/munin09:30:252
17413992912,14cyclictest17605-21/usr/sbin/munin07:10:232
1741399291,18cyclictest19643-21ntp_states12:05:222
1741399291,17cyclictest8876-21df09:15:132
1741399291,17cyclictest8876-21df09:15:132
1741399291,17cyclictest29291-21diskstats11:15:152
1741399291,16cyclictest0-21swapper/211:30:182
1741399291,14cyclictest31144-21/usr/sbin/munin11:20:252
17413992911,15cyclictest4929-21expr07:50:232
17413992910,16cyclictest28998-21cat08:45:252
17413992910,16cyclictest11212-21cat11:45:252
1741399290,19cyclictest0-21swapper/210:45:112
1740699294,17cyclictest27365-21sshd08:43:441
1740199291,2cyclictest0-21swapper/010:02:180
1741899289,3cyclictest32284-21fschecks_time12:35:153
1741899289,16cyclictest31612-21cpuspeed_turbos07:40:123
1741899288,3cyclictest3307-21tr11:30:103
1741899288,17cyclictest3864-21sh07:50:133
1741899282,5cyclictest28772-21ntp_states08:45:213
1741899281,5cyclictest0-21swapper/310:32:003
1741899281,5cyclictest0-21swapper/310:32:003
1741899281,3cyclictest0-21swapper/312:25:153
1741899281,2cyclictest0-21swapper/309:25:163
17418992811,14cyclictest19497-21fw_packets08:25:163
1741899280,2cyclictest86750irq/128-enp1s0-11:08:323
1741399289,16cyclictest23221-21latency11:00:172
1741399288,17cyclictest1504-21crond08:35:012
1741399287,18cyclictest19447-21unixbench_multi10:50:242
1741399284,22cyclictest6091-21fschecks_count07:55:132
1741399284,21cyclictest15773-21smart_sda08:15:242
1741399281,24cyclictest13778-21/usr/sbin/munin08:10:252
1741399281,23cyclictest30391-21/usr/sbin/munin07:35:212
17413992812,13cyclictest21639-21grep07:15:222
1741399281,17cyclictest86750irq/128-enp1s0-07:25:172
17413992810,14cyclictest25604-21grep08:40:132
1741399280,25cyclictest16710-21/usr/sbin/munin08:20:142
1740199281,13cyclictest7703-21perf08:00:010
1740199281,12cyclictest0-21swapper/009:45:340
1741899272,4cyclictest7258-21memory09:10:173
1741399271,3cyclictest11968-21/usr/sbin/munin11:50:212
1741399271,23cyclictest8549-21/usr/sbin/munin10:25:262
1741399271,23cyclictest30864-21df_inode10:05:142
1741399271,23cyclictest2665-21if_enp1s009:00:172
1741399271,23cyclictest1311-21/usr/sbin/munin07:45:112
1741399271,23cyclictest12028-21hddtemp_smartct10:35:152
1741399271,22cyclictest23935-21/usr/sbin/munin07:20:242
17413992710,14cyclictest8107-21expr08:00:122
1740199271,12cyclictest7024-21kworker/0:011:40:000
1740199270,2cyclictest0-21swapper/010:55:070
1741899268,3cyclictest2120-21cstates09:00:123
1741399269,14cyclictest833-21runrttasks07:30:022
1740699263,7cyclictest4268-40dnf10:16:491
1740199261,12cyclictest28720-21munin-run11:15:020
1740199261,12cyclictest0-21swapper/008:24:230
1740199261,10cyclictest0-21swapper/011:22:030
1740199261,10cyclictest0-21swapper/011:01:510
1740699252,3cyclictest12095-21munin-run08:10:011
1740699251,7cyclictest19713-21cat07:15:021
1740699251,5cyclictest0-21swapper/111:12:081
17406992514,3cyclictest28051-21diskstats08:45:151
1740699251,14cyclictest0-21swapper/110:53:591
1740199254,14cyclictest12050irq/123-ahci[0012:06:020
1740199251,12cyclictest12710-21munin-run09:25:000
1740199251,11cyclictest0-21swapper/009:35:300
1740199251,10cyclictest0-21swapper/010:19:190
1740199250,12cyclictest0-21swapper/012:21:220
1740199250,12cyclictest0-21swapper/009:59:540
1740199250,12cyclictest0-21swapper/008:35:310
1740199250,10cyclictest0-21swapper/009:03:300
254192240,7sleep011-21rcuc/009:50:250
1740699248,4cyclictest0-21swapper/112:10:591
1740699242,12cyclictest1625-21smart_sda10:10:221
1740699241,6cyclictest0-21swapper/112:16:541
1740699241,17cyclictest0-21swapper/108:26:131
1740699241,11cyclictest0-21swapper/111:32:081
1740699241,10cyclictest0-21swapper/110:56:351
17401992417,4cyclictest27363-21/usr/sbin/munin12:25:220
1740199241,11cyclictest0-21swapper/011:25:350
1740199241,11cyclictest0-21swapper/008:11:510
1740199240,9cyclictest0-21swapper/009:16:420
1740199240,9cyclictest0-21swapper/009:16:420
1740199240,11cyclictest0-21swapper/011:49:060
1740199240,11cyclictest0-21swapper/011:40:060
1740199240,11cyclictest0-21swapper/011:40:060
1740199240,10cyclictest0-21swapper/011:15:590
1740699233,4cyclictest24458-21sshd07:22:311
1740699232,17cyclictest26529-21munin-run11:10:021
17406992317,3cyclictest14128-21perl10:40:151
1740699231,6cyclictest0-21swapper/112:38:421
1740699231,17cyclictest0-21swapper/109:11:031
1740699231,16cyclictest0-21swapper/111:56:171
1740699231,16cyclictest0-21swapper/111:46:121
1740699231,16cyclictest0-21swapper/110:31:471
1740699231,16cyclictest0-21swapper/110:31:471
1740699231,12cyclictest0-21swapper/111:36:461
1740699231,12cyclictest0-21swapper/109:26:511
1740699231,11cyclictest0-21swapper/111:41:371
1740699231,11cyclictest0-21swapper/111:41:371
1740699231,10cyclictest0-21swapper/112:20:031
17406992310,11cyclictest17428-21kworker/u8:011:50:051
1740699230,6cyclictest0-21swapper/110:36:321
1740699230,5cyclictest0-21swapper/109:15:211
1740699230,5cyclictest0-21swapper/109:15:211
1740199236,3cyclictest0-21swapper/012:31:580
1740199231,9cyclictest0-21swapper/011:50:420
1740199231,9cyclictest0-21swapper/007:39:470
1740199231,10cyclictest0-21swapper/010:52:150
1740199231,10cyclictest0-21swapper/008:29:430
1740199230,10cyclictest0-21swapper/007:45:590
1740699226,13cyclictest27383-21smart_sda09:55:241
1740699222,6cyclictest22126-40dnf07:15:501
1740699221,4cyclictest8661-21kworker/u8:210:45:211
17406992213,3cyclictest0-21swapper/108:11:181
1740699221,2cyclictest0-21swapper/108:03:271
1740699221,18cyclictest28422-21ntpq12:25:211
1740699221,18cyclictest12325-21users09:20:261
1740699221,15cyclictest0-21swapper/110:08:111
1740699221,15cyclictest0-21swapper/108:24:171
1740699221,14cyclictest0-21swapper/109:49:371
1740699221,13cyclictest0-21swapper/110:20:221
1740699221,12cyclictest1328-21kworker/u8:110:28:101
1740699221,11cyclictest2346-21netstat07:45:191
1740699221,11cyclictest0-21swapper/110:00:041
1740699221,11cyclictest0-21swapper/109:54:061
1740699221,10cyclictest0-21swapper/112:01:001
1740699220,6cyclictest0-21swapper/111:24:161
1740699220,17cyclictest0-21swapper/111:02:271
1740199221,9cyclictest0-21swapper/008:07:430
1740199221,8cyclictest0-21swapper/007:25:270
1740699219,3cyclictest1075-21cpuspeed_turbos11:25:141
1740699218,3cyclictest0-21swapper/109:41:181
1740699211,5cyclictest0-21swapper/112:33:321
1740699211,4cyclictest0-21swapper/112:05:391
17406992113,3cyclictest0-21swapper/108:35:561
17406992113,3cyclictest0-21swapper/107:33:451
1740699211,15cyclictest0-21swapper/108:55:061
1740699211,15cyclictest0-21swapper/108:32:501
1740699211,15cyclictest0-21swapper/107:38:111
1740699211,15cyclictest0-21swapper/107:28:001
1740699211,14cyclictest0-21swapper/109:35:131
1740699211,11cyclictest30811-21memory08:50:161
1740699211,10cyclictest0-21swapper/111:16:131
1740699211,10cyclictest0-21swapper/109:32:041
1740699210,15cyclictest0-21swapper/107:55:131
1740199217,3cyclictest0-21swapper/010:35:110
1740199217,3cyclictest0-21swapper/010:31:470
1740199217,3cyclictest0-21swapper/010:31:470
1740199217,2cyclictest0-21swapper/007:33:510
1740199212,6cyclictest5719-1kworker/u9:211:30:530
1740199212,10cyclictest27705-21/usr/sbin/munin08:45:170
1740199211,9cyclictest0-21swapper/009:32:460
1740199211,5cyclictest5719-1kworker/u9:209:25:020
1740199210,8cyclictest0-21swapper/009:12:060
1740199210,10cyclictest0-21swapper/007:52:190
17406992013,2cyclictest0-21swapper/107:40:531
1740699201,11cyclictest15425-21memory08:15:191
1740699201,10cyclictest4332-21irqstats07:50:161
1740199209,3cyclictest29898-21/usr/sbin/munin08:50:140
1740199202,5cyclictest0-21swapper/010:26:380
1740199201,6cyclictest5719-1kworker/u9:210:24:200
17401992010,3cyclictest86750irq/128-enp1s0-12:01:280
1740699191,10cyclictest3913-21latency_hist09:05:021
1740199198,3cyclictest0-21swapper/008:30:430
1740199196,3cyclictest0-21swapper/011:56:120
1740199195,3cyclictest0-21swapper/009:43:060
1740199195,2cyclictest0-21swapper/012:37:060
1740199194,3cyclictest0-21swapper/010:45:350
1740199191,6cyclictest0-21swapper/011:05:170
17401991912,4cyclictest23571-21head12:15:140
321632181,2sleep00-21swapper/010:05:560
1740699181,16cyclictest0-21swapper/109:05:591
1740199185,3cyclictest0-21swapper/007:10:310
1740199180,10cyclictest0-21swapper/012:14:380
1740199172,3cyclictest937-21sshd07:43:270
1740199171,9cyclictest0-21swapper/008:00:170
17401991712,3cyclictest21846-21sort07:15:240
1740199171,14cyclictest0-21swapper/008:58:100
17401991611,3cyclictest16214-21seq08:16:430
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional