You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-12 - 11:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Jul 12, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
274812238163,24sleep10-21swapper/119:09:071
274782238166,24sleep30-21swapper/319:09:063
272312233161,59sleep00-21swapper/019:05:520
272232203166,24sleep20-21swapper/219:05:462
1613121630,4sleep22764399cyclictest22:20:222
27643991091,3cyclictest0-21swapper/222:47:242
95222590,1sleep00-21swapper/019:40:120
27643995955,3cyclictest0-21swapper/220:41:422
27643995653,2cyclictest0-21swapper/220:35:572
309392510,5sleep12763899cyclictest22:55:161
65102500,2sleep30-21swapper/319:30:273
46872420,4sleep1221ktimersoftd/120:40:241
27649994210,25cyclictest463-21/usr/sbin/munin19:20:273
2764399421,30cyclictest22198-21/usr/sbin/munin21:20:232
2764399412,24cyclictest25541-21grep23:55:232
2764399403,31cyclictest30832-40dnf20:26:362
2764999381,26cyclictest18163-21users21:10:273
2764399381,27cyclictest18773-21/usr/sbin/munin22:25:272
2764999371,32cyclictest31989-21sh19:15:253
2764999371,23cyclictest0-21swapper/300:00:173
2764399371,23cyclictest11323-21latency_hist19:45:022
2764399370,27cyclictest10596-21/usr/sbin/munin22:10:202
2764399370,27cyclictest10596-21/usr/sbin/munin22:10:202
2764999363,29cyclictest30832-40dnf20:26:373
2764999361,32cyclictest24405-21/usr/sbin/munin23:55:243
27643993634,1cyclictest0-21swapper/222:07:002
2764399361,25cyclictest7258-21ntpq22:00:222
2764399361,22cyclictest20352-21/usr/sbin/munin20:05:232
2764399361,22cyclictest15956-21/usr/sbin/munin23:35:142
2764999351,30cyclictest5261-21/usr/sbin/munin00:25:263
2764999351,21cyclictest24348-21fschecks_time22:40:173
2764999351,20cyclictest9430-21latency_hist00:35:013
2764999351,20cyclictest9430-21latency_hist00:35:013
2764399351,23cyclictest4947-21missed_timers21:55:222
2764399351,23cyclictest13415-21/usr/sbin/munin23:30:152
2764399351,21cyclictest804-21abrt-dump-journ23:05:012
2764999342,29cyclictest17810-21/usr/sbin/munin23:40:163
2764399341,23cyclictest24076-21memory21:25:172
141692340,1sleep10-21swapper/122:15:231
27649993313,14cyclictest0-21swapper/323:15:193
2764999331,29cyclictest27558-21/usr/sbin/munin21:35:213
2763199335,13cyclictest0-21swapper/020:55:020
2764999321,28cyclictest25982-21/usr/sbin/munin22:45:143
2764999321,27cyclictest19392-21/usr/sbin/munin22:30:223
2764999321,22cyclictest1228-21/usr/sbin/munin23:00:193
2764399324,19cyclictest22595-21netstat22:35:202
2764399322,22cyclictest32-21ksoftirqd/200:00:072
291812310,1sleep30-21swapper/319:10:253
2764999311,27cyclictest30384-21/usr/sbin/munin22:55:273
2764999311,26cyclictest0-21swapper/323:28:343
2764999311,22cyclictest580-21sshd20:31:343
2764999311,18cyclictest12171-21/usr/sbin/munin21:00:263
2764399312,25cyclictest27287-21diskstats20:20:142
2764399311,26cyclictest32418-21ntp_states20:30:232
2764399311,26cyclictest31581-21tune2fs00:10:152
2764399311,26cyclictest16573-21/usr/sbin/munin21:10:282
2764399311,26cyclictest14819-21ntp_states19:50:222
2764399311,18cyclictest9806-21if_err_enp2s023:20:162
2764399311,18cyclictest9806-21if_err_enp2s023:20:162
2764399311,17cyclictest87350irq/126-enp1s0-21:45:222
2764399311,17cyclictest827-21runrttasks00:20:352
27643993110,17cyclictest21352-21crond22:35:012
2764399310,20cyclictest0-21swapper/223:10:232
85462300,5sleep12763899cyclictest22:05:121
2764999309,18cyclictest4811-21timerwakeupswit20:40:253
2764999309,18cyclictest11735-21cstates19:45:133
2764999302,24cyclictest1461-21grep00:15:153
27649993013,14cyclictest31957-21/usr/sbin/munin21:45:153
27649993012,15cyclictest4623-21/usr/sbin/munin23:10:243
2764999301,13cyclictest0-21swapper/322:07:123
27649993011,16cyclictest21310-21df_inode21:20:143
27649993010,17cyclictest27617-21hddtemp_smartct20:20:173
27649993010,17cyclictest17916-21if_enp1s022:25:183
2764999300,28cyclictest0-21swapper/320:03:123
2764399307,20cyclictest18758-21head20:00:162
2764399301,26cyclictest20036-21smart_sda21:15:272
2764399301,25cyclictest31341-21/usr/sbin/munin21:40:252
27643993012,14cyclictest13185-21turbostat.cron23:30:012
2764399301,17cyclictest20007-21/usr/sbin/munin23:45:122
2764399301,16cyclictest0-21swapper/222:15:112
2764999299,17cyclictest24926-21cstates20:15:123
2764999299,17cyclictest14161-21timerandwakeup22:15:233
2764999298,18cyclictest20439-21cpuspeed_turbos20:05:133
2764999298,18cyclictest10924-21unixbench_singl19:40:253
2764999292,23cyclictest29890-21ntp_states00:05:213
2764999292,23cyclictest16685-21ntp_states23:35:223
2764999291,25cyclictest4105-21/usr/sbin/munin00:20:213
2764999291,25cyclictest31029-21/usr/sbin/munin00:10:183
2764999291,24cyclictest4438-40dnf19:26:223
2764999291,20cyclictest0-21swapper/322:50:273
2764999290,27cyclictest0-21swapper/323:05:243
2764399292,23cyclictest8528-21ntp_states00:30:212
2764399292,23cyclictest8528-21ntp_states00:30:212
2764399292,23cyclictest3115-21diskstats19:25:172
2764399291,25cyclictest29007-21/usr/sbin/munin22:50:192
2764399291,25cyclictest28188-21/usr/sbin/munin21:35:152
2764399291,24cyclictest31554-21uname19:15:192
2764399291,24cyclictest31373-21strings22:55:212
2764399291,24cyclictest28503-21cut19:10:152
2764399291,24cyclictest24582-21/usr/sbin/munin22:40:182
2764399291,24cyclictest18724-21/usr/sbin/munin23:40:182
2764399291,24cyclictest1463-21tail00:15:152
2764399291,20cyclictest0-21swapper/221:53:032
2764399291,17cyclictest87650irq/129-enp1s0-20:45:222
2764399291,16cyclictest87550irq/128-enp1s0-20:15:242
27631992911,15cyclictest13199-21munin-run23:30:020
2764999289,16cyclictest14521-21cpuspeed_turbos21:05:153
2764999289,16cyclictest11718-21pmu-power22:10:213
2764999289,16cyclictest11718-21pmu-power22:10:213
2764999282,23cyclictest7819-21grep19:35:163
2764999281,24cyclictest9976-21/usr/sbin/munin20:55:263
2764999281,24cyclictest2445-21smart_sda20:35:243
2764999281,24cyclictest22542-21/usr/sbin/munin20:10:103
2764999281,24cyclictest20007-21/usr/sbin/munin23:45:123
2764999281,24cyclictest18775-21/usr/sbin/munin21:15:263
2764999281,24cyclictest14100-21/usr/sbin/munin19:50:153
2764999281,23cyclictest9843-21/usr/sbin/munin23:20:173
2764999281,23cyclictest9843-21/usr/sbin/munin23:20:173
2764999281,23cyclictest7777-21/usr/sbin/munin20:50:253
27649992812,13cyclictest25706-21df_inode21:30:133
27649992812,13cyclictest1800-21/usr/sbin/munin21:50:183
27649992810,15cyclictest23909-21if_enp2s021:25:153
2764399282,23cyclictest29191-21uniq00:05:142
2764399282,22cyclictest24182-21munin-run23:55:012
27643992813,12cyclictest12171-21/usr/sbin/munin21:00:192
2764399281,24cyclictest6316-21/usr/sbin/munin00:25:212
2764399281,24cyclictest24659-21grep20:15:012
2764399281,24cyclictest14988-21tail21:05:182
2764399281,23cyclictest9746-21turbostat20:55:002
2764399281,23cyclictest8347-21grep23:15:272
2764399281,23cyclictest7820-21sed19:35:172
2764399281,23cyclictest10343-21grep20:55:162
2764399281,18cyclictest87350irq/126-enp1s0-21:30:222
2764399280,25cyclictest11553-21/usr/sbin/munin19:45:262
2763199287,10cyclictest0-21swapper/020:20:010
2763199282,9cyclictest31014-21kworker/0:020:35:150
2763199282,11cyclictest26111-21kworker/0:300:10:260
2763199281,10cyclictest27812-21kworker/0:123:10:260
2764999277,17cyclictest21586-21/usr/sbin/munin22:35:153
2764999276,19cyclictest14001-21tr23:30:153
2764999271,23cyclictest9663-21/usr/sbin/munin00:35:193
2764999271,23cyclictest6365-21/usr/sbin/munin22:00:133
2764999271,23cyclictest5033-21/usr/sbin/munin21:55:233
2764999271,23cyclictest29759-21/usr/sbin/munin21:40:163
2764999271,23cyclictest22205-21/usr/sbin/munin23:50:133
2764999271,23cyclictest15949-21/usr/sbin/munin19:55:133
2764999271,22cyclictest15412-21sort22:20:153
2764399271,23cyclictest5385-21uniq19:30:172
2764399271,23cyclictest11192-21tail00:35:292
2764399271,22cyclictest2960-21cut23:05:162
2764399271,22cyclictest15949-21/usr/sbin/munin19:55:132
2763199272,9cyclictest2219-21kworker/0:319:29:340
2763199272,18cyclictest14624-21kworker/0:222:22:500
2764999261,22cyclictest5579-21/usr/sbin/munin20:45:113
2764399268,15cyclictest659-21cstates19:20:132
27638992615,3cyclictest13046-40dnf23:28:241
2763199263,11cyclictest16361-21perf21:10:020
2763199262,9cyclictest5988-21kworker/0:222:05:260
2763199261,9cyclictest11210-21kworker/0:119:53:020
2763199252,11cyclictest25080-21kworker/0:219:15:110
2763199251,9cyclictest29260-21kworker/0:121:46:420
2763199251,9cyclictest11961-21kworker/0:221:14:310
2763199251,8cyclictest2592-21kworker/0:000:27:180
2763199251,14cyclictest15314-21kworker/0:123:35:270
2763199251,12cyclictest2592-21kworker/0:000:30:180
2763199251,12cyclictest2592-21kworker/0:000:30:180
2763899241,5cyclictest808-21snmpd22:10:371
2763899241,5cyclictest808-21snmpd22:10:371
2763199249,2cyclictest13082-21kworker/0:023:40:470
2763199242,8cyclictest17626-21kworker/0:200:03:020
2763199242,11cyclictest17558-21kworker/0:219:57:500
2763199242,10cyclictest10518-21kworker/0:119:10:020
2763199241,11cyclictest27812-21kworker/0:123:23:340
2763199241,11cyclictest27812-21kworker/0:123:23:340
2763199241,10cyclictest29260-21kworker/0:121:50:140
2763899238,7cyclictest19158-21latency_hist22:30:021
2763899231,9cyclictest0-21swapper/123:45:001
2763899231,3cyclictest827-21runrttasks21:40:441
2763899231,17cyclictest0-21swapper/122:40:101
2763899231,16cyclictest0-21swapper/121:02:071
2763899231,16cyclictest0-21swapper/120:30:511
2763899231,15cyclictest16343-21diskmemload23:39:551
2763899231,15cyclictest0-21swapper/122:50:051
2763899231,10cyclictest0-21swapper/119:45:221
2763199236,9cyclictest2219-21kworker/0:319:46:420
2763199236,3cyclictest19178-21kworker/0:122:35:180
2763199235,3cyclictest87350irq/126-enp1s0-21:40:210
2763199231,9cyclictest20661-21kworker/0:121:26:440
2763199231,6cyclictest0-21swapper/000:15:260
2763199231,15cyclictest29260-21kworker/0:122:00:140
2763199230,11cyclictest22955-21perf21:25:020
2763899228,5cyclictest29961-21munin-node19:15:011
2763899221,3cyclictest8803-21perf23:20:011
27638992213,3cyclictest0-21swapper/121:31:411
2763899221,15cyclictest16343-21diskmemload22:03:321
2763899221,15cyclictest0-21swapper/119:36:111
2763899221,10cyclictest29465-21gltestperf00:05:161
2763899220,6cyclictest20729-21taskset21:20:001
2763899220,16cyclictest0-21swapper/123:08:371
2763899220,15cyclictest0-21swapper/120:26:241
2763199222,8cyclictest23960-21kworker/0:023:52:510
2763199222,11cyclictest5156-21kworker/0:220:47:110
2763199222,11cyclictest26546-21kworker/0:120:21:340
2763199221,9cyclictest19178-21kworker/0:122:32:020
2763199221,6cyclictest0-21swapper/023:46:180
2763199221,12cyclictest32371-21kworker/0:223:03:060
2763199221,11cyclictest25080-21kworker/0:219:20:060
27631992210,3cyclictest28626-21ntp_states21:35:200
27631992210,2cyclictest26111-21kworker/0:300:09:420
2763899216,4cyclictest0-21swapper/121:37:401
2763899212,11cyclictest20306-21memory22:30:181
27638992113,3cyclictest0-21swapper/123:53:381
27638992113,3cyclictest0-21swapper/122:23:521
27638992113,3cyclictest0-21swapper/121:48:371
2763899211,17cyclictest15710-21runrttasks19:54:591
2763899211,15cyclictest0-21swapper/123:30:401
2763899211,15cyclictest0-21swapper/120:51:251
2763899211,15cyclictest0-21swapper/120:49:091
2763899211,15cyclictest0-21swapper/120:04:201
2763899211,15cyclictest0-21swapper/119:59:431
2763899211,12cyclictest0-21swapper/100:14:021
2763899210,6cyclictest0-21swapper/122:48:261
2763899210,5cyclictest1248-21kworker/u8:021:55:511
2763899210,16cyclictest0-21swapper/100:22:311
2763199218,2cyclictest27812-21kworker/0:122:52:380
2763199214,15cyclictest87350irq/126-enp1s0-23:55:160
2763199213,10cyclictest31345-21/usr/sbin/munin20:30:150
2763199212,17cyclictest17558-21kworker/0:220:10:220
2763199212,10cyclictest2219-21kworker/0:319:39:540
2763199212,10cyclictest11961-21kworker/0:221:17:150
2763199211,9cyclictest24338-21kworker/0:021:34:500
2763199211,8cyclictest9092-21kworker/0:100:36:140
2763199211,6cyclictest0-21swapper/021:59:360
2763199211,5cyclictest0-21swapper/023:32:030
2763199211,4cyclictest0-21swapper/022:11:160
2763199211,4cyclictest0-21swapper/022:11:160
2763199211,11cyclictest26328-21diskstats22:45:150
2763199211,10cyclictest27812-21kworker/0:123:19:300
2763899208,6cyclictest0-21swapper/100:15:021
2763899206,3cyclictest0-21swapper/123:55:191
2763899204,9cyclictest23-21ksoftirqd/121:27:231
2763899201,8cyclictest0-21swapper/119:16:381
27638992013,2cyclictest0-21swapper/120:05:561
27638992012,3cyclictest0-21swapper/121:10:221
27638992012,3cyclictest0-21swapper/119:33:181
2763899201,17cyclictest0-21swapper/119:43:291
2763899201,15cyclictest0-21swapper/123:00:521
2763899201,15cyclictest0-21swapper/121:07:431
2763899201,14cyclictest0-21swapper/121:51:011
2763899201,14cyclictest0-21swapper/120:57:501
2763899201,14cyclictest0-21swapper/119:26:081
2763899201,14cyclictest0-21swapper/100:35:091
2763899201,14cyclictest0-21swapper/100:00:491
2763899201,11cyclictest9299-21expr23:20:121
2763899201,11cyclictest9299-21expr23:20:121
2763899201,11cyclictest867-21df_inode19:20:161
2763899201,11cyclictest22891-21sshd21:24:341
2763899201,10cyclictest9040-21grep00:30:271
2763899201,10cyclictest9040-21grep00:30:271
2763899200,15cyclictest0-21swapper/123:47:021
2763899200,15cyclictest0-21swapper/120:17:131
2763899200,15cyclictest0-21swapper/100:25:071
2763199207,2cyclictest19178-21kworker/0:122:40:300
2763199202,8cyclictest12370-21kworker/0:122:16:540
2763199201,9cyclictest27812-21kworker/0:122:57:220
2763199201,11cyclictest31014-21kworker/0:020:55:350
2763199201,10cyclictest3602-21fschecks_time00:20:160
2763199201,10cyclictest2219-21kworker/0:319:32:300
2763199201,10cyclictest19829-21kworker/0:120:02:100
2763199200,11cyclictest14624-21kworker/0:222:28:380
2763899191,16cyclictest0-21swapper/122:35:551
27638991911,3cyclictest0-21swapper/123:11:441
2763899190,17cyclictest0-21swapper/120:10:201
2763199192,9cyclictest26546-21kworker/0:120:28:380
2763199191,10cyclictest31014-21kworker/0:021:01:270
2763899180,1cyclictest0-21swapper/120:20:181
2763899180,14cyclictest0-21swapper/120:36:211
2763199181,10cyclictest0-21swapper/023:06:080
2763199171,14cyclictest0-21swapper/020:40:470
2763199161,2cyclictest0-21swapper/020:05:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional