You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-03 - 05:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jun 03, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
124302237163,24sleep30-21swapper/319:08:343
125062226196,20sleep20-21swapper/219:09:332
124322226155,58sleep00-21swapper/019:08:360
122342201164,25sleep10-21swapper/119:06:031
2729221720,5sleep11263099cyclictest21:05:131
12635991091,106cyclictest0-21swapper/200:02:232
1263599105102,2cyclictest0-21swapper/221:04:552
126359910097,2cyclictest0-21swapper/222:26:222
305442640,1sleep10-21swapper/122:25:161
273612520,1sleep30-21swapper/321:05:153
9432500,3sleep19-21rcu_preempt20:05:181
207592500,2sleep20-21swapper/200:30:192
1264099431,32cyclictest15016-21/usr/sbin/munin23:05:233
1264099431,27cyclictest18175-21df00:25:133
1264099431,27cyclictest14845-21ntp_states20:35:233
1264099396,21cyclictest87350irq/126-enp1s0-22:40:013
1264099375,21cyclictest31154-21if_enp2s023:40:163
1264099373,22cyclictest0-21swapper/323:35:123
1264099371,26cyclictest23286-21sshd23:21:303
1264099371,25cyclictest20042-21/usr/sbin/munin00:30:243
1264099361,24cyclictest939-21latency_hist21:20:013
12623993611,12cyclictest22338-21munin-run20:55:010
1264099352,29cyclictest30885-21cut22:25:213
1264099351,7cyclictest14905-21cat19:15:023
1264099351,7cyclictest14905-21cat19:15:013
1264099351,23cyclictest16584-21/usr/sbin/munin21:55:223
1264099342,28cyclictest9459-40dnf21:36:233
1264099342,28cyclictest9459-40dnf21:36:233
1264099341,6cyclictest26737-21latency_hist21:05:023
12640993415,16cyclictest10752-21grep22:55:143
12640993414,17cyclictest21372-21fschecks_count22:05:163
1264099341,25cyclictest0-21swapper/322:15:373
1264099341,21cyclictest4623-21/usr/sbin/munin23:55:143
1264099334,15cyclictest23798-21/usr/sbin/munin23:25:163
1264099333,4cyclictest15239-21latency21:50:193
1264099332,3cyclictest909-21sssd_nss22:30:143
1264099331,3cyclictest13515-21cpuspeed_turbos00:15:143
1264099324,18cyclictest1123-21missed_timers23:45:183
1264099324,16cyclictest401ktimersoftd/300:05:223
1264099324,15cyclictest12050irq/123-ahci[0020:35:023
1264099323,25cyclictest5947-21diskstats21:30:213
1264099321,6cyclictest30745-21/usr/sbin/munin20:00:233
1264099321,3cyclictest29513-21diskstats21:10:153
1264099321,21cyclictest0-21swapper/300:20:263
1264099321,17cyclictest22234-21/usr/sbin/munin00:35:143
1264099320,22cyclictest0-21swapper/323:00:123
1264099314,3cyclictest16327-21/usr/sbin/munin19:20:123
1264099313,15cyclictest5348-21awk22:40:243
1264099312,25cyclictest26224-21grep19:50:013
1264099311,3cyclictest87650irq/129-enp1s0-20:45:183
1264099311,26cyclictest6361-21cpuspeed_turbos22:45:133
1264099311,26cyclictest6361-21cpuspeed_turbos22:45:133
1264099311,20cyclictest24527-21latency_hist21:00:023
1264099311,20cyclictest21725-21latency_hist19:40:013
1262399311,15cyclictest8179-21munin-run22:50:010
1262399311,15cyclictest8179-21munin-run22:50:010
1264099309,18cyclictest19875-21grep23:15:143
1264099308,19cyclictest4126-21timerandwakeup20:10:243
1264099302,3cyclictest26347-21/usr/sbin/munin23:30:143
1264099302,24cyclictest26491-21grep19:50:113
1264099301,9cyclictest4827-21timerwakeupswit21:25:253
1264099301,4cyclictest0-21swapper/321:23:043
1264099301,2cyclictest87650irq/129-enp1s0-20:20:153
12640993011,16cyclictest28849-21expr19:55:133
1264099298,18cyclictest15313-21open_files19:15:023
1264099298,18cyclictest15313-21open_files19:15:023
1264099297,19cyclictest827-21runrttasks20:28:533
1264099297,19cyclictest23709-21/usr/sbin/munin22:10:153
1264099295,16cyclictest87650irq/129-enp1s0-20:50:213
1264099292,3cyclictest6051-21ntp_states20:15:223
1264099292,24cyclictest10600-21head21:40:153
1264099291,6cyclictest16690-21kworker/u8:023:14:303
1264099291,5cyclictest0-21swapper/323:51:443
1264099291,3cyclictest8621-21cstates22:50:143
1264099291,3cyclictest20302-21unixbench_multi22:00:263
1264099291,25cyclictest11264-21egrep00:10:133
1262399291,13cyclictest17961-21basename20:45:010
1264099282,3cyclictest13257-21ntp_states21:45:213
1264099281,4cyclictest28607-21/usr/sbin/munin22:20:213
1264099281,4cyclictest22693-21/usr/sbin/munin19:40:133
1264099281,3cyclictest15975-21/usr/sbin/munin20:40:193
1264099281,19cyclictest0-21swapper/300:03:293
1263599288,9cyclictest15177-21cpuspeed_turbos23:05:142
1264099271,4cyclictest18785-21sshd19:28:283
1264099271,3cyclictest19308-21/usr/sbin/munin19:30:083
12640992710,14cyclictest687-21cstates20:05:163
1263599277,2cyclictest21551-21wc23:20:012
12635992711,7cyclictest21950-21/usr/sbin/munin19:40:122
1262399271,15cyclictest23358-21kworker/0:200:16:550
1263099262,17cyclictest0-21swapper/123:37:271
1263099261,10cyclictest0-21swapper/122:52:271
1263099260,5cyclictest0-21swapper/123:23:491
1263099260,3cyclictest0-21swapper/122:21:161
1263099260,11cyclictest0-21swapper/100:13:541
12623992617,5cyclictest13225-21/usr/sbin/munin21:45:200
1263599259,7cyclictest1808-21/usr/sbin/munin22:35:132
1263599255,6cyclictest87550irq/128-enp1s0-00:30:022
1263099251,17cyclictest0-21swapper/100:37:161
1262399256,16cyclictest3787-21munin-run22:40:000
1262399251,5cyclictest0-21swapper/022:12:130
1263599249,7cyclictest8859-21sh21:35:212
1263599249,7cyclictest8859-21sh21:35:212
1263599249,3cyclictest0-21swapper/220:39:062
1263599246,8cyclictest25581-21expr22:15:132
1263599245,3cyclictest19366-21gltestperf22:00:162
1263599244,8cyclictest28864-21hddtemp_smartct23:35:152
12635992412,4cyclictest87350irq/126-enp1s0-23:20:182
1263099247,3cyclictest14173-21awk21:50:021
1263099242,5cyclictest20713-21hddtemp_smartct00:30:171
1263099241,12cyclictest0-21swapper/122:39:551
1263099240,9cyclictest0-21swapper/123:12:291
1263099240,5cyclictest0-21swapper/123:26:471
1263099240,11cyclictest0-21swapper/122:40:351
1262399249,4cyclictest0-21swapper/023:25:140
1262399245,10cyclictest0-21swapper/023:00:130
1263599237,9cyclictest0-21swapper/221:34:552
1263599237,7cyclictest11246-21/usr/sbin/munin00:10:282
1263599236,7cyclictest7371-21/usr/sbin/munin22:45:242
1263599236,7cyclictest7371-21/usr/sbin/munin22:45:242
1263599234,9cyclictest22575-21users22:05:292
1263599232,7cyclictest31129-21latency_hist21:15:012
12635992313,2cyclictest87650irq/129-enp1s0-23:50:242
1263099238,8cyclictest0-21swapper/100:27:521
1263099238,3cyclictest22352-21awk20:55:011
1263099237,2cyclictest24655-1kworker/u9:023:00:431
1263099235,11cyclictest23487-21cstates22:10:141
1263099234,15cyclictest21353-40dnf19:35:251
1263099232,3cyclictest1270-21ntp_states23:45:211
1263099232,11cyclictest0-21swapper/123:55:381
1263099231,8cyclictest0-21swapper/100:23:061
1263099231,19cyclictest20763-21munin-run22:05:021
1263099231,19cyclictest15385-40dnf20:35:291
1263099231,10cyclictest0-21swapper/121:38:121
1263099231,10cyclictest0-21swapper/121:38:111
1263099230,9cyclictest0-21swapper/123:40:401
1262399231,4cyclictest0-21swapper/022:20:560
1263599228,5cyclictest4820-21irqstats22:40:182
1263599227,3cyclictest579-21systemd-journal00:10:012
1263599227,3cyclictest11338-21awk22:55:172
1263599226,8cyclictest17295-21cpuspeed_turbos23:10:122
1263599225,7cyclictest808-21snmpd19:19:112
1263599225,7cyclictest808-21snmpd19:19:112
1263599224,7cyclictest16848-21latency20:40:182
1263599222,9cyclictest17349-21if_enp2s021:55:162
1263599222,17cyclictest18337-21tr20:45:132
1263599221,12cyclictest0-21swapper/221:28:022
12635992210,3cyclictest1511-21switchtime23:45:232
1263099227,7cyclictest0-21swapper/122:06:161
1263099226,2cyclictest0-21swapper/119:40:161
1263099221,6cyclictest0-21swapper/122:47:431
1263099221,6cyclictest0-21swapper/122:47:431
12630992214,3cyclictest0-21swapper/121:54:021
1263099221,3cyclictest13889-21ntp_states19:10:201
1263099221,3cyclictest13889-21ntp_states19:10:191
1263099221,16cyclictest0-21swapper/100:19:381
1263099221,11cyclictest0-21swapper/100:06:161
1262399222,11cyclictest30745-21/usr/sbin/munin20:00:170
1262399221,6cyclictest0-21swapper/023:47:130
1262399221,11cyclictest31362-21/usr/sbin/munin21:15:140
1263599216,3cyclictest22561-21/usr/sbin/munin20:55:142
1263599214,9cyclictest87550irq/128-enp1s0-00:20:152
1263599214,7cyclictest32717-21latency_hist20:05:012
1263599212,12cyclictest12870-21egrep23:00:112
12635992112,2cyclictest87350irq/126-enp1s0-23:40:132
12635992112,2cyclictest87350irq/126-enp1s0-19:20:092
12635992110,4cyclictest311ktimersoftd/221:24:362
1263599210,13cyclictest11576-21/usr/sbin/munin20:30:182
1263099218,4cyclictest2879-21cpuspeed_turbos20:10:121
1263099215,13cyclictest15489-21munin-node19:15:031
1263099215,13cyclictest15489-21munin-node19:15:031
1263099211,8cyclictest0-21swapper/122:18:121
1263099211,3cyclictest27023-21hddtemp_smartct19:50:161
1263099211,3cyclictest11715-21taskset21:42:051
1263099211,18cyclictest0-21swapper/121:34:201
1263099211,17cyclictest19924-21fschecks_time23:15:141
1263099211,15cyclictest0-21swapper/119:29:091
12630992111,3cyclictest0-21swapper/120:20:161
1263099211,11cyclictest25214-21ntp_states19:45:211
1263099210,14cyclictest0-21swapper/121:24:361
1262399215,4cyclictest0-21swapper/023:35:260
1262399212,11cyclictest22561-21/usr/sbin/munin20:55:230
1262399211,4cyclictest0-21swapper/000:36:530
1262399211,11cyclictest26347-21/usr/sbin/munin19:50:210
1262399211,11cyclictest12826-21/usr/sbin/munin19:10:230
1262399211,11cyclictest12826-21/usr/sbin/munin19:10:230
12623992110,3cyclictest87550irq/128-enp1s0-00:29:460
251842200,2sleep230-21rcuc/222:15:022
1263599208,2cyclictest87550irq/128-enp1s0-22:50:242
1263599204,8cyclictest16355-21latency_hist21:55:012
1263599204,7cyclictest87350irq/126-enp1s0-19:29:282
1263599203,8cyclictest87650irq/129-enp1s0-20:05:182
1263599203,8cyclictest26347-21/usr/sbin/munin19:50:172
1263599202,9cyclictest27585-21users23:30:262
1263599202,8cyclictest24483-21/usr/sbin/munin23:25:162
1263599201,13cyclictest0-21swapper/221:15:232
1263599200,8cyclictest22813-21chrt00:35:162
1263099209,3cyclictest25931-21ps21:00:231
1263099207,4cyclictest8818-21munin-run00:05:021
1263099207,2cyclictest0-21swapper/123:00:011
12630992012,3cyclictest0-21swapper/119:30:291
1263099201,15cyclictest0-21swapper/119:56:151
1263099201,14cyclictest0-21swapper/121:16:331
1263099201,14cyclictest0-21swapper/121:14:311
1263099201,14cyclictest0-21swapper/120:18:521
1263099201,11cyclictest3600-21ls21:25:121
1262399201,9cyclictest773-21NetworkManager21:07:240
1262399201,9cyclictest4007-21/usr/sbin/munin22:40:200
1262399201,6cyclictest31359-21users22:25:250
12623992013,4cyclictest6085-21grep23:55:250
1262399201,13cyclictest0-21swapper/023:10:200
1262399201,10cyclictest808-21snmpd23:40:030
1262399201,10cyclictest808-21snmpd21:59:430
1262399201,10cyclictest21950-21/usr/sbin/munin19:40:110
1262399200,4cyclictest0-21swapper/021:35:490
1262399200,4cyclictest0-21swapper/021:35:490
9162191,3sleep0938-21fschecks_count20:05:180
294282191,2sleep20-21swapper/222:23:382
1263599195,6cyclictest87550irq/128-enp1s0-21:40:132
1263599193,7cyclictest13257-21ntp_states21:45:212
12635991912,4cyclictest5276-21gltestperf23:55:162
12635991910,2cyclictest87350irq/126-enp1s0-21:10:012
1263099191,3cyclictest11361-21awk20:30:011
12630991912,4cyclictest26332-21df23:30:131
1263099191,17cyclictest0-21swapper/120:55:411
1263099191,16cyclictest0-21swapper/120:40:281
1263099191,11cyclictest0-21swapper/122:30:061
1263099190,8cyclictest0-21swapper/123:06:171
1263099190,14cyclictest0-21swapper/123:51:221
1262399191,8cyclictest21088-21kworker/0:123:18:520
1262399191,6cyclictest11714-21fschecks_time00:10:160
1262399191,6cyclictest0-21swapper/022:03:550
1262399191,4cyclictest22673-21ntp_states23:20:230
1262399191,4cyclictest0-21swapper/000:00:300
1262399191,15cyclictest29790-21/usr/sbin/munin21:10:170
1262399191,10cyclictest28548-21/usr/sbin/munin19:55:150
12623991910,3cyclictest8987-21gltestperf22:50:150
25682181,3sleep20-21swapper/220:10:022
1263599185,4cyclictest32530-21grep22:30:162
1263599181,11cyclictest0-21swapper/220:19:252
1263599181,10cyclictest0-21swapper/219:30:222
1263599181,10cyclictest0-21swapper/200:19:362
1263099189,2cyclictest17261-21hddtemp_smartct21:55:171
1263099182,13cyclictest18290-21cpuspeed_turbos20:45:131
1263099181,3cyclictest13183-21vmstat20:30:271
1263099181,16cyclictest0-21swapper/120:01:241
1262399186,3cyclictest28939-21diskmemload23:54:360
1262399186,3cyclictest10270-21cat20:25:170
1262399183,14cyclictest0-21swapper/000:25:010
1262399182,7cyclictest28018-21latency_hist23:35:010
1262399181,6cyclictest0-21swapper/022:31:240
1262399181,6cyclictest0-21swapper/021:28:460
1262399181,4cyclictest0-21swapper/021:42:540
12623991811,4cyclictest827-21runrttasks22:15:250
1262399181,10cyclictest0-21swapper/020:34:270
27832170,14sleep02843-21cpu20:10:110
215112171,2sleep20-21swapper/219:37:232
154292170,3sleep00-21swapper/020:36:040
1263099171,15cyclictest0-21swapper/119:20:291
1262399174,5cyclictest7063-21runrttasks21:30:350
1262399172,12cyclictest4982-21/usr/sbin/munin20:15:200
1262399172,11cyclictest0-21swapper/019:35:250
1262399171,9cyclictest0-21swapper/021:20:180
1262399171,5cyclictest0-21swapper/000:05:510
1262399171,2cyclictest0-21swapper/022:07:090
12623991711,3cyclictest827-21runrttasks23:05:190
12623991710,4cyclictest20644-21gltestperf00:30:170
1263599161,8cyclictest0-21swapper/220:22:232
1263599161,6cyclictest0-21swapper/219:10:182
1263599161,6cyclictest0-21swapper/219:10:182
1263599161,2cyclictest28548-21/usr/sbin/munin19:55:182
1263599161,14cyclictest0-21swapper/219:49:412
1263599160,2cyclictest89250irq/135-enp2s0-20:50:132
1262399166,3cyclictest10601-21/usr/sbin/munin22:55:120
1262399161,5cyclictest24286-21cpuspeed_turbos19:45:110
1262399161,5cyclictest0-21swapper/021:53:380
12623991612,3cyclictest0-21swapper/019:23:480
12623991611,3cyclictest19666-21cut19:30:120
1262399161,12cyclictest18172-21/usr/sbin/munin20:45:190
12623991610,3cyclictest8036-21sort20:20:180
12623991610,3cyclictest24761-21/usr/sbin/munin21:00:180
185982150,2sleep00-21swapper/019:26:060
12635991510,3cyclictest9717-21df_inode20:25:142
1262399140,8cyclictest0-21swapper/019:18:290
1262399140,8cyclictest0-21swapper/019:18:280
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional