You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-20 - 08:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed May 20, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
45742241162,25sleep00-21swapper/019:08:020
44602238162,24sleep10-21swapper/119:06:351
45542203166,24sleep20-21swapper/219:07:492
45872201164,24sleep30-21swapper/319:08:153
197521470,3sleep2482399cyclictest21:25:532
854921460,4sleep1481899cyclictest00:10:151
4823991101,107cyclictest0-21swapper/219:57:592
171092540,2sleep317113-21if_enp2s023:15:193
118172540,4sleep0481199cyclictest00:15:230
75442470,2sleep339-21rcuc/320:25:233
482399393,22cyclictest87550irq/128-enp1s0-20:15:242
482399374,19cyclictest30499-21missed_timers23:45:202
482399371,32cyclictest26609-21grep23:35:252
482399371,26cyclictest19843-21chrt22:09:252
482399371,26cyclictest19843-21chrt22:09:252
482399371,25cyclictest0-21swapper/222:15:152
4811993711,12cyclictest29311-21munin-run23:45:020
482399361,21cyclictest6701-21latency_hist21:40:022
481199368,14cyclictest8305-21basename20:30:000
482399356,20cyclictest12050irq/123-ahci[0000:15:002
482399352,29cyclictest24959-40dnf19:51:232
4811993510,12cyclictest24914-21munin-run23:35:020
168782350,4sleep1481899cyclictest00:30:021
482399343,21cyclictest28283-21latency_hist21:15:012
481199347,12cyclictest27466-21perf20:00:010
482399335,16cyclictest12050irq/123-ahci[0000:20:022
482399332,27cyclictest22048-21diskstats21:00:132
482399332,26cyclictest32181-21ntp_states22:35:222
482399331,3cyclictest25889-21perf21:10:012
482399331,3cyclictest0-21swapper/222:30:262
482399331,22cyclictest4114-21/usr/sbin/munin20:20:152
482399331,19cyclictest27397-21/usr/sbin/munin22:25:162
482399322,27cyclictest31025-21head20:05:232
482399322,26cyclictest28748-21ntp_states20:00:222
482399322,26cyclictest16369-21smart_sda20:45:232
482399321,6cyclictest9989-21latency21:45:212
482399321,22cyclictest22939-21/usr/sbin/munin23:30:142
482399321,22cyclictest14343-21irqstats21:55:182
482399312,25cyclictest13854-21smart_sda00:20:262
482399311,18cyclictest6425-21ntp_states22:50:252
482399311,18cyclictest16992-21smartctl22:00:272
4823993110,18cyclictest11734-21pluginstate23:05:002
482399310,22cyclictest87350irq/126-enp1s0-20:30:232
481199318,11cyclictest4519-21perf21:35:020
481199311,14cyclictest24295-21munin-run22:20:010
482399305,13cyclictest87550irq/128-enp1s0-00:30:162
482399301,25cyclictest378-21/usr/sbin/munin20:10:192
482399301,17cyclictest11958-21/usr/sbin/munin23:05:182
482399301,15cyclictest0-21swapper/223:25:182
481199304,13cyclictest13305-21munin-run21:55:000
482999299,9cyclictest39-21rcuc/322:25:243
482999291,3cyclictest39-21rcuc/323:20:173
482999291,3cyclictest39-21rcuc/323:20:173
482399295,21cyclictest22129-21ntp_kernel_pll_19:45:212
482399292,24cyclictest909-21sssd_nss20:55:142
482399292,24cyclictest909-21sssd_nss20:55:142
482399292,23cyclictest19948-21ntp_states19:40:212
482399292,23cyclictest11779-21ntp_states20:35:202
482399291,5cyclictest0-21swapper/223:20:062
482399291,5cyclictest0-21swapper/223:20:062
482399291,4cyclictest5891-21/usr/sbin/munin19:10:212
482399291,3cyclictest87550irq/128-enp1s0-19:30:212
482399291,3cyclictest87350irq/126-enp1s0-22:10:042
482399291,3cyclictest31774-21/usr/sbin/munin21:20:242
482399291,3cyclictest15213-21/usr/sbin/munin23:10:192
482399291,25cyclictest909-21sssd_nss23:40:212
482399291,24cyclictest19526-21diskstats00:35:152
482399291,24cyclictest12397-21ntp_states21:50:222
4823992910,16cyclictest7552-21/usr/sbin/munin22:55:162
4823992910,16cyclictest26637-21sort22:25:022
482399290,19cyclictest0-21swapper/221:20:012
481199290,22cyclictest18114-21kworker/0:023:29:050
482399289,17cyclictest6346-21cpu20:25:112
482399289,16cyclictest10354-21expr19:20:112
482399282,23cyclictest763-21dbus-daemon22:40:062
482399282,23cyclictest17975-21hddtemp_smartct20:50:152
482399281,3cyclictest12912-21/usr/sbin/munin20:40:222
482399281,24cyclictest827-21runrttasks23:57:042
482399281,24cyclictest4304-21fschecks_time00:00:162
482399281,23cyclictest9373-21grep19:15:252
482399281,23cyclictest6121-21cpuspeed_turbos00:05:172
482399281,23cyclictest2930-21cut21:30:152
482399281,23cyclictest17087-21cut19:35:162
482399281,23cyclictest16770-21fschecks_count23:15:162
482399281,23cyclictest14876-21cpuspeed_turbos00:25:132
482399281,23cyclictest14280-21basename19:30:012
482399271,3cyclictest7614-21hddtemp_smartct21:40:162
482399271,3cyclictest3147-21/usr/sbin/munin22:45:122
482399271,2cyclictest31730-21/usr/sbin/munin23:50:172
481199279,10cyclictest0-21swapper/021:20:270
481199270,12cyclictest18672-21idleruntime-cro19:40:000
481899267,12cyclictest0-21swapper/122:36:321
481899265,6cyclictest7084-40dnf22:52:081
482999251,12cyclictest1229-21memory21:25:173
481899259,13cyclictest0-21swapper/123:35:221
481899251,16cyclictest0-21swapper/122:42:211
4818992510,13cyclictest12327-21kworker/u8:221:02:041
482999248,3cyclictest5107-21cat22:49:563
482999241,12cyclictest839-21lldpd23:12:113
482999241,11cyclictest5931-21memory19:10:223
482999241,11cyclictest28918-21/usr/sbin/munin22:30:223
482999241,11cyclictest16081-21users19:30:243
481899243,11cyclictest0-21swapper/122:12:251
481899242,9cyclictest0-21swapper/119:45:001
4818992418,3cyclictest20522-21munin-run23:25:011
4818992418,3cyclictest20522-21munin-run23:25:011
481899241,17cyclictest0-21swapper/121:38:451
481899241,14cyclictest0-21swapper/123:15:481
481899240,18cyclictest18028-21kworker/u8:023:28:331
481899240,17cyclictest8012-21fschecks_count22:55:131
481899240,16cyclictest24977-21/usr/sbin/munin21:05:211
481199241,12cyclictest633-21munin-run22:40:010
482999238,3cyclictest29533-21/usr/sbin/munin23:45:133
482999231,3cyclictest0-21swapper/319:25:233
482999231,16cyclictest0-21swapper/323:05:053
482999231,15cyclictest0-21swapper/321:10:023
482999231,10cyclictest1919-21/usr/sbin/munin20:15:273
481899238,9cyclictest0-21swapper/123:53:341
481899235,11cyclictest0-21swapper/122:04:221
481899234,3cyclictest0-21swapper/100:19:161
481899232,12cyclictest21150-21munin-run00:40:021
4818992315,2cyclictest0-21swapper/120:09:571
481899231,16cyclictest0-21swapper/123:14:021
481899231,16cyclictest0-21swapper/122:07:241
481899231,16cyclictest0-21swapper/122:07:241
481899231,16cyclictest0-21swapper/121:11:291
481899231,16cyclictest0-21swapper/119:10:181
482999228,3cyclictest25879-21diskmemload00:39:543
482999222,11cyclictest25484-21diskstats23:35:143
482999221,5cyclictest0-21swapper/300:22:523
4829992215,4cyclictest2020-21ntp_states22:40:213
481899229,11cyclictest12327-21kworker/u8:220:51:311
481899228,4cyclictest0-21swapper/100:31:001
481899227,4cyclictest0-21swapper/123:59:271
481899225,9cyclictest0-21swapper/121:41:441
4818992214,3cyclictest0-21swapper/119:34:161
481899221,16cyclictest0-21swapper/123:04:241
481899221,15cyclictest0-21swapper/123:30:171
481899221,15cyclictest0-21swapper/100:08:231
481899221,11cyclictest0-21swapper/123:42:201
481899221,11cyclictest0-21swapper/100:24:381
481899221,10cyclictest0-21swapper/122:15:251
481899220,16cyclictest0-21swapper/120:56:491
481899220,16cyclictest0-21swapper/120:56:481
481899220,15cyclictest0-21swapper/122:30:031
481899220,15cyclictest0-21swapper/121:45:281
481199221,8cyclictest0-21swapper/023:51:220
482999216,3cyclictest29952-21sshd21:16:023
482999213,2cyclictest0-21swapper/320:10:223
482999212,4cyclictest21812-21ntp_states23:25:223
482999212,2cyclictest87450irq/127-enp1s0-23:42:093
482999211,9cyclictest0-21swapper/322:50:213
482999211,4cyclictest0-21swapper/300:18:303
482999211,2cyclictest20132-21/usr/sbin/munin22:10:113
482999211,11cyclictest21893-21irqstats19:45:173
482999210,2cyclictest16981-21/usr/sbin/munin00:30:243
4818992116,3cyclictest0-21swapper/120:01:321
4818992114,2cyclictest0-21swapper/120:19:241
4818992113,3cyclictest0-21swapper/120:29:171
4818992113,3cyclictest0-21swapper/119:25:031
4818992112,6cyclictest962-21smart_sda20:10:251
481899211,17cyclictest0-21swapper/122:45:141
481899211,14cyclictest0-21swapper/120:36:261
4818992111,2cyclictest0-21swapper/121:59:211
481899211,11cyclictest17033-21df_inode19:35:151
481899211,11cyclictest12599-21gltestperf23:05:141
481899211,10cyclictest30851-21diskstats21:20:151
481899211,10cyclictest0-21swapper/121:30:481
481899210,16cyclictest0-21swapper/121:53:051
481899210,15cyclictest0-21swapper/120:30:151
481199212,11cyclictest8200-21/usr/sbin/munin00:10:260
481199211,5cyclictest2388-1kworker/u9:121:39:010
481199211,11cyclictest0-21swapper/022:52:080
481199211,10cyclictest12702-21perf20:40:010
482999203,9cyclictest0-21swapper/321:32:543
482999201,6cyclictest1-21systemd23:00:013
482999201,5cyclictest0-21swapper/322:15:183
482999201,3cyclictest4824-21kworker/u8:123:02:543
482999201,10cyclictest31738-21gltestperf22:35:153
482999200,10cyclictest0-21swapper/322:02:393
481899207,3cyclictest0-21swapper/122:27:581
481899207,3cyclictest0-21swapper/122:22:541
481899205,10cyclictest0-21swapper/121:26:281
481899201,17cyclictest0-21swapper/119:56:011
481899201,14cyclictest0-21swapper/120:41:581
481899201,14cyclictest0-21swapper/119:52:141
481899201,14cyclictest0-21swapper/119:15:091
481899201,12cyclictest0-21swapper/121:15:381
481899201,10cyclictest22434-21timerandwakeup19:45:241
481899201,10cyclictest0-21swapper/123:47:061
4818992010,3cyclictest3773-21/usr/sbin/munin00:00:141
481199209,3cyclictest21101-21cpu19:45:110
481199201,4cyclictest0-21swapper/023:55:030
481199201,4cyclictest0-21swapper/021:47:180
481199201,10cyclictest482450kworker/u8:123:06:380
4811992010,3cyclictest87550irq/128-enp1s0-00:30:000
482999196,2cyclictest39-21rcuc/320:40:113
482999194,8cyclictest0-21swapper/322:24:283
482999193,3cyclictest21381-21sshd20:58:583
482999193,3cyclictest21381-21sshd20:58:583
482999191,9cyclictest20148-21smart_sda19:40:233
482999191,4cyclictest0-21swapper/300:06:403
482999191,3cyclictest5413-21hddtemp_smartct21:35:153
482999191,3cyclictest0-21swapper/321:57:433
482999191,15cyclictest9474-21grep19:15:263
482999191,10cyclictest24870-21users19:50:263
482999191,10cyclictest15365-21perl00:25:153
482999191,10cyclictest11678-21df_inode21:50:143
482999191,10cyclictest0-21swapper/321:25:013
4818991912,2cyclictest0-21swapper/120:21:021
481899190,14cyclictest0-21swapper/120:48:391
481899190,14cyclictest0-21swapper/119:21:261
481199197,4cyclictest235502expr19:50:130
481199194,7cyclictest755-21rngd21:55:200
481199191,6cyclictest0-21swapper/021:13:040
481199191,10cyclictest19178-21/usr/sbin/munin00:35:170
110032191,2sleep30-21swapper/321:48:553
482999188,3cyclictest0-21swapper/300:00:083
482999186,3cyclictest87650irq/129-enp1s0-21:40:033
482999186,3cyclictest0-21swapper/320:49:403
482999185,3cyclictest0-21swapper/323:32:083
482999183,4cyclictest31860-21munin-run20:10:023
482999181,9cyclictest0-21swapper/323:56:063
482999181,5cyclictest31730-21/usr/sbin/munin23:50:233
482999181,4cyclictest0-21swapper/322:09:123
482999181,4cyclictest0-21swapper/322:09:123
4829991812,3cyclictest19186-21sshd20:53:583
482999181,15cyclictest0-21swapper/320:39:093
482999181,14cyclictest4450-21df20:20:133
482999181,10cyclictest0-21swapper/300:12:053
481199188,3cyclictest808-21snmpd20:59:370
481199188,3cyclictest808-21snmpd20:59:370
481199187,3cyclictest808-21snmpd21:44:440
481199186,2cyclictest14201-21kworker/0:119:34:370
481199181,7cyclictest0-21swapper/023:47:370
481199181,7cyclictest0-21swapper/022:02:200
481199181,5cyclictest2601-21taskset20:15:160
481199181,5cyclictest0-21swapper/022:32:240
481199181,4cyclictest0-21swapper/022:45:310
481199181,4cyclictest0-21swapper/022:20:160
481199181,3cyclictest0-21swapper/019:20:210
481199181,15cyclictest0-21swapper/020:31:080
4811991811,4cyclictest0-21swapper/021:05:250
481199181,10cyclictest20431-21sshd23:24:050
481199181,10cyclictest20431-21sshd23:24:050
482999171,14cyclictest0-21swapper/321:02:303
482999171,13cyclictest8518-21/usr/sbin/munin20:30:193
4829991711,3cyclictest827-21runrttasks20:00:303
4829991711,3cyclictest25567-21cpuspeed_turbos19:55:113
4829991711,3cyclictest10098-21/usr/sbin/munin19:20:233
482999170,11cyclictest0-21swapper/321:05:163
481199175,4cyclictest25879-21diskmemload22:40:360
481199173,6cyclictest21951-21runrttasks22:13:150
481199172,2cyclictest87350irq/126-enp1s0-00:30:130
481199172,12cyclictest18816-21grep20:50:260
481199171,6cyclictest0-21swapper/000:05:070
481199171,6cyclictest0-21swapper/000:02:320
481199171,5cyclictest0-21swapper/000:24:110
481199171,3cyclictest0-21swapper/021:25:560
4811991712,3cyclictest26719-21/usr/sbin/munin22:25:140
481199170,6cyclictest30879-21needreboot20:05:190
179872171,2sleep00-21swapper/023:15:560
4829991612,3cyclictest0-21swapper/319:39:503
481199169,4cyclictest20395-21grep19:40:240
481199162,5cyclictest0-21swapper/023:10:200
481199161,6cyclictest6100-21perf20:25:010
481199161,6cyclictest28297-21tail20:00:150
481199161,6cyclictest0-21swapper/022:55:430
481199161,6cyclictest0-21swapper/021:16:400
481199161,5cyclictest14266-21latency_hist19:30:000
481199161,4cyclictest7907-21/usr/sbin/munin19:15:150
4811991611,3cyclictest6568-21awk19:10:280
4811991611,3cyclictest32147-21ls20:10:130
4811991611,3cyclictest21699-21/usr/sbin/munin21:00:220
481199161,12cyclictest17936-21/usr/sbin/munin22:05:150
481199161,12cyclictest17936-21/usr/sbin/munin22:05:150
4811991610,3cyclictest9750-21/usr/sbin/munin23:00:220
4811991610,3cyclictest15110-21/usr/sbin/munin20:45:160
481199160,10cyclictest0-21swapper/023:35:270
481199159,3cyclictest14876-21latency_hist20:45:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional