You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-22 - 20:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat Aug 22, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
113582237164,24sleep10-21swapper/107:09:421
112472208170,25sleep30-21swapper/307:08:173
113712204166,25sleep20-21swapper/207:09:532
112342200162,26sleep00-21swapper/007:08:040
122862510,2sleep10-21swapper/110:50:151
1148499421,29cyclictest23438-21ntp_offset10:00:223
1148499421,29cyclictest23438-21ntp_offset10:00:213
1147999411,29cyclictest20185-21/usr/sbin/munin09:55:162
1148499401,29cyclictest4688-21ls11:45:193
11467994018,4cyclictest23022-21/usr/sbin/munin08:45:220
1147999398,20cyclictest124432latency_hist12:05:002
1147999393,31cyclictest13223-40dnf09:38:252
11467993813,12cyclictest17151-21munin-run08:35:010
1148499371,22cyclictest29331-21diskstats10:15:143
1147999371,6cyclictest14468-21latency09:40:212
220612360,4sleep123-21ksoftirqd/112:25:181
220612360,4sleep123-21ksoftirqd/112:25:181
1148499360,27cyclictest0-21swapper/310:05:163
1147999361,24cyclictest87650irq/129-enp1s0-10:40:122
1147999361,23cyclictest1655-21cpu11:40:112
1147999361,21cyclictest10532-21/usr/sbin/munin09:30:252
1148499351,31cyclictest14213-21/usr/sbin/munin10:55:123
1147999351,25cyclictest0-21swapper/212:36:502
1147999351,24cyclictest28018-21sendmail_mailqu10:10:222
1147999351,22cyclictest7748-21sshd11:51:562
1148499341,21cyclictest0-21swapper/309:40:213
1148499341,20cyclictest17379-21users09:45:273
1147999345,18cyclictest87650irq/129-enp1s0-09:03:562
1147999341,20cyclictest28246-21latency_hist09:00:022
1147999341,20cyclictest2819-21sshd10:27:562
1147999341,20cyclictest19011-21awk09:50:182
1146799344,14cyclictest15575-21awk09:45:000
1148499331,29cyclictest20759-21tail12:20:263
1148499331,20cyclictest909-21sssd_nss10:50:253
1147999331,21cyclictest26570-21awk10:10:002
1148499321,4cyclictest27745-21/usr/sbin/munin07:45:233
1148499321,3cyclictest87350irq/126-enp1s0-09:15:023
1148499321,3cyclictest8579-21/usr/sbin/munin08:15:163
1148499321,27cyclictest15697-21grep08:30:153
1148499321,20cyclictest3220-21/usr/sbin/munin10:30:173
1148499321,17cyclictest11145-21cat12:00:183
11484993211,18cyclictest28126-21latency_hist09:00:013
1147999321,17cyclictest17661-21gltestperf12:15:152
1146799323,13cyclictest3960-21munin-run08:05:020
1148499319,19cyclictest16348-21unixbench_multi12:10:263
1148499312,26cyclictest9810-21/usr/sbin/munin10:45:213
1148499311,26cyclictest10707-21grep09:30:263
1148499311,17cyclictest87350irq/126-enp1s0-10:20:213
11484993111,17cyclictest18994-21latency_hist12:20:013
11484993110,18cyclictest22794-21ntp_kernel_err08:45:183
1147999311,26cyclictest1326-40dnf11:39:232
1147999311,13cyclictest0-21swapper/208:45:202
1148499309,18cyclictest29506-21pmu-power09:00:213
1148499303,23cyclictest8366-21tr11:55:133
11484993013,14cyclictest27091-21grep07:40:273
11484993013,14cyclictest27091-21grep07:40:273
1148499301,26cyclictest5414-21/usr/sbin/munin10:35:123
1148499301,26cyclictest30571-21/usr/sbin/munin09:05:123
1148499301,17cyclictest23026-21sshd07:33:573
11484993011,16cyclictest30197-21cat07:50:133
1147999304,3cyclictest0-21swapper/211:20:002
1147999304,3cyclictest0-21swapper/211:20:002
1147999301,25cyclictest14560-21diskstats10:55:152
1147999301,21cyclictest22583-21proc_pri12:25:242
1147999301,21cyclictest22583-21proc_pri12:25:242
1147999301,19cyclictest5015-21sshd10:32:552
1147999301,15cyclictest21154-21diskstats11:10:132
1147999301,12cyclictest0-21swapper/209:07:562
1146799305,12cyclictest25594-21munin-run12:35:020
1148499298,17cyclictest24346-21if_enp1s012:30:173
1148499292,23cyclictest20216-21smart_sda07:25:223
1148499292,23cyclictest14956-21awk08:30:003
1148499291,25cyclictest23124-21tr11:15:143
1148499291,25cyclictest23124-21tr11:15:143
1148499291,25cyclictest0-21swapper/310:10:203
1148499291,24cyclictest3029-21uniq09:15:163
1148499291,24cyclictest1213-21sshd11:37:573
1148499291,21cyclictest0-21swapper/307:37:163
11484992911,15cyclictest2306-21expr08:00:133
11484992910,16cyclictest6960-21perl08:10:153
11484992910,16cyclictest11488-21idleruntime08:20:163
1147999299,3cyclictest0-21swapper/208:03:552
1147999299,17cyclictest18743-21timerandwakeup08:35:272
1147999292,22cyclictest13863-21df_inode07:15:022
1147999292,22cyclictest1-21systemd08:30:202
1147999291,5cyclictest909-21sssd_nss11:58:562
1147999291,25cyclictest8101-21/usr/sbin/munin09:25:242
1147999291,25cyclictest26181-21gltestperf07:40:182
1147999291,25cyclictest26181-21gltestperf07:40:182
1147999291,25cyclictest23080-21/usr/sbin/munin10:00:172
1147999291,25cyclictest23080-21/usr/sbin/munin10:00:172
1147999291,23cyclictest31016-21ntp_states07:50:212
1147999291,17cyclictest6056-21sshd08:08:552
1147999291,16cyclictest87650irq/129-enp1s0-10:15:152
1147999291,16cyclictest0-21swapper/211:48:432
1147999291,13cyclictest0-21swapper/211:21:162
1147999291,12cyclictest17889-21pmu-power07:20:242
11479992910,16cyclictest12897-21latency_hist08:25:012
1148499288,17cyclictest18545-21cut11:05:013
1148499282,23cyclictest7346-21diskstats09:25:163
1148499282,23cyclictest18636-21smart_sda08:35:253
1148499282,23cyclictest13878-21wc07:15:033
1148499281,4cyclictest16916-21expr07:20:133
1148499281,24cyclictest3708-21cat11:45:023
1148499281,24cyclictest30747-21pmu-power11:30:243
1148499281,24cyclictest14026-21grep07:15:023
1148499281,24cyclictest1001-21/usr/sbin/munin10:25:203
1148499281,23cyclictest4663-21uniq08:05:163
1148499281,23cyclictest26662-21chrt12:35:183
1148499281,23cyclictest25307-21grep11:20:143
1148499281,18cyclictest0-21swapper/309:35:453
11484992811,14cyclictest6510-21cut11:50:163
11484992810,3cyclictest0-21swapper/311:26:363
11484992810,16cyclictest18167-21cstates09:50:123
1147999288,17cyclictest27991-21gltestperf11:25:182
1147999287,18cyclictest20637-21latency_hist11:10:012
1147999284,13cyclictest87350irq/126-enp1s0-12:20:242
1147999282,23cyclictest19808-21grep07:25:182
1147999282,22cyclictest16805-21munin-run12:15:002
1147999282,22cyclictest14951-21munin-run08:30:002
1147999281,4cyclictest0-21swapper/210:45:172
1147999281,24cyclictest8690-21cpuspeed_turbos08:15:112
1147999281,24cyclictest32293-21grep07:55:122
1147999281,24cyclictest305-21/usr/sbin/munin10:20:252
1147999281,24cyclictest22194-21ntp_kernel_pll_07:30:212
1147999281,23cyclictest6547-21pmu-power10:35:222
1147999281,23cyclictest3028-21sort09:15:162
11479992810,15cyclictest23645-21cat07:35:152
1146799280,13cyclictest18748-21idleruntime-cro07:25:010
1148499276,18cyclictest0-21swapper/311:10:183
1148499274,20cyclictest910-21systemd-logind10:40:123
1148499272,22cyclictest5153-21/usr/sbin/munin09:20:153
1148499272,22cyclictest13814-21processes12:05:243
1148499272,21cyclictest19710-21tr08:40:123
1148499271,23cyclictest24110-21grep08:50:123
1148499271,23cyclictest24110-21grep08:50:123
1148499271,23cyclictest20312-21cpuspeed_turbos09:55:143
1148499271,22cyclictest32295-21tr07:55:133
11484992711,3cyclictest21765-21df_inode12:25:153
11484992711,3cyclictest21765-21df_inode12:25:153
1147999279,16cyclictest1055-21grep09:10:162
1147999279,15cyclictest20928-21cat08:40:252
1147999274,19cyclictest13004-21/usr/sbin/munin12:05:152
1147999272,22cyclictest17052-21sed11:00:162
1147999271,23cyclictest5418-21tail09:20:162
1147999271,23cyclictest24315-21diskstats08:50:152
1147999271,23cyclictest24315-21diskstats08:50:152
1147999271,23cyclictest16824-21grep09:45:202
1147999271,23cyclictest15231-21cut07:15:152
1147999271,23cyclictest12830-21/usr/sbin/munin10:50:202
1147999271,22cyclictest30749-21pmu-power11:30:242
1147999271,22cyclictest28382-21grep07:45:152
1146799278,16cyclictest28134-21munin-run09:00:000
1146799274,11cyclictest17766-21munin-run09:50:010
1146799273,16cyclictest14636-21kworker/0:008:28:480
1146799272,14cyclictest16540-21munin-run07:20:010
1146799272,13cyclictest29370-21munin-run11:30:010
1146799272,12cyclictest5809-21munin-run11:50:020
11467992710,4cyclictest31510-21kworker/0:311:35:190
1148499261,21cyclictest18783-21/usr/sbin/munin11:05:133
1147999268,15cyclictest6397-21cpu08:10:112
1147999261,22cyclictest24259-21cut12:30:162
11473992621,3cyclictest0-21swapper/107:10:141
11467992612,2cyclictest24120-21kworker/0:210:04:200
11467992612,2cyclictest24120-21kworker/0:210:04:200
1146799261,13cyclictest19355-21awk08:40:010
1147399259,5cyclictest13223-40dnf09:38:251
1147399255,10cyclictest11428-21gltestperf08:20:161
11473992514,3cyclictest15282-21ntp_states10:55:231
1147399250,14cyclictest0-21swapper/112:01:461
173392240,2sleep00-21swapper/012:15:140
1146799242,10cyclictest31758-21kworker/0:207:59:520
1146799241,11cyclictest16817-21perf12:15:010
28642230,5sleep121-21rcuc/111:40:221
1147399235,2cyclictest87550irq/128-enp1s0-11:13:301
1147399231,12cyclictest8080-21fschecks_count10:40:161
1147399231,11cyclictest0-21swapper/110:14:021
11473992310,4cyclictest18444-21ntp_states08:35:231
1146799238,3cyclictest23575-21kworker/0:109:00:510
1146799237,3cyclictest7874-21kworker/0:011:56:120
1146799236,14cyclictest30969-21awk10:20:020
1146799235,15cyclictest6159-21munin-run08:10:020
1147399222,11cyclictest7333-21df09:25:151
1147399221,4cyclictest15943-21kworker/u8:211:34:161
11473992214,3cyclictest0-21swapper/108:16:101
1147399221,3cyclictest808-21snmpd09:30:331
1147399221,3cyclictest5237-21fschecks_count09:20:141
1147399221,18cyclictest1689-21switchtime09:10:251
1147399221,16cyclictest0-21swapper/112:22:031
1147399221,16cyclictest0-21swapper/112:07:531
1147399221,15cyclictest0-21swapper/110:25:441
1147399221,10cyclictest0-21swapper/109:48:471
11473992210,2cyclictest0-21swapper/108:30:081
1147399220,6cyclictest0-21swapper/112:38:061
1147399220,6cyclictest0-21swapper/112:31:281
1147399220,6cyclictest0-21swapper/109:41:361
1146799228,3cyclictest24967-21kworker/0:107:36:430
1146799222,4cyclictest11791-21munin-run10:50:000
1146799222,12cyclictest667-21df_inode09:10:130
1146799222,10cyclictest19619-21kworker/0:109:56:240
11467992216,3cyclictest18458-21grep09:50:150
1146799221,11cyclictest11005-21kworker/0:109:38:470
11467992210,2cyclictest24967-21kworker/0:107:45:230
39552211,3sleep14170-21df11:45:131
1147399219,3cyclictest32711-21memory11:35:181
1147399218,5cyclictest0-21swapper/110:05:481
1147399212,11cyclictest4517-21switchtime10:30:261
11473992114,3cyclictest0-21swapper/109:07:251
11473992114,2cyclictest0-21swapper/109:51:011
1147399211,3cyclictest20590-21perf11:10:011
1147399211,15cyclictest0-21swapper/112:19:271
1147399211,15cyclictest0-21swapper/111:50:071
1147399211,15cyclictest0-21swapper/110:16:361
1147399211,15cyclictest0-21swapper/109:01:581
1147399211,15cyclictest0-21swapper/108:26:291
1147399211,14cyclictest0-21swapper/111:15:021
1147399211,14cyclictest0-21swapper/111:15:021
1147399211,11cyclictest30616-21hddtemp_smartct07:50:151
1147399211,11cyclictest25101-40dnf07:37:221
1147399211,11cyclictest16096-21smart_sda12:10:241
1147399211,11cyclictest0-21swapper/108:45:221
1147399210,15cyclictest0-21swapper/111:25:201
1146799218,3cyclictest6079-21kworker/0:208:13:360
1146799217,3cyclictest7874-21kworker/0:012:06:240
1146799217,3cyclictest4248-21kworker/0:209:27:030
1146799217,3cyclictest24967-21kworker/0:107:41:190
1146799217,3cyclictest24967-21kworker/0:107:41:190
1146799217,3cyclictest14592-21kworker/0:212:26:480
1146799217,3cyclictest14592-21kworker/0:212:26:480
1146799216,3cyclictest26344-21kworker/0:110:40:120
1146799212,8cyclictest8564-1kworker/u9:110:07:170
1146799211,8cyclictest8639-21kworker/0:108:22:400
1146799211,6cyclictest0-21swapper/010:25:290
1146799211,5cyclictest0-21swapper/011:17:140
1146799211,5cyclictest0-21swapper/011:17:140
1146799211,11cyclictest32537-21diskmemload12:04:560
1146799211,11cyclictest21809-21kworker/u8:010:10:180
11467992110,7cyclictest0-21swapper/011:13:180
1146799210,11cyclictest13880-21kworker/0:211:04:150
1147399202,13cyclictest87650irq/129-enp1s0-10:23:031
1147399201,3cyclictest27955-21sshd08:57:561
11473992012,2cyclictest0-21swapper/111:03:011
1147399201,15cyclictest0-21swapper/108:52:501
1147399201,15cyclictest0-21swapper/108:52:501
1147399201,15cyclictest0-21swapper/107:21:481
1147399201,14cyclictest0-21swapper/110:38:321
1147399201,14cyclictest0-21swapper/109:17:081
1147399201,14cyclictest0-21swapper/107:42:021
1147399201,14cyclictest0-21swapper/107:42:021
1147399201,11cyclictest25725-21grep11:20:151
1147399201,11cyclictest20530-21df_inode09:55:141
1147399201,10cyclictest7295-21memory08:10:191
1147399200,15cyclictest0-21swapper/108:06:111
1147399200,14cyclictest0-21swapper/107:45:021
1146799207,3cyclictest14592-21kworker/0:212:20:360
1146799206,3cyclictest11108-21kworker/0:107:12:030
1146799206,2cyclictest4248-21kworker/0:209:30:350
1146799205,3cyclictest16332-21kworker/0:007:26:230
1146799202,10cyclictest4801-21/usr/sbin/munin09:20:190
1146799201,4cyclictest0-21swapper/010:52:010
1146799201,4cyclictest0-21swapper/009:16:330
1146799201,11cyclictest7287-40dnf10:38:580
1146799201,11cyclictest19571-21/usr/sbin/munin08:40:200
1146799201,11cyclictest0-21swapper/012:35:270
1146799201,11cyclictest0-21swapper/010:55:070
1146799200,5cyclictest8564-1kworker/u9:110:22:480
1147399197,3cyclictest10496-21hwlatdetect10:45:181
1147399191,5cyclictest0-21swapper/110:00:171
1147399191,5cyclictest0-21swapper/110:00:171
1147399191,16cyclictest0-21swapper/108:44:071
1147399191,14cyclictest0-21swapper/111:55:281
1147399191,14cyclictest0-21swapper/108:01:291
1147399191,11cyclictest0-21swapper/107:20:011
1147399191,10cyclictest20933-21munin-run07:30:011
1147399190,14cyclictest0-21swapper/107:33:491
1146799197,2cyclictest25825-21kworker/0:208:54:510
1146799197,2cyclictest25825-21kworker/0:208:54:510
1146799195,3cyclictest24800-21kworker/0:211:43:400
1146799191,10cyclictest25204-21/usr/sbin/munin11:20:160
1146799191,10cyclictest0-21swapper/011:53:450
1146799191,10cyclictest0-21swapper/011:05:350
1146799191,10cyclictest0-21swapper/010:31:130
1146799184,2cyclictest88950irq/132-enp2s0-08:15:200
1146799180,9cyclictest0-21swapper/011:32:020
11473991713,3cyclictest0-21swapper/107:55:171
1146799172,6cyclictest22223-21ntp_states07:30:210
11467991712,3cyclictest31072-21cut09:05:150
1146799161,6cyclictest30285-21df_inode07:50:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional