You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-23 - 14:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 23, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
43002232160,59sleep00-21swapper/007:07:290
41932203166,24sleep30-21swapper/307:06:073
41972202164,25sleep20-21swapper/207:06:102
41612202166,23sleep10-21swapper/107:05:411
67221800,3sleep1458799cyclictest09:25:151
4593991330,116cyclictest0-21swapper/209:09:472
4593991171,114cyclictest0-21swapper/210:01:482
4593991171,114cyclictest0-21swapper/210:01:482
299822520,2sleep30-21swapper/310:30:223
62622510,7sleep232-21ksoftirqd/208:25:122
70472480,2sleep30-21swapper/309:40:143
460099432,35cyclictest3288-40dnf11:59:263
4593994340,2cyclictest0-21swapper/212:33:342
460099423,33cyclictest21021-40dnf08:57:243
460099421,30cyclictest28006-21latency11:40:183
167902410,1sleep10-21swapper/112:30:121
460099376,17cyclictest808-21snmpd10:08:283
460099373,22cyclictest87350irq/126-enp1s0-12:15:203
460099371,28cyclictest5794-21/usr/sbin/munin12:05:143
460099371,27cyclictest5362-21fschecks_time07:10:143
460099371,27cyclictest24351-21hddtemp_smartct09:05:163
460099371,18cyclictest27218-21dump-pmu-power08:00:013
460099361,25cyclictest24914-21latency_hist11:35:023
460099361,25cyclictest24102-21latency_hist10:20:023
460099358,19cyclictest808-21snmpd12:03:343
460099356,16cyclictest31542-21/usr/sbin/munin11:50:263
460099353,21cyclictest87550irq/128-enp1s0-08:05:263
460099351,26cyclictest808-21snmpd09:19:283
458399358,13cyclictest11-21rcuc/011:50:010
4583993510,12cyclictest9642-21munin-run07:20:010
460099340,24cyclictest0-21swapper/307:45:253
460099339,16cyclictest16455-21/usr/sbin/munin07:35:233
460099332,3cyclictest12050irq/123-ahci[0008:50:013
460099331,3cyclictest87350irq/126-enp1s0-09:55:143
460099330,24cyclictest0-21swapper/307:40:133
460099330,24cyclictest0-21swapper/307:40:133
458399338,12cyclictest4278-21munin-run09:35:020
460099325,15cyclictest12378-21/usr/sbin/munin12:20:193
460099325,15cyclictest12378-21/usr/sbin/munin12:20:183
460099321,5cyclictest0-21swapper/310:25:213
460099321,27cyclictest808-21snmpd09:23:033
458399327,11cyclictest0-21swapper/008:10:010
460099311,6cyclictest0-21swapper/311:25:163
460099311,4cyclictest27131-21smart_sda09:10:213
460099311,26cyclictest14609-21diskstats07:30:143
460099311,22cyclictest16246-21sshd12:26:173
458799311,27cyclictest21196-21dnf07:45:121
458799311,27cyclictest10537-21diskstats12:15:151
460099306,21cyclictest3353-21ntpq09:30:213
460099304,14cyclictest23563-21cut07:50:153
460099302,5cyclictest7094-21diskstats07:15:033
460099302,25cyclictest9202-40dnf10:58:273
460099301,6cyclictest26708-21sshd11:37:263
460099301,3cyclictest20979-21ntp_states10:10:203
460099301,25cyclictest12410-21diskstats07:25:143
460099301,22cyclictest808-21snmpd12:38:413
4600993012,15cyclictest2359-21vmstat10:40:253
460099301,17cyclictest8119-21cpuspeed_turbos12:10:113
460099300,5cyclictest0-21swapper/308:10:143
460099300,5cyclictest0-21swapper/308:00:243
458799306,7cyclictest0-21swapper/110:47:461
460099296,20cyclictest11896-21interrupts09:50:183
460099296,12cyclictest87350irq/126-enp1s0-08:15:143
460099294,13cyclictest87350irq/126-enp1s0-09:35:153
460099293,15cyclictest87350irq/126-enp1s0-08:50:143
460099292,24cyclictest7282-21ls10:55:003
460099291,7cyclictest1565-21unixbench-2d09:25:243
460099291,5cyclictest0-21swapper/310:03:023
460099291,5cyclictest0-21swapper/310:03:023
460099291,17cyclictest30714-21timerandwakeup11:45:263
460099291,13cyclictest5912-21latency_hist08:25:013
460099290,3cyclictest87350irq/126-enp1s0-08:40:123
458799291,25cyclictest25644-21diskmemload09:19:191
458799290,18cyclictest0-21swapper/107:59:591
460099287,18cyclictest11773-21/usr/sbin/munin11:05:253
460099284,20cyclictest10155-21grep11:00:153
460099284,13cyclictest87350irq/126-enp1s0-08:35:173
460099283,21cyclictest8951-21cut08:30:143
460099281,6cyclictest0-21swapper/311:23:523
460099281,6cyclictest0-21swapper/310:45:143
460099281,5cyclictest25644-21diskmemload10:20:103
460099281,4cyclictest9954-21ntp_offset09:45:213
460099281,3cyclictest7624-21/usr/sbin/munin08:25:253
460099281,3cyclictest16901-21/usr/sbin/munin11:15:173
460099281,3cyclictest16901-21/usr/sbin/munin11:15:173
460099281,3cyclictest15032-21/usr/sbin/munin11:10:203
4600992813,12cyclictest21578-21tr09:00:123
459399288,11cyclictest0-21swapper/212:10:202
458799288,3cyclictest21243-21munin-run09:00:011
458799281,12cyclictest8070-21awk08:30:011
458799281,11cyclictest13967-21users12:20:251
458799281,11cyclictest13967-21users12:20:251
460099271,3cyclictest30927-21/usr/sbin/munin10:35:263
460099271,3cyclictest10573-21/usr/sbin/munin07:20:163
460099271,2cyclictest16774-21/usr/sbin/munin12:30:203
4593992724,2cyclictest0-21swapper/208:45:372
458399270,20cyclictest0-21swapper/010:14:250
459399261,10cyclictest0-21swapper/209:21:572
458799261,2cyclictest0-21swapper/112:06:081
458799261,2cyclictest0-21swapper/111:12:331
458799261,2cyclictest0-21swapper/107:29:441
458799261,14cyclictest0-21swapper/110:25:061
458799260,2cyclictest0-21swapper/111:17:001
458799260,2cyclictest0-21swapper/111:17:001
458399264,15cyclictest0-21swapper/009:57:240
13072260,1sleep00-21swapper/009:25:220
4593992512,3cyclictest87650irq/129-enp1s0-09:55:252
458799252,3cyclictest29799-21ntp_states10:30:201
458799252,3cyclictest10628-21ntp_states11:00:211
458799251,4cyclictest19040-21latency_hist08:55:011
458799251,2cyclictest11831-21crond07:25:011
458399253,14cyclictest0-21swapper/012:09:570
459399243,9cyclictest32447-21gltestperf08:10:152
459399241,8cyclictest0-21swapper/208:57:242
459399241,12cyclictest6799-21munin-node07:15:022
458799243,2cyclictest87650irq/129-enp1s0-10:10:151
458799241,3cyclictest8430-21ntp_states10:55:211
458799241,3cyclictest6508-21cut09:40:011
458799241,3cyclictest20756-21cat07:45:011
458799241,3cyclictest20756-21cat07:45:011
458799241,11cyclictest4350-21sed09:35:021
458799241,11cyclictest30914-21smart_sda08:05:241
458799241,11cyclictest24625-21expr10:20:121
458799240,2cyclictest0-21swapper/111:44:221
458799240,14cyclictest0-21swapper/110:41:511
4583992418,3cyclictest808-21snmpd10:53:210
458399241,18cyclictest0-21swapper/012:31:420
458399241,18cyclictest0-21swapper/012:01:300
458399241,17cyclictest0-21swapper/011:38:470
458399241,16cyclictest808-21snmpd07:42:430
458399241,16cyclictest808-21snmpd07:42:430
458399241,16cyclictest0-21swapper/009:51:200
458399241,14cyclictest909-21sssd_nss10:05:150
231242240,2sleep10-21swapper/107:50:121
459399233,2cyclictest87350irq/126-enp1s0-11:10:172
459399232,7cyclictest30049-21munin-run09:20:012
459399231,9cyclictest434750kworker/u8:212:15:262
459399231,14cyclictest0-21swapper/211:49:292
459399231,10cyclictest11740-21smart_sda08:35:232
458799235,7cyclictest0-21swapper/110:05:321
4587992317,3cyclictest15646-21ntp_states12:25:221
458799231,3cyclictest597-21cat10:40:021
458799231,19cyclictest17031-21grep07:35:141
4587992310,3cyclictest11442-21df_inode09:50:141
458399239,3cyclictest808-21snmpd10:38:180
458399231,18cyclictest0-21swapper/012:22:590
458399231,18cyclictest0-21swapper/012:22:590
458399231,18cyclictest0-21swapper/012:18:370
458399231,18cyclictest0-21swapper/011:59:100
458399231,18cyclictest0-21swapper/010:00:520
458399231,18cyclictest0-21swapper/010:00:520
458399230,21cyclictest29767-21kworker/u8:009:46:150
458399230,18cyclictest0-21swapper/011:08:580
458399230,18cyclictest0-21swapper/010:23:200
459399228,3cyclictest87350irq/126-enp1s0-08:30:112
459399225,9cyclictest14205-1kworker/u9:110:30:382
459399225,5cyclictest9367-21latency_hist11:00:012
459399221,7cyclictest11652-21perl09:50:162
459399221,7cyclictest10068-21ps09:45:232
459399221,3cyclictest0-21swapper/211:35:132
459399221,15cyclictest0-21swapper/212:20:402
459399221,15cyclictest0-21swapper/212:20:402
459399221,12cyclictest28499-21latency_hist10:30:022
459399221,12cyclictest1-21systemd07:56:262
459399221,10cyclictest0-21swapper/211:59:352
459399220,16cyclictest0-21swapper/207:42:562
459399220,16cyclictest0-21swapper/207:42:562
458799228,4cyclictest31077-21irqstats09:20:181
458799227,2cyclictest827-21runrttasks09:46:291
458799227,13cyclictest12354-21kworker/u8:108:40:091
458799221,8cyclictest0-21swapper/111:23:471
458799221,20cyclictest0-21swapper/112:15:011
458799220,14cyclictest0-21swapper/110:03:171
458799220,14cyclictest0-21swapper/110:03:171
4583992217,3cyclictest21779-21kworker/u8:310:29:520
458399221,18cyclictest23332-21fw_packets11:30:140
458399221,17cyclictest0-21swapper/011:50:250
458399221,17cyclictest0-21swapper/011:26:180
458399221,15cyclictest0-21swapper/009:11:410
458399221,10cyclictest8899-21sshd10:55:260
229942220,1sleep00-21swapper/010:15:180
459399215,8cyclictest87350irq/126-enp1s0-10:05:242
459399215,8cyclictest12050irq/123-ahci[0007:40:012
459399215,4cyclictest32531-21needreboot11:50:192
459399215,3cyclictest25505-21kworker/2:010:20:252
459399214,4cyclictest10254-21fschecks_count07:20:142
459399212,6cyclictest23721-21users10:15:262
459399212,10cyclictest16972-21irqstats11:15:172
459399212,10cyclictest16972-21irqstats11:15:172
459399211,7cyclictest5569-21ntp_states09:35:202
459399211,6cyclictest0-21swapper/210:11:052
459399211,5cyclictest0-21swapper/212:06:352
459399211,4cyclictest0-21swapper/209:44:552
459399211,12cyclictest5866-21munin-run08:25:012
459399211,12cyclictest0-21swapper/210:38:042
459399211,11cyclictest30721-21ntp_states08:05:212
459399210,15cyclictest0-21swapper/212:38:012
4587992119,1cyclictest0-21swapper/109:10:121
458799211,7cyclictest0-21swapper/111:46:211
4587992117,2cyclictest1437-21dump-pmu-power08:15:011
458799211,18cyclictest0-21swapper/111:50:561
458799211,17cyclictest0-21swapper/109:40:191
458799211,13cyclictest0-21swapper/110:51:031
458799211,12cyclictest0-21swapper/108:30:401
458799211,11cyclictest4493-21memory12:00:181
4587992110,9cyclictest16850-21awk08:50:011
4587992110,10cyclictest0-21swapper/111:27:191
458399219,3cyclictest1485-21if_enp2s010:40:160
458399217,3cyclictest20112-21grep08:55:170
458399216,9cyclictest0-21swapper/012:12:200
458399212,12cyclictest0-21swapper/011:15:060
458399212,12cyclictest0-21swapper/011:15:050
458399212,12cyclictest0-21swapper/011:00:090
458399211,9cyclictest19036-21hddtemp_smartct11:20:160
458399211,7cyclictest13964-21/usr/sbin/munin11:10:220
4583992116,3cyclictest30267-21/usr/sbin/munin09:20:220
458399211,17cyclictest4528-21grep10:45:270
458399211,17cyclictest19959-21needreboot12:35:190
458399211,13cyclictest0-21swapper/011:40:200
458399210,3cyclictest0-21swapper/009:35:490
458399210,16cyclictest0-21swapper/009:43:310
458399210,10cyclictest0-21swapper/010:30:200
459399206,3cyclictest27267-21timerandwakeup09:10:222
459399204,4cyclictest5513-21diskstats10:50:122
459399204,4cyclictest25644-21diskmemload11:44:202
459399202,8cyclictest21447-21grep07:45:132
459399202,10cyclictest0-21swapper/211:00:122
459399202,10cyclictest0-21swapper/209:29:002
459399201,9cyclictest0-21swapper/208:00:252
459399201,8cyclictest23430-21smartctl11:30:152
459399201,7cyclictest18708-21diskstats11:20:142
459399201,7cyclictest15838-21smart_sda12:25:242
459399201,5cyclictest8842-21ps07:15:232
459399201,4cyclictest0-21swapper/209:33:402
4593992014,3cyclictest21187-21cut11:25:152
459399201,2cyclictest87350irq/126-enp1s0-10:43:532
459399201,12cyclictest0-21swapper/211:09:332
459399201,10cyclictest3652-21latency_hist08:20:022
458799205,3cyclictest0-21swapper/111:35:311
458799201,4cyclictest8589-21memory07:15:201
458799201,4cyclictest22856-21cpuspeed_turbos11:30:101
458799201,4cyclictest20558-21users12:35:261
458799201,18cyclictest0-21swapper/110:15:211
458799201,18cyclictest0-21swapper/109:05:031
458799201,18cyclictest0-21swapper/109:02:551
458799201,11cyclictest11446-21missed_timers08:35:181
458399206,3cyclictest15935-21timerandwakeup12:25:250
458399201,9cyclictest808-21snmpd08:50:280
458399201,17cyclictest0-21swapper/008:00:170
458399201,14cyclictest0-21swapper/009:17:170
458399201,11cyclictest31847-21/usr/sbin/munin08:10:250
459399193,3cyclictest14874-21gltestperf07:30:162
459399191,9cyclictest0-21swapper/208:50:342
459399191,7cyclictest0-21swapper/210:47:352
459399191,6cyclictest21817-21runrttasks09:00:142
459399191,12cyclictest0-21swapper/212:04:092
458799196,4cyclictest21-21rcuc/111:55:151
458799193,3cyclictest87550irq/128-enp1s0-08:15:261
458799191,9cyclictest0-21swapper/108:00:241
4587991913,4cyclictest14743-21taskset09:55:251
458799191,16cyclictest0-21swapper/108:21:431
458799191,16cyclictest0-21swapper/107:12:261
458799190,17cyclictest0-21swapper/111:05:041
458399196,3cyclictest9762-21sed08:30:240
458399196,3cyclictest4592-21idleruntime08:20:170
458399194,3cyclictest15491-21gzip07:30:250
458399191,6cyclictest808-21snmpd07:22:390
458399191,4cyclictest773-21NetworkManager08:36:340
458399191,10cyclictest17903-21sed07:35:240
459399183,4cyclictest13538-21unixbench-2d07:25:252
459399181,8cyclictest0-21swapper/207:52:462
458399185,3cyclictest6287-21expr08:25:120
458399184,7cyclictest755-21rngd07:45:140
458399184,2cyclictest5541-21if_enp2s007:10:160
458399182,9cyclictest23269-21cstates07:50:120
458399181,5cyclictest27224-21munin-run08:00:010
458399181,5cyclictest17033-21cat08:50:010
458399181,5cyclictest12677-21/usr/sbin/munin08:40:250
4583991813,3cyclictest21469-21/usr/sbin/munin09:00:240
458399180,16cyclictest0-21swapper/009:05:270
459399172,3cyclictest12050irq/123-ahci[0008:40:142
458799175,3cyclictest14963-21basename07:30:171
458399172,7cyclictest12751-21/usr/sbin/munin07:25:160
458399170,15cyclictest0-21swapper/008:15:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional