You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-03 - 18:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Jan 03, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
207852236163,25sleep20-21swapper/207:09:072
206552234161,24sleep00-21swapper/007:07:250
208382202166,24sleep10-21swapper/107:09:471
206162201165,24sleep30-21swapper/307:06:543
947121440,4sleep22095499cyclictest07:55:132
2095499110102,2cyclictest0-21swapper/208:34:552
20954999491,2cyclictest0-21swapper/207:31:162
20954997774,2cyclictest0-21swapper/207:23:202
272202670,4sleep02094499cyclictest09:50:020
192612450,1sleep20-21swapper/211:55:242
20959993910,14cyclictest0-21swapper/308:36:003
2094499389,13cyclictest13435-21munin-run08:05:010
145482370,3sleep314531-21latency08:05:173
2095999364,22cyclictest751-21rngd08:45:263
20954993613,3cyclictest16467-21/usr/sbin/munin09:25:202
2095999341,24cyclictest0-21swapper/309:18:063
20954993411,3cyclictest13659-21/usr/sbin/munin08:05:242
20944993410,11cyclictest6503-21idleruntime-cro11:30:020
167722340,2sleep00-21swapper/009:25:130
20954993310,4cyclictest23108-21crond07:15:012
20954993310,4cyclictest0-21swapper/210:10:122
2095999321,12cyclictest0-21swapper/307:18:063
2095499329,3cyclictest1130-210anacron10:01:012
2095499321,7cyclictest7465-21/usr/sbin/munin10:15:242
20954993212,3cyclictest31400-21cut11:10:212
20954993211,3cyclictest21398-21sort10:50:022
20954993211,3cyclictest21398-21sort10:50:012
20954993211,3cyclictest11111-21/usr/sbin/munin11:40:242
20954993210,3cyclictest23058-21/usr/sbin/munin09:40:162
2094499327,12cyclictest26003-21munin-run07:20:010
2095999311,12cyclictest0-21swapper/307:24:383
2095499312,11cyclictest30057-21kworker/2:009:46:332
20954993112,4cyclictest21798-21grep08:20:262
20954993112,4cyclictest21798-21grep08:20:262
20954993111,3cyclictest6190-21cat07:45:242
20954993111,3cyclictest21334-21cut09:35:162
20954993111,3cyclictest21334-21cut09:35:162
20954993111,3cyclictest15151-21expr10:35:112
20954993110,3cyclictest9181-21cut07:55:012
2095999301,13cyclictest0-21swapper/312:19:503
2095999301,12cyclictest0-21swapper/309:50:293
2095999301,12cyclictest0-21swapper/308:12:453
2095499306,20cyclictest29298-21gltestperf12:20:142
2095499301,7cyclictest16195-21hwlatdetect11:50:162
20954993011,3cyclictest23260-21munin-node07:15:032
20954993010,3cyclictest31377-21tr08:45:132
2095499300,12cyclictest30057-21kworker/2:009:50:232
2095999295,10cyclictest18661-21/usr/sbin/munin09:30:203
2095999292,10cyclictest0-21swapper/311:15:183
2095999291,9cyclictest0-21swapper/308:55:343
2095999291,13cyclictest0-21swapper/311:42:453
2095999291,13cyclictest0-21swapper/309:03:263
2095999291,11cyclictest0-21swapper/307:50:453
2095999291,10cyclictest0-21swapper/307:46:453
2095999290,27cyclictest0-21swapper/310:09:373
2095499292,22cyclictest30057-21kworker/2:010:54:252
2095499291,6cyclictest7501-21cat09:05:012
2095499291,4cyclictest0-21swapper/212:01:462
2095499291,4cyclictest0-21swapper/212:01:452
20954992911,3cyclictest31222-21ls12:25:142
20954992911,3cyclictest22189-21cat12:05:132
2095499290,15cyclictest30057-21kworker/2:011:45:252
2094999293,14cyclictest9000-40dnf07:54:581
2095999282,15cyclictest30625-21/usr/sbin/munin07:30:133
2095999281,19cyclictest18761-21ntp_states11:55:193
2095999281,14cyclictest0-21swapper/309:11:363
2095999281,13cyclictest0-21swapper/308:43:343
2095999281,11cyclictest0-21swapper/310:53:343
2095999281,11cyclictest0-21swapper/310:30:223
2095999281,11cyclictest0-21swapper/307:29:443
2095999280,22cyclictest0-21swapper/309:22:463
2095999280,12cyclictest0-21swapper/308:33:583
2095499289,3cyclictest26896-21cut12:15:172
2095499282,10cyclictest7334-21gltestperf11:30:162
2095499281,2cyclictest87250irq/129-enp1s0-12:31:222
20954992811,3cyclictest23128-21sh08:25:182
2094999280,21cyclictest0-21swapper/111:06:171
2095999276,11cyclictest16467-21/usr/sbin/munin09:25:163
2095999271,9cyclictest0-21swapper/310:18:533
2095999271,6cyclictest0-21swapper/310:12:213
2095999271,11cyclictest0-21swapper/312:14:133
2095999271,11cyclictest0-21swapper/311:05:223
2095999271,11cyclictest0-21swapper/307:43:573
2095999270,13cyclictest0-21swapper/310:35:023
2095999270,10cyclictest0-21swapper/311:25:303
2095499276,3cyclictest577-21systemd-journal11:05:012
2095499274,3cyclictest0-21swapper/210:58:532
2095499274,2cyclictest86950irq/126-enp1s0-09:21:522
2095499271,5cyclictest9883-21/usr/sbin/munin09:10:262
20954992712,3cyclictest0-21swapper/212:37:082
2095499271,15cyclictest30-21rcuc/212:13:522
20954992710,3cyclictest2001-21expr10:05:142
2095999267,3cyclictest0-21swapper/309:49:333
2095999265,13cyclictest15502-21/usr/sbin/munin11:50:163
2095999262,11cyclictest31357-21kworker/3:012:08:313
2095999261,12cyclictest0-21swapper/312:36:143
2095999261,11cyclictest0-21swapper/312:27:063
2095999261,11cyclictest0-21swapper/309:43:273
20959992610,4cyclictest751-21rngd09:05:103
2095999260,9cyclictest0-21swapper/311:03:503
2095499263,2cyclictest88850irq/135-enp2s0-07:40:172
2095499261,8cyclictest30067-21awk11:10:012
20954992610,3cyclictest9134-21expr11:35:132
20954992610,3cyclictest27657-21kernelversion08:35:172
2095499260,3cyclictest0-21swapper/209:34:492
2095499260,14cyclictest0-21swapper/208:04:342
2094499261,12cyclictest2438-21munin-run07:40:020
2095999254,5cyclictest0-21swapper/312:30:193
2095999251,11cyclictest0-21swapper/311:39:413
2095999251,10cyclictest0-21swapper/312:21:103
2095999251,10cyclictest0-21swapper/311:32:143
2095999251,10cyclictest0-21swapper/311:22:543
2095999251,10cyclictest0-21swapper/310:45:503
2095999251,10cyclictest0-21swapper/310:45:503
2095999251,10cyclictest0-21swapper/308:29:093
2095999251,10cyclictest0-21swapper/307:56:213
2095999250,9cyclictest0-21swapper/307:11:013
2095999250,10cyclictest5352-21kworker/3:210:29:173
2095499254,4cyclictest10549-21cpu10:25:142
20954992512,3cyclictest0-21swapper/209:08:192
2095999245,9cyclictest13473-21cpuspeed_turbos11:45:123
2095999244,2cyclictest5352-21kworker/3:210:22:453
2095999242,9cyclictest0-21swapper/308:53:103
2095999241,13cyclictest0-21swapper/308:19:173
2095999241,10cyclictest0-21swapper/307:36:453
2095999240,9cyclictest0-21swapper/310:59:103
2095999240,11cyclictest0-21swapper/309:38:453
2095999240,11cyclictest0-21swapper/309:38:453
20954992410,3cyclictest30057-21kworker/2:011:28:432
2094999242,11cyclictest25035-21munin-run09:45:021
2094999241,16cyclictest15614-21timerandwakeup09:20:241
38902230,5sleep23894-21unixbench_singl11:20:242
2095999234,8cyclictest31348-21sh11:10:213
2095999231,9cyclictest0-21swapper/308:20:093
2095999231,9cyclictest0-21swapper/308:20:093
2095999231,11cyclictest0-21swapper/308:02:013
2095499236,2cyclictest30057-21kworker/2:008:10:192
2094999236,10cyclictest0-21swapper/112:00:011
2094999231,16cyclictest0-21swapper/110:11:101
2094999231,16cyclictest0-21swapper/108:45:251
2094999231,16cyclictest0-21swapper/107:45:061
2094999231,15cyclictest30626-21chrt12:24:281
2094999231,12cyclictest0-21swapper/112:15:161
2094999231,12cyclictest0-21swapper/110:21:491
2095999223,7cyclictest659-21sendmail_mailtr10:00:253
2095999222,2cyclictest0-21swapper/310:40:183
2095999220,9cyclictest0-21swapper/312:02:213
2095999220,9cyclictest0-21swapper/312:02:213
2095499221,2cyclictest87050irq/127-enp1s0-07:35:112
2094999228,8cyclictest0-21swapper/112:00:181
2094999228,8cyclictest0-21swapper/112:00:181
2094999228,12cyclictest1783-21kworker/u8:211:36:191
2094999227,7cyclictest801-21snmpd11:00:161
20949992215,4cyclictest12805-21tr10:30:131
2094999221,2cyclictest3000-21smartctl11:20:151
2094999221,18cyclictest18896-21expr09:30:121
2094999221,15cyclictest0-21swapper/108:59:561
2094999221,12cyclictest1468-21idleruntime-cro10:05:021
2094999220,16cyclictest0-21swapper/107:40:051
2094499221,6cyclictest20816-21memory12:00:190
2094499221,6cyclictest20816-21memory12:00:190
20944992215,4cyclictest27511-21tail08:35:140
2095499213,15cyclictest30057-21kworker/2:009:15:172
2095499211,10cyclictest30057-21kworker/2:009:58:192
2094999216,4cyclictest0-21swapper/112:14:021
2094999216,3cyclictest0-21swapper/110:15:251
2094999216,3cyclictest0-21swapper/107:16:571
2094999212,4cyclictest25352-40dnf10:56:211
2094999212,11cyclictest4205-21smart_sda12:35:241
20949992114,4cyclictest12596-21grep08:00:231
2094999211,3cyclictest17830-21perf08:15:011
2094999211,18cyclictest0-21swapper/110:05:191
2094999211,15cyclictest0-21swapper/108:54:581
2094999211,15cyclictest0-21swapper/107:29:111
2094999211,14cyclictest0-21swapper/110:46:311
2094999211,14cyclictest0-21swapper/110:46:311
2094999211,14cyclictest0-21swapper/108:21:171
2094999211,14cyclictest0-21swapper/108:21:171
2094999211,11cyclictest9601-21diskstats07:55:161
20949992110,4cyclictest22165-21hwlatdetect10:50:181
2094499212,11cyclictest26229-21/usr/sbin/munin07:20:120
20944992110,3cyclictest13642-21sshd10:30:190
2095999206,3cyclictest0-21swapper/309:55:493
2095499208,3cyclictest30057-21kworker/2:010:22:362
2095499207,3cyclictest20162-21cat08:20:022
2095499206,2cyclictest30057-21kworker/2:007:28:592
2095499205,2cyclictest30057-21kworker/2:010:44:482
20954992014,3cyclictest1714-21cut08:50:182
2094999206,8cyclictest0-21swapper/109:13:541
2094999202,10cyclictest7736-21cpu09:05:091
20949992016,3cyclictest0-21swapper/111:30:441
20949992014,1cyclictest0-21swapper/109:35:161
20949992014,1cyclictest0-21swapper/109:35:151
20949992012,3cyclictest0-21swapper/110:40:051
20949992012,3cyclictest0-21swapper/108:35:341
20949992012,2cyclictest0-21swapper/109:15:211
2094999201,14cyclictest0-21swapper/110:25:271
2094999201,14cyclictest0-21swapper/109:58:131
2094999201,14cyclictest0-21swapper/108:26:181
2094999201,14cyclictest0-21swapper/107:37:231
2094999201,14cyclictest0-21swapper/107:12:031
2094999201,13cyclictest0-21swapper/111:10:101
20949992011,2cyclictest27793-21df09:50:161
2094999201,11cyclictest31174-21expr12:25:131
2094999201,11cyclictest26396-21expr07:20:121
2094999201,11cyclictest16814-21diskstats09:25:131
2094999201,11cyclictest1069-21diskstats12:30:141
2094999201,10cyclictest25064-21fschecks_count08:30:141
20949992010,3cyclictest353-21df_inode11:15:121
2094999200,15cyclictest0-21swapper/109:48:561
2094499205,6cyclictest0-21swapper/010:54:020
2094499201,7cyclictest0-21swapper/010:38:480
2094499201,6cyclictest24321-21kworker/u8:010:56:210
2094499201,11cyclictest22223-21timerandwakeup09:35:270
2094499201,11cyclictest22223-21timerandwakeup09:35:270
2094499201,11cyclictest0-21swapper/010:28:060
2094499200,13cyclictest0-21swapper/011:16:120
2095499194,3cyclictest30057-21kworker/2:008:55:312
2094999192,8cyclictest18051-21/usr/sbin/munin08:15:181
2094999191,8cyclictest0-21swapper/110:36:411
20949991915,3cyclictest0-21swapper/111:44:081
2094999191,3cyclictest0-21swapper/108:08:141
2094999191,10cyclictest15522-21egrep11:50:131
2094499198,3cyclictest9253-21/usr/sbin/munin07:55:200
2094499197,3cyclictest0-21swapper/012:32:360
2094499193,5cyclictest0-21swapper/012:35:250
2094499193,14cyclictest17466-21idleruntime-cro11:55:010
2094499191,9cyclictest25878-21/usr/sbin/munin11:00:130
2094499191,7cyclictest0-21swapper/012:18:020
2094499191,7cyclictest0-21swapper/010:22:030
2094499191,6cyclictest0-21swapper/011:42:330
2094499191,5cyclictest0-21swapper/011:20:450
2094499191,2cyclictest0-21swapper/009:44:450
2094499191,11cyclictest0-21swapper/009:31:110
2094499191,10cyclictest7051-21/usr/sbin/munin07:50:150
2094499191,10cyclictest27724-21expr09:50:160
2094499191,10cyclictest18310-21gunzip10:40:220
2094499191,10cyclictest0-21swapper/012:06:100
2094499191,10cyclictest0-21swapper/009:22:000
2094499190,6cyclictest0-21swapper/010:07:140
2094499190,5cyclictest0-21swapper/011:14:290
322512180,2sleep00-21swapper/007:30:450
2095499184,2cyclictest30057-21kworker/2:010:30:142
2095499181,9cyclictest30057-21kworker/2:011:15:172
2094999181,8cyclictest4672-21expr11:25:111
20949991814,3cyclictest0-21swapper/108:40:101
2094999181,16cyclictest0-21swapper/111:48:001
2094999180,16cyclictest23-21ksoftirqd/109:00:211
2094999180,14cyclictest0-21swapper/107:31:361
2094499186,7cyclictest0-21swapper/011:46:230
2094499186,6cyclictest0-21swapper/009:18:420
2094499186,2cyclictest9647-21diskmemload11:05:270
2094499181,9cyclictest801-21snmpd08:58:010
2094499181,8cyclictest28430-21/usr/sbin/munin07:25:160
20944991814,2cyclictest3308-21head07:40:160
2094499180,7cyclictest0-21swapper/011:30:210
13632181,2sleep00-21swapper/010:03:580
20954991711,3cyclictest29260-21cat08:40:142
2094999172,12cyclictest23178-21ntp_states12:05:221
2094499178,3cyclictest0-21swapper/010:45:040
2094499178,3cyclictest0-21swapper/010:45:040
2094499176,3cyclictest0-21swapper/009:10:340
2094499173,3cyclictest0-21swapper/010:18:010
2094499172,12cyclictest30906-21/usr/sbin/munin12:25:130
2094499172,10cyclictest0-21swapper/009:57:380
2094499171,9cyclictest9647-21diskmemload12:10:190
2094499171,9cyclictest0-21swapper/011:55:500
2094499171,6cyclictest23546-21open_inodes08:25:220
2094499171,5cyclictest788-21NetworkManager08:48:010
2094499171,3cyclictest5286-21sed10:10:240
20944991712,3cyclictest3040-21idleruntime-cro08:55:010
20944991711,3cyclictest9256-21df_inode11:35:150
2094499171,11cyclictest0-21swapper/012:20:370
71942161,2sleep00-21swapper/009:01:500
2094499169,4cyclictest18177-21cpuspeed_turbos08:15:130
2094499169,4cyclictest0-21swapper/008:20:140
2094499169,4cyclictest0-21swapper/008:20:140
2094499161,9cyclictest0-21swapper/009:05:160
2094499161,6cyclictest0-21swapper/007:10:210
2094499161,3cyclictest24644-21/usr/sbin/munin08:30:140
2094499161,2cyclictest0-21swapper/008:05:350
2094499161,12cyclictest4855-21/usr/sbin/munin07:45:170
20944991510,3cyclictest29389-21diskstats08:40:140
20944991510,3cyclictest16676-21cut08:10:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional