You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 14:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Mar 12, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
195672184111,25sleep10-21swapper/107:06:051
197742182105,25sleep00-21swapper/007:08:460
197242181108,23sleep20-21swapper/207:08:072
195542175101,25sleep30-21swapper/307:05:553
1526021350,1sleep30-21swapper/309:25:023
199709910399,3cyclictest0-21swapper/309:29:283
208572690,3sleep188350irq/128-enp1s0-09:35:181
96912540,4sleep09694-21timerwakeupswit07:55:250
175242400,4sleep011-21rcuc/008:15:150
19954993810,13cyclictest2681-21awk10:10:000
1995499358,12cyclictest15869-21idleruntime-cro10:40:010
19970993330,2cyclictest0-21swapper/308:12:053
1995499331,5cyclictest0-21swapper/012:31:120
1995499331,26cyclictest0-21swapper/011:52:440
1995499331,14cyclictest0-21swapper/007:35:220
1995499331,12cyclictest0-21swapper/011:00:000
1995499321,7cyclictest0-21swapper/011:35:420
1995499321,7cyclictest0-21swapper/011:35:420
1995499321,23cyclictest0-21swapper/011:59:280
1995499321,22cyclictest0-21swapper/009:34:280
1995499321,14cyclictest0-21swapper/010:15:520
1995499321,13cyclictest0-21swapper/010:03:000
1995499313,13cyclictest18056-21munin-run10:45:010
1995499311,6cyclictest0-21swapper/009:38:040
1995499311,14cyclictest26708-21processes12:15:220
1995499311,12cyclictest0-21swapper/010:47:580
1995499311,10cyclictest0-21swapper/011:47:180
118452310,8sleep07-21ksoftirqd/010:30:110
1995499301,20cyclictest0-21swapper/008:49:000
1995499301,13cyclictest0-21swapper/012:14:060
1995499301,13cyclictest0-21swapper/010:27:320
1995499301,12cyclictest0-21swapper/009:44:430
1995499291,6cyclictest0-21swapper/010:52:540
1995499291,17cyclictest0-21swapper/008:32:220
1995499291,12cyclictest0-21swapper/009:50:590
1995499291,12cyclictest0-21swapper/009:28:380
1995499291,12cyclictest0-21swapper/007:42:400
1995499291,12cyclictest0-21swapper/007:12:030
1995499291,11cyclictest0-21swapper/012:39:120
1995499291,11cyclictest0-21swapper/012:04:560
1995499291,11cyclictest0-21swapper/008:40:340
1995499291,11cyclictest0-21swapper/007:49:140
1995499290,13cyclictest0-21swapper/011:25:170
19970992814,4cyclictest23296-21gltestperf10:55:163
1995499281,12cyclictest0-21swapper/011:06:000
1995499281,11cyclictest0-21swapper/011:40:240
1995499281,11cyclictest0-21swapper/010:11:160
1995499281,11cyclictest0-21swapper/008:38:220
1997099279,4cyclictest18232-21wc08:15:233
1995499278,3cyclictest0-21swapper/008:55:140
1995499276,4cyclictest0-21swapper/007:31:360
1995499271,9cyclictest0-21swapper/011:23:530
1995499271,13cyclictest0-21swapper/012:06:440
1995499271,12cyclictest0-21swapper/009:01:480
1995499271,11cyclictest0-21swapper/012:25:360
1995499271,11cyclictest0-21swapper/011:00:380
1995499271,11cyclictest0-21swapper/010:21:200
1995499271,11cyclictest0-21swapper/009:55:090
1995499271,11cyclictest0-21swapper/008:04:230
84222260,1sleep30-21swapper/307:55:113
1996699262,6cyclictest24889-21users12:10:252
1996099261,21cyclictest23073-21munin-run12:10:011
1995499261,13cyclictest0-21swapper/011:12:490
1995499261,12cyclictest0-21swapper/007:53:140
1995499261,11cyclictest0-21swapper/011:30:110
1995499261,11cyclictest0-21swapper/009:17:200
1995499261,11cyclictest0-21swapper/009:09:040
1995499261,11cyclictest0-21swapper/007:15:220
1995499261,10cyclictest0-21swapper/009:12:400
1996099251,18cyclictest0-21swapper/112:39:091
1995499259,3cyclictest0-21swapper/009:45:230
1995499251,13cyclictest0-21swapper/012:20:500
1995499251,12cyclictest0-21swapper/008:27:060
199982240,2sleep3401ktimersoftd/312:00:203
1997099248,4cyclictest12439-21munin-run08:05:013
1997099245,3cyclictest7058-21needreboot07:50:193
1996099248,6cyclictest29475-21cat11:10:111
1996099242,13cyclictest24625-21needreboot08:30:171
1995499246,3cyclictest0-21swapper/008:20:180
1995499241,13cyclictest0-21swapper/009:22:180
71872230,4sleep121-21rcuc/110:20:021
1997099238,4cyclictest10864-21idleruntime-cro09:15:023
1997099234,10cyclictest5834-21timerandwakeup09:00:243
1996699234,10cyclictest32173-21ls11:15:152
1996099236,7cyclictest0-21swapper/107:50:191
19960992310,3cyclictest6052-21diskstats11:30:131
1996099230,4cyclictest25901-21kworker/u8:009:50:021
1995499238,3cyclictest0-21swapper/008:14:090
1995499234,2cyclictest0-21swapper/008:50:320
1995499232,3cyclictest0-21swapper/011:17:330
1995499230,11cyclictest0-21swapper/008:08:250
1995499230,11cyclictest0-21swapper/008:08:250
1997099221,8cyclictest31609-21munin-run07:35:023
1996699227,4cyclictest373-21awk10:05:002
1996099226,8cyclictest0-21swapper/112:31:121
1996099224,4cyclictest2303-21needreboot11:20:191
1996099224,10cyclictest30993-21timerandwakeup07:30:231
1996099221,3cyclictest369-21perf10:05:001
1996099221,11cyclictest28222-21cut12:20:151
1996099220,15cyclictest0-21swapper/108:50:121
1995499227,3cyclictest0-21swapper/007:24:240
1995499221,14cyclictest0-21swapper/007:27:340
255172211,2sleep225567-21wc09:45:242
1997099213,10cyclictest0-21swapper/308:20:203
1997099211,13cyclictest0-21swapper/309:40:263
1997099211,12cyclictest0-21swapper/311:25:173
1997099211,10cyclictest0-21swapper/310:17:083
1996699218,3cyclictest14176-21sed09:20:172
1996699211,6cyclictest22124-21munin-plugin-st07:15:012
1996099215,6cyclictest21-21rcuc/107:40:241
1996099213,9cyclictest23-21ksoftirqd/109:33:051
1996099212,10cyclictest823-21snmpd10:48:191
1996099212,10cyclictest28227-21expr08:40:121
1996099211,6cyclictest14178-21runrttasks11:49:031
1996099211,3cyclictest27461-21latency_hist12:20:021
1996099211,11cyclictest9532-21swap07:55:241
1996099211,11cyclictest4326-21/usr/sbin/munin10:10:251
1997099208,3cyclictest0-21swapper/307:22:463
1997099203,9cyclictest0-21swapper/307:25:133
1997099201,9cyclictest0-21swapper/309:18:003
1997099201,8cyclictest0-21swapper/307:14:043
1997099201,6cyclictest0-21swapper/312:30:183
1997099201,11cyclictest0-21swapper/311:10:163
1996699206,4cyclictest1704-21awk12:35:012
1996699204,3cyclictest88450irq/129-enp1s0-10:50:012
1996699203,9cyclictest0-21swapper/212:08:062
1996699203,9cyclictest0-21swapper/210:08:522
1996699202,5cyclictest25617-21crond08:35:012
1996699201,11cyclictest4651-21cstates09:00:132
1996699201,10cyclictest20635-40dnf12:02:062
1996099209,2cyclictest14510-21egrep11:50:111
1996099208,3cyclictest23501-21cat09:40:241
1996099208,3cyclictest12428-21sh08:05:011
1996099207,4cyclictest16812-21kworker/1:207:45:211
1996099205,7cyclictest0-21swapper/109:26:411
1996099202,4cyclictest89750irq/132-enp2s0-10:55:151
19960992016,2cyclictest1428-21perf07:40:011
1996099201,5cyclictest17864-21irqstats08:15:171
1996099201,10cyclictest9427-21cat10:25:021
1996099201,10cyclictest5512-21ntpq09:00:191
1996099201,10cyclictest21701-21rt-features10:50:231
1996099201,10cyclictest20871-21latency_hist12:05:011
1996099201,10cyclictest18654-21latency_hist12:00:021
1996099201,10cyclictest15169-21ls08:10:121
1996099201,10cyclictest0-21swapper/111:25:171
29462191,2sleep22984-21ntpq12:35:202
1997099198,2cyclictest8654-21diskmemload09:45:253
1997099196,2cyclictest479-21unixbench-2d11:15:243
1997099195,3cyclictest0-21swapper/311:50:213
1997099194,3cyclictest88450irq/129-enp1s0-10:10:243
1997099193,7cyclictest28039-21/usr/sbin/munin08:40:153
1997099192,6cyclictest28426-21munin-run09:55:013
1997099191,9cyclictest9489-21/usr/sbin/munin10:25:183
1997099191,9cyclictest0-21swapper/310:40:233
1997099191,6cyclictest24366-21ntp_states12:10:223
19970991915,3cyclictest0-21swapper/311:39:493
19970991915,3cyclictest0-21swapper/311:39:493
1997099191,10cyclictest32691-21sshd10:03:213
1997099191,10cyclictest31831-21/usr/sbin/munin07:35:173
1997099191,10cyclictest1961-21smart_sda10:05:233
1997099191,10cyclictest12294-21/usr/sbin/munin11:45:103
1996699194,3cyclictest88450irq/129-enp1s0-11:10:212
1996699192,4cyclictest29719-21ntp_states09:55:212
1996699191,9cyclictest0-21swapper/210:11:322
1996699191,8cyclictest0-21swapper/209:52:242
1996699191,6cyclictest0-21swapper/210:39:332
1996699191,5cyclictest22984-21unixbench_multi08:25:262
1996699191,5cyclictest1424-21munin-run07:40:012
19966991911,6cyclictest22985-21kworker/u8:112:19:422
1996099197,4cyclictest0-21swapper/109:05:261
1996099194,5cyclictest498-21unixbench-2d11:15:241
1996099194,3cyclictest0-21swapper/111:35:111
1996099194,3cyclictest0-21swapper/111:35:111
1996099192,9cyclictest13886-21acpi10:35:111
1996099191,5cyclictest3086-21irqstats08:55:171
1996099191,16cyclictest0-21swapper/109:15:151
1996099191,11cyclictest0-21swapper/110:28:151
1996099191,10cyclictest9735-21latency09:10:191
1996099191,10cyclictest1568-21latency10:05:181
1996099191,10cyclictest10396-21cstates11:40:131
19960991910,3cyclictest0-21swapper/111:00:171
1996099190,6cyclictest0-21swapper/109:56:531
321992181,14sleep20-21swapper/208:50:022
1997099187,3cyclictest88150irq/126-enp1s0-09:35:023
1997099186,3cyclictest88450irq/129-enp1s0-11:40:123
1997099185,7cyclictest0-21swapper/308:27:573
1997099181,8cyclictest0-21swapper/312:38:383
1997099181,7cyclictest22270-21munin-node11:05:273
1997099181,7cyclictest0-21swapper/312:25:093
1997099181,7cyclictest0-21swapper/312:17:423
1997099181,7cyclictest0-21swapper/308:33:593
1997099181,6cyclictest0-21swapper/310:30:423
1997099181,4cyclictest3220-21sshd11:24:213
19970991813,3cyclictest24616-21grep07:15:243
19970991811,4cyclictest17277-21grep11:55:173
1997099181,12cyclictest0-21swapper/310:47:373
1996699183,4cyclictest823-21snmpd12:28:142
1996699183,3cyclictest17437-21crond09:30:012
1996699182,6cyclictest6397-21df07:50:122
1996699182,3cyclictest26233-21grep07:20:192
1996699182,3cyclictest14654-21cpuspeed_turbos11:50:122
1996699181,9cyclictest0-21swapper/209:31:282
1996699181,7cyclictest0-21swapper/210:57:402
1996699181,6cyclictest22174-21cut09:40:112
1996699181,5cyclictest19026-21awk08:20:012
1996699181,4cyclictest3031-21awk08:55:182
19966991812,3cyclictest1293-21/usr/sbin/munin11:20:172
19966991810,6cyclictest28765-21kworker/u8:108:00:232
1996099186,4cyclictest0-21swapper/109:21:151
1996099183,3cyclictest27627-21ps09:50:221
1996099181,4cyclictest31157-21memory08:45:201
1996099181,4cyclictest16093-21cpu10:40:101
1996099181,4cyclictest14638-21awk08:10:011
1996099181,4cyclictest14638-21awk08:10:011
1996099181,3cyclictest27620-21cstates07:25:131
1996099181,3cyclictest22948-21memory07:15:051
1996099180,12cyclictest0-21swapper/112:25:011
1997099176,3cyclictest88150irq/126-enp1s0-07:46:393
1997099172,5cyclictest19108-1kworker/u9:009:55:273
1997099171,7cyclictest8914-21sshd10:20:393
1997099171,4cyclictest0-21swapper/311:33:063
1997099171,4cyclictest0-21swapper/311:01:113
1997099171,3cyclictest825-21sshd08:49:213
19970991712,3cyclictest6794-21cpuspeed_turbos09:05:123
19970991712,3cyclictest21189-21sort10:50:183
19970991711,3cyclictest833-21cat08:50:163
19970991711,3cyclictest14009-21grep10:35:113
19970991711,3cyclictest12940-21ls08:05:123
19970991711,3cyclictest12940-21ls08:05:123
19970991710,4cyclictest913-21sssd_nss12:20:193
19970991710,4cyclictest2167-21cut07:40:153
19970991710,4cyclictest0-21swapper/308:55:153
1996699174,10cyclictest0-21swapper/207:25:252
1996699173,4cyclictest10231-21cpuspeed_turbos11:40:122
1996699171,9cyclictest0-21swapper/211:57:052
1996699171,9cyclictest0-21swapper/211:30:112
1996699171,9cyclictest0-21swapper/210:18:382
1996699171,7cyclictest0-21swapper/211:26:042
1996699171,7cyclictest0-21swapper/210:30:152
1996699171,7cyclictest0-21swapper/207:42:452
1996699171,6cyclictest8499-21expr07:55:122
1996699171,5cyclictest775-21NetworkManager08:43:072
1996699171,5cyclictest0-21swapper/210:45:022
1996699171,4cyclictest22436-21gltestperf07:15:022
1996699171,4cyclictest0-21swapper/211:49:032
19966991712,3cyclictest27893-21irqstats11:05:172
19966991712,3cyclictest10400-21memory10:25:182
1996699171,14cyclictest0-21swapper/211:35:042
1996699171,14cyclictest0-21swapper/211:35:042
19966991711,3cyclictest9840-21cat09:10:202
19966991711,3cyclictest29659-21dump-pmu-power12:25:002
1996699171,10cyclictest0-21swapper/211:01:062
19966991710,4cyclictest0-21swapper/210:20:152
1996099178,2cyclictest0-21swapper/108:20:141
1996099171,2cyclictest23360-21grep12:10:111
1996099171,15cyclictest0-21swapper/108:37:321
19960991711,3cyclictest20439-21expr07:10:121
1996099171,13cyclictest12033-21diskstats10:30:131
1996099171,12cyclictest21470-21cpu08:25:131
1996099171,10cyclictest0-21swapper/111:05:371
1997099166,3cyclictest0-21swapper/312:05:213
1997099161,13cyclictest401ktimersoftd/309:35:173
1997099160,15cyclictest0-21swapper/308:39:593
1996699164,3cyclictest88450irq/129-enp1s0-09:05:242
1996699163,3cyclictest0-21swapper/209:35:062
1996699161,9cyclictest0-21swapper/207:49:552
1996699161,6cyclictest20822-21df_inode10:50:132
1996699161,6cyclictest11075-21/usr/sbin/munin09:15:202
1996699161,5cyclictest0-21swapper/208:20:192
19966991612,3cyclictest0-21swapper/208:35:402
19966991612,3cyclictest0-21swapper/207:31:292
1996699161,1cyclictest0-21swapper/208:51:542
19966991611,3cyclictest13583-21expr08:05:182
19966991611,3cyclictest13583-21expr08:05:182
1996099161,14cyclictest0-21swapper/107:21:471
1996699149,3cyclictest14870-21cpu08:10:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional