You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 15:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Sep 17, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
138802240162,24sleep00-21swapper/007:06:380
140912239163,25sleep30-21swapper/307:09:233
141422229153,24sleep20-21swapper/207:09:582
121822207168,26sleep10-21swapper/107:05:101
14246999482,7cyclictest0-21swapper/211:35:472
150112620,3sleep10-21swapper/107:10:161
261902550,1sleep011-21rcuc/011:15:150
174242490,4sleep20-21swapper/209:40:232
174242490,4sleep20-21swapper/209:40:232
14246994946,2cyclictest0-21swapper/212:25:132
1425199391,12cyclictest0-21swapper/311:54:003
1425199391,12cyclictest0-21swapper/311:54:003
1425199372,13cyclictest0-21swapper/311:13:243
1425199370,25cyclictest0-21swapper/310:11:143
14251993620,8cyclictest0-21swapper/310:38:323
1425199359,9cyclictest0-21swapper/312:20:423
1425199351,14cyclictest0-21swapper/309:17:403
1425199343,11cyclictest0-21swapper/310:29:403
1424199340,4cyclictest0-21swapper/112:23:531
1423799348,12cyclictest18380-21awk09:45:010
1423799348,12cyclictest18380-21awk09:45:010
1423799346,13cyclictest32242-21munin-run11:30:020
1425199331,13cyclictest27872-21pmu-power08:50:303
1425199325,3cyclictest0-21swapper/311:00:003
1425199321,19cyclictest4256-21ntpq09:10:223
1425199321,12cyclictest0-21swapper/312:35:283
1425199311,23cyclictest0-21swapper/311:43:183
14251993110,11cyclictest0-21swapper/311:05:523
1424199312,12cyclictest18372-21munin-run09:45:011
1424199312,12cyclictest18372-21munin-run09:45:011
14241993115,14cyclictest29353-21kworker/u8:010:30:191
14251993015,3cyclictest22759-40dnf09:55:023
14251993015,10cyclictest0-21swapper/311:00:423
1425199301,13cyclictest0-21swapper/312:04:403
1425199300,13cyclictest0-21swapper/311:23:023
1424199307,3cyclictest836-21runrttasks11:32:461
1424199301,12cyclictest0-21swapper/109:24:421
1425199297,11cyclictest0-21swapper/310:08:323
14251992914,4cyclictest0-21swapper/310:33:373
1425199291,13cyclictest14634-21snmpd10:40:143
1425199291,13cyclictest0-21swapper/310:01:423
1425199291,12cyclictest0-21swapper/309:38:563
1425199290,13cyclictest0-21swapper/308:17:413
134342290,5sleep11424199cyclictest08:20:011
14251992812,10cyclictest0-21swapper/309:21:463
14251992811,4cyclictest30503-21expr11:25:163
1425199281,13cyclictest0-21swapper/312:30:223
1425199281,13cyclictest0-21swapper/310:50:223
1425199281,11cyclictest0-21swapper/308:57:503
1425199281,10cyclictest32142-21strings09:00:223
1425199280,12cyclictest0-21swapper/309:55:263
1425199280,12cyclictest0-21swapper/308:43:583
1425199280,11cyclictest0-21swapper/308:04:383
1424699286,12cyclictest21926-21cat12:20:022
1424699282,12cyclictest32062-21kworker/u8:011:54:412
1424699282,12cyclictest32062-21kworker/u8:011:54:412
1424699282,11cyclictest22298-21grep09:50:282
1423799285,2cyclictest0-21swapper/011:50:090
1423799285,2cyclictest0-21swapper/011:50:090
1423799281,12cyclictest0-21swapper/008:35:360
1423799281,12cyclictest0-21swapper/008:35:360
14251992715,3cyclictest0-21swapper/312:13:143
1425199271,12cyclictest0-21swapper/312:16:303
1425199271,12cyclictest0-21swapper/310:49:023
1425199271,12cyclictest0-21swapper/310:49:013
1425199271,12cyclictest0-21swapper/309:47:263
1425199271,10cyclictest10150-21strings08:10:213
1425199270,13cyclictest0-21swapper/310:15:503
1425199270,12cyclictest0-21swapper/311:35:333
1424699278,8cyclictest16892-21date10:55:012
1424699273,13cyclictest22119-21/usr/sbin/munin12:20:242
14246992714,8cyclictest28675-40dnf08:54:362
1424699271,17cyclictest0-21swapper/208:06:402
1423799271,13cyclictest24089-21munin-run12:25:010
14251992615,3cyclictest0-21swapper/307:28:403
1425199261,21cyclictest0-21swapper/311:30:183
1425199261,15cyclictest21189-21kworker/u8:111:15:143
1425199261,14cyclictest0-21swapper/307:50:263
1425199261,12cyclictest0-21swapper/307:48:463
1425199261,11cyclictest0-21swapper/308:32:383
1425199261,10cyclictest20596-21ntp_states07:20:223
1425199261,10cyclictest0-21swapper/308:46:223
1424699263,14cyclictest784-21NetworkManager12:35:202
1424699261,10cyclictest11916-21cut11:55:192
1424699260,15cyclictest29353-21kworker/u8:010:40:142
1424199260,9cyclictest0-21swapper/110:50:481
256532251,20sleep125846-21grep12:25:261
1425199259,9cyclictest0-21swapper/308:22:503
1425199257,3cyclictest0-21swapper/309:41:443
1425199257,3cyclictest0-21swapper/309:41:443
14251992510,9cyclictest0-21swapper/312:07:563
1424699257,7cyclictest11246-21cat08:15:022
1424699256,9cyclictest0-21swapper/211:16:242
1424699255,9cyclictest10186-21latency_hist10:40:012
1424699255,10cyclictest14351-21ntpq12:00:212
1424699254,10cyclictest0-21swapper/209:23:442
1424699251,13cyclictest28489-21open_files10:05:242
1424699251,11cyclictest1738-21cpuspeed_turbos10:20:152
1424199258,14cyclictest15489-21systemd-journal08:24:381
1424199252,4cyclictest21254-21munin-run11:05:011
14241992520,3cyclictest6737-21munin-run08:05:011
1424199251,6cyclictest0-21swapper/110:00:071
1424199251,4cyclictest0-21swapper/111:14:051
1424199251,4cyclictest0-21swapper/110:55:281
1424199251,14cyclictest0-21swapper/111:25:311
1424199251,10cyclictest0-21swapper/111:50:101
1424199251,10cyclictest0-21swapper/111:50:101
1424199250,4cyclictest0-21swapper/111:35:221
1425199243,3cyclictest0-21swapper/311:48:083
1425199241,13cyclictest0-21swapper/310:20:223
1425199241,11cyclictest0-21swapper/312:29:183
1425199241,11cyclictest0-21swapper/307:18:383
1425199241,10cyclictest0-21swapper/307:56:213
1425199240,12cyclictest0-21swapper/309:09:303
1424699248,9cyclictest0-21swapper/208:41:162
1424699245,10cyclictest28953-21/usr/sbin/munin08:55:232
1424699243,9cyclictest792-21avahi-daemon12:10:222
1424699243,6cyclictest5877-21latency_hist10:30:012
1424699243,5cyclictest21504-21perf07:25:012
1424699242,10cyclictest22130-21cut11:05:192
1424699241,9cyclictest0-21swapper/212:33:122
1424699241,6cyclictest0-21swapper/211:23:452
1424699241,11cyclictest780-21rngd09:25:142
1424699241,11cyclictest0-21swapper/207:51:522
1424699241,10cyclictest0-21swapper/208:35:042
1424699241,10cyclictest0-21swapper/208:35:042
14246992410,11cyclictest11990-21/usr/sbin/munin09:30:302
1424199249,2cyclictest2977-21timerandwakeup10:20:271
1424199242,11cyclictest30667-21ntp_states10:10:221
1424199241,5cyclictest0-21swapper/111:55:181
1424199241,11cyclictest30669-21latency_hist12:40:011
1424199240,5cyclictest0-21swapper/110:39:351
1423799240,11cyclictest0-21swapper/007:40:010
1425199235,8cyclictest0-21swapper/307:36:363
1425199231,6cyclictest0-21swapper/309:27:043
1425199231,11cyclictest0-21swapper/307:10:383
1424699239,3cyclictest0-21swapper/207:10:032
1424699237,7cyclictest25193-21/usr/sbin/munin10:00:162
1424699236,8cyclictest11356-21/usr/sbin/munin08:15:172
1424699236,4cyclictest0-21swapper/210:49:392
1424699236,4cyclictest0-21swapper/210:49:392
1424699233,6cyclictest28320-21/usr/sbin/munin07:40:172
1424699232,6cyclictest0-21swapper/210:12:252
1424699232,5cyclictest0-21swapper/211:42:072
1424699231,7cyclictest0-21swapper/209:14:592
1424699231,6cyclictest0-21swapper/209:56:142
1424699231,14cyclictest0-21swapper/208:20:482
1424199231,4cyclictest0-21swapper/110:18:491
1424199231,19cyclictest18941-21diskstats09:45:151
1424199230,6cyclictest0-21swapper/109:30:471
14237992312,3cyclictest8467-21unixbench_multi08:05:260
1425199228,8cyclictest0-21swapper/311:57:563
1425199228,3cyclictest0-21swapper/309:33:163
1425199220,3cyclictest0-21swapper/308:39:043
1425199220,3cyclictest0-21swapper/308:39:043
1424699227,13cyclictest20553-21crond09:50:012
1424699225,14cyclictest6010-21gltestperf09:15:162
1424699223,7cyclictest20148-21cat11:00:202
1424699223,10cyclictest2969-21diskmemload10:19:102
1424699221,9cyclictest0-21swapper/207:55:142
1424699221,6cyclictest0-21swapper/210:55:072
1424699221,6cyclictest0-21swapper/208:04:342
1424699221,13cyclictest31502-21chrt09:00:162
14246992211,3cyclictest0-21swapper/211:25:452
1424699221,11cyclictest0-21swapper/211:12:482
1424699221,11cyclictest0-21swapper/207:46:182
1424699221,10cyclictest18761-21kworker/2:107:40:022
1424699221,10cyclictest0-21swapper/207:30:002
1424199226,3cyclictest86850irq/129-enp1s0-07:20:211
1424199225,3cyclictest21863-21ntp_states09:50:241
14241992216,3cyclictest21845-21taskset12:19:391
1424199221,3cyclictest10202-21kworker/u8:211:44:381
1424199221,11cyclictest0-21swapper/112:01:411
1424199221,10cyclictest9790-21/usr/sbin/munin09:25:211
1424199221,10cyclictest26565-21cat08:50:011
1424199220,5cyclictest0-21swapper/110:26:231
1423799225,3cyclictest86750irq/128-enp1s0-10:10:130
1423799222,5cyclictest24751-21ntp_states11:10:230
1423799221,7cyclictest0-21swapper/009:49:350
1423799221,3cyclictest0-21swapper/008:41:320
1423799221,14cyclictest0-21swapper/011:05:110
1423799221,10cyclictest0-21swapper/010:06:030
1425199211,10cyclictest8304-21timerandwakeup08:05:263
1424699215,9cyclictest0-21swapper/208:26:002
1424699212,12cyclictest19604-21chrt08:30:582
1424699211,7cyclictest2969-21diskmemload12:05:212
1424699211,6cyclictest32502-21cpu11:30:142
1424699211,5cyclictest0-21swapper/209:37:422
1424699211,10cyclictest0-21swapper/208:46:262
1424699211,10cyclictest0-21swapper/207:16:402
1424199216,4cyclictest0-21swapper/111:18:381
1424199216,3cyclictest28889-21irqstats11:20:181
1424199214,2cyclictest86750irq/128-enp1s0-09:09:501
1424199212,16cyclictest22126-21dump-pmu-power08:40:011
1424199212,16cyclictest22126-21dump-pmu-power08:40:011
1424199211,3cyclictest0-21swapper/107:50:231
1424199211,2cyclictest15520-21/usr/sbin/munin12:05:171
1424199211,18cyclictest27854-21grep12:30:261
1424199211,17cyclictest30137-21processes08:55:241
1424199211,17cyclictest22397-21memory11:05:221
1424199211,11cyclictest0-21swapper/108:50:171
14241992110,4cyclictest20097-21cat08:35:011
1423799217,9cyclictest0-21swapper/012:31:330
1423799215,3cyclictest23697-21awk07:30:010
1423799212,7cyclictest11356-21/usr/sbin/munin08:15:220
1423799211,8cyclictest28296-21sshd08:50:330
1423799211,8cyclictest21396-21grep09:50:160
1423799211,7cyclictest13609-21strings10:45:200
1423799211,7cyclictest13609-21strings10:45:200
1423799211,6cyclictest0-21swapper/011:36:270
1423799211,6cyclictest0-21swapper/010:35:220
1423799211,5cyclictest29774-21ntp_states12:35:230
14237992111,3cyclictest29068-21head11:20:200
1423799211,12cyclictest0-21swapper/010:21:250
1423799211,11cyclictest31580-21ntp_states07:45:230
1423799211,11cyclictest27120-21sshd10:04:320
14237992110,3cyclictest15933-21grep07:10:270
1425199209,3cyclictest28320-21/usr/sbin/munin07:40:193
1425199207,3cyclictest0-21swapper/307:33:283
1424699208,3cyclictest0-21swapper/211:47:082
1424699201,8cyclictest86550irq/126-enp1s0-09:05:262
1424699201,10cyclictest0-21swapper/207:30:402
1424199209,3cyclictest677-21latency_hist09:05:021
1424199205,5cyclictest29354-21systemd-journal10:09:591
1424199205,4cyclictest0-21swapper/111:45:471
14241992017,2cyclictest0-21swapper/107:34:391
1424199201,6cyclictest4646-21cat09:10:261
1424199201,5cyclictest17326-21/usr/sbin/munin07:15:241
14241992013,4cyclictest14793-21grep09:35:151
1424199201,2cyclictest0-21swapper/108:06:201
1424199201,18cyclictest0-21swapper/112:11:341
1424199201,18cyclictest0-21swapper/110:42:551
1424199201,18cyclictest0-21swapper/109:56:161
1424199201,18cyclictest0-21swapper/108:41:561
1424199201,18cyclictest0-21swapper/107:45:341
1424199201,10cyclictest16938-21kworker/u8:008:25:321
1424199200,19cyclictest0-21swapper/108:13:391
1423799209,3cyclictest15760-21/usr/sbin/munin08:25:220
1423799205,7cyclictest780-21rngd12:25:520
1423799205,5cyclictest0-21swapper/011:30:280
1423799204,3cyclictest86750irq/128-enp1s0-09:30:280
1423799204,2cyclictest86750irq/128-enp1s0-12:08:510
1423799202,9cyclictest19546-21users08:30:250
1423799202,9cyclictest0-21swapper/010:51:330
1423799201,6cyclictest11445-21diskstats11:55:140
1423799201,4cyclictest0-21swapper/010:30:150
14237992014,1cyclictest0-21swapper/007:55:200
1423799201,12cyclictest0-21swapper/010:42:170
1423799201,11cyclictest3832-21/usr/sbin/munin10:25:290
1423799201,10cyclictest20166-21memory11:00:200
1423799200,5cyclictest0-21swapper/009:11:430
1425199195,3cyclictest86550irq/126-enp1s0-08:25:183
1424699191,5cyclictest6030-21/usr/sbin/munin10:30:192
1424199191,6cyclictest0-21swapper/107:28:071
14241991915,3cyclictest0-21swapper/107:40:581
1424199191,17cyclictest0-21swapper/110:48:261
1424199191,17cyclictest0-21swapper/110:48:261
1424199191,17cyclictest0-21swapper/107:35:471
1424199190,2cyclictest0-21swapper/109:15:501
1423799197,2cyclictest0-21swapper/012:17:330
1423799195,3cyclictest0-21swapper/007:54:090
1423799194,3cyclictest11-21rcuc/009:55:220
1423799193,3cyclictest86750irq/128-enp1s0-09:35:110
1423799191,6cyclictest0-21swapper/009:00:170
1423799191,5cyclictest5494-21needreboot11:40:200
1423799191,3cyclictest86750irq/128-enp1s0-09:26:240
1423799191,15cyclictest29555-21gltestperf08:55:160
1423799191,11cyclictest26162-21sshd08:45:320
1423799191,11cyclictest0-21swapper/009:19:340
1423799191,10cyclictest6603-21sshd08:03:320
1423799191,10cyclictest0-21swapper/012:00:590
14241991814,3cyclictest0-21swapper/107:55:091
1423799186,1cyclictest0-21swapper/008:20:180
1423799182,3cyclictest0-21swapper/012:10:150
1423799181,9cyclictest18288-21/usr/sbin/munin10:55:280
1423799181,4cyclictest0-21swapper/009:20:510
1423799181,14cyclictest20789-21smart_sda07:20:240
1423799181,11cyclictest0-21swapper/011:46:590
300352171,2sleep00-21swapper/007:41:550
1423799176,10cyclictest0-21swapper/007:35:010
1423799173,2cyclictest86750irq/128-enp1s0-08:10:210
1423799171,4cyclictest0-21swapper/007:16:080
14237991710,1cyclictest0-21swapper/010:15:230
1423799165,3cyclictest0-21swapper/009:07:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional