You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-23 - 00:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jun 22, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
117252236161,25sleep20-21swapper/207:09:422
116152230156,25sleep10-21swapper/107:08:161
116912204166,26sleep00-21swapper/007:09:160
115442203167,24sleep30-21swapper/307:07:213
11850991112,107cyclictest0-21swapper/208:20:332
11850997665,10cyclictest0-21swapper/208:49:002
192222600,3sleep31185699cyclictest08:35:263
320732490,1sleep10-21swapper/110:20:141
225482490,2sleep122549-21sort08:45:151
33072450,4sleep02343-21/usr/sbin/munin08:00:200
1185099389,8cyclictest3977-21awk11:45:012
1185099389,8cyclictest3977-21awk11:45:012
1185099369,16cyclictest12477-21chrt12:01:172
11850993611,11cyclictest432-21diskmemload09:58:082
11850993522,8cyclictest18015-40dnf11:00:252
11850993513,5cyclictest29757-21sa109:00:202
11850993512,13cyclictest7248-21sort11:50:182
1183999359,12cyclictest19385-21idleruntime-cro12:20:000
1185699341,26cyclictest0-21swapper/310:28:373
11850993411,6cyclictest19325-21/usr/sbin/munin07:25:152
1185699331,14cyclictest0-21swapper/310:00:213
1185099339,8cyclictest15422-21tune2fs07:15:152
11850993323,7cyclictest30391-21ntp_kernel_pll_10:15:202
11856993220,7cyclictest0-21swapper/312:29:493
1185099329,14cyclictest29089-21taskset10:14:482
1185099327,16cyclictest20357-21sed09:55:022
11850993211,10cyclictest27551-21basename11:25:012
267222310,5sleep31185699cyclictest07:40:163
1185699311,19cyclictest0-21swapper/309:22:053
1185099319,13cyclictest2349-21sed10:25:192
11850993122,5cyclictest1042-21smart_sda11:35:252
11850993117,5cyclictest29960-21sshd07:48:402
11850993114,9cyclictest5967-21irqstats09:20:172
11850993112,11cyclictest13004-21/usr/sbin/munin12:05:152
11850993110,17cyclictest662-21grep10:20:252
11856993020,3cyclictest0-21swapper/307:59:573
1185699301,21cyclictest0-21swapper/311:59:213
1185099308,8cyclictest17060-21memory09:45:182
1185099308,4cyclictest0-21swapper/208:55:172
1185099308,13cyclictest0-21swapper/209:30:012
1185099307,9cyclictest22740-21/usr/sbin/munin10:00:162
11850993012,7cyclictest19594-21/usr/sbin/munin12:20:122
11850993011,6cyclictest432-21diskmemload11:18:072
11850993011,12cyclictest9562-21users10:40:262
11850993010,11cyclictest0-21swapper/211:35:012
11850993010,10cyclictest24326-21df_inode12:30:132
1185099299,12cyclictest22327-21grep12:25:152
1185099296,7cyclictest0-21swapper/212:15:022
11850992920,5cyclictest17746-21diskstats12:15:142
11850992910,7cyclictest29731-21latency_hist11:30:012
1183999291,13cyclictest28505-21awk09:00:010
90722280,3sleep11184499cyclictest10:40:211
73052280,4sleep339-21rcuc/311:50:183
11856992812,8cyclictest0-21swapper/312:11:133
1185099289,6cyclictest432-21diskmemload09:18:502
1185099288,16cyclictest10167-21gunzip08:15:222
1185099285,9cyclictest0-21swapper/209:34:102
11850992820,5cyclictest9660-21ntp_states11:55:222
11850992819,6cyclictest2072-40dnf07:59:412
11850992819,5cyclictest3603-21smart_sda08:00:242
11850992819,5cyclictest15016-21ntp_states09:40:212
11850992812,5cyclictest87650irq/129-enp1s0-08:06:402
11850992811,9cyclictest25866-21memory10:05:192
147542270,4sleep31185699cyclictest10:55:123
11856992717,3cyclictest0-21swapper/309:59:333
11856992712,8cyclictest18233-21unixbench-2d11:00:243
1185699271,12cyclictest0-21swapper/310:32:053
1185699271,12cyclictest0-21swapper/309:29:573
1185699270,20cyclictest0-21swapper/312:15:233
1185699270,19cyclictest0-21swapper/310:17:053
1185099279,5cyclictest21582-21cpuspeed_turbos07:30:122
1185099278,6cyclictest28149-21latency_hist12:40:012
1185099277,7cyclictest6260-21cat11:50:012
1185099277,7cyclictest6260-21cat11:50:012
1185099277,7cyclictest15644-21ntpq10:55:202
1185099277,7cyclictest0-21swapper/208:35:162
1185099276,9cyclictest87350irq/126-enp1s0-10:54:052
11850992719,5cyclictest31274-21diskstats09:05:142
1184499271,17cyclictest4534-21df_inode11:45:131
1184499271,17cyclictest4534-21df_inode11:45:131
1183999273,12cyclictest28148-21idleruntime-cro12:40:010
1185699269,3cyclictest0-21swapper/311:24:293
1185699268,10cyclictest0-21swapper/311:10:533
11856992621,3cyclictest0-21swapper/308:42:213
11856992613,3cyclictest0-21swapper/312:24:213
1185699261,18cyclictest0-21swapper/309:15:413
1185699261,10cyclictest0-21swapper/307:19:413
11856992610,3cyclictest0-21swapper/309:00:473
1185699260,12cyclictest0-21swapper/312:06:293
1185099268,4cyclictest87350irq/126-enp1s0-10:47:162
1185099267,12cyclictest87650irq/129-enp1s0-09:35:132
1185099266,8cyclictest19929-21/usr/sbin/munin08:40:192
1185099266,5cyclictest87650irq/129-enp1s0-10:35:122
1185099266,5cyclictest87650irq/129-enp1s0-10:35:122
11850992619,4cyclictest8005-21smart_sda08:10:252
11850992619,4cyclictest8005-21smart_sda08:10:252
11850992619,4cyclictest31342-21/usr/sbin/munin07:50:212
11850992617,6cyclictest19699-21/usr/sbin/munin11:05:152
1184499262,17cyclictest87650irq/129-enp1s0-10:17:501
1184499261,18cyclictest0-21swapper/111:41:081
1184499261,18cyclictest0-21swapper/111:41:081
1185699251,12cyclictest0-21swapper/312:35:053
1185699251,12cyclictest0-21swapper/310:09:293
1185699251,11cyclictest27882-21cpuspeed_turbos11:25:113
1185699251,11cyclictest0-21swapper/312:00:413
1185099257,11cyclictest87650irq/129-enp1s0-08:50:182
1185099253,13cyclictest0-21swapper/210:30:192
11850992518,4cyclictest18039-21memory07:20:192
11850992518,4cyclictest1095-21diskstats09:10:142
11850992516,6cyclictest23713-21/usr/sbin/munin07:35:212
11850992516,6cyclictest15884-21/usr/sbin/munin08:30:142
11850992516,5cyclictest26821-21memory07:40:172
11850992516,5cyclictest22245-21ntp_states11:10:212
11850992515,6cyclictest13408-21/usr/sbin/munin08:25:132
1184499252,12cyclictest2072-40dnf07:59:421
1185699248,2cyclictest0-21swapper/307:25:133
1185699246,2cyclictest0-21swapper/307:45:533
1185699241,9cyclictest24878-21perl08:50:153
11856992417,4cyclictest9554-21/usr/sbin/munin09:30:183
11856992415,2cyclictest0-21swapper/312:30:253
1185699241,13cyclictest0-21swapper/311:05:153
1185699241,12cyclictest0-21swapper/310:45:453
1185699240,13cyclictest0-21swapper/309:12:293
11850992415,6cyclictest13168-21/usr/sbin/munin07:10:232
1184499241,16cyclictest0-21swapper/107:41:381
11844992410,12cyclictest3093-21kworker/u8:210:30:421
149602230,3sleep339-21rcuc/309:40:183
1185699238,3cyclictest0-21swapper/308:30:233
1185699237,8cyclictest0-21swapper/307:37:133
1185699237,3cyclictest0-21swapper/308:46:333
1185699237,3cyclictest0-21swapper/307:14:513
11856992312,2cyclictest0-21swapper/311:35:253
1185699230,13cyclictest0-21swapper/309:35:053
1185699230,12cyclictest0-21swapper/311:18:173
1184499232,11cyclictest18015-40dnf11:00:231
1184499231,16cyclictest0-21swapper/109:45:251
1184499231,16cyclictest0-21swapper/107:27:151
1184499230,17cyclictest0-21swapper/111:29:581
1183999232,10cyclictest23802-40dnf10:00:210
1183999231,11cyclictest3981-21awk11:45:010
1183999231,11cyclictest3981-21awk11:45:010
1185699228,8cyclictest0-21swapper/310:39:253
1185699228,8cyclictest0-21swapper/310:39:253
1185699228,8cyclictest0-21swapper/310:20:413
1185699228,3cyclictest6762-21cpu08:10:133
1185699228,3cyclictest6762-21cpu08:10:133
1185699228,3cyclictest2566-21cstates08:00:133
1185699225,11cyclictest0-21swapper/311:47:333
1185699225,11cyclictest0-21swapper/311:47:333
1185699223,4cyclictest8784-21/usr/sbin/munin10:40:173
1185699221,14cyclictest0-21swapper/309:09:093
1185699220,11cyclictest0-21swapper/309:52:013
1184499223,2cyclictest87650irq/129-enp1s0-10:52:311
1184499221,18cyclictest30272-21head09:00:261
1184499221,16cyclictest0-21swapper/109:18:471
1184499221,15cyclictest0-21swapper/109:30:161
1184499221,15cyclictest0-21swapper/108:30:371
1184499220,4cyclictest0-21swapper/111:24:541
1184499220,17cyclictest0-21swapper/110:25:491
1184499220,17cyclictest0-21swapper/110:09:431
1183999222,15cyclictest0-21swapper/007:59:430
1183999220,10cyclictest0-21swapper/011:09:480
247692211,16sleep024785-21ntp_states07:35:200
120352210,1sleep30-21swapper/308:20:173
1185699218,8cyclictest0-21swapper/311:33:333
1185699218,4cyclictest0-21swapper/307:31:453
1185699217,8cyclictest0-21swapper/311:40:573
1185699217,8cyclictest0-21swapper/311:40:563
1185699217,3cyclictest0-21swapper/310:13:193
1184499213,4cyclictest773-21NetworkManager08:57:451
11844992117,2cyclictest1041-21expr09:10:141
11844992114,2cyclictest0-21swapper/108:10:401
11844992114,2cyclictest0-21swapper/108:10:401
1184499211,3cyclictest8513-21cat11:55:021
11844992113,5cyclictest27747-21grep12:35:241
11844992113,5cyclictest14597-21vmstat12:05:261
1184499211,18cyclictest0-21swapper/109:05:261
1184499211,18cyclictest0-21swapper/108:15:161
1184499211,17cyclictest200122sleep109:51:131
1184499211,15cyclictest0-21swapper/110:58:481
1184499211,15cyclictest0-21swapper/110:10:091
1184499211,14cyclictest0-21swapper/108:08:441
1184499211,14cyclictest0-21swapper/107:36:341
1184499211,11cyclictest20508-21memory12:20:181
1183999215,13cyclictest7-21ksoftirqd/010:20:010
1183999211,8cyclictest9700-21taskset10:42:020
1183999211,4cyclictest0-21swapper/012:05:130
1185699209,3cyclictest16147-21/usr/sbin/munin09:45:113
1185699209,3cyclictest12371-21/usr/sbin/munin10:50:163
1185699208,3cyclictest0-21swapper/308:16:213
1184499205,4cyclictest0-21swapper/109:41:011
11844992015,2cyclictest14985-21cpu07:15:121
11844992013,2cyclictest0-21swapper/107:51:151
1184499201,14cyclictest0-21swapper/111:35:081
1184499201,14cyclictest0-21swapper/109:57:041
1184499201,14cyclictest0-21swapper/109:25:261
1184499201,14cyclictest0-21swapper/108:23:291
1184499201,14cyclictest0-21swapper/107:20:161
1184499201,14cyclictest0-21swapper/107:11:171
1184499201,11cyclictest2364-21egrep08:00:121
1184499201,10cyclictest24434-21cpuspeed_turbos08:50:121
1184499201,10cyclictest12818-21ntp_states09:35:211
1184499200,3cyclictest432-21diskmemload11:56:191
1183999206,5cyclictest0-21swapper/012:27:560
1183999206,3cyclictest0-21swapper/010:51:200
1183999205,6cyclictest0-21swapper/010:24:180
1183999205,3cyclictest0-21swapper/010:29:540
1183999204,5cyclictest10613-21ntpq09:30:190
1183999202,9cyclictest0-21swapper/011:00:260
1183999202,10cyclictest19325-21/usr/sbin/munin07:25:240
1183999202,10cyclictest11-21rcuc/010:45:200
11839992016,3cyclictest87350irq/126-enp1s0-11:45:240
11839992016,3cyclictest87350irq/126-enp1s0-11:45:240
1183999201,12cyclictest0-21swapper/009:35:110
1183999201,11cyclictest20540-21/usr/sbin/munin09:55:110
1183999201,11cyclictest15201-21/usr/sbin/munin12:10:130
215152191,2sleep00-21swapper/011:10:150
1185699195,3cyclictest0-21swapper/307:21:133
1185699192,3cyclictest0-21swapper/308:59:573
1185699192,3cyclictest0-21swapper/308:29:093
1185699191,10cyclictest4537-21sshd08:05:033
1184499198,3cyclictest22140-21diskstats12:25:141
1184499198,3cyclictest20999-21ntp_states08:40:211
1184499194,9cyclictest0-21swapper/111:12:551
11844991912,3cyclictest0-21swapper/110:46:141
11844991912,3cyclictest0-21swapper/109:22:521
11844991912,2cyclictest0-21swapper/111:32:201
1184499191,14cyclictest0-21swapper/110:36:141
1184499191,14cyclictest0-21swapper/110:36:141
1184499191,14cyclictest0-21swapper/108:25:431
1184499191,14cyclictest0-21swapper/107:31:331
1184499191,10cyclictest24472-21awk11:15:221
1184499191,10cyclictest19684-21cat08:39:561
1183999199,2cyclictest432-21diskmemload11:31:340
11839991912,4cyclictest9072-21tr08:15:110
1183999191,12cyclictest0-21swapper/010:30:200
1183999191,12cyclictest0-21swapper/009:14:320
1183999191,10cyclictest5774-21/usr/sbin/munin10:35:160
1183999191,10cyclictest5774-21/usr/sbin/munin10:35:160
1183999191,10cyclictest27147-21/usr/sbin/munin10:10:120
1183999191,10cyclictest13951-21/usr/sbin/munin09:40:240
1183999191,10cyclictest0-21swapper/011:50:380
1183999191,10cyclictest0-21swapper/010:58:360
1183999190,9cyclictest0-21swapper/011:39:120
256242181,2sleep00-21swapper/012:30:530
11856991811,3cyclictest0-21swapper/307:52:173
1184499187,3cyclictest19790-21irqstats11:05:171
1184499182,4cyclictest23804-21ntp_states10:00:211
1184499181,9cyclictest432-21diskmemload12:30:101
1184499181,16cyclictest0-21swapper/112:16:431
11844991811,2cyclictest0-21swapper/107:45:351
1184499180,6cyclictest0-21swapper/112:00:411
1184499180,3cyclictest0-21swapper/112:12:451
1183999189,2cyclictest0-21swapper/007:20:160
1183999186,3cyclictest0-21swapper/012:00:050
1183999186,3cyclictest0-21swapper/010:09:230
1183999182,7cyclictest21517-21/usr/sbin/munin07:30:220
1183999181,4cyclictest1441-21kworker/u8:212:20:190
1183999181,11cyclictest0-21swapper/009:45:530
1183999180,8cyclictest0-21swapper/009:27:590
1183999178,3cyclictest0-21swapper/011:25:260
1183999176,4cyclictest87350irq/126-enp1s0-08:20:100
1183999174,11cyclictest7-21ksoftirqd/011:20:090
1183999172,2cyclictest87350irq/126-enp1s0-08:40:210
1183999171,9cyclictest24360-21strings11:15:190
1183999171,9cyclictest0-21swapper/007:11:400
1183999171,7cyclictest17129-21users08:30:260
1183999171,7cyclictest0-21swapper/011:55:200
1183999171,14cyclictest308-21sshd09:08:400
1183999171,14cyclictest0-21swapper/009:20:300
1183999171,11cyclictest0-21swapper/009:52:590
11839991710,4cyclictest23267-21head08:45:230
11839991710,4cyclictest17864-21cut08:35:130
11839991612,3cyclictest0-21swapper/007:50:000
11839991612,3cyclictest0-21swapper/007:43:560
11839991612,2cyclictest30303-21/usr/sbin/munin07:50:150
11839991611,3cyclictest6746-21acpi08:10:130
11839991611,3cyclictest6746-21acpi08:10:130
11839991611,3cyclictest4549-21/usr/sbin/munin08:05:230
11839991611,3cyclictest3547-21gltestperf09:15:160
11839991611,3cyclictest24323-21/usr/sbin/munin08:50:160
11839991611,3cyclictest14157-21munin-node07:15:030
11839991611,3cyclictest13517-21cat08:25:130
1183999149,3cyclictest28743-21grep09:00:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional