You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 22:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat May 23, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
103942236163,24sleep30-21swapper/307:09:483
102712236164,59sleep20-21swapper/207:08:102
103972201165,23sleep10-21swapper/107:09:501
102172201164,25sleep00-21swapper/007:07:270
1145121590,5sleep21050299cyclictest09:35:212
1134421580,5sleep11049799cyclictest12:03:571
1134421580,5sleep11049799cyclictest12:03:571
415421550,5sleep21050299cyclictest11:45:242
7862790,9sleep011-21rcuc/011:40:120
299992710,1sleep30-21swapper/307:50:203
318052660,4sleep230-21rcuc/210:20:282
64772560,1sleep30-21swapper/308:10:233
44102500,1sleep24388-21processes08:05:242
29572500,1sleep10-21swapper/110:30:181
212472500,2sleep20-21swapper/210:00:022
1048699399,14cyclictest9573-21munin-run08:20:010
250462380,1sleep20-21swapper/212:35:122
1050799383,4cyclictest28737-21latency_hist07:50:023
1050799363,5cyclictest13679-40dnf08:26:253
10507993513,19cyclictest17018-21/usr/sbin/munin09:50:153
1050799342,5cyclictest22034-21/usr/sbin/munin11:15:273
1050799341,4cyclictest1791-21NetworkManager10:27:143
10507993411,20cyclictest9801-21acpi08:20:113
1050299341,13cyclictest0-21swapper/207:50:372
1048699346,13cyclictest24014-21idleruntime-cro11:20:020
1050799339,21cyclictest24579-21/usr/sbin/munin07:40:183
1050799331,5cyclictest3745-21sort11:45:183
10507993312,18cyclictest8222-21/usr/sbin/munin09:30:183
10507993312,18cyclictest29591-21/usr/sbin/munin09:05:173
1050799331,13cyclictest0-21swapper/308:00:213
10507993311,19cyclictest8836-21/usr/sbin/munin10:45:193
10507993311,19cyclictest8836-21/usr/sbin/munin10:45:193
10507993311,19cyclictest7256-21/usr/sbin/munin11:55:173
10507993311,19cyclictest24850-21/usr/sbin/munin12:35:243
10507993311,19cyclictest17988-21/usr/sbin/munin07:25:223
10507993310,20cyclictest21415-21/usr/sbin/munin10:00:193
1050299331,11cyclictest0-21swapper/210:13:332
1050799327,4cyclictest25630-21timerwakeupswit11:20:253
1050799326,3cyclictest31559-21diskmemload09:55:163
1050799322,5cyclictest11686-21smart_sda09:35:263
1050799321,6cyclictest31559-21diskmemload10:30:163
10507993213,16cyclictest9922-21grep12:00:153
10507993213,16cyclictest9922-21grep12:00:153
10507993212,17cyclictest23512-21sort12:30:193
10507993211,18cyclictest28270-21sort09:00:183
10507993211,18cyclictest13041-21sort07:15:023
1050799320,11cyclictest0-21swapper/312:10:243
1050299321,12cyclictest31559-21diskmemload10:02:132
1050299320,24cyclictest0-21swapper/211:53:212
1050299320,24cyclictest0-21swapper/210:16:152
1050299320,23cyclictest0-21swapper/209:09:432
162922310,3sleep29-21rcu_preempt11:00:172
1050799314,3cyclictest12050irq/123-ahci[0008:50:013
1050799311,7cyclictest29315-21switchtime10:15:263
10507993117,9cyclictest0-21swapper/311:09:003
10507993111,17cyclictest827-21runrttasks12:25:303
10507993111,17cyclictest11667-21/usr/sbin/munin12:05:173
10507993110,3cyclictest31751-21gltestperf07:55:163
10507993110,18cyclictest827-21runrttasks07:35:073
1050799310,13cyclictest0-21swapper/310:55:123
1050299311,13cyclictest0-21swapper/212:21:532
1050299310,22cyclictest0-21swapper/209:32:352
1050299310,12cyclictest0-21swapper/209:15:132
1048699314,12cyclictest2034-21awk10:30:010
10486993111,17cyclictest7028-21munin-run11:55:000
1050799308,3cyclictest3214-21/usr/sbin/munin08:05:253
1050799308,2cyclictest827-21runrttasks08:32:183
1050799308,2cyclictest827-21runrttasks08:32:183
1050799307,4cyclictest5685-21/usr/sbin/munin10:35:273
1050799307,3cyclictest25818-21/usr/sbin/munin10:10:173
1050799306,3cyclictest30213-21/usr/sbin/munin10:20:143
1050799304,3cyclictest87650irq/129-enp1s0-12:20:163
1050799302,6cyclictest0-21swapper/309:45:133
1050799301,5cyclictest11767-21ntp_states07:10:213
10507993011,16cyclictest16138-21sh07:20:153
10507993011,16cyclictest16138-21sh07:20:143
10507993010,3cyclictest0-21swapper/311:25:133
10507993010,2cyclictest7603-21/usr/sbin/munin08:15:223
1050799300,10cyclictest0-21swapper/310:05:163
1050299301,12cyclictest0-21swapper/210:36:532
1050799297,3cyclictest30825-21/usr/sbin/munin11:35:183
1050799296,3cyclictest16875-21fschecks_time08:35:173
1050799296,3cyclictest0-21swapper/312:15:153
1050799295,3cyclictest21194-21threads11:10:393
1050799293,3cyclictest12050irq/123-ahci[0009:20:003
1050799291,7cyclictest5236-21cat11:50:133
1050799291,5cyclictest30591-21latency_hist11:35:023
1050799291,4cyclictest11043-21/usr/sbin/munin10:50:273
1050799291,3cyclictest5636-21sshd09:23:063
10507992910,16cyclictest6290-21cat09:25:133
1050299291,9cyclictest0-21swapper/208:30:392
1050299291,9cyclictest0-21swapper/208:30:392
1050299291,12cyclictest0-21swapper/212:31:002
1050299291,11cyclictest0-21swapper/208:36:122
1050299290,23cyclictest0-21swapper/211:28:232
1050299290,12cyclictest0-21swapper/210:42:492
1050299290,11cyclictest0-21swapper/207:23:512
1050299290,11cyclictest0-21swapper/207:23:512
10486992910,16cyclictest15834-21munin-run12:15:010
1050799289,3cyclictest8332-21seq10:41:373
1050799288,3cyclictest26904-21seq08:56:553
1050799287,2cyclictest827-21runrttasks08:53:373
1050799281,3cyclictest2475-21sshd11:43:073
1050799281,3cyclictest12737-21tr09:40:123
1050799281,3cyclictest12737-21tr09:40:113
1050299281,12cyclictest0-21swapper/207:16:302
1050299281,10cyclictest0-21swapper/208:50:042
1050299281,10cyclictest0-21swapper/207:12:232
1050299280,20cyclictest0-21swapper/210:47:462
1050299280,20cyclictest0-21swapper/210:47:462
1050299280,12cyclictest0-21swapper/211:57:172
1050299280,11cyclictest0-21swapper/209:29:232
1048699282,14cyclictest30601-21munin-run11:35:010
1050799278,3cyclictest21499-21sort07:30:243
1050799277,3cyclictest32081-21expr09:10:133
1050799276,3cyclictest19385-21if_err_enp2s008:40:183
1050299271,9cyclictest827-21runrttasks09:22:552
1050299271,9cyclictest0-21swapper/211:17:472
1050299271,12cyclictest0-21swapper/210:07:412
1050299271,11cyclictest0-21swapper/210:53:252
1050299271,10cyclictest0-21swapper/212:00:312
1050299271,10cyclictest0-21swapper/212:00:312
1050299271,10cyclictest0-21swapper/211:37:172
1050299271,10cyclictest0-21swapper/207:32:512
1050299270,9cyclictest0-21swapper/212:10:492
1050299270,13cyclictest0-21swapper/211:08:172
1050299270,12cyclictest0-21swapper/208:57:082
1050299270,12cyclictest0-21swapper/208:43:322
1050299270,11cyclictest0-21swapper/210:34:212
1050299270,10cyclictest0-21swapper/211:23:112
1048699273,3cyclictest88950irq/132-enp2s0-09:09:050
1048699271,13cyclictest18992-21munin-run09:55:010
1050799267,3cyclictest15434-21/usr/sbin/munin11:00:133
1050299261,12cyclictest0-21swapper/209:46:132
1050299261,12cyclictest0-21swapper/207:48:132
1050299261,11cyclictest0-21swapper/212:27:512
1050299261,10cyclictest0-21swapper/211:41:552
1050299260,9cyclictest0-21swapper/208:20:152
1050299260,12cyclictest0-21swapper/211:13:212
1050299260,12cyclictest0-21swapper/207:36:152
1050299260,11cyclictest0-21swapper/208:01:252
1050299260,10cyclictest0-21swapper/209:53:052
1050299260,10cyclictest0-21swapper/209:11:252
1050299260,10cyclictest0-21swapper/207:57:472
1049799261,17cyclictest0-21swapper/109:45:161
326882250,4sleep011-21rcuc/009:10:190
1050299251,9cyclictest0-21swapper/210:56:352
1050299251,9cyclictest0-21swapper/208:10:012
1050299251,8cyclictest0-21swapper/208:25:092
1050299251,11cyclictest19575-21users07:25:272
1050299250,9cyclictest0-21swapper/211:34:092
1050299250,11cyclictest0-21swapper/212:08:172
1049799257,8cyclictest10732-21expr09:35:151
1048699251,12cyclictest338-21idleruntime-cro11:40:000
285182240,5sleep230-21rcuc/209:00:212
1050299246,3cyclictest0-21swapper/207:40:512
1050299240,12cyclictest0-21swapper/209:43:552
1050299240,12cyclictest0-21swapper/209:43:552
1050299240,11cyclictest0-21swapper/208:45:182
1050299240,10cyclictest0-21swapper/210:25:472
1049799242,15cyclictest87650irq/129-enp1s0-07:25:071
1049799241,5cyclictest5972-21memory11:50:191
1050299230,10cyclictest0-21swapper/212:19:512
1049799233,2cyclictest87650irq/129-enp1s0-10:49:531
1049799233,2cyclictest87650irq/129-enp1s0-10:49:531
1049799232,19cyclictest0-21swapper/110:05:231
1049799231,5cyclictest0-21swapper/112:06:081
1049799231,3cyclictest0-21swapper/110:29:091
1049799231,3cyclictest0-21swapper/107:17:371
1049799231,11cyclictest31445-21expr07:55:131
1049799230,6cyclictest0-21swapper/112:30:551
1049799230,6cyclictest0-21swapper/110:24:581
1049799230,5cyclictest0-21swapper/112:28:091
1049799230,5cyclictest0-21swapper/111:05:151
1049799230,5cyclictest0-21swapper/110:10:041
1050299221,10cyclictest0-21swapper/208:15:592
1049799223,8cyclictest87550irq/128-enp1s0-08:10:001
1049799222,2cyclictest87650irq/129-enp1s0-11:03:481
10497992218,3cyclictest0-21swapper/107:35:241
1049799221,3cyclictest30654-21ntp_states09:05:211
10497992211,3cyclictest4121-21smart_sda11:45:241
1049799221,11cyclictest0-21swapper/109:59:341
1049799220,6cyclictest0-21swapper/112:39:011
1049799220,5cyclictest0-21swapper/111:57:591
1049799220,5cyclictest0-21swapper/110:57:171
1049799220,5cyclictest0-21swapper/110:42:351
1049799220,5cyclictest0-21swapper/110:01:531
1049799220,5cyclictest0-21swapper/109:54:001
1049799220,5cyclictest0-21swapper/109:28:171
1049799220,5cyclictest0-21swapper/109:22:481
1049799220,4cyclictest0-21swapper/111:33:151
1049799220,2cyclictest0-21swapper/112:16:581
1049799220,15cyclictest0-21swapper/107:34:071
1048699221,18cyclictest18523-21grep11:05:190
1049799214,8cyclictest87550irq/128-enp1s0-09:40:121
1049799214,8cyclictest87550irq/128-enp1s0-09:40:121
1049799212,3cyclictest0-21swapper/110:35:321
10497992117,2cyclictest16783-21strings07:20:211
10497992117,2cyclictest16783-21strings07:20:211
1049799211,18cyclictest0-21swapper/108:45:051
1049799211,18cyclictest0-21swapper/107:10:391
1049799210,5cyclictest0-21swapper/111:43:261
1049799210,5cyclictest0-21swapper/111:21:571
1049799210,5cyclictest0-21swapper/109:12:231
1049799210,4cyclictest0-21swapper/112:12:471
1049799210,4cyclictest0-21swapper/111:27:261
1049799210,4cyclictest0-21swapper/109:31:401
1049799210,3cyclictest0-21swapper/111:39:081
1049799210,3cyclictest0-21swapper/109:15:241
1048699216,8cyclictest0-21swapper/009:41:320
1048699216,8cyclictest0-21swapper/009:41:320
1048699216,8cyclictest0-21swapper/009:24:330
1048699216,3cyclictest0-21swapper/012:04:100
1048699216,3cyclictest0-21swapper/012:04:090
1048699215,3cyclictest0-21swapper/010:52:140
1048699215,3cyclictest0-21swapper/009:34:390
10486992112,4cyclictest0-21swapper/009:15:120
1049799205,3cyclictest28356-21df_inode10:15:141
1049799205,3cyclictest18286-21sshd08:39:071
1049799201,6cyclictest0-21swapper/111:19:291
1049799201,3cyclictest0-21swapper/112:21:411
1049799201,3cyclictest0-21swapper/108:18:061
1049799201,2cyclictest12030-21strings10:50:201
1049799201,18cyclictest0-21swapper/108:53:111
1049799201,18cyclictest0-21swapper/108:42:131
1049799201,17cyclictest0-21swapper/108:55:571
1049799200,3cyclictest0-21swapper/111:13:081
1049799200,2cyclictest0-21swapper/107:42:071
1048699208,3cyclictest0-21swapper/012:37:550
1048699205,6cyclictest0-21swapper/011:28:280
1048699204,3cyclictest0-21swapper/011:14:510
1048699203,3cyclictest0-21swapper/010:57:540
1048699203,15cyclictest29994-21awk10:20:020
1048699201,8cyclictest0-21swapper/009:45:230
1048699201,3cyclictest0-21swapper/011:20:420
1048699201,13cyclictest0-21swapper/010:44:540
1048699201,13cyclictest0-21swapper/010:13:310
1048699201,11cyclictest19217-21/usr/sbin/munin09:55:240
229032190,3sleep01048699cyclictest12:30:130
10497991915,3cyclictest0-21swapper/107:45:151
10497991913,3cyclictest10145-21df_inode08:20:121
1049799191,17cyclictest0-21swapper/109:01:571
1049799191,17cyclictest0-21swapper/108:25:571
1049799191,17cyclictest0-21swapper/108:04:031
1049799191,16cyclictest0-21swapper/107:50:401
1049799191,14cyclictest6013-21gltestperf08:10:161
1049799190,17cyclictest0-21swapper/108:30:161
1049799190,17cyclictest0-21swapper/108:30:151
1048699199,2cyclictest0-21swapper/010:31:050
1048699196,3cyclictest24655-1kworker/u9:010:38:320
1048699194,9cyclictest0-21swapper/011:59:340
1048699191,6cyclictest0-21swapper/012:05:160
1048699191,5cyclictest0-21swapper/011:04:220
10486991914,4cyclictest0-21swapper/010:00:070
10486991912,2cyclictest0-21swapper/007:50:140
1048699191,10cyclictest20188-21/usr/sbin/munin07:30:110
1048699191,10cyclictest18599-21/usr/sbin/munin08:40:230
1048699190,12cyclictest0-21swapper/012:19:220
1048699188,3cyclictest11488-21ntp_states09:35:230
1048699184,5cyclictest0-21swapper/010:46:350
1048699184,5cyclictest0-21swapper/010:46:350
1048699183,8cyclictest0-21swapper/009:25:150
1048699181,9cyclictest20795-21/usr/sbin/munin08:45:160
1048699181,6cyclictest2861-21/usr/sbin/munin11:45:240
1048699181,5cyclictest28737-21latency_hist07:50:010
1048699181,10cyclictest20187-21kworker/0:012:29:250
291422170,2sleep00-21swapper/009:02:180
1048699179,7cyclictest7-21ksoftirqd/008:31:130
1048699179,7cyclictest7-21ksoftirqd/008:31:130
1048699178,7cyclictest1632-21sed08:00:170
1048699173,11cyclictest1-21systemd12:20:080
1048699172,12cyclictest6043-21tail08:10:170
1048699171,5cyclictest0-21swapper/008:55:070
1048699171,4cyclictest31559-21diskmemload10:20:170
10486991712,3cyclictest14153-21wc08:30:010
1048699170,4cyclictest0-21swapper/008:35:260
1048699170,3cyclictest0-21swapper/010:05:470
1048699169,4cyclictest3828-21grep08:05:170
1048699165,2cyclictest0-21swapper/007:20:230
1048699165,2cyclictest0-21swapper/007:20:230
1048699161,8cyclictest22383-21/usr/sbin/munin07:35:280
1048699161,5cyclictest31349-21cstates07:55:130
10486991611,3cyclictest24848-21expr07:40:130
1048699161,12cyclictest13594-21/usr/sbin/munin07:15:150
10486991610,4cyclictest19216-21/usr/sbin/munin07:25:230
1048699160,3cyclictest87550irq/128-enp1s0-07:10:220
1048699151,3cyclictest0-21swapper/008:20:130
1048699151,2cyclictest0-21swapper/008:50:070
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional