You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-24 - 19:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu Sep 24, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
297112251165,22sleep10-21swapper/107:08:151
295242238162,23sleep30-21swapper/307:05:483
295752236163,23sleep20-21swapper/207:06:272
297772234161,23sleep00-21swapper/007:09:060
1367721730,2sleep12993999cyclictest08:55:261
29943998683,2cyclictest0-21swapper/211:07:032
29943997370,2cyclictest0-21swapper/209:26:332
200432550,2sleep341-21ksoftirqd/309:10:213
280082540,4sleep230-21rcuc/209:30:152
274442540,4sleep10-21swapper/109:30:011
179082470,4sleep230-21rcuc/211:35:122
2994999310,13cyclictest11771-21fschecks_count11:20:133
2994999300,13cyclictest0-21swapper/311:08:173
2994999300,11cyclictest0-21swapper/312:03:583
2993999309,15cyclictest7296-21crond12:25:001
2994999291,13cyclictest0-21swapper/312:11:303
2994999291,13cyclictest0-21swapper/309:52:223
2994999290,13cyclictest0-21swapper/312:24:103
2994999290,13cyclictest0-21swapper/308:35:333
2993199290,20cyclictest0-21swapper/012:37:500
17762290,2sleep20-21swapper/207:15:212
2994999281,9cyclictest0-21swapper/309:25:423
2994999281,12cyclictest0-21swapper/311:35:143
2994999281,11cyclictest18637-21diskmemload12:19:223
2994999281,11cyclictest0-21swapper/310:04:143
2994999281,11cyclictest0-21swapper/307:36:303
2994999281,10cyclictest0-21swapper/311:41:423
2994999281,10cyclictest0-21swapper/311:17:293
2994999281,10cyclictest0-21swapper/311:13:013
2994999281,10cyclictest0-21swapper/308:59:373
2994999280,12cyclictest0-21swapper/312:08:023
2994999280,12cyclictest0-21swapper/311:26:413
2994999280,12cyclictest0-21swapper/310:30:413
2994999280,11cyclictest0-21swapper/310:59:493
2994999280,11cyclictest0-21swapper/309:06:263
2994999280,10cyclictest0-21swapper/309:04:573
2994999280,10cyclictest0-21swapper/309:04:573
2994399288,12cyclictest0-21swapper/210:00:242
2994399281,6cyclictest0-21swapper/210:38:102
2994399281,5cyclictest0-21swapper/210:19:492
2993999282,11cyclictest9507-21munin-run12:30:001
2993199283,12cyclictest23655-21munin-run10:35:020
29931992814,5cyclictest0-21swapper/011:39:050
29931992812,14cyclictest81ktimersoftd/007:40:000
2993199281,13cyclictest8902-21awk11:15:000
2994999274,3cyclictest17014-1kworker/3:0H08:02:523
2994999271,12cyclictest0-21swapper/311:01:413
2994999270,11cyclictest0-21swapper/311:59:223
2994999270,10cyclictest0-21swapper/310:43:533
2994399273,5cyclictest0-21swapper/209:11:372
2994399271,4cyclictest0-21swapper/212:30:042
2993999270,24cyclictest0-21swapper/112:30:041
2994999261,9cyclictest0-21swapper/310:26:173
2994999261,11cyclictest0-21swapper/307:45:503
2994999261,11cyclictest0-21swapper/307:45:503
2994999261,10cyclictest0-21swapper/311:50:583
2994999261,10cyclictest0-21swapper/307:13:023
2994999260,9cyclictest0-21swapper/307:33:343
2994999260,11cyclictest0-21swapper/307:21:573
2994999260,10cyclictest0-21swapper/308:52:013
2994999260,10cyclictest0-21swapper/308:34:493
2994999260,10cyclictest0-21swapper/307:42:423
2994399261,7cyclictest13613-21crond07:45:012
29939992621,3cyclictest29849-21grep10:45:251
29939992621,3cyclictest20222-21perf08:00:011
2993999260,23cyclictest0-21swapper/109:11:361
29931992615,2cyclictest11-21rcuc/008:50:130
2993199260,17cyclictest0-21swapper/009:44:540
283502260,5sleep011-21rcuc/009:30:160
2994999258,3cyclictest0-21swapper/311:45:483
2994999251,9cyclictest0-21swapper/307:58:223
2994999251,12cyclictest0-21swapper/310:11:253
29949992510,3cyclictest0-21swapper/310:08:133
2994999250,13cyclictest0-21swapper/309:56:183
2994999250,11cyclictest0-21swapper/312:28:343
2994999250,11cyclictest0-21swapper/307:26:213
2994999250,10cyclictest0-21swapper/309:30:103
2994999250,10cyclictest0-21swapper/309:24:263
2994999250,10cyclictest0-21swapper/308:21:253
2994399257,5cyclictest4505-21perf11:05:012
2994399251,6cyclictest0-21swapper/210:23:082
2994399251,5cyclictest0-21swapper/210:33:312
2994399251,4cyclictest0-21swapper/209:57:242
29939992520,2cyclictest9223-21sh07:35:001
29939992519,3cyclictest32645-21tail09:40:141
2993999251,6cyclictest0-21swapper/110:25:461
2994999248,3cyclictest0-21swapper/309:39:423
2994999240,9cyclictest0-21swapper/311:31:413
2994999240,9cyclictest0-21swapper/310:16:053
2994999240,9cyclictest0-21swapper/308:26:413
2994999240,11cyclictest0-21swapper/312:36:223
2994399247,4cyclictest12050irq/123-ahci[0007:55:002
2994399244,7cyclictest30-21rcuc/212:20:202
2994399242,6cyclictest21817-21if_enp2s009:15:172
2994399241,5cyclictest0-21swapper/209:35:332
2994399241,14cyclictest0-21swapper/208:40:162
2993999249,3cyclictest87550irq/128-enp1s0-11:10:201
2993999241,14cyclictest24687-21/usr/sbin/munin10:35:171
2993999241,14cyclictest0-21swapper/109:31:401
2993999241,12cyclictest0-21swapper/111:15:141
2993999240,16cyclictest0-21swapper/110:51:191
2993199246,15cyclictest16441-21munin-run09:05:000
2993199246,15cyclictest16441-21munin-run09:05:000
29931992417,2cyclictest0-21swapper/010:55:150
29931992411,7cyclictest145542sleep010:11:140
2993199241,16cyclictest0-21swapper/010:04:310
29931992410,8cyclictest0-21swapper/012:07:190
2994999239,3cyclictest0-21swapper/308:19:333
2994999235,3cyclictest0-21swapper/308:12:593
2994999231,6cyclictest0-21swapper/307:15:093
2994999230,9cyclictest0-21swapper/310:45:253
2994999230,9cyclictest0-21swapper/308:42:373
2994999230,12cyclictest0-21swapper/310:39:533
2994399233,7cyclictest30-21rcuc/207:10:172
2994399232,8cyclictest14577-21ntp_states11:25:212
2994399231,7cyclictest1022-21idleruntime-cro08:30:012
2994399231,4cyclictest0-21swapper/210:10:462
2994399231,3cyclictest0-21swapper/210:29:162
2994399231,11cyclictest0-21swapper/207:35:172
2993999236,4cyclictest0-21swapper/107:37:141
2993999235,10cyclictest2945-21sshd08:31:151
2993999231,9cyclictest0-21swapper/109:50:231
29939992319,3cyclictest0-21swapper/112:02:461
29939992319,3cyclictest0-21swapper/109:15:211
29939992318,2cyclictest21-21rcuc/111:40:011
29939992317,3cyclictest31774-21df08:25:131
29939992317,3cyclictest15086-21/usr/sbin/munin10:15:211
2993999231,4cyclictest1830-21tail10:55:271
2993999231,19cyclictest8679-21irqstats08:45:171
2993999231,19cyclictest16672-21missed_timers11:30:221
2993999230,5cyclictest0-21swapper/110:12:121
2993999230,4cyclictest0-21swapper/110:41:101
2993999230,4cyclictest0-21swapper/109:38:261
2993199239,6cyclictest0-21swapper/011:27:270
2993199237,9cyclictest0-21swapper/009:10:230
2993199237,10cyclictest0-21swapper/011:49:530
2993199236,5cyclictest10796-21ntp_states12:30:240
29931992312,7cyclictest0-21swapper/009:54:310
2993199231,20cyclictest0-21swapper/010:07:360
29931992310,8cyclictest0-21swapper/009:38:330
29931992310,7cyclictest0-21swapper/012:14:160
29931992310,7cyclictest0-21swapper/011:00:560
29931992310,7cyclictest0-21swapper/009:45:060
2993199230,15cyclictest25530-21cpuspeed_turbos09:25:120
2993199230,10cyclictest0-21swapper/007:40:210
2994999228,3cyclictest0-21swapper/307:51:163
2994999221,6cyclictest0-21swapper/309:47:383
2994999220,9cyclictest0-21swapper/312:30:143
2994399229,3cyclictest30-21rcuc/209:20:132
2994399225,5cyclictest30-21rcuc/209:50:012
2994399221,8cyclictest29190-21memory10:45:172
2994399221,8cyclictest0-21swapper/211:58:422
2994399221,7cyclictest18637-21diskmemload10:05:142
2993999227,3cyclictest28670-21munin-run12:00:011
2993999225,8cyclictest87550irq/128-enp1s0-12:35:141
29939992219,2cyclictest0-21swapper/108:55:011
29939992218,3cyclictest0-21swapper/108:21:071
29939992218,2cyclictest0-21swapper/111:20:151
2993999221,6cyclictest0-21swapper/107:49:301
2993999221,6cyclictest0-21swapper/107:49:301
2993999221,15cyclictest87650irq/129-enp1s0-12:05:221
2993999221,13cyclictest0-21swapper/109:55:011
2993999220,5cyclictest0-21swapper/110:22:391
2993199228,3cyclictest11095-21awk11:20:000
2993199228,3cyclictest0-21swapper/009:17:140
2993199224,4cyclictest0-21swapper/012:15:190
29931992217,3cyclictest21086-21vmstat10:25:260
29931992210,8cyclictest0-21swapper/011:31:380
29931992210,7cyclictest0-21swapper/012:25:430
29931992210,7cyclictest0-21swapper/009:56:550
2993199220,20cyclictest0-21swapper/010:40:190
2994999215,3cyclictest0-21swapper/308:47:153
2994999211,10cyclictest0-21swapper/309:15:283
2994399219,3cyclictest32-21ksoftirqd/209:50:182
2994399217,3cyclictest12050irq/123-ahci[0012:20:012
2994399214,6cyclictest30-21rcuc/212:25:102
2994399213,6cyclictest25307-21irqstats11:50:192
2994399213,5cyclictest0-21swapper/208:15:122
2994399212,7cyclictest30-21rcuc/208:50:232
2994399212,7cyclictest30-21rcuc/208:35:252
2994399211,7cyclictest929-21/usr/sbin/munin12:10:252
2994399211,7cyclictest30723-21cat10:50:182
2994399211,7cyclictest24262-21latency_hist11:50:022
2994399211,6cyclictest19091-21grep07:55:182
2994399211,6cyclictest16085-21fschecks_count11:30:192
2994399211,6cyclictest15590-21cut09:00:252
2994399211,6cyclictest15590-21cut09:00:252
29943992115,3cyclictest21106-21cut08:00:152
2994399211,4cyclictest27114-21ntpq10:40:192
2994399211,4cyclictest0-21swapper/212:04:362
2993999218,8cyclictest0-21swapper/110:00:081
2993999217,3cyclictest3254-21fschecks_count07:20:161
2993999216,3cyclictest0-21swapper/111:02:411
2993999216,10cyclictest0-21swapper/111:54:041
29939992117,3cyclictest0-21swapper/111:05:081
29939992117,3cyclictest0-21swapper/108:12:241
29939992117,3cyclictest0-21swapper/108:02:211
29939992117,2cyclictest0-21swapper/109:45:121
2993999211,6cyclictest0-21swapper/110:06:251
2993999211,3cyclictest0-21swapper/111:45:041
2993999211,10cyclictest3136-21/usr/sbin/munin12:15:201
2993999210,5cyclictest0-21swapper/109:23:541
2993199218,8cyclictest0-21swapper/008:28:170
2993199218,7cyclictest0-21swapper/012:20:110
2993199216,9cyclictest0-21swapper/008:35:160
2993199216,3cyclictest0-21swapper/011:55:140
2993199214,8cyclictest87550irq/128-enp1s0-08:15:250
29931992117,3cyclictest0-21swapper/009:24:390
2993199211,18cyclictest0-21swapper/010:53:430
29931992111,3cyclictest0-21swapper/012:02:340
29931992111,3cyclictest0-21swapper/010:22:140
2993199211,12cyclictest0-21swapper/008:40:100
29931992110,6cyclictest0-21swapper/010:19:450
2993199210,9cyclictest0-21swapper/011:20:200
2994999206,3cyclictest0-21swapper/310:23:373
2994399208,3cyclictest0-21swapper/208:49:352
2994399203,4cyclictest0-21swapper/211:10:212
2994399203,3cyclictest0-21swapper/212:07:592
2994399202,5cyclictest14235-21munin-run09:00:012
2994399201,7cyclictest20099-21/usr/sbin/munin11:40:142
2994399201,7cyclictest0-21swapper/212:39:072
2994399201,7cyclictest0-21swapper/211:22:232
2994399201,7cyclictest0-21swapper/208:20:252
2994399201,6cyclictest7591-21df07:30:132
2994399201,6cyclictest0-21swapper/210:55:192
2994399201,5cyclictest26810-21sh08:15:012
2994399201,4cyclictest0-21swapper/209:41:212
2994399201,11cyclictest0-21swapper/207:25:142
29939992017,2cyclictest0-21swapper/107:43:051
29939992016,3cyclictest0-21swapper/108:35:091
29939992016,3cyclictest0-21swapper/108:09:501
29939992016,3cyclictest0-21swapper/107:52:311
29939992016,2cyclictest5412-21dnf07:25:151
2993999201,2cyclictest0-21swapper/111:42:351
2993999201,17cyclictest0-21swapper/107:15:211
2993999201,14cyclictest0-21swapper/110:34:341
2993199209,7cyclictest0-21swapper/011:52:280
29931992017,2cyclictest11-21rcuc/011:40:150
29931992016,3cyclictest0-21swapper/011:05:010
29931992016,2cyclictest0-21swapper/008:00:230
29931992014,4cyclictest0-21swapper/010:45:190
2993199201,15cyclictest309492chrt07:10:190
29931992010,6cyclictest0-21swapper/010:36:160
2994999195,3cyclictest0-21swapper/309:42:343
2994399196,2cyclictest87350irq/126-enp1s0-09:05:152
2994399192,3cyclictest87350irq/126-enp1s0-11:15:182
2994399191,6cyclictest2942-21cat07:20:132
2994399190,10cyclictest0-21swapper/207:45:172
2994399190,10cyclictest0-21swapper/207:45:162
29939991916,2cyclictest0-21swapper/109:06:521
2993999191,5cyclictest28859-21sshd08:18:151
29939991914,3cyclictest2908-21latency_hist12:15:011
29939991913,4cyclictest6740-21ntp_states08:40:201
2993999191,15cyclictest13624-21cpuspeed_turbos11:25:121
2993199191,6cyclictest8924-21sshd07:31:150
2993199191,6cyclictest571-21/usr/sbin/munin07:15:150
2994999181,6cyclictest0-21swapper/310:51:013
2994399185,3cyclictest87350irq/126-enp1s0-08:30:242
2994399181,5cyclictest0-21swapper/208:05:102
2993999181,5cyclictest14462-21/usr/sbin/munin09:00:221
2993999181,5cyclictest14462-21/usr/sbin/munin09:00:221
29939991815,2cyclictest0-21swapper/107:14:471
2993199189,1cyclictest0-21swapper/007:55:120
2993199185,2cyclictest18487-21kworker/0:009:08:200
29931991815,2cyclictest3270-21kworker/0:007:25:140
29931991813,3cyclictest30236-21strings08:20:200
29931991813,3cyclictest18017-21ls07:55:010
2993199181,10cyclictest7889-21cpu08:45:110
155792181,2sleep00-21swapper/007:47:080
155792181,2sleep00-21swapper/007:47:070
2994999173,2cyclictest87350irq/126-enp1s0-08:07:303
2993199176,2cyclictest0-21swapper/008:55:220
29931991712,3cyclictest3557-21hwlatdetect07:20:170
265512170,2sleep00-21swapper/008:11:480
29931991612,3cyclictest0-21swapper/008:32:190
29931991610,4cyclictest23493-21latency08:05:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional