You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 14:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Mar 25, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
286052235163,24sleep20-21swapper/207:08:072
286092204167,24sleep10-21swapper/107:08:111
285882201164,25sleep00-21swapper/007:07:540
284422201163,25sleep30-21swapper/307:05:583
987821850,5sleep02884199cyclictest12:30:220
475821710,4sleep12884799cyclictest07:25:161
1859221520,4sleep02884199cyclictest07:55:250
1025021390,5sleep02884199cyclictest10:05:140
28858991091,106cyclictest0-21swapper/307:23:463
28858999895,2cyclictest0-21swapper/308:01:503
28858997167,3cyclictest0-21swapper/308:05:583
25542520,6sleep1221ktimersoftd/107:20:171
5222500,4sleep011-21rcuc/010:55:240
38842500,2sleep10-21swapper/111:05:111
180262490,3sleep1221ktimersoftd/111:35:221
28858994845,2cyclictest0-21swapper/310:10:213
244402460,4sleep339-21rcuc/308:10:163
28858994541,3cyclictest0-21swapper/309:19:043
148062450,1sleep20-21swapper/207:50:012
2884799330,30cyclictest0-21swapper/109:40:131
2884799321,19cyclictest0-21swapper/108:26:291
2884799320,14cyclictest0-21swapper/112:05:571
28858993122,3cyclictest0-21swapper/310:54:073
2884799311,23cyclictest0-21swapper/111:58:211
2884799311,15cyclictest851-21sssd_nss09:59:251
2884799300,23cyclictest0-21swapper/112:15:391
2884799291,12cyclictest0-21swapper/107:13:271
2884799291,11cyclictest0-21swapper/108:03:491
2884799291,10cyclictest17553-21diskmemload09:23:481
2884799290,14cyclictest0-21swapper/109:26:411
2884799290,14cyclictest0-21swapper/109:26:401
2884799290,12cyclictest0-21swapper/108:38:231
2884799281,13cyclictest0-21swapper/112:00:091
2884799281,13cyclictest0-21swapper/110:16:491
2884799281,13cyclictest0-21swapper/109:33:231
2884799281,13cyclictest0-21swapper/109:13:401
2884799281,11cyclictest0-21swapper/110:34:111
2884799281,10cyclictest0-21swapper/110:14:091
2884799280,13cyclictest0-21swapper/110:54:521
2884799271,14cyclictest0-21swapper/109:45:171
2884799271,13cyclictest0-21swapper/112:28:571
2884799271,12cyclictest0-21swapper/109:54:051
2884799271,11cyclictest0-21swapper/112:11:351
2884799271,11cyclictest0-21swapper/110:46:481
2884799271,10cyclictest0-21swapper/110:28:291
2884799271,10cyclictest0-21swapper/108:07:471
2884799270,12cyclictest0-21swapper/107:36:411
2884799270,11cyclictest0-21swapper/111:28:091
2884799270,11cyclictest0-21swapper/110:20:291
2884199270,12cyclictest0-21swapper/007:40:020
2884799267,8cyclictest0-21swapper/108:21:371
2884799261,9cyclictest0-21swapper/111:17:061
2884799261,13cyclictest0-21swapper/111:22:561
2884799261,12cyclictest0-21swapper/111:45:091
2884799261,11cyclictest0-21swapper/111:51:251
2884799261,11cyclictest0-21swapper/111:51:251
2884799260,17cyclictest0-21swapper/109:09:081
2884799260,13cyclictest0-21swapper/108:55:121
2884799260,12cyclictest0-21swapper/111:31:151
2884799260,12cyclictest0-21swapper/110:06:591
2884799260,12cyclictest0-21swapper/107:46:111
2884799260,11cyclictest0-21swapper/112:35:251
2884799260,11cyclictest0-21swapper/112:31:251
2884199261,6cyclictest0-21swapper/009:50:200
2885899253,7cyclictest0-21swapper/310:01:483
2885299255,7cyclictest771-21rngd09:01:292
2885299251,6cyclictest0-21swapper/212:04:242
2884799251,9cyclictest0-21swapper/110:00:251
2884799251,8cyclictest0-21swapper/111:03:161
2884799251,8cyclictest0-21swapper/108:54:061
2884799251,12cyclictest0-21swapper/112:22:411
2884799251,12cyclictest0-21swapper/108:40:531
2884799251,10cyclictest0-21swapper/107:40:271
2884799250,13cyclictest0-21swapper/107:19:331
2884799250,12cyclictest0-21swapper/110:57:241
2884799250,11cyclictest0-21swapper/110:35:161
2884799250,11cyclictest0-21swapper/109:15:261
2884199251,14cyclictest22557-21latency_hist10:35:010
2885299249,3cyclictest26323-21threads10:40:242
2885299241,5cyclictest0-21swapper/210:52:322
28852992410,3cyclictest86750irq/128-enp1s0-09:11:562
2884799241,4cyclictest27923-21munin-run08:20:011
2884799241,4cyclictest27923-21munin-run08:20:011
2884799241,15cyclictest0-21swapper/107:50:251
2884799241,14cyclictest6868-21cpuspeed_turbos08:45:131
2884799241,11cyclictest0-21swapper/111:14:481
2884799241,10cyclictest0-21swapper/109:39:151
2884799241,10cyclictest0-21swapper/109:03:341
2884799240,12cyclictest0-21swapper/107:57:311
2885899231,11cyclictest0-21swapper/311:56:463
2885899231,11cyclictest0-21swapper/310:20:103
28858992310,6cyclictest833-21runrttasks11:44:203
2885299231,3cyclictest31024-21hddtemp_smartct08:25:182
2884799231,9cyclictest0-21swapper/107:32:321
2884799231,10cyclictest0-21swapper/111:43:131
2884799231,10cyclictest0-21swapper/110:44:241
2884199234,3cyclictest0-21swapper/010:46:480
2884199231,11cyclictest0-21swapper/011:46:170
2884799221,10cyclictest0-21swapper/108:30:091
2884199226,8cyclictest0-21swapper/008:54:160
2884199226,6cyclictest0-21swapper/011:14:570
2884199223,8cyclictest991750kworker/u8:111:39:220
2884199222,11cyclictest3966-21/usr/sbin/munin07:25:140
2884199221,5cyclictest0-21swapper/012:12:170
2884199221,15cyclictest0-21swapper/009:30:580
185652220,3sleep32885899cyclictest10:25:133
2885899218,5cyclictest401ktimersoftd/311:35:013
2885899216,5cyclictest0-21swapper/312:39:553
2885899210,11cyclictest0-21swapper/312:08:133
2885299213,5cyclictest20914-40dnf11:44:192
2884799211,11cyclictest0-21swapper/108:13:171
2884199218,4cyclictest0-21swapper/011:30:250
2884199216,5cyclictest0-21swapper/010:14:080
2884199216,4cyclictest0-21swapper/010:15:220
2884199216,3cyclictest0-21swapper/007:20:050
2884199216,14cyclictest0-21swapper/010:00:020
2884199211,6cyclictest2154-21cat09:45:250
2884199211,6cyclictest11137-21tune2fs07:40:150
2884199211,12cyclictest12418-21/usr/sbin/munin11:25:120
2884199211,12cyclictest0-21swapper/010:40:160
2884199211,11cyclictest6783-21/usr/sbin/munin08:45:130
65462200,1sleep00-21swapper/008:45:010
2885899208,4cyclictest0-21swapper/309:50:223
2885899206,4cyclictest0-21swapper/312:14:503
2885899206,4cyclictest0-21swapper/309:32:263
2885899204,9cyclictest86550irq/126-enp1s0-08:25:253
2885899202,3cyclictest86550irq/126-enp1s0-09:45:183
2885899202,10cyclictest1405-21/usr/sbin/munin11:00:193
2885899201,4cyclictest0-21swapper/310:06:273
2885899200,9cyclictest0-21swapper/310:46:113
2885899200,9cyclictest0-21swapper/309:42:383
2885899200,11cyclictest0-21swapper/312:21:103
2885899200,10cyclictest0-21swapper/311:16:563
2885299204,3cyclictest6363-21expr07:30:112
2885299202,8cyclictest29414-21sensors_temp08:20:252
2885299202,5cyclictest26917-21diskstats09:30:142
2885299201,8cyclictest25771-1kworker/u9:012:17:302
2885299201,7cyclictest31119-21cpuspeed_turbos09:40:112
2885299201,6cyclictest5648-21ntp_states08:40:212
2885299201,6cyclictest31930-21df_inode10:55:132
2885299201,6cyclictest12418-21/usr/sbin/munin11:25:142
2885299201,6cyclictest0-21swapper/211:00:522
2885299201,5cyclictest10617-21fschecks_count11:20:132
28852992010,8cyclictest18594-21timerwakeupswit07:55:252
2884199209,3cyclictest10842-21/usr/sbin/munin12:35:220
2884199209,3cyclictest0-21swapper/011:24:290
2884199205,3cyclictest0-21swapper/012:00:290
2884199202,10cyclictest3631-21/usr/sbin/munin11:05:220
2884199202,10cyclictest14959-21/usr/sbin/munin07:50:240
2884199201,6cyclictest27929-21perf08:20:010
2884199201,6cyclictest27929-21perf08:20:000
28841992014,3cyclictest7398-21find07:30:230
2884199201,12cyclictest0-21swapper/012:05:210
2884199201,10cyclictest9939-21sshd11:19:250
2884199200,10cyclictest0-21swapper/011:45:010
80132190,3sleep38016-21sendmail_mailqu08:45:243
6252191,14sleep20-21swapper/212:10:192
2885899197,5cyclictest39-21rcuc/307:37:483
2885899194,6cyclictest0-21swapper/311:28:043
2885899192,9cyclictest24324-21memory11:50:193
2885899192,9cyclictest24324-21memory11:50:183
2885899192,14cyclictest0-21swapper/312:00:133
2885899191,9cyclictest0-21swapper/309:57:243
2885899191,8cyclictest0-21swapper/309:37:113
2885899191,8cyclictest0-21swapper/309:29:193
2885899191,8cyclictest0-21swapper/309:29:193
2885899191,7cyclictest0-21swapper/310:35:133
28858991914,3cyclictest15523-21grep10:15:263
28858991913,4cyclictest14927-21grep09:00:263
2885899191,10cyclictest0-21swapper/311:35:133
28858991910,3cyclictest5841-21cat11:10:123
2885299193,6cyclictest3631-21/usr/sbin/munin11:05:182
2885299191,9cyclictest0-21swapper/211:14:212
2885299191,8cyclictest6226-21munin-run12:25:022
2885299191,5cyclictest23136-21diskstats10:35:142
28852991914,3cyclictest1143-21users07:15:272
28852991913,3cyclictest15286-21sed11:30:142
28852991911,6cyclictest10251-21gltestperf10:05:152
2884199197,3cyclictest21482-21unixbench_multi09:15:250
2884199195,8cyclictest0-21swapper/011:51:170
2884199195,8cyclictest0-21swapper/011:51:170
2884199195,6cyclictest0-21swapper/010:21:480
2884199193,8cyclictest0-21swapper/011:00:430
2884199191,6cyclictest30647-21sensors_temp10:50:230
2884199191,12cyclictest0-21swapper/009:44:460
2884199191,10cyclictest3376-21grep08:35:180
2884199191,10cyclictest31613-21sshd08:25:250
2884199191,10cyclictest21558-21/usr/sbin/munin08:05:200
237802190,2sleep00-21swapper/009:20:300
42662181,2sleep20-21swapper/208:39:022
297702180,3sleep00-21swapper/008:20:370
2885899186,3cyclictest86850irq/129-enp1s0-08:15:223
2885899186,3cyclictest86850irq/129-enp1s0-08:15:223
2885899185,4cyclictest781-21sshd07:15:253
2885899184,5cyclictest25132-21kworker/3:112:25:033
2885899183,5cyclictest4240-21grep11:05:153
2885899182,5cyclictest41-21ksoftirqd/312:30:153
2885899182,2cyclictest86550irq/126-enp1s0-11:45:253
2885899181,9cyclictest26643-21taskset10:41:073
28858991814,3cyclictest0-21swapper/309:24:303
28858991814,3cyclictest0-21swapper/309:11:463
28858991811,4cyclictest2161-21ls08:35:003
2885299188,2cyclictest86750irq/128-enp1s0-09:59:592
2885299184,11cyclictest24884-21sed08:10:222
2885299183,2cyclictest9020-21grep11:15:192
2885299181,5cyclictest20342-21needreboot08:00:182
2885299181,5cyclictest15469-21cut07:50:152
2885299181,5cyclictest10842-21/usr/sbin/munin12:35:192
2885299181,4cyclictest0-21swapper/210:25:102
2885299181,4cyclictest0-21swapper/210:01:202
2885299181,3cyclictest0-21swapper/209:21:332
28852991812,3cyclictest16202-21/usr/sbin/munin10:20:152
2885299181,15cyclictest0-21swapper/207:28:072
2885299181,14cyclictest7555-21ls12:25:232
2885299181,14cyclictest23074-21unixbench_multi08:05:272
28852991811,4cyclictest16174-21tail09:05:162
2885299181,13cyclictest0-21swapper/207:20:052
2884199189,2cyclictest11176-21/usr/sbin/munin08:55:210
2884199185,4cyclictest31249-21cat07:15:020
2884199183,9cyclictest26340-21latency_hist09:30:020
2884199183,9cyclictest26340-21latency_hist09:30:020
2884199183,3cyclictest0-21swapper/012:20:050
2884199181,9cyclictest20307-21sshd10:29:240
2884199181,9cyclictest0-21swapper/008:00:160
2884199181,8cyclictest0-21swapper/012:27:390
2884199181,6cyclictest0-21swapper/008:30:200
2884199181,5cyclictest0-21swapper/009:11:520
2884199181,11cyclictest0-21swapper/011:56:540
2884199181,11cyclictest0-21swapper/007:46:370
123442181,14sleep312345-40dnf07:42:483
293202171,2sleep30-21swapper/308:20:243
2885899177,2cyclictest39-21rcuc/308:35:213
2885899176,8cyclictest31011-21munin-plugin-st07:15:003
2885899174,4cyclictest11176-21/usr/sbin/munin08:55:273
2885899172,6cyclictest0-21swapper/308:40:233
2885899171,9cyclictest0-21swapper/312:15:183
2885899171,8cyclictest0-21swapper/310:33:363
2885899171,6cyclictest0-21swapper/310:55:593
2885899171,6cyclictest0-21swapper/309:05:153
2885899171,3cyclictest0-21swapper/311:21:373
28858991712,3cyclictest9104-21grep08:50:143
28858991712,3cyclictest13895-21pmu-power07:45:213
2885899171,15cyclictest0-21swapper/307:30:273
2885899171,15cyclictest0-21swapper/307:28:503
2885299177,7cyclictest27715-21fschecks_time10:45:142
2885299176,3cyclictest86550irq/126-enp1s0-08:50:162
2885299171,7cyclictest14001-21/usr/sbin/munin10:15:152
2885299171,7cyclictest0-21swapper/210:30:342
2885299171,6cyclictest21039-21ntp_states09:15:212
2885299171,6cyclictest0-21swapper/209:38:282
2885299171,15cyclictest0-21swapper/208:33:132
2885299171,14cyclictest0-21swapper/207:11:432
28852991711,3cyclictest26863-21memory08:15:192
28852991711,3cyclictest26863-21memory08:15:192
28852991711,3cyclictest1565-21awk09:45:172
28852991711,3cyclictest12095-21expr10:10:122
28852991710,4cyclictest24739-21fschecks_count09:25:142
28852991710,4cyclictest24739-21fschecks_count09:25:142
28852991710,4cyclictest21344-21cut11:45:122
2885299170,2cyclictest11444-21expr08:55:122
2885299170,11cyclictest0-21swapper/211:50:492
2885299170,11cyclictest0-21swapper/211:50:492
2884199179,5cyclictest851-21sssd_nss10:35:150
2884199172,6cyclictest0-21swapper/010:00:190
2884199171,9cyclictest0-21swapper/009:36:090
2884199171,14cyclictest0-21swapper/009:00:060
2884199171,13cyclictest2055-21/usr/sbin/munin12:15:160
174262170,4sleep230-21rcuc/211:35:142
28858991612,2cyclictest39-21rcuc/307:55:143
28858991611,3cyclictest14959-21/usr/sbin/munin07:50:173
2885299162,4cyclictest26211-21grep11:55:152
2885299161,6cyclictest0-21swapper/209:53:002
2885299161,14cyclictest0-21swapper/207:42:462
2885299161,13cyclictest30343-21expr12:05:122
2885299161,12cyclictest9325-21hddtemp_smartct12:30:162
28852991610,4cyclictest9012-21tail07:35:152
28852991610,4cyclictest7083-21expr08:45:132
28841991611,3cyclictest15671-21wc09:05:130
2884199161,13cyclictest0-21swapper/008:10:070
2884199160,5cyclictest86550irq/126-enp1s0-07:15:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional