You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-03 - 19:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Sep 03, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
330952621260,3sleep20-21swapper/210:35:122
330740921130,3sleep33307400-21turbostat10:30:003
330099921040,4sleep10-21swapper/110:10:171
139099790,66rtkit-daemon0-21swapper/307:06:393
323641426843,16sleep20-21swapper/207:05:282
323668126641,16sleep10-21swapper/107:07:281
323653126443,14sleep00-21swapper/007:05:300
32832572610,3sleep03056-21PK-Backend09:20:120
33126392530,4sleep03056-21PK-Backend10:45:010
33272272410,4sleep13056-21PK-Backend11:25:121
323702699360,26cyclictest3303947-21perf10:20:003
323702699342,22cyclictest0-21swapper/307:54:393
323702699340,18cyclictest3265697-21latency_hist08:30:003
323702699320,24cyclictest0-21swapper/311:17:393
323702699320,23cyclictest0-21swapper/309:50:373
323702699320,18cyclictest978-21snmpd09:09:423
323702699320,18cyclictest0-21swapper/307:40:493
323702699320,18cyclictest0-21swapper/307:18:403
323702699320,18cyclictest0-21swapper/307:18:403
323702699310,19cyclictest0-21swapper/307:56:453
323702699310,18cyclictest3281553-21expr09:15:133
323702699310,17cyclictest0-21swapper/312:00:103
323702699310,17cyclictest0-21swapper/308:43:383
323702699300,18cyclictest32615931timerandwakeup08:15:213
323702699300,18cyclictest0-21swapper/310:50:363
323702699300,18cyclictest0-21swapper/310:36:533
323702699300,18cyclictest0-21swapper/310:20:373
323702699300,17cyclictest0-21swapper/308:38:483
323702699300,16cyclictest0-21swapper/311:30:173
32802042290,10sleep342-21ksoftirqd/309:10:163
323702699290,17cyclictest0-21swapper/311:09:013
323702699290,17cyclictest0-21swapper/310:42:303
323702699290,17cyclictest0-21swapper/307:38:303
32666922280,8sleep126-21ksoftirqd/108:30:201
323702699281,25cyclictest3289681-21users09:35:223
323702699270,25cyclictest3308613-21find10:30:253
323702699270,25cyclictest3299801-21/usr/sbin/munin10:05:203
139099270,18rtkit-daemon3322529-21ntp_states11:10:182
323702699262,22cyclictest3348259-21cstates12:25:113
323702699262,19cyclictest3321794-21apt11:10:103
323702699261,23cyclictest3275590-21taskset08:55:343
323702699261,23cyclictest3256902-21unin-run08:05:003
323702699261,23cyclictest0-21swapper/311:00:013
323702699260,25cyclictest0-21swapper/307:30:423
323702699260,24cyclictest3292446-21diskstats09:45:133
323702699260,15cyclictest3286110-21runrttasks09:27:123
323702699252,20cyclictest0-21swapper/308:05:113
323702699251,13cyclictest3317803-21munin-run11:00:003
3237026992511,12cyclictest3237591-21df_inode07:10:143
323702699250,23cyclictest0-21swapper/310:45:173
32981042240,3chrt0-21swapper/210:00:202
32555072240,3chrt3056-21PK-Backend08:00:131
323702699247,15cyclictest883-21dbus-daemon09:34:353
323702699242,18cyclictest0-21swapper/308:45:113
323702699241,21cyclictest3262938-21memory08:20:193
323702699241,20cyclictest0-21swapper/309:20:113
323702699240,22cyclictest3331929-21latency_hist11:40:013
323702699240,22cyclictest3241383-21cstates07:20:113
323702699239,12cyclictest3259061-21grep08:10:113
323702699232,19cyclictest0-21swapper/311:40:113
323702699232,18cyclictest3344583-21apt12:15:113
323702699232,18cyclictest0-21swapper/312:20:113
323702699232,18cyclictest0-21swapper/311:45:103
323702699232,18cyclictest0-21swapper/311:45:103
3237026992315,5cyclictest3352992-21sh12:40:003
323702699231,20cyclictest1-21systemd08:55:013
323702699231,19cyclictest3326989-21awk11:25:103
3237026992311,10cyclictest1095-21runrttasks10:00:493
323702699230,21cyclictest3350459-21grep12:30:223
323702699230,21cyclictest3341277-21tr12:05:123
323702699230,12cyclictest3242915-21apt07:25:133
323702699230,11cyclictest33356422find11:50:143
323702099230,21cyclictest3056-21PK-Backend11:13:191
3237017992321,1cyclictest3056-21PK-Backend12:11:310
323701799230,21cyclictest3056-21PK-Backend08:50:210
139099230,3rtkit-daemon0-21swapper/208:55:172
33356402220,3sleep10-21swapper/111:50:141
323702699226,14cyclictest3266081-21expr08:30:123
323702699222,17cyclictest0-21swapper/311:20:103
323702699221,19cyclictest3342989-21cpuspeed_turbos12:10:163
323702699221,18cyclictest3337533-21cpuspeed_turbos11:55:093
323702699220,21cyclictest0-21swapper/307:45:203
323702699220,20cyclictest3296382-21awk09:55:203
323702699220,20cyclictest3290863-21latency09:40:173
323702699220,20cyclictest3277130-21wc09:00:213
3237024992213,7cyclictest3287179-21expr09:30:162
3237020992220,1cyclictest3056-21PK-Backend08:16:021
3237020992219,2cyclictest3056-21PK-Backend09:52:491
3237020992219,2cyclictest3056-21PK-Backend07:11:401
3237020992214,6cyclictest3298733-21cron10:05:011
323702099220,20cyclictest0-21swapper/108:35:151
3237017992220,1cyclictest3056-21PK-Backend10:25:520
3237017992220,1cyclictest0-21swapper/011:23:590
3237017992220,1cyclictest0-21swapper/010:35:590
3237017992220,1cyclictest0-21swapper/009:51:450
3237017992220,1cyclictest0-21swapper/009:41:180
3237017992219,2cyclictest0-21swapper/007:51:470
323701799220,21cyclictest3056-21PK-Backend11:16:420
323701799220,21cyclictest0-21swapper/010:31:200
323701799220,21cyclictest0-21swapper/007:12:130
323702699211,18cyclictest3300895-21cpuspeed_turbos10:10:183
3237020992119,1cyclictest3056-21PK-Backend12:14:501
3237020992119,1cyclictest3056-21PK-Backend11:39:081
3237020992119,1cyclictest3056-21PK-Backend11:33:541
3237020992119,1cyclictest3056-21PK-Backend11:20:361
3237020992119,1cyclictest3056-21PK-Backend10:33:311
3237020992119,1cyclictest3056-21PK-Backend10:16:351
3237020992119,1cyclictest3056-21PK-Backend09:35:451
3237020992119,1cyclictest0-21swapper/108:45:091
3237020992118,2cyclictest3056-21PK-Backend12:35:591
3237020992118,2cyclictest3056-21PK-Backend10:54:201
3237020992118,1cyclictest3056-21PK-Backend12:22:281
323702099210,19cyclictest3056-21PK-Backend12:19:031
323702099210,19cyclictest3056-21PK-Backend10:21:011
323702099210,19cyclictest3056-21PK-Backend09:55:251
323702099210,19cyclictest3056-21PK-Backend09:16:461
3237017992119,1cyclictest0-21swapper/012:18:200
3237017992119,1cyclictest0-21swapper/012:08:060
3237017992119,1cyclictest0-21swapper/011:38:320
3237017992119,1cyclictest0-21swapper/010:02:260
3237017992119,1cyclictest0-21swapper/009:32:390
3237017992119,1cyclictest0-21swapper/009:07:110
3237017992119,1cyclictest0-21swapper/009:01:090
3237017992119,1cyclictest0-21swapper/008:57:440
3237017992119,1cyclictest0-21swapper/008:27:220
3237017992119,1cyclictest0-21swapper/008:17:080
3237017992119,1cyclictest0-21swapper/008:10:400
3237017992119,1cyclictest0-21swapper/007:46:330
3237017992118,2cyclictest3056-21PK-Backend10:45:190
3237017992114,6cyclictest3317792-21apt-get11:00:000
323701799210,20cyclictest0-21swapper/010:08:420
323701799210,20cyclictest0-21swapper/009:56:110
323701799210,20cyclictest0-21swapper/008:31:420
323701799210,1cyclictest3056-21PK-Backend08:40:340
323701799210,19cyclictest3056-21PK-Backend08:20:060
323701799210,19cyclictest3056-21PK-Backend07:30:110
139099210,15rtkit-daemon0-21swapper/208:35:012
32520812200,3sleep20-21swapper/207:50:152
3237024992018,1cyclictest0-21swapper/211:37:442
3237024992018,1cyclictest0-21swapper/210:32:552
3237024992018,1cyclictest0-21swapper/207:13:552
3237024992014,4cyclictest978-21snmpd07:18:362
3237024992014,4cyclictest978-21snmpd07:18:362
3237024992014,4cyclictest34-21ksoftirqd/208:00:012
3237024992012,6cyclictest436-20systemd-journal11:35:002
3237024992012,6cyclictest3347181-21python312:20:202
3237024992012,6cyclictest3334074-21expr11:45:112
3237024992012,6cyclictest3334074-21expr11:45:112
323702499200,19cyclictest0-21swapper/212:30:022
323702499200,19cyclictest0-21swapper/212:08:192
323702499200,19cyclictest0-21swapper/211:09:112
323702499200,19cyclictest0-21swapper/210:40:252
323702499200,19cyclictest0-21swapper/210:21:462
323702499200,19cyclictest0-21swapper/210:10:512
323702499200,19cyclictest0-21swapper/210:07:132
323702499200,19cyclictest0-21swapper/209:19:542
323702499200,19cyclictest0-21swapper/208:52:232
323702499200,19cyclictest0-21swapper/208:16:342
323702499200,19cyclictest0-21swapper/208:13:362
323702499200,19cyclictest0-21swapper/207:47:202
323702499200,19cyclictest0-21swapper/207:26:452
323702499200,19cyclictest0-21swapper/207:24:222
323702499200,18cyclictest3056-21PK-Backend07:32:202
3237020992018,1cyclictest3056-21PK-Backend12:31:401
3237020992018,1cyclictest3056-21PK-Backend09:42:341
3237020992018,1cyclictest0-21swapper/111:16:581
3237020992018,1cyclictest0-21swapper/107:20:531
3237020992017,2cyclictest3056-21PK-Backend09:09:571
323702099200,1cyclictest3056-21PK-Backend12:05:381
323702099200,1cyclictest0-21swapper/110:25:271
323702099200,19cyclictest3056-21PK-Backend11:07:521
323702099200,19cyclictest3056-21PK-Backend11:04:131
323702099200,19cyclictest3056-21PK-Backend07:33:101
323702099200,19cyclictest3056-21PK-Backend07:17:571
323702099200,19cyclictest3056-21PK-Backend07:17:571
323702099200,19cyclictest0-21swapper/112:26:401
323702099200,19cyclictest0-21swapper/112:00:381
323702099200,18cyclictest3056-21PK-Backend07:56:221
3237017992018,1cyclictest0-21swapper/012:39:150
3237017992018,1cyclictest0-21swapper/011:34:540
3237017992018,1cyclictest0-21swapper/011:29:260
3237017992018,1cyclictest0-21swapper/009:38:400
3237017992018,1cyclictest0-21swapper/009:29:140
3237017992018,1cyclictest0-21swapper/007:43:080
3237017992018,1cyclictest0-21swapper/007:27:470
3237017992018,1cyclictest0-21swapper/007:17:540
3237017992018,1cyclictest0-21swapper/007:17:540
323701799200,19cyclictest3056-21PK-Backend12:26:580
323701799200,19cyclictest0-21swapper/012:20:570
323701799200,19cyclictest0-21swapper/011:56:160
323701799200,19cyclictest0-21swapper/010:22:140
323701799200,19cyclictest0-21swapper/010:18:350
323701799200,19cyclictest0-21swapper/010:14:570
323701799200,19cyclictest0-21swapper/008:09:520
323701799200,19cyclictest0-21swapper/008:03:300
323701799200,19cyclictest0-21swapper/007:56:130
323701799200,19cyclictest0-21swapper/007:37:200
139099200,14rtkit-daemon0-21swapper/209:20:062
3237024991917,1cyclictest0-21swapper/211:26:492
3237024991917,1cyclictest0-21swapper/211:16:412
3237024991917,1cyclictest0-21swapper/209:39:352
3237024991917,1cyclictest0-21swapper/208:46:352
3237024991917,1cyclictest0-21swapper/208:00:532
3237024991914,3cyclictest3280350-21strings09:10:182
3237024991911,6cyclictest3345996-21seq12:19:322
3237024991911,6cyclictest3338136-21ntpq11:55:162
3237024991911,6cyclictest3306322-21awk10:25:132
323702499190,1cyclictest0-21swapper/211:42:102
323702499190,1cyclictest0-21swapper/209:58:072
323702499190,1cyclictest0-21swapper/208:22:292
323702499190,18cyclictest0-21swapper/212:39:422
323702499190,18cyclictest0-21swapper/212:00:422
323702499190,18cyclictest0-21swapper/209:51:522
323702499190,18cyclictest0-21swapper/209:26:452
323702499190,18cyclictest0-21swapper/209:05:142
323702499190,18cyclictest0-21swapper/208:27:362
3237020991916,2cyclictest3056-21PK-Backend08:24:411
3237020991916,2cyclictest3056-21PK-Backend07:29:591
323702099190,1cyclictest3056-21PK-Backend10:39:461
323702099190,1cyclictest3056-21PK-Backend09:30:451
323702099190,1cyclictest3056-21PK-Backend09:14:091
323702099190,1cyclictest3056-21PK-Backend09:01:311
323702099190,1cyclictest3056-21PK-Backend07:51:081
323702099190,1cyclictest3056-21PK-Backend07:47:431
323702099190,1cyclictest3056-21PK-Backend07:39:321
3237017991917,1cyclictest0-21swapper/011:53:380
3237017991917,1cyclictest0-21swapper/009:46:310
3237017991917,1cyclictest0-21swapper/008:37:370
3237017991912,6cyclictest883-21dbus-daemon11:05:000
323701799190,1cyclictest0-21swapper/011:47:240
323701799190,1cyclictest0-21swapper/011:47:240
323701799190,1cyclictest0-21swapper/011:10:130
323701799190,18cyclictest0-21swapper/007:24:370
3237024991816,1cyclictest0-21swapper/212:14:212
3237024991816,1cyclictest0-21swapper/211:21:212
3237024991816,1cyclictest0-21swapper/210:54:242
3237024991816,1cyclictest0-21swapper/208:09:172
3237024991814,3cyclictest978-21snmpd08:37:382
3237024991813,4cyclictest3317457-21sshd10:55:472
3237024991812,5cyclictest34-21ksoftirqd/211:50:262
3237024991812,4cyclictest3249688-21munin-plugin-st07:45:002
3237024991810,6cyclictest3313864-21sed10:45:222
3237020991816,1cyclictest3056-21PK-Backend10:46:151
3237020991816,1cyclictest3056-21PK-Backend10:40:481
3237020991815,2cyclictest3056-21PK-Backend09:24:571
323701799184,4cyclictest13-21ksoftirqd/010:50:350
323702499179,6cyclictest3348046-21awk12:25:102
323702499179,6cyclictest3292072-21apt09:45:112
3237024991715,1cyclictest0-21swapper/209:41:242
3237024991712,4cyclictest34-21ksoftirqd/211:00:012
3237024991711,5cyclictest3247548-21file07:35:332
3237024991710,5cyclictest3303181-21ntpq10:15:182
323702099179,6cyclictest3249682-21apt07:45:011
3237020991715,1cyclictest3056-21PK-Backend11:49:361
3237020991715,1cyclictest3056-21PK-Backend11:49:361
3237020991715,1cyclictest3056-21PK-Backend08:41:581
3237020991714,2cyclictest3056-21PK-Backend08:12:321
3237020991714,2cyclictest3056-21PK-Backend08:07:371
3237017991715,1cyclictest0-21swapper/012:04:410
3237017991715,1cyclictest0-21swapper/009:12:240
3237017991711,5cyclictest3332568-21diskstats11:40:130
3237024991612,3cyclictest3276992-21proc_pri09:00:192
3237020991613,2cyclictest3318102-21apt-config11:00:011
3237020991611,4cyclictest3285765-21grep09:25:211
3237020991611,3cyclictest3300474-21idleruntime-cro10:10:001
323702099161,13cyclictest3264755-21needreboot08:25:171
3237020991610,4cyclictest3338398-21grep11:55:191
3237020991610,4cyclictest3292663-21ls09:45:161
3237017991614,1cyclictest0-21swapper/012:34:080
3237017991612,3cyclictest13-21ksoftirqd/011:10:000
323702499158,5cyclictest3269591-21cat08:40:132
3237020991513,1cyclictest3056-21PK-Backend11:40:571
3237020991512,2cyclictest3056-21PK-Backend08:54:561
323701799154,10cyclictest3272474-21cron08:50:010
3237017991510,2cyclictest13-21ksoftirqd/009:15:220
323702099136,6cyclictest3275134-21needreboot08:55:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional