You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-15 - 00:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Jul 14, 2026 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
393989921240,3sleep33056-21PK-Backend12:15:243
392230521100,3sleep33056-21PK-Backend11:30:013
386577421100,4sleep20-21swapper/208:45:192
386577421100,4sleep20-21swapper/208:45:192
389796921000,4sleep20-21swapper/210:20:112
38375872980,3sleep23056-21PK-Backend07:25:222
38728162900,4sleep30-21swapper/309:05:213
383059326842,21sleep30-21swapper/307:09:293
383045726343,14sleep00-21swapper/007:07:460
383036726140,14sleep20-21swapper/207:06:352
383029826040,14sleep10-21swapper/107:05:411
38893312420,4sleep20-21swapper/209:55:132
383077299323,14cyclictest3901247-21taskset10:27:571
383077299313,8cyclictest0-21swapper/111:35:011
383077299304,10cyclictest3832473-21taskset07:14:581
383077299294,15cyclictest3846830-21taskset07:50:511
383077299294,14cyclictest3873342-21taskset09:08:581
383077299293,13cyclictest3941791-21taskset12:21:411
383077299285,14cyclictest3885627-21latency_hist09:45:001
38914862270,3sleep00-21swapper/010:00:160
383077299278,11cyclictest38512511cstates08:05:121
383077299277,17cyclictest3912813-21ntp_kernel_pll_11:00:191
383077299274,8cyclictest3914936-21taskset11:05:431
139099270,4rtkit-daemon1389-21rtkit-daemon11:27:470
139099260,4rtkit-daemon1389-21rtkit-daemon12:39:460
383077299254,10cyclictest3947241-21runrttasks12:36:261
383077299253,8cyclictest3838416-21runrttasks07:30:031
3830772992518,4cyclictest3876905-21sh09:20:001
3830772992514,9cyclictest3849740-21load08:00:161
383077799240,22cyclictest3056-21PK-Backend09:32:133
383077499241,21cyclictest3884952-21cut09:40:202
383077299245,6cyclictest3934601-21apt-get12:05:021
383077299245,6cyclictest3882120-21perf09:35:001
383077299244,9cyclictest3890415-21taskset09:55:411
383077299241,9cyclictest0-21swapper/111:25:111
3830772992415,7cyclictest3904880-21cron10:40:001
3830772992410,9cyclictest3856328-21apt-get08:20:041
139099240,3rtkit-daemon1389-21rtkit-daemon09:46:510
139099240,3rtkit-daemon1389-21rtkit-daemon09:46:510
38873392230,4chrt3056-21PK-Backend09:50:003
38873392230,4chrt3056-21PK-Backend09:50:003
383077499230,21cyclictest3931125-21/usr/sbin/munin11:55:142
383077299231,9cyclictest0-21swapper/107:40:111
383077299230,21cyclictest3870537-21cstates09:00:121
39214742220,2sleep23056-21PK-Backend11:25:192
38415042220,3sleep33056-21PK-Backend07:35:233
3830777992220,1cyclictest3056-21PK-Backend08:25:553
3830777992213,7cyclictest3897695-21cron10:20:003
3830774992220,1cyclictest0-21swapper/212:32:122
3830774992220,1cyclictest0-21swapper/212:06:492
3830774992220,1cyclictest0-21swapper/211:24:532
3830774992220,1cyclictest0-21swapper/211:05:532
3830774992220,1cyclictest0-21swapper/209:50:022
383077499220,5cyclictest0-21swapper/207:15:462
383077499220,21cyclictest0-21swapper/210:51:542
383077499220,20cyclictest3918761-21munin-plugin-st11:20:002
383077499220,20cyclictest1788598-21PressureMonitor10:35:592
383077299226,15cyclictest3939667-21ps12:15:221
383077299224,6cyclictest3892767-21apt10:05:111
3830772992220,1cyclictest0-21swapper/112:02:231
3830772992217,4cyclictest0-21swapper/108:55:001
3830767992220,1cyclictest0-21swapper/009:03:490
3830767992220,1cyclictest0-21swapper/008:38:070
3830767992214,6cyclictest3914212-21fw_packets11:05:150
383076799220,21cyclictest0-21swapper/011:33:400
383077799217,4cyclictest42-21ksoftirqd/308:23:303
3830777992119,1cyclictest3056-21PK-Backend07:52:213
3830777992118,2cyclictest3056-21PK-Backend11:35:413
383077799210,9cyclictest3056-21PK-Backend12:00:113
383077799210,2cyclictest0-21swapper/308:06:423
383077799210,20cyclictest3056-21PK-Backend09:18:203
383077799210,20cyclictest3056-21PK-Backend08:51:163
383077799210,19cyclictest3056-21PK-Backend12:34:013
383077799210,19cyclictest3056-21PK-Backend11:50:163
383077799210,19cyclictest3056-21PK-Backend10:40:403
383077799210,19cyclictest3056-21PK-Backend08:03:243
3830774992119,1cyclictest3056-21PK-Backend10:32:212
3830774992119,1cyclictest0-21swapper/212:36:592
3830774992119,1cyclictest0-21swapper/212:27:252
3830774992119,1cyclictest0-21swapper/211:02:492
3830774992119,1cyclictest0-21swapper/209:11:092
3830774992119,1cyclictest0-21swapper/208:50:202
3830774992119,1cyclictest0-21swapper/208:26:002
3830774992119,1cyclictest0-21swapper/207:30:442
3830774992118,2cyclictest3056-21PK-Backend08:21:552
383077499210,20cyclictest0-21swapper/210:48:502
383077499210,20cyclictest0-21swapper/210:48:492
383077499210,20cyclictest0-21swapper/209:35:322
383077499210,19cyclictest0-21swapper/209:45:122
383077499210,19cyclictest0-21swapper/209:45:112
383077299216,13cyclictest3863358-21cpuspeed08:40:121
3830772992119,1cyclictest0-21swapper/112:31:431
3830772992119,1cyclictest0-21swapper/112:13:251
3830772992119,1cyclictest0-21swapper/111:47:021
3830772992119,1cyclictest0-21swapper/111:47:011
3830772992119,1cyclictest0-21swapper/111:43:371
3830772992119,1cyclictest0-21swapper/111:37:221
3830772992119,1cyclictest0-21swapper/109:49:201
3830772992119,1cyclictest0-21swapper/109:49:201
3830772992119,1cyclictest0-21swapper/109:37:441
3830772992119,1cyclictest0-21swapper/108:39:101
3830772992118,2cyclictest0-21swapper/111:50:261
3830772992110,9cyclictest3896626-21sshd10:15:171
3830772992110,9cyclictest3848924-21awk08:00:001
383077299210,20cyclictest0-21swapper/112:26:221
383077299210,20cyclictest0-21swapper/108:11:321
383077299210,20cyclictest0-21swapper/107:27:051
3830767992119,1cyclictest0-21swapper/012:08:020
3830767992119,1cyclictest0-21swapper/011:01:230
3830767992119,1cyclictest0-21swapper/010:21:560
3830767992119,1cyclictest0-21swapper/009:56:140
3830767992119,1cyclictest0-21swapper/009:54:520
3830767992119,1cyclictest0-21swapper/009:20:520
3830767992119,1cyclictest0-21swapper/008:17:510
3830767992119,1cyclictest0-21swapper/008:01:430
3830767992119,1cyclictest0-21swapper/007:54:200
3830767992119,1cyclictest0-21swapper/007:35:410
3830767992119,1cyclictest0-21swapper/007:12:490
3830767992112,7cyclictest3857100-21ntp_states08:20:180
383076799210,20cyclictest0-21swapper/012:27:580
383076799210,20cyclictest0-21swapper/010:35:540
383076799210,20cyclictest0-21swapper/010:29:470
383076799210,20cyclictest0-21swapper/010:06:000
383076799210,1cyclictest0-21swapper/012:17:210
383076799210,1cyclictest0-21swapper/007:25:470
3830777992018,1cyclictest3056-21PK-Backend11:13:183
3830777992018,1cyclictest3056-21PK-Backend09:29:423
3830777992017,2cyclictest3056-21PK-Backend07:49:043
3830777992016,2cyclictest42-21ksoftirqd/309:04:413
3830777992013,6cyclictest3903174-21cron10:35:013
3830777992013,5cyclictest3900911-21wc10:25:223
383077799200,7cyclictest42-21ksoftirqd/310:05:103
383077799200,19cyclictest3056-21PK-Backend10:22:423
383077799200,19cyclictest3056-21PK-Backend10:02:143
383077799200,19cyclictest3056-21PK-Backend08:19:533
383077799200,18cyclictest3056-21PK-Backend08:47:583
383077799200,18cyclictest3056-21PK-Backend08:47:573
3830774992018,1cyclictest3056-21PK-Backend12:14:272
3830774992018,1cyclictest0-21swapper/210:42:072
3830774992018,1cyclictest0-21swapper/210:16:582
3830774992018,1cyclictest0-21swapper/208:36:072
3830774992018,1cyclictest0-21swapper/207:49:232
383077499200,19cyclictest0-21swapper/212:20:562
383077499200,19cyclictest0-21swapper/210:11:252
383077499200,19cyclictest0-21swapper/210:00:162
383077499200,19cyclictest0-21swapper/209:07:512
383077499200,19cyclictest0-21swapper/208:59:062
383077499200,19cyclictest0-21swapper/208:42:092
383077499200,19cyclictest0-21swapper/208:09:382
383077499200,19cyclictest0-21swapper/207:53:292
383077499200,19cyclictest0-21swapper/207:37:202
383077299209,8cyclictest3905163-21apt10:40:121
383077299208,10cyclictest3901686-21basename10:30:121
3830772992018,1cyclictest0-21swapper/111:13:091
3830772992018,1cyclictest0-21swapper/110:02:381
3830772992018,1cyclictest0-21swapper/108:33:091
3830772992015,4cyclictest0-21swapper/109:50:201
3830772992013,3cyclictest3906892-21apt10:45:101
3830772992013,3cyclictest3906892-21apt10:45:091
3830772992012,6cyclictest3917386-21cat11:15:121
3830772992011,7cyclictest3855861-21http08:20:001
3830772992011,7cyclictest3833646-21seq07:15:151
3830772992010,9cyclictest3932755-21sed12:00:011
3830772992010,8cyclictest3878909-21apt09:25:131
3830772992010,8cyclictest3845458-21awk07:50:001
383077299200,19cyclictest0-21swapper/111:22:541
383077299200,19cyclictest0-21swapper/108:57:291
383077299200,19cyclictest0-21swapper/108:49:181
383077299200,19cyclictest0-21swapper/108:49:181
3830767992018,1cyclictest0-21swapper/012:23:500
3830767992018,1cyclictest0-21swapper/011:21:310
3830767992018,1cyclictest0-21swapper/010:32:170
3830767992018,1cyclictest0-21swapper/009:27:140
3830767992018,1cyclictest0-21swapper/009:19:300
3830767992018,1cyclictest0-21swapper/008:46:520
3830767992018,1cyclictest0-21swapper/008:46:510
3830767992018,1cyclictest0-21swapper/008:32:050
3830767992018,1cyclictest0-21swapper/008:07:450
3830767992018,1cyclictest0-21swapper/007:48:180
3830767992013,6cyclictest3916285-21ntpq11:10:190
3830767992012,6cyclictest3884172-21cat09:40:110
383076799200,1cyclictest0-21swapper/007:55:420
383076799200,19cyclictest0-21swapper/010:49:550
383076799200,19cyclictest0-21swapper/010:49:550
383076799200,19cyclictest0-21swapper/010:12:090
383076799200,19cyclictest0-21swapper/009:35:320
383076799200,19cyclictest0-21swapper/009:31:400
383076799200,19cyclictest0-21swapper/008:29:210
383076799200,19cyclictest0-21swapper/008:13:470
139099200,3rtkit-daemon1389-21rtkit-daemon11:19:340
383077799194,4cyclictest42-21ksoftirqd/308:55:263
383077799194,2cyclictest141rcu_preempt09:42:203
3830777991917,1cyclictest3056-21PK-Backend10:49:393
3830777991917,1cyclictest3056-21PK-Backend10:49:393
3830777991917,1cyclictest3056-21PK-Backend09:20:163
3830777991917,1cyclictest3056-21PK-Backend09:11:233
383077799190,8cyclictest3056-21PK-Backend07:45:003
383077799190,1cyclictest3056-21PK-Backend10:56:423
383077799190,1cyclictest3056-21PK-Backend09:37:133
383077799190,1cyclictest3056-21PK-Backend08:40:493
383077799190,1cyclictest3056-21PK-Backend07:11:463
383077799190,18cyclictest3056-21PK-Backend11:23:593
383077799190,18cyclictest3056-21PK-Backend11:19:133
383077799190,18cyclictest3056-21PK-Backend09:51:393
383077799190,18cyclictest3056-21PK-Backend07:26:133
383077799190,18cyclictest3056-21PK-Backend07:24:513
383077799190,17cyclictest3056-21PK-Backend11:07:563
3830774991917,1cyclictest3056-21PK-Backend09:29:212
3830774991917,1cyclictest0-21swapper/210:56:072
3830774991917,1cyclictest0-21swapper/208:30:052
3830774991917,1cyclictest0-21swapper/207:13:272
3830774991915,3cyclictest978-21snmpd07:55:132
3830774991914,3cyclictest3870389-21tr09:00:122
383077499190,1cyclictest0-21swapper/209:23:322
383077499190,1cyclictest0-21swapper/209:18:192
383077499190,1cyclictest0-21swapper/208:16:272
383077499190,1cyclictest0-21swapper/208:12:212
383077499190,1cyclictest0-21swapper/207:20:512
383077499190,18cyclictest0-21swapper/211:33:072
383077499190,18cyclictest0-21swapper/210:25:342
383077499190,18cyclictest0-21swapper/208:02:142
383077299199,9cyclictest3899503-21cmp10:25:011
383077299199,8cyclictest3873680-21/usr/sbin/munin09:10:161
383077299197,11cyclictest3840601-21tr07:35:121
3830772991913,3cyclictest3858099-21apt08:25:131
3830772991912,3cyclictest3908628-21apt10:50:121
3830772991910,7cyclictest3895116-21grep10:10:171
3830772991910,6cyclictest3910728-21df_inode10:55:161
383077299190,1cyclictest0-21swapper/107:21:511
3830767991917,1cyclictest0-21swapper/010:57:100
3830767991917,1cyclictest0-21swapper/007:32:230
3830767991913,4cyclictest13-21ksoftirqd/007:15:150
3830767991912,6cyclictest3875168-21sh09:14:590
3830767991912,6cyclictest3871967-21awk09:05:120
3830767991912,5cyclictest3869203-21ntp_kernel_err08:55:190
3830767991910,7cyclictest3905528-21df_inode10:40:140
383076799190,18cyclictest0-21swapper/008:54:160
383077799189,3cyclictest141rcu_preempt08:35:173
383077799188,4cyclictest42-21ksoftirqd/312:09:393
383077799188,4cyclictest42-21ksoftirqd/311:44:293
383077799187,4cyclictest42-21ksoftirqd/311:55:383
383077799187,4cyclictest42-21ksoftirqd/307:59:133
3830777991815,2cyclictest3056-21PK-Backend11:33:253
3830777991815,2cyclictest3056-21PK-Backend08:14:393
3830777991810,6cyclictest3839922-21apt-get07:35:013
383077799180,2cyclictest3056-21PK-Backend11:45:213
383077799180,2cyclictest3056-21PK-Backend11:45:213
383077799180,1cyclictest3056-21PK-Backend07:18:283
3830774991816,1cyclictest0-21swapper/211:50:072
3830774991816,1cyclictest0-21swapper/209:33:132
3830774991811,5cyclictest28650irq/123-ahci[0011:35:232
3830772991811,5cyclictest3877534-21df09:20:131
3830767991816,1cyclictest0-21swapper/010:52:590
3830767991816,1cyclictest0-21swapper/010:18:180
3830767991816,1cyclictest0-21swapper/008:40:510
3830767991815,2cyclictest3930884-21latency_hist11:55:010
3830767991814,3cyclictest13-21ksoftirqd/012:01:380
3830767991811,6cyclictest3945005-21sed12:30:210
3830767991811,6cyclictest3931889-21ntp_states11:55:190
139099180,15rtkit-daemon0-21swapper/007:22:590
383077799177,4cyclictest42-21ksoftirqd/310:36:433
383077799176,2cyclictest3947467-21chrt12:39:173
383077799172,2cyclictest141rcu_preempt11:00:483
3830777991715,1cyclictest3056-21PK-Backend08:31:093
3830777991712,3cyclictest39098192sleep310:51:443
3830774991715,1cyclictest0-21swapper/210:05:172
3830774991715,1cyclictest0-21swapper/207:44:432
3830774991713,3cyclictest3915678-21cpuspeed_turbos11:10:122
3830774991711,5cyclictest3934608-21cron12:05:002
3830767991715,1cyclictest3056-21PK-Backend07:40:210
383077799169,2cyclictest141rcu_preempt12:14:393
3830777991614,1cyclictest3056-21PK-Backend09:58:013
383077799160,10cyclictest3941816-21chrt12:22:023
3830774991611,4cyclictest3927872-21cpuspeed_turbos11:45:122
3830774991611,4cyclictest3927872-21cpuspeed_turbos11:45:112
383076799169,5cyclictest3928096-21ls11:45:150
383076799169,5cyclictest3928096-21ls11:45:150
3830767991613,2cyclictest1095-21runrttasks11:42:570
3830767991611,4cyclictest3937220-21ls12:10:130
383077799159,4cyclictest3943247-21grep12:25:223
3830777991511,2cyclictest42-21ksoftirqd/310:12:183
3830774991513,1cyclictest0-21swapper/211:44:592
383077499151,13cyclictest3940347-21grep12:20:012
383076799142,11cyclictest1095-21runrttasks11:35:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional