You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-31 - 22:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Aug 31, 2026 12:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
158961521340,3sleep33056-21PK-Backend09:35:223
162770721310,4sleep20-21swapper/211:25:012
154945121200,3sleep342-21ksoftirqd/307:40:183
157904821160,3sleep33056-21PK-Backend09:05:193
157853321150,3sleep00-21swapper/009:05:120
164000421120,3sleep00-21swapper/012:00:110
155830821030,3sleep20-21swapper/208:05:202
155830821030,3sleep20-21swapper/208:05:192
16013122970,2sleep10-21swapper/110:10:011
153707528240,14sleep00-21swapper/007:07:070
153710927739,33sleep20-21swapper/207:07:342
153715127540,13sleep30-21swapper/307:08:063
153704626140,14sleep10-21swapper/107:06:441
15978332580,1chrt1597600-21/usr/sbin/munin10:00:122
16212542470,3sleep33056-21PK-Backend11:05:173
15686862440,3sleep3411rcuc/308:35:193
16338612430,5sleep23056-21PK-Backend11:40:202
15637262420,3sleep23056-21PK-Backend08:20:232
16318032380,3sleep30-21swapper/311:35:163
16068532350,2sleep00-21swapper/010:25:120
15629312270,17sleep00-21swapper/008:20:140
15816022260,4sleep21552281-21kworker/u8:109:15:002
139099260,4rtkit-daemon1389-21rtkit-daemon08:11:430
139099250,4rtkit-daemon1389-21rtkit-daemon09:32:031
139099250,4rtkit-daemon1389-21rtkit-daemon09:06:441
153744899243,3cyclictest26-21ksoftirqd/111:50:011
1537448992419,4cyclictest26-21ksoftirqd/112:06:171
1537448992418,5cyclictest26-21ksoftirqd/112:01:161
1537448992416,6cyclictest1599033-21sshd10:04:311
153744899241,4cyclictest26-21ksoftirqd/111:35:211
1537448992411,8cyclictest1618947-21taskset11:00:111
153744899240,6cyclictest3056-21PK-Backend11:25:001
153744899240,22cyclictest1609765-21munin-run10:35:001
139099240,4rtkit-daemon1389-21rtkit-daemon08:59:590
139099240,4rtkit-daemon1389-21rtkit-daemon08:17:291
139099240,3rtkit-daemon1389-21rtkit-daemon12:09:570
1537455992314,7cyclictest1637985-21cron11:55:002
1537448992321,1cyclictest0-21swapper/107:33:441
1537443992321,1cyclictest0-21swapper/007:37:350
1537443992320,2cyclictest3056-21PK-Backend11:13:250
153744399230,21cyclictest1561345-21df_inode08:15:140
139099230,4rtkit-daemon1389-21rtkit-daemon12:11:391
139099230,4rtkit-daemon1389-21rtkit-daemon10:22:341
1537458992220,1cyclictest0-21swapper/307:23:333
153745899220,21cyclictest3056-21PK-Backend10:19:013
153745899220,21cyclictest3056-21PK-Backend10:04:483
153745899220,20cyclictest3056-21PK-Backend12:07:163
1537455992220,1cyclictest0-21swapper/212:07:192
1537455992220,1cyclictest0-21swapper/212:03:072
1537455992220,1cyclictest0-21swapper/211:13:392
1537455992220,1cyclictest0-21swapper/208:54:072
1537455992220,1cyclictest0-21swapper/208:38:202
1537455992220,1cyclictest0-21swapper/208:04:252
1537455992220,1cyclictest0-21swapper/207:45:262
153745599220,21cyclictest0-21swapper/212:13:072
153745599220,21cyclictest0-21swapper/210:23:302
153745599220,21cyclictest0-21swapper/209:36:192
1537448992220,1cyclictest0-21swapper/110:41:151
153744899221,8cyclictest15637322wc08:20:231
153744899220,20cyclictest15389902chrt07:13:451
1537443992220,1cyclictest3056-21PK-Backend11:25:290
1537443992220,1cyclictest0-21swapper/011:54:010
1537443992220,1cyclictest0-21swapper/010:53:520
1537443992220,1cyclictest0-21swapper/010:13:230
1537443992220,1cyclictest0-21swapper/010:06:330
1537443992220,1cyclictest0-21swapper/009:32:540
1537443992220,1cyclictest0-21swapper/009:29:360
1537443992220,1cyclictest0-21swapper/007:21:550
153745899218,12cyclictest1558967-21sh08:10:003
153745899218,12cyclictest1558967-21sh08:10:003
1537458992119,1cyclictest3056-21PK-Backend11:45:543
1537458992113,6cyclictest1540730-21python307:15:223
153745899210,20cyclictest0-21swapper/307:25:503
153745899210,19cyclictest3056-21PK-Backend11:50:543
153745899210,19cyclictest0-21swapper/312:20:143
1537455992119,1cyclictest0-21swapper/212:38:222
1537455992119,1cyclictest0-21swapper/212:18:072
1537455992119,1cyclictest0-21swapper/211:59:422
1537455992119,1cyclictest0-21swapper/211:48:402
1537455992119,1cyclictest0-21swapper/211:25:422
1537455992119,1cyclictest0-21swapper/211:01:352
1537455992119,1cyclictest0-21swapper/210:17:152
1537455992119,1cyclictest0-21swapper/208:18:382
1537455992119,1cyclictest0-21swapper/207:59:392
1537455992119,1cyclictest0-21swapper/207:50:262
153745599210,20cyclictest0-21swapper/212:25:242
153745599210,20cyclictest0-21swapper/208:56:172
153744899219,4cyclictest26-21ksoftirqd/111:31:441
1537448992119,1cyclictest3056-21PK-Backend12:20:591
1537448992119,1cyclictest0-21swapper/112:30:051
1537448992119,1cyclictest0-21swapper/111:57:401
1537448992119,1cyclictest0-21swapper/111:52:401
1537448992119,1cyclictest0-21swapper/111:43:281
1537448992119,1cyclictest0-21swapper/111:06:041
1537448992119,1cyclictest0-21swapper/110:47:311
1537448992119,1cyclictest0-21swapper/110:08:511
1537448992119,1cyclictest0-21swapper/108:57:121
1537448992119,1cyclictest0-21swapper/108:42:461
1537448992119,1cyclictest0-21swapper/108:14:421
153744899210,20cyclictest0-21swapper/112:39:451
153744899210,20cyclictest0-21swapper/110:17:021
153744899210,20cyclictest0-21swapper/109:28:151
153744899210,20cyclictest0-21swapper/108:02:441
1537443992119,1cyclictest0-21swapper/012:24:500
1537443992119,1cyclictest0-21swapper/011:59:010
1537443992119,1cyclictest0-21swapper/011:09:400
1537443992119,1cyclictest0-21swapper/010:32:430
1537443992119,1cyclictest0-21swapper/009:58:220
1537443992119,1cyclictest0-21swapper/009:45:310
1537443992119,1cyclictest0-21swapper/008:40:010
1537443992119,1cyclictest0-21swapper/008:36:430
1537443992119,1cyclictest0-21swapper/007:34:040
153744399210,20cyclictest0-21swapper/012:16:590
153744399210,20cyclictest0-21swapper/007:18:360
1537458992018,1cyclictest0-21swapper/307:10:013
1537458992017,2cyclictest3056-21PK-Backend12:29:203
1537458992017,2cyclictest3056-21PK-Backend10:34:493
1537458992017,2cyclictest3056-21PK-Backend08:46:003
153745899200,1cyclictest3056-21PK-Backend10:53:353
153745899200,1cyclictest3056-21PK-Backend08:43:163
153745899200,19cyclictest3056-21PK-Backend11:01:053
153745899200,19cyclictest3056-21PK-Backend08:19:373
1537455992018,1cyclictest3056-21PK-Backend10:05:332
1537455992018,1cyclictest0-21swapper/211:37:042
1537455992018,1cyclictest0-21swapper/210:37:362
1537455992018,1cyclictest0-21swapper/209:42:202
1537455992018,1cyclictest0-21swapper/209:33:002
1537455992018,1cyclictest0-21swapper/209:21:242
1537455992018,1cyclictest0-21swapper/208:45:082
1537455992018,1cyclictest0-21swapper/207:42:562
1537455992018,1cyclictest0-21swapper/207:30:052
1537455992018,1cyclictest0-21swapper/207:25:322
1537455992018,1cyclictest0-21swapper/207:19:582
1537455992013,5cyclictest1548222-21sshd07:36:212
153745599200,19cyclictest3056-21PK-Backend08:25:412
153745599200,19cyclictest0-21swapper/212:24:232
153745599200,19cyclictest0-21swapper/211:09:062
153745599200,19cyclictest0-21swapper/210:56:152
153745599200,19cyclictest0-21swapper/210:27:012
153745599200,19cyclictest0-21swapper/209:53:492
1537448992018,1cyclictest0-21swapper/110:51:021
1537448992018,1cyclictest0-21swapper/109:17:071
1537448992018,1cyclictest0-21swapper/108:05:551
1537448992018,1cyclictest0-21swapper/108:05:551
1537448992016,3cyclictest26-21ksoftirqd/111:10:351
1537448992016,3cyclictest251rcuc/109:10:151
153744899200,7cyclictest1611475-21cron10:40:001
153744899200,2cyclictest3056-21PK-Backend07:45:011
153744899200,1cyclictest3056-21PK-Backend07:19:121
153744899200,19cyclictest0-21swapper/109:54:381
153744899200,19cyclictest0-21swapper/109:21:191
153744899200,19cyclictest0-21swapper/109:02:541
153744899200,19cyclictest0-21swapper/108:34:141
1537443992018,1cyclictest3056-21PK-Backend07:10:250
1537443992018,1cyclictest0-21swapper/012:37:340
1537443992018,1cyclictest0-21swapper/010:55:270
1537443992018,1cyclictest0-21swapper/009:52:560
1537443992018,1cyclictest0-21swapper/009:40:510
1537443992018,1cyclictest0-21swapper/008:47:250
1537443992018,1cyclictest0-21swapper/008:04:390
1537443992018,1cyclictest0-21swapper/007:46:070
1537443992012,6cyclictest1582347-21memory09:15:160
153744399200,19cyclictest0-21swapper/012:33:560
153744399200,19cyclictest0-21swapper/011:20:560
153744399200,19cyclictest0-21swapper/010:41:080
153744399200,19cyclictest0-21swapper/009:36:120
153744399200,19cyclictest0-21swapper/009:23:490
153744399200,19cyclictest0-21swapper/007:42:220
153744399200,18cyclictest1611492-21perf10:40:000
139099200,2rtkit-daemon1389-21rtkit-daemon08:31:000
1537458991917,1cyclictest3056-21PK-Backend12:38:463
1537458991917,1cyclictest3056-21PK-Backend10:43:483
1537458991917,1cyclictest3056-21PK-Backend10:07:323
1537458991917,1cyclictest3056-21PK-Backend08:13:493
1537458991917,1cyclictest0-21swapper/311:16:063
1537458991916,2cyclictest3056-21PK-Backend10:10:153
1537458991916,2cyclictest3056-21PK-Backend09:40:073
1537458991916,2cyclictest3056-21PK-Backend08:22:143
1537458991912,5cyclictest1095-21runrttasks11:10:183
1537458991911,6cyclictest1554412-21df_inode07:55:143
153745899190,1cyclictest3056-21PK-Backend12:31:093
153745899190,1cyclictest3056-21PK-Backend09:54:413
153745899190,1cyclictest3056-21PK-Backend09:27:113
153745899190,1cyclictest3056-21PK-Backend09:19:053
153745899190,1cyclictest3056-21PK-Backend09:14:193
153745899190,1cyclictest3056-21PK-Backend08:30:523
153745899190,1cyclictest3056-21PK-Backend07:34:433
153745899190,1cyclictest0-21swapper/308:25:393
153745899190,18cyclictest3056-21PK-Backend11:57:293
153745899190,18cyclictest0-21swapper/311:22:013
1537455991917,1cyclictest0-21swapper/210:49:322
1537455991917,1cyclictest0-21swapper/209:45:042
1537455991917,1cyclictest0-21swapper/209:25:512
1537455991917,1cyclictest0-21swapper/208:41:022
1537455991917,1cyclictest0-21swapper/208:10:132
1537455991917,1cyclictest0-21swapper/207:23:152
1537455991910,7cyclictest1625656-21sshd11:17:312
153745599190,1cyclictest0-21swapper/212:31:532
153745599190,1cyclictest0-21swapper/210:54:392
153745599190,1cyclictest0-21swapper/210:42:292
153745599190,1cyclictest0-21swapper/210:10:592
153744899197,4cyclictest26-21ksoftirqd/107:55:351
153744899197,4cyclictest26-21ksoftirqd/107:20:541
153744899196,8cyclictest26-21ksoftirqd/110:58:361
153744899196,4cyclictest26-21ksoftirqd/109:46:351
153744899194,9cyclictest26-21ksoftirqd/107:30:001
1537448991916,2cyclictest0-21swapper/112:26:271
1537448991912,5cyclictest1625137-21cat11:15:211
1537448991910,7cyclictest1596285-21fw_packets09:55:151
1537448991910,2cyclictest141rcu_preempt12:19:491
153744899190,6cyclictest3056-21PK-Backend07:40:001
153744899190,3cyclictest1606551-21apt10:25:111
153744899190,1cyclictest0-21swapper/108:28:191
153744899190,18cyclictest0-21swapper/109:35:251
153744899190,18cyclictest0-21swapper/108:53:411
1537443991917,1cyclictest0-21swapper/011:40:020
1537443991917,1cyclictest0-21swapper/011:02:580
1537443991917,1cyclictest0-21swapper/009:12:190
1537443991917,1cyclictest0-21swapper/009:02:530
1537443991917,1cyclictest0-21swapper/007:56:080
153744399190,1cyclictest0-21swapper/012:28:280
153744399190,1cyclictest0-21swapper/011:45:360
153744399190,1cyclictest0-21swapper/008:28:050
153744399190,1cyclictest0-21swapper/007:25:130
1537458991816,1cyclictest3056-21PK-Backend09:32:573
1537458991816,1cyclictest0-21swapper/307:45:303
1537458991812,5cyclictest1628172-21expr11:25:143
1537458991810,6cyclictest1645201-21find12:15:113
1537458991810,5cyclictest1630464-21python311:30:223
153745899180,1cyclictest0-21swapper/309:45:213
153745599187,7cyclictest1567649-21latency_hist08:35:002
1537455991813,4cyclictest1579755-21cron09:10:012
153745599181,13cyclictest1581770-21apt09:15:112
153745599180,1cyclictest0-21swapper/207:11:332
1537448991816,1cyclictest0-21swapper/109:40:051
1537448991811,6cyclictest1572431-21sed08:45:241
153744899181,13cyclictest1568777-21sshd08:35:201
153744899180,6cyclictest3056-21PK-Backend07:45:201
1537443991816,1cyclictest0-21swapper/012:12:130
1537443991816,1cyclictest0-21swapper/010:02:280
1537443991816,1cyclictest0-21swapper/007:51:070
1537443991814,3cyclictest1603692-21missed_timers10:15:170
1537443991812,4cyclictest1629897-21cat11:30:130
153745899171,4cyclictest1553078-21/usr/sbin/munin07:50:213
1537458991714,2cyclictest3056-21PK-Backend08:59:053
1537458991714,2cyclictest3056-21PK-Backend08:52:493
1537458991712,3cyclictest1547797-21ntp_states07:35:193
1537458991711,5cyclictest42-21ksoftirqd/311:40:133
153745899171,14cyclictest0-21swapper/310:40:013
153745899170,15cyclictest1595869-21/usr/sbin/munin09:55:183
1537455991715,1cyclictest0-21swapper/210:34:042
1537455991715,1cyclictest0-21swapper/209:01:242
153744899173,7cyclictest1628244-21chrt11:25:161
1537443991715,1cyclictest0-21swapper/010:49:320
1537458991614,1cyclictest3056-21PK-Backend10:55:103
1537458991614,1cyclictest3056-21PK-Backend09:01:143
1537458991614,1cyclictest3056-21PK-Backend08:02:413
153745599162,8cyclictest1629600-21apt11:30:122
153744899160,5cyclictest26-21ksoftirqd/107:51:091
1537443991614,1cyclictest0-21swapper/011:15:010
153744399161,11cyclictest1604823-21apt10:20:130
153744399161,11cyclictest1557511-21apt08:05:110
153744399161,11cyclictest1557511-21apt08:05:110
153745899159,5cyclictest1615418-21cat10:50:013
1537458991513,1cyclictest3056-21PK-Backend09:21:353
1537458991512,2cyclictest3056-21PK-Backend10:25:163
1537458991511,3cyclictest3056-21PK-Backend10:24:143
1537455991513,1cyclictest0-21swapper/209:55:592
153745899149,4cyclictest3056-21PK-Backend12:11:173
153744399148,5cyclictest1632008-21grep11:35:180
1537443991412,1cyclictest0-21swapper/008:52:180
1537458991310,2cyclictest3056-21PK-Backend12:01:313
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional