You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-18 - 18:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Thu Jun 18, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
166241222420,7sleep21662422-21munin-plugin-st07:10:002
170935621080,3sleep03056-21PK-Backend09:20:180
175480821030,4sleep13056-21PK-Backend11:30:111
16963932980,4sleep33056-21PK-Backend08:45:013
166218927742,13sleep30-21swapper/307:07:263
166134027453,14sleep00-21swapper/007:05:180
166223726343,13sleep10-21swapper/107:08:031
16989922540,4sleep33056-21PK-Backend08:50:183
17726322460,10sleep234-21ksoftirqd/212:20:092
17313292440,4sleep00-21swapper/010:20:220
16630952380,3sleep10-21swapper/107:10:111
17322482370,4sleep33056-21PK-Backend10:25:143
17234192350,3sleep30-21swapper/310:00:163
17165532300,4sleep30-21swapper/309:40:183
17573982290,4chrt3056-21PK-Backend11:35:211
1662551992516,7cyclictest1722475-21basename10:00:013
1662541992511,12cyclictest436-20systemd-journal09:45:000
1662543992420,3cyclictest251rcuc/107:33:151
1662543992415,7cyclictest1695610-21python308:40:191
1662543992411,4cyclictest26-21ksoftirqd/111:07:451
166254399240,6cyclictest3056-21PK-Backend11:25:201
1662541992413,9cyclictest1761720-21apt-get11:50:020
16850942230,3sleep13056-21PK-Backend08:10:181
1662547992321,1cyclictest0-21swapper/210:53:512
1662547992321,1cyclictest0-21swapper/207:52:222
166254399233,4cyclictest26-21ksoftirqd/111:55:011
1662543992320,2cyclictest251rcuc/108:22:001
1662543992320,2cyclictest251rcuc/107:49:321
166254399230,6cyclictest3056-21PK-Backend09:25:141
166254199230,4cyclictest1773514-21python312:20:190
17539142220,3sleep30-21swapper/311:25:213
1662551992220,1cyclictest3056-21PK-Backend10:15:463
1662551992218,2cyclictest3056-21PK-Backend07:10:183
166255199220,20cyclictest3056-21PK-Backend11:45:163
166255199220,20cyclictest3056-21PK-Backend08:35:023
1662547992220,1cyclictest0-21swapper/212:38:282
1662547992220,1cyclictest0-21swapper/212:13:072
1662547992220,1cyclictest0-21swapper/211:03:532
1662547992220,1cyclictest0-21swapper/210:59:322
1662547992220,1cyclictest0-21swapper/210:24:452
1662547992220,1cyclictest0-21swapper/209:57:412
1662547992220,1cyclictest0-21swapper/209:54:092
1662547992220,1cyclictest0-21swapper/209:47:542
1662547992220,1cyclictest0-21swapper/208:58:052
1662547992220,1cyclictest0-21swapper/208:44:412
1662547992220,1cyclictest0-21swapper/208:24:192
166254799220,21cyclictest0-21swapper/212:08:542
166254799220,21cyclictest0-21swapper/212:08:542
166254399224,7cyclictest26-21ksoftirqd/108:45:091
166254399223,2cyclictest3056-21PK-Backend09:50:021
1662543992219,2cyclictest3056-21PK-Backend11:02:291
1662543992219,2cyclictest251rcuc/107:37:191
1662543992218,3cyclictest251rcuc/112:25:431
1662543992211,3cyclictest26-21ksoftirqd/109:54:401
166254399220,7cyclictest26-21ksoftirqd/110:30:031
166254399220,20cyclictest3056-21PK-Backend08:19:531
1662541992213,7cyclictest883-21dbus-daemon07:33:570
139099220,15rtkit-daemon0-21swapper/209:01:462
139099220,15rtkit-daemon0-21swapper/208:35:112
17750602210,16sleep00-21swapper/012:25:160
166255199216,14cyclictest1686128-21cpuspeed08:15:113
166255199212,16cyclictest1726202-21unattended-upgr10:06:223
1662551992119,1cyclictest3056-21PK-Backend07:32:363
1662551992119,1cyclictest0-21swapper/311:23:463
1662551992119,1cyclictest0-21swapper/309:46:263
1662551992119,1cyclictest0-21swapper/309:04:143
1662551992119,1cyclictest0-21swapper/308:00:283
1662551992114,5cyclictest436-20systemd-journal10:55:013
166255199210,20cyclictest3056-21PK-Backend12:19:103
166255199210,20cyclictest3056-21PK-Backend08:48:123
166255199210,20cyclictest0-21swapper/311:02:443
166255199210,20cyclictest0-21swapper/307:58:383
166255199210,1cyclictest0-21swapper/309:27:403
166255199210,19cyclictest3056-21PK-Backend10:45:063
1662547992119,1cyclictest0-21swapper/212:04:422
1662547992119,1cyclictest0-21swapper/209:36:592
1662547992119,1cyclictest0-21swapper/209:34:152
1662547992119,1cyclictest0-21swapper/209:19:092
1662547992119,1cyclictest0-21swapper/209:10:512
1662547992119,1cyclictest0-21swapper/209:10:502
1662547992119,1cyclictest0-21swapper/209:08:542
1662547992119,1cyclictest0-21swapper/208:45:292
1662547992119,1cyclictest0-21swapper/208:10:482
1662547992119,1cyclictest0-21swapper/208:02:152
1662547992119,1cyclictest0-21swapper/207:16:132
1662547992113,6cyclictest1757279-21ntp_states11:35:202
1662547992113,6cyclictest1725119-21fw_packets10:05:162
1662547992111,8cyclictest1762368-21/usr/sbin/munin11:50:132
166254799210,20cyclictest0-21swapper/211:57:182
166254799210,20cyclictest0-21swapper/211:31:372
166254799210,20cyclictest0-21swapper/207:39:312
166254799210,20cyclictest0-21swapper/207:27:142
166254799210,20cyclictest0-21swapper/207:13:292
166254799210,19cyclictest0-21swapper/208:05:062
166254399219,4cyclictest26-21ksoftirqd/108:37:561
166254399215,4cyclictest26-21ksoftirqd/110:35:111
1662543992119,1cyclictest3056-21PK-Backend11:57:451
1662543992119,1cyclictest0-21swapper/112:15:431
1662543992119,1cyclictest0-21swapper/110:46:061
1662543992118,2cyclictest3056-21PK-Backend11:24:121
1662543992118,2cyclictest251rcuc/111:41:451
1662543992113,6cyclictest26-21ksoftirqd/108:55:211
1662543992111,4cyclictest26-21ksoftirqd/110:57:141
1662543992110,3cyclictest26-21ksoftirqd/111:18:231
166254399210,4cyclictest26-21ksoftirqd/110:28:571
166254399210,20cyclictest0-21swapper/112:24:281
166254399210,20cyclictest0-21swapper/111:10:551
166254399210,20cyclictest0-21swapper/107:52:281
166254399210,20cyclictest0-21swapper/107:25:181
166254399210,19cyclictest3056-21PK-Backend10:01:181
166254399210,19cyclictest3056-21PK-Backend09:30:021
1662541992119,1cyclictest0-21swapper/009:09:310
166254199210,1cyclictest0-21swapper/007:41:560
166254199210,19cyclictest0-21swapper/012:15:330
139099210,14rtkit-daemon3056-21PK-Backend08:34:443
1662551992018,1cyclictest3056-21PK-Backend11:09:333
1662551992018,1cyclictest3056-21PK-Backend09:20:573
1662551992018,1cyclictest3056-21PK-Backend07:27:013
1662551992018,1cyclictest0-21swapper/312:00:443
1662551992018,1cyclictest0-21swapper/308:22:033
166255199200,1cyclictest3056-21PK-Backend12:07:063
166255199200,1cyclictest3056-21PK-Backend12:07:063
166255199200,19cyclictest3056-21PK-Backend11:44:283
166255199200,19cyclictest0-21swapper/311:57:343
166255199200,19cyclictest0-21swapper/311:35:363
1662547992018,1cyclictest0-21swapper/211:45:082
1662547992018,1cyclictest0-21swapper/211:29:132
1662547992018,1cyclictest0-21swapper/211:08:122
1662547992018,1cyclictest0-21swapper/210:36:002
1662547992018,1cyclictest0-21swapper/210:01:332
1662547992018,1cyclictest0-21swapper/209:41:062
1662547992018,1cyclictest0-21swapper/208:16:562
1662547992018,1cyclictest0-21swapper/207:43:442
166254799200,19cyclictest0-21swapper/212:32:402
166254799200,19cyclictest0-21swapper/211:22:312
166254799200,19cyclictest0-21swapper/208:50:352
166254399202,10cyclictest26-21ksoftirqd/109:35:101
1662543992018,1cyclictest0-21swapper/112:34:491
1662543992018,1cyclictest0-21swapper/108:30:471
1662543992018,1cyclictest0-21swapper/108:03:091
1662543992017,2cyclictest251rcuc/107:56:541
1662543992016,3cyclictest251rcuc/112:07:191
1662543992016,3cyclictest251rcuc/112:07:181
1662543992016,3cyclictest251rcuc/107:25:011
1662543992014,4cyclictest26-21ksoftirqd/112:15:001
1662543992012,6cyclictest1760924-21grep11:45:171
1662543992010,3cyclictest26-21ksoftirqd/108:05:521
166254399200,1cyclictest0-21swapper/108:50:211
166254399200,18cyclictest3056-21PK-Backend12:00:081
1662541992018,1cyclictest0-21swapper/010:35:280
1662541992018,1cyclictest0-21swapper/010:05:540
1662541992018,1cyclictest0-21swapper/008:37:190
1662541992018,1cyclictest0-21swapper/008:17:380
1662541992018,1cyclictest0-21swapper/007:11:140
1662541992016,3cyclictest1095-21runrttasks07:45:030
1662541992012,6cyclictest1727486-21awk10:10:170
1662541992012,6cyclictest1727486-21awk10:10:170
1662541992012,6cyclictest1668607-21rm07:25:020
166254199200,1cyclictest0-21swapper/010:26:220
166254199200,1cyclictest0-21swapper/009:28:370
166254199200,19cyclictest0-21swapper/010:31:360
166254199200,19cyclictest0-21swapper/008:47:530
166254199200,19cyclictest0-21swapper/007:50:210
166254199200,18cyclictest0-21swapper/010:45:080
139099200,13rtkit-daemon0-21swapper/308:12:373
17557772190,4sleep30-21swapper/311:30:213
1662551991918,1cyclictest0-21swapper/307:20:183
1662551991917,1cyclictest3056-21PK-Backend11:16:573
1662551991917,1cyclictest3056-21PK-Backend11:12:173
1662551991917,1cyclictest3056-21PK-Backend08:25:553
1662551991917,1cyclictest3056-21PK-Backend07:46:073
1662551991917,1cyclictest3056-21PK-Backend07:42:563
1662551991917,1cyclictest0-21swapper/308:09:203
166255199191,4cyclictest1742661-21apt10:55:123
1662551991912,5cyclictest1701627-21sed09:00:013
166255199190,7cyclictest1704274-21python309:05:203
166255199190,1cyclictest3056-21PK-Backend10:40:543
166255199190,1cyclictest0-21swapper/309:34:373
166255199190,18cyclictest3056-21PK-Backend10:34:113
166255199190,18cyclictest3056-21PK-Backend09:11:123
166255199190,18cyclictest3056-21PK-Backend09:11:113
166255199190,18cyclictest3056-21PK-Backend07:36:283
1662547991918,1cyclictest0-21swapper/210:43:442
1662547991917,1cyclictest0-21swapper/212:27:552
1662547991917,1cyclictest0-21swapper/210:15:392
1662547991917,1cyclictest0-21swapper/210:14:182
1662547991917,1cyclictest0-21swapper/210:14:172
1662547991917,1cyclictest0-21swapper/209:24:222
1662547991917,1cyclictest0-21swapper/207:31:542
1662547991910,6cyclictest436-20systemd-journal09:25:102
166254799190,1cyclictest0-21swapper/210:28:022
166254799190,1cyclictest0-21swapper/207:56:482
166254799190,1cyclictest0-21swapper/207:22:482
166254799190,18cyclictest0-21swapper/210:34:392
166254399197,4cyclictest26-21ksoftirqd/108:27:061
166254399193,6cyclictest26-21ksoftirqd/107:40:191
1662543991917,1cyclictest3056-21PK-Backend10:06:181
1662543991917,1cyclictest3056-21PK-Backend09:02:581
1662543991917,1cyclictest0-21swapper/110:21:321
1662543991917,1cyclictest0-21swapper/110:17:411
1662543991917,1cyclictest0-21swapper/109:15:211
1662543991917,1cyclictest0-21swapper/107:16:591
1662543991916,2cyclictest251rcuc/109:22:231
166254399190,1cyclictest3056-21PK-Backend09:58:481
166254399190,18cyclictest3056-21PK-Backend09:41:511
166254399190,18cyclictest0-21swapper/112:35:371
1662541991917,1cyclictest0-21swapper/012:05:450
1662541991917,1cyclictest0-21swapper/012:05:450
1662541991917,1cyclictest0-21swapper/011:31:320
1662541991917,1cyclictest0-21swapper/011:23:150
1662541991917,1cyclictest0-21swapper/011:11:170
1662541991917,1cyclictest0-21swapper/010:55:490
1662541991917,1cyclictest0-21swapper/010:00:540
1662541991917,1cyclictest0-21swapper/009:54:520
1662541991917,1cyclictest0-21swapper/009:45:190
1662541991917,1cyclictest0-21swapper/009:39:510
1662541991917,1cyclictest0-21swapper/008:56:460
1662541991917,1cyclictest0-21swapper/008:40:500
1662541991917,1cyclictest0-21swapper/008:09:070
1662541991914,3cyclictest1688004-21cstates08:20:110
1662541991913,5cyclictest1760704-21cat11:45:140
1662541991913,4cyclictest2156-21gnome-shell08:25:100
166254199190,1cyclictest0-21swapper/012:33:030
166254199190,18cyclictest0-21swapper/012:01:330
166254199190,18cyclictest0-21swapper/011:55:110
166254199190,18cyclictest0-21swapper/010:41:160
166254199190,18cyclictest0-21swapper/010:19:120
166254199190,18cyclictest0-21swapper/009:58:590
166254199190,18cyclictest0-21swapper/009:31:550
166254199190,18cyclictest0-21swapper/009:16:260
166254199190,18cyclictest0-21swapper/009:00:320
166254199190,18cyclictest0-21swapper/008:53:000
166254199190,18cyclictest0-21swapper/008:31:520
166254199190,18cyclictest0-21swapper/008:14:210
139099190,3rtkit-daemon0-21swapper/210:48:132
139099190,13rtkit-daemon3056-21PK-Backend12:34:293
139099190,13rtkit-daemon0-21swapper/310:11:343
139099190,13rtkit-daemon0-21swapper/310:11:343
1662551991816,1cyclictest3056-21PK-Backend10:36:413
1662551991816,1cyclictest0-21swapper/311:51:243
1662551991813,3cyclictest1714548-21irqstats09:35:153
1662547991816,1cyclictest0-21swapper/211:14:402
1662547991816,1cyclictest0-21swapper/208:29:122
1662547991812,5cyclictest883-21dbus-daemon11:18:032
166254799180,16cyclictest883-21dbus-daemon07:50:012
166254399181,7cyclictest26-21ksoftirqd/110:50:211
1662543991816,1cyclictest0-21swapper/110:42:151
1662543991816,1cyclictest0-21swapper/109:13:251
1662543991816,1cyclictest0-21swapper/109:13:251
1662543991811,5cyclictest1389-21rtkit-daemon10:13:351
1662543991811,5cyclictest1389-21rtkit-daemon10:13:341
166254399180,6cyclictest3056-21PK-Backend09:05:431
1662541991816,1cyclictest0-21swapper/011:39:160
1662541991816,1cyclictest0-21swapper/011:25:230
1662541991816,1cyclictest0-21swapper/011:06:240
1662541991816,1cyclictest0-21swapper/011:02:520
1662541991816,1cyclictest0-21swapper/007:55:360
1662541991816,1cyclictest0-21swapper/007:23:380
1662551991715,1cyclictest3056-21PK-Backend07:18:223
166255199170,5cyclictest1720310-21ls09:50:213
1662547991715,1cyclictest0-21swapper/212:19:432
1662541991715,1cyclictest0-21swapper/010:52:190
1662541991715,1cyclictest0-21swapper/009:11:260
1662541991715,1cyclictest0-21swapper/009:11:260
1662541991715,1cyclictest0-21swapper/008:01:510
1662541991710,5cyclictest1779597-21cron12:40:000
1662541991710,5cyclictest1769848-21tail12:10:180
1662551991614,1cyclictest3056-21PK-Backend07:54:393
166255199160,14cyclictest1708141-21runrttasks09:17:383
1662547991611,4cyclictest1692736-21cron08:35:002
166254199169,5cyclictest1673388-21sshd07:37:100
1662541991610,5cyclictest1759576-21sed11:40:210
1662541991610,4cyclictest1749769-21grep11:15:110
1662551991513,1cyclictest3056-21PK-Backend12:21:323
1662551991512,2cyclictest3056-21PK-Backend10:22:293
1662551991510,3cyclictest1776114-21latency_hist12:30:013
166254799156,2cyclictest141rcu_preempt11:40:122
1662541991510,3cyclictest1095-21runrttasks07:15:050
1662551991410,3cyclictest3056-21PK-Backend12:11:323
166255199130,12cyclictest3056-21PK-Backend12:35:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional