You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-26 - 20:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue May 26, 2026 12:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
125049222700,9sleep30-21swapper/307:05:143
132785321290,3sleep10-21swapper/110:40:231
131962421200,2sleep20-21swapper/210:20:102
128472321140,2sleep0181irq_work/008:40:110
139099760,61rtkit-daemon0-21swapper/107:08:211
125040127358,9sleep20-21swapper/207:05:142
125173826344,14sleep00-21swapper/007:08:410
12689472500,4sleep33056-21PK-Backend07:55:123
12711302480,2sleep33056-21PK-Backend08:00:173
13547892450,2sleep00-21swapper/012:00:130
12586102440,3sleep33056-21PK-Backend07:25:163
13370142430,4sleep00-21swapper/011:10:000
13305522430,3sleep10-21swapper/110:50:121
12871562390,3sleep10-21swapper/108:45:181
12640802390,4sleep23056-21PK-Backend07:40:152
13633332360,4sleep2331rcuc/212:25:022
125196199365,18cyclictest0-21swapper/209:15:062
13341842340,4chrt34-21ksoftirqd/211:00:172
13628712310,4chrt0-21swapper/112:20:221
1251961993014,14cyclictest1306027-21cut09:40:152
125196499292,24cyclictest1266846-21sh07:49:593
1251964992910,17cyclictest1308706-21cron09:50:003
13459052280,8sleep013-21ksoftirqd/011:35:110
1251964992811,15cyclictest1296519-21sshd09:14:273
13571992270,16chrt0-21swapper/012:05:230
125196499275,6cyclictest1337896-21grep11:10:183
1251964992710,15cyclictest1297598-21ntpq09:15:193
13445472260,4sleep33056-21PK-Backend11:30:123
125196499261,8cyclictest1275516-21idleruntime-cro08:14:593
125196199262,3cyclictest1280379-21python308:25:202
125196199261,23cyclictest1321114-21apt-get10:25:032
1251959992611,13cyclictest1334183-21/usr/sbin/munin11:00:171
139099250,17rtkit-daemon1268899-21/usr/sbin/munin07:55:111
12989782250,9sleep126-21ksoftirqd/109:20:141
12786512250,4sleep03056-21PK-Backend08:20:190
125196499258,3cyclictest1327053-21cstates10:40:133
125196499258,15cyclictest0-21swapper/310:25:023
125196499256,5cyclictest1276921-21grep08:15:203
1251964992511,13cyclictest1030-21sshd11:10:003
125196199259,15cyclictest1309467-21uname09:50:152
125196199251,4cyclictest1357799-21perf12:10:002
125196199250,23cyclictest1278516-21ntp_offset08:20:162
125196199250,1cyclictest0-21swapper/207:35:352
1251959992511,11cyclictest1255905-21unin-run07:20:011
1251955992515,9cyclictest1342502-21/usr/sbin/munin11:25:110
125196499246,3cyclictest1278430-21needreboot08:20:163
125196499241,9cyclictest1322015-21/usr/sbin/munin10:25:133
125196499241,8cyclictest436-20systemd-journal09:44:593
125196499241,7cyclictest1254544-21cpuspeed_turbos07:15:113
125196499241,7cyclictest1252362-21cpuspeed_turbos07:10:113
125196499241,6cyclictest1298731-21cpuspeed_turbos09:20:123
125196499241,5cyclictest1357801-21idleruntime-cro12:10:013
125196499240,22cyclictest13681942sleep312:36:403
125196499240,22cyclictest13626542sleep312:20:203
125196199249,13cyclictest1312707-21expr10:00:132
125196199248,14cyclictest1332229-21tr10:55:112
1251961992410,12cyclictest1298677-21cpuspeed_turbos09:20:112
125196199240,23cyclictest0-21swapper/207:51:572
125195999240,22cyclictest1314197-21grep10:05:131
1251955992415,7cyclictest883-21dbus-daemon10:42:240
139099230,14rtkit-daemon0-21swapper/108:30:161
12873302230,2sleep00-21swapper/008:45:210
125196499230,9cyclictest1310472-21idleruntime-cro09:55:003
125196499230,6cyclictest1312596-21cpuspeed_turbos10:00:133
125196499230,1cyclictest3056-21PK-Backend09:02:053
125196199238,13cyclictest1337831-21missed_timers11:10:172
125196199237,14cyclictest1336163-21needreboot11:05:172
125196199236,15cyclictest436-20systemd-journal07:25:002
125196199236,15cyclictest1308897-21cat09:50:002
125196199236,15cyclictest1293825-21awk09:05:142
125196199233,5cyclictest1254222-21sed07:15:022
1251961992317,4cyclictest0-21swapper/208:00:092
125196199231,6cyclictest1253065-21cat07:10:212
125196199230,22cyclictest0-21swapper/209:14:232
125196199230,21cyclictest1366178-21sed12:30:212
1251955992315,7cyclictest1317888-21/usr/sbin/munin10:15:090
139099220,14rtkit-daemon0-21swapper/011:14:550
125196499228,3cyclictest3056-21PK-Backend11:46:533
1251964992219,2cyclictest3056-21PK-Backend11:00:133
1251964992219,2cyclictest3056-21PK-Backend10:19:173
125196499221,8cyclictest1326574-21cron10:39:593
125196499221,8cyclictest1287921-21dump-pmu-power08:49:593
125196499221,6cyclictest1312200-21perf10:00:003
1251964992210,3cyclictest3056-21PK-Backend11:23:223
125196499220,20cyclictest3056-21PK-Backend09:38:483
125196499220,20cyclictest3056-21PK-Backend07:35:333
1251961992217,4cyclictest0-21swapper/211:49:102
125196199220,9cyclictest82332-21update-manager11:32:262
125196199220,7cyclictest1340466-21basename11:20:002
125196199220,4cyclictest1312203-21basename09:59:592
125195999220,21cyclictest0-21swapper/107:25:331
125195599222,17cyclictest1367675-21cut12:35:180
1251955992220,1cyclictest0-21swapper/010:55:080
1251955992220,1cyclictest0-21swapper/010:38:590
1251955992220,1cyclictest0-21swapper/010:30:410
1251955992220,1cyclictest0-21swapper/008:27:530
1251955992220,1cyclictest0-21swapper/007:28:460
1251955992218,3cyclictest1095-21runrttasks08:17:340
125195599220,3cyclictest1364595-21users12:25:190
125195599220,21cyclictest3056-21PK-Backend09:18:500
125195599220,21cyclictest0-21swapper/010:45:430
125195599220,21cyclictest0-21swapper/010:45:430
139099210,15rtkit-daemon0-21swapper/007:41:180
139099210,13rtkit-daemon0-21swapper/011:45:300
125196499218,3cyclictest3056-21PK-Backend12:27:423
1251964992119,1cyclictest3056-21PK-Backend12:30:103
1251964992119,1cyclictest3056-21PK-Backend12:15:243
1251964992119,1cyclictest3056-21PK-Backend10:46:563
1251964992119,1cyclictest3056-21PK-Backend10:46:553
1251964992119,1cyclictest3056-21PK-Backend09:32:323
1251964992119,1cyclictest3056-21PK-Backend08:31:573
1251964992119,1cyclictest3056-21PK-Backend07:24:233
125196499211,7cyclictest1358249-21gpg-connect-age12:10:023
1251964992117,3cyclictest3056-21PK-Backend09:25:443
125196499211,5cyclictest1352605-21idleruntime-cro11:54:593
125196499210,20cyclictest3056-21PK-Backend07:43:153
125196499210,19cyclictest3056-21PK-Backend11:35:013
125196499210,19cyclictest3056-21PK-Backend11:19:063
125196499210,19cyclictest3056-21PK-Backend10:32:293
125196199217,12cyclictest1316686-21kernelversion10:10:152
125196199217,12cyclictest1272690-21runrttasks08:05:172
125196199216,13cyclictest1326957-21cpuspeed_turbos10:40:122
125196199215,3cyclictest1343299-21rt-features11:25:202
125196199214,4cyclictest1325701-21wc10:35:192
125196199214,3cyclictest1330983-21needreboot10:50:172
125196199214,3cyclictest0-21swapper/208:10:032
125196199212,5cyclictest1259601-21perf07:30:002
1251961992116,4cyclictest0-21swapper/212:12:012
125196199211,3cyclictest1365015-21munin-run12:30:012
1251961992110,9cyclictest1345902-21/usr/sbin/munin11:35:142
1251961992110,10cyclictest1330053-21/usr/sbin/munin10:49:592
1251961992110,10cyclictest1330053-21/usr/sbin/munin10:49:592
125196199210,7cyclictest1276755-21awk08:15:172
125196199210,6cyclictest1315983-21find10:10:012
125196199210,20cyclictest0-21swapper/211:20:452
1251959992119,1cyclictest3056-21PK-Backend07:14:521
1251959992119,1cyclictest0-21swapper/112:03:151
1251959992119,1cyclictest0-21swapper/111:27:461
1251959992119,1cyclictest0-21swapper/111:10:421
1251959992119,1cyclictest0-21swapper/111:05:291
1251959992119,1cyclictest0-21swapper/110:36:211
1251959992119,1cyclictest0-21swapper/110:17:571
1251959992119,1cyclictest0-21swapper/109:57:411
1251959992119,1cyclictest0-21swapper/109:50:591
1251959992119,1cyclictest0-21swapper/109:46:471
1251959992119,1cyclictest0-21swapper/108:20:291
1251959992119,1cyclictest0-21swapper/108:01:561
1251959992119,1cyclictest0-21swapper/107:48:451
1251959992119,1cyclictest0-21swapper/107:38:441
1251959992113,6cyclictest1298318-21cron09:20:011
125195999210,20cyclictest0-21swapper/112:28:161
125195999210,20cyclictest0-21swapper/111:41:101
125195999210,1cyclictest0-21swapper/109:27:061
1251955992119,1cyclictest0-21swapper/011:40:040
1251955992119,1cyclictest0-21swapper/011:17:270
1251955992119,1cyclictest0-21swapper/011:02:520
1251955992119,1cyclictest0-21swapper/010:50:430
1251955992119,1cyclictest0-21swapper/010:27:240
1251955992119,1cyclictest0-21swapper/010:02:290
1251955992119,1cyclictest0-21swapper/009:08:350
1251955992119,1cyclictest0-21swapper/008:54:030
1251955992119,1cyclictest0-21swapper/007:33:250
1251955992119,1cyclictest0-21swapper/007:20:410
1251955992119,1cyclictest0-21swapper/007:18:510
1251955992119,1cyclictest0-21swapper/007:12:150
1251955992116,3cyclictest13-21ksoftirqd/008:30:170
1251955992114,6cyclictest1345442-21http11:32:260
1251955992114,5cyclictest436-20systemd-journal08:04:210
1251955992114,5cyclictest1360761-21ntp_kernel_err12:15:170
125195599210,2cyclictest1788620-21WebKitWebProces09:43:230
125195599210,20cyclictest0-21swapper/009:29:320
125195599210,1cyclictest0-21swapper/009:53:520
125195599210,1cyclictest0-21swapper/008:58:020
139099200,14rtkit-daemon0-21swapper/009:00:350
125196499209,3cyclictest3056-21PK-Backend11:57:553
1251964992018,1cyclictest3056-21PK-Backend11:28:183
1251964992018,1cyclictest3056-21PK-Backend10:56:213
1251964992018,1cyclictest3056-21PK-Backend07:34:313
125196499201,6cyclictest1294850-21apt09:09:593
1251964992016,3cyclictest3056-21PK-Backend10:54:263
125196499200,1cyclictest3056-21PK-Backend08:35:413
125196199206,12cyclictest978-21snmpd07:32:412
125196199204,3cyclictest1304298-21cut09:35:172
1251961992018,1cyclictest0-21swapper/210:16:292
125196199201,5cyclictest1290781-21python308:55:202
125196199200,7cyclictest1368455-21cron12:39:592
125196199200,7cyclictest1323860-21grep10:30:172
125196199200,7cyclictest1302777-21awk09:30:192
125196199200,6cyclictest1351792-21head11:50:182
125196199200,6cyclictest1285305-21grep08:40:172
125196199200,6cyclictest1265554-21grep07:45:132
125196199200,19cyclictest0-21swapper/212:15:272
125196199200,19cyclictest0-21swapper/211:56:402
125196199200,19cyclictest0-21swapper/211:44:372
125196199200,19cyclictest0-21swapper/209:25:192
125196199200,19cyclictest0-21swapper/208:47:132
125196199200,19cyclictest0-21swapper/208:37:542
125196199200,19cyclictest0-21swapper/208:33:212
1251959992018,1cyclictest3056-21PK-Backend10:23:101
1251959992018,1cyclictest0-21swapper/111:22:041
1251959992018,1cyclictest0-21swapper/110:11:351
1251959992018,1cyclictest0-21swapper/109:38:161
1251959992018,1cyclictest0-21swapper/109:00:291
1251959992018,1cyclictest0-21swapper/108:17:311
1251959992018,1cyclictest0-21swapper/107:40:481
1251959992017,2cyclictest0-21swapper/108:10:011
1251959992012,6cyclictest26-21ksoftirqd/112:05:211
1251959992012,6cyclictest1344894-21ntp_kernel_pll_11:30:181
1251959992012,6cyclictest1283087-21sort08:35:131
125195999200,1cyclictest3056-21PK-Backend09:41:471
125195999200,19cyclictest0-21swapper/111:49:151
125195999200,19cyclictest0-21swapper/111:37:261
125195999200,19cyclictest0-21swapper/110:45:351
125195999200,19cyclictest0-21swapper/110:45:341
125195999200,19cyclictest0-21swapper/110:03:161
125195999200,19cyclictest0-21swapper/109:11:121
1251955992018,1cyclictest0-21swapper/010:21:020
1251955992018,1cyclictest0-21swapper/010:11:210
1251955992018,1cyclictest0-21swapper/008:37:460
1251955992018,1cyclictest0-21swapper/007:50:080
1251955992018,1cyclictest0-21swapper/007:47:030
1251955992011,6cyclictest436-20systemd-journal08:09:590
125195599200,19cyclictest0-21swapper/012:31:070
125195599200,19cyclictest0-21swapper/010:07:290
125196499197,3cyclictest3056-21PK-Backend08:52:183
125196499192,5cyclictest1317641-21sh10:15:003
1251964991917,1cyclictest3056-21PK-Backend10:06:323
1251964991917,1cyclictest3056-21PK-Backend08:05:533
1251964991915,3cyclictest3056-21PK-Backend12:00:023
125196199195,12cyclictest0-21swapper/212:00:122
1251961991917,1cyclictest0-21swapper/209:03:362
125196199190,6cyclictest1289040-21grep08:50:192
125196199190,6cyclictest1269261-21sort07:55:142
1251959991917,1cyclictest0-21swapper/111:16:231
1251959991917,1cyclictest0-21swapper/110:31:071
1251959991917,1cyclictest0-21swapper/110:26:281
1251959991917,1cyclictest0-21swapper/109:07:061
125195999190,1cyclictest0-21swapper/111:50:021
125195999190,1cyclictest0-21swapper/110:56:561
125195999190,1cyclictest0-21swapper/107:51:491
125195999190,1cyclictest0-21swapper/107:24:321
125195999190,18cyclictest0-21swapper/108:26:231
125195599199,7cyclictest436-20systemd-journal11:25:000
1251955991917,1cyclictest0-21swapper/012:13:440
1251955991917,1cyclictest0-21swapper/009:37:550
1251955991917,1cyclictest0-21swapper/009:34:240
1251955991917,1cyclictest0-21swapper/009:22:080
1251955991915,3cyclictest13-21ksoftirqd/012:25:000
1251955991911,6cyclictest883-21dbus-daemon09:15:000
125195599190,1cyclictest0-21swapper/009:56:560
125195599190,1cyclictest0-21swapper/008:13:530
1251964991816,1cyclictest3056-21PK-Backend08:43:113
1251964991815,2cyclictest3056-21PK-Backend11:43:543
1251959991816,1cyclictest0-21swapper/112:10:031
1251959991816,1cyclictest0-21swapper/108:50:091
1251959991812,5cyclictest1360400-21expr12:15:121
1251959991812,5cyclictest1290494-21seq08:55:151
1251955991816,1cyclictest0-21swapper/011:59:100
1251955991816,1cyclictest0-21swapper/011:51:530
1251955991816,1cyclictest0-21swapper/007:37:160
1251964991715,1cyclictest3056-21PK-Backend07:50:263
1251964991712,4cyclictest3056-21PK-Backend08:27:243
125195999179,6cyclictest1275507-21apt-get08:15:011
1251959991715,1cyclictest0-21swapper/109:34:431
1251959991715,1cyclictest0-21swapper/108:42:261
1251959991714,2cyclictest1260037-21cpuspeed_turbos07:30:121
125195599171,12cyclictest1268847-21apt07:55:110
125195999169,5cyclictest1365572-21cat12:30:131
1251955991612,3cyclictest1308719-21sh09:50:000
125195999158,5cyclictest883-21dbus-daemon12:00:011
125195999158,5cyclictest1367447-21cat12:35:161
1251964991310,2cyclictest3056-21PK-Backend08:55:433
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional