You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 12:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Tue Nov 18, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10558849912530,1249cyclictest11090082sleep319:49:563
10558849910971,1092cyclictest11413542sleep320:14:563
1055884992421,1cyclictest0-21swapper/300:24:593
190898220,19rtkit-daemon1456656-21ssh23:37:592
1055884992118,1cyclictest0-21swapper/320:30:133
105588499198,7cyclictest1115646-21turbostat19:55:003
1055884991815,1cyclictest0-21swapper/321:05:253
190898170,5rtkit-daemon1522987-21ssh00:19:520
123570921713,1sleep30-21swapper/321:25:003
12196612172,5sleep11219695-21apt-get21:15:131
105588499177,6cyclictest13901582sleep322:56:453
1055884991715,1cyclictest0-21swapper/323:20:173
105588499162,11cyclictest1244185-21taskset21:30:133
1055884991613,1cyclictest0-21swapper/323:50:163
1055884991613,1cyclictest0-21swapper/322:06:393
1055884991513,1cyclictest0-21swapper/321:55:563
1055884991513,1cyclictest0-21swapper/300:33:413
1055884991513,0cyclictest0-21swapper/321:37:283
1055884991512,1cyclictest0-21swapper/322:40:183
1055884991512,1cyclictest0-21swapper/322:31:213
1055884991512,1cyclictest0-21swapper/319:55:263
1055884991512,1cyclictest0-21swapper/319:40:253
1055884991512,1cyclictest0-21swapper/300:05:153
1055884991512,1cyclictest0-21swapper/300:03:203
1055884991512,1cyclictest0-21swapper/300:00:003
105588499151,11cyclictest1466337-21taskset23:45:133
105588499151,11cyclictest1380760-21taskset22:50:233
105588499151,11cyclictest1380760-21taskset22:50:223
1055884991511,1cyclictest0-21swapper/323:35:133
105588499151,10cyclictest1331686-21taskset22:20:223
13597452140,3sleep11360628-21grep22:40:201
12004402140,5sleep21202516-21kthreadcore21:00:202
11283492141,4sleep01128631-21gpgv20:05:010
107399921411,1sleep01059552-21kworker/0:1+kacpid19:20:210
105588499144,8cyclictest1179486-21turbostat20:49:563
105588499144,8cyclictest1179486-21turbostat20:49:553
1055884991412,1cyclictest0-21swapper/322:16:323
1055884991412,1cyclictest0-21swapper/320:17:133
1055884991412,1cyclictest0-21swapper/319:19:273
1055884991412,1cyclictest0-21swapper/300:17:053
1055884991412,0cyclictest0-21swapper/322:45:583
1055884991412,0cyclictest0-21swapper/322:14:573
1055884991412,0cyclictest0-21swapper/321:52:433
1055884991411,1cyclictest0-21swapper/323:32:353
1055884991411,1cyclictest0-21swapper/320:35:243
105588499141,10cyclictest1424537-21turbostat23:20:003
105588499141,10cyclictest1355755-21taskset22:35:213
105588499141,10cyclictest1243176-21chrt21:28:333
105588499141,10cyclictest1154354-21taskset20:25:143
105588499141,10cyclictest1130324-21taskset20:05:183
105588499141,10cyclictest1070941-21chrt19:20:153
105588499138,3cyclictest1334399-21taskset22:25:143
105588499137,4cyclictest1192165-21turbostat20:55:013
105588499131,9cyclictest1532152-21taskset00:25:163
105588499131,9cyclictest1128375-21turbostat20:05:003
1055884991312,1cyclictest0-21swapper/323:03:003
1055884991311,1cyclictest0-21swapper/321:16:223
1055884991311,1cyclictest0-21swapper/319:13:373
105588499131,10cyclictest1555182-21taskset00:38:053
105588499131,10cyclictest1276474-21turbostat21:50:003
1055884991311,0cyclictest0-21swapper/323:44:183
1055884991310,1cyclictest0-21swapper/320:55:103
1055884991310,1cyclictest0-21swapper/320:45:013
1055884991310,1cyclictest0-21swapper/320:20:203
1055884991310,1cyclictest0-21swapper/319:25:553
105588499130,10cyclictest1512485-21taskset00:10:203
105588499130,10cyclictest1211400-21taskset21:10:013
12586932121,8sleep21258701-21ssh21:36:452
105588499129,1cyclictest0-21swapper/323:10:233
105588499129,1cyclictest0-21swapper/321:00:163
105588499123,7cyclictest1265909-21chrt21:40:233
105588499121,9cyclictest1090231-21taskset19:35:143
105588499121,8cyclictest1400276-21taskset23:05:023
105588499120,9cyclictest1439204-21taskset23:25:293
105588499120,9cyclictest1293029-21taskset22:00:013
105588499120,9cyclictest1089643-21taskset19:31:133
190898110,6rtkit-daemon1134142-21ntp_states20:05:231
190898110,4rtkit-daemon1062342-21munin-node00:10:340
15151952111,6sleep11515227-21apt-get00:15:131
10496852111,4sleep20-21swapper/219:05:132
190898100,6rtkit-daemon1540790-21sed00:30:161
190898100,6rtkit-daemon1217590-21taskset21:10:521
190898100,3rtkit-daemon81364-21packagekitd23:25:462
190898100,1rtkit-daemon0-21swapper/023:17:170
14807052101,7sleep11480713-21ssh23:51:291
14429232105,2sleep10-21swapper/123:30:161
13915762101,6sleep21391957-21gpgv23:00:012
13907832100,1sleep01390785-21ssh22:58:110
13640542100,7sleep01364192-21/usr/sbin/munin22:40:230
13425962101,6sleep11342700-21cpuspeed_turbos22:30:141
13097722100,6sleep01309932-21df_inode22:10:140
12514832104,3sleep11251490-21sh21:33:571
12361212101,5sleep01236284-21diskstats21:25:170
11352552101,5sleep11135277-21apt-get20:10:141
19089890,5rtkit-daemon1468653-21kthreadcore23:45:172
19089890,5rtkit-daemon1337524-21cut22:25:202
19089890,5rtkit-daemon1091374-21journal-offline19:35:171
19089890,5rtkit-daemon1062342-21munin-node19:19:191
19089890,4rtkit-daemon1332727-21ssh22:21:562
19089890,4rtkit-daemon1083337-21apt-get19:30:011
19089890,3rtkit-daemon667-21snmpd20:42:461
19089890,1rtkit-daemon0-21swapper/021:21:430
1500798291,5sleep01502841-21kthreadcore00:05:190
1497588291,5sleep21497596-21ssh00:02:482
1490642290,6sleep11490652-21ssh00:00:031
1457876292,3sleep21457913-21apt-get23:40:162
1394006291,5sleep01396069-21kthreadcore23:00:190
1382862290,6sleep11382870-21ssh22:54:231
1382862290,6sleep11382870-21ssh22:54:221
1378093290,1sleep21380296-21kthreadcore22:50:192
1378093290,1sleep21380296-21kthreadcore22:50:182
1375276292,3sleep01375309-21apt-get22:50:130
1375276292,3sleep01375309-21apt-get22:50:130
1332853291,5sleep01332861-21ssh22:22:120
1129056290,5sleep01129168-21tune2fs20:05:170
1049868290,6sleep01049916-21fschecks_count19:05:140
19089880,4rtkit-daemon81364-21packagekitd23:21:550
19089880,4rtkit-daemon81364-21packagekitd19:47:561
19089880,4rtkit-daemon697-21runrttasks21:35:380
19089880,4rtkit-daemon671-21ntpd23:00:221
19089880,4rtkit-daemon671-21ntpd19:58:051
19089880,4rtkit-daemon671-21ntpd19:40:241
19089880,4rtkit-daemon667-21snmpd20:29:581
19089880,4rtkit-daemon667-21snmpd19:50:211
19089880,4rtkit-daemon1496855-21ssh00:01:040
19089880,4rtkit-daemon1358830-21kworker/u16:4-kvfree_rcu_reclaim22:45:251
19089880,4rtkit-daemon1300194-21ssh22:03:131
19089880,4rtkit-daemon1273886-21grep21:45:191
19089880,4rtkit-daemon1250357-21ssh21:31:190
19089880,4rtkit-daemon1223-21cron22:12:011
19089880,4rtkit-daemon1186445-21hddtemp_smartct20:50:161
19089880,4rtkit-daemon1075168-21cut19:20:211
19089880,4rtkit-daemon1062342-21munin-node23:05:041
19089880,4rtkit-daemon1062342-21munin-node22:31:142
19089880,4rtkit-daemon1062342-21munin-node21:05:451
19089880,4rtkit-daemon1061810-21users19:10:261
19089880,4rtkit-daemon0-21swapper/123:13:441
19089880,3rtkit-daemon438-21wpa_supplicant22:37:211
19089880,2rtkit-daemon1928-21sendmail-mta19:29:331
19089880,1rtkit-daemon0-21swapper/121:41:281
1529615281,4sleep21529621-21ssh00:21:192
1488163281,4sleep01488374-21users23:55:250
1474463281,3sleep01474497-21apt-get23:50:160
1434820281,4sleep11436307-21cut23:25:201
1350587280,5sleep21350593-21ssh22:35:052
1318265280,6sleep11318460-21if_eno122:15:191
1308551280,6sleep11308559-21ssh22:07:581
1301656281,4sleep21301799-21df22:05:152
1295659280,2sleep21297769-21tr22:00:172
1290359281,4sleep11290492-21python321:55:241
1285036281,4sleep01285193-21df_inode21:55:150
1283583280,6sleep01283590-21sh21:52:300
1268582280,2sleep20-21swapper/221:45:122
1226517281,4sleep21226525-21ssh21:17:472
1226437283,3sleep0151ktimers/021:17:370
1211511281,3sleep01211550-21apt-get21:10:140
1199151280,5sleep01199705-21irqcore21:00:170
1191247281,4sleep21191306-21ntp_states20:50:222
1179916280,6sleep11180068-21head20:45:161
1179916280,6sleep11180068-21head20:45:151
1161110281,4sleep01162103-21irqstats20:30:170
1123236281,4sleep21124139-21grep20:00:182
1121919282,1sleep0171rcu_preempt19:59:150
1109533281,3sleep01109566-21apt-get19:50:140
1085982281,4sleep01088110-21cut19:30:180
1082456281,4sleep01082508-21/usr/sbin/munin19:25:220
19089870,4rtkit-daemon81364-21packagekitd20:30:461
19089870,4rtkit-daemon473-21gmain23:47:350
19089870,4rtkit-daemon1062342-21munin-node00:26:061
19089870,3rtkit-daemon667-21snmpd21:04:291
19089870,3rtkit-daemon667-21snmpd20:15:091
19089870,3rtkit-daemon1928-21sendmail-mta20:55:221
19089870,3rtkit-daemon1524420-21irqrtprio00:20:160
19089870,3rtkit-daemon1282796-21ssh21:50:432
19089870,3rtkit-daemon1223-21cron23:35:002
19089870,3rtkit-daemon1223-21cron23:24:002
19089870,3rtkit-daemon1062342-21munin-node00:36:342
19089870,3rtkit-daemon1017352-21munin-node19:06:101
19089870,2rtkit-daemon671-21ntpd22:20:021
19089870,1rtkit-daemon0-21swapper/122:55:531
1539349270,1sleep20-21swapper/200:29:252
1529864270,2sleep11529872-21ssh00:21:591
1498860271,3sleep21498899-21apt-get00:05:132
1472589271,3sleep11472597-21ssh23:46:231
1455067271,4sleep11449620-21/usr/sbin/munin23:35:221
1441439271,2sleep01441473-21apt-get23:30:110
1424113271,4sleep11424121-21ssh23:19:111
1418146271,3sleep21419852-21grep23:15:192
1400577270,4sleep21400711-21grep23:05:172
1340230270,1sleep10-21swapper/122:25:301
1282878270,4sleep11282886-21ssh21:50:551
1242779270,1sleep20-21swapper/221:27:372
1228934272,2sleep11230040-21tr21:20:181
1228010270,4sleep21228059-21cut21:20:142
1192145271,4sleep01192260-21date20:55:010
1169527271,3sleep10-21swapper/120:35:221
1153296270,4sleep01153382-21awk20:20:230
1148715270,4sleep21149693-21grep20:20:162
1141358271,3sleep21141368-21munin-plugin-st20:15:002
1122255271,3sleep11122288-21apt-get20:00:131
1109012271,3sleep21109026-21apt-get19:50:002
1096609270,4sleep01096761-21df_inode19:40:170
1051096270,4sleep30-21swapper/319:05:183
19089860,3rtkit-daemon473-21gmain00:34:202
19089860,3rtkit-daemon421-21avahi-daemon21:40:162
1549659260,4sleep11551325-21ssh00:35:201
1531795261,2sleep00-21swapper/000:25:130
1514081260,1sleep10-21swapper/100:13:081
1513215260,3sleep20-21swapper/200:11:092
1480954261,3sleep21480962-21ssh23:52:102
1433328261,2sleep00-21swapper/023:25:150
1425277260,4sleep11425393-21fschecks_time23:20:161
1415514260,1sleep00-21swapper/023:13:130
1414367260,1sleep20-21swapper/223:10:312
1406892260,4sleep01406900-21ssh23:06:590
1392592261,2sleep20-21swapper/223:00:172
1374458260,1sleep20-21swapper/222:48:592
1367499260,3sleep01368032-21sed22:45:190
1350669261,2sleep01350701-21apt-get22:35:150
1348535260,1sleep00-21swapper/022:30:530
1340248261,2sleep01340256-21ssh22:25:330
1308286260,3sleep01308294-21ssh22:07:250
1293570260,4sleep01294125-21irqrtprio22:00:160
1284951260,4sleep21284958-21diskmemload21:55:092
1266351260,1sleep00-21swapper/021:40:320
1259006260,1sleep10-21swapper/121:37:241
1198383260,1sleep00-21swapper/020:58:540
1179760260,3sleep01179915-21diskstats20:45:150
1179760260,3sleep01179915-21diskstats20:45:140
1174195261,3sleep01175153-21cut20:40:190
1160390260,4sleep01160547-21latency_hist20:30:000
1148386260,3sleep11148714-21irqcore20:20:151
1129332260,3sleep21130323-21irqstats20:05:192
1112225260,3sleep21114424-21kthreadcore19:50:202
1104927260,4sleep01107035-21kthreadcore19:45:200
1090388260,3sleep01090525-21grep19:35:160
1090196261,2sleep20-21swapper/219:35:142
1076078260,3sleep21076155-21/usr/sbin/munin19:20:232
1064923261,2sleep00-21swapper/019:15:160
1548195251,2sleep00-21swapper/000:35:160
1515170250,1sleep20-21swapper/200:15:102
1484727251,2sleep20-21swapper/223:55:182
1482498250,3sleep10-21swapper/123:55:091
1464059251,2sleep1667-21snmpd23:41:191
1450633251,2sleep00-21swapper/023:35:170
1320584251,2sleep00-21swapper/022:15:230
1318463251,2sleep20-21swapper/222:15:202
1317250250,3sleep20-21swapper/222:14:132
1244141250,2sleep20-21swapper/221:30:132
1243046250,1sleep10-21swapper/121:28:151
1210456250,3sleep21210610-21sed21:05:252
1206307251,2sleep00-21swapper/021:05:190
1172888250,1sleep20-21swapper/220:37:232
1168433250,2sleep00-21swapper/020:35:210
1147569250,1sleep20-21swapper/220:18:132
1102471250,1sleep20-21swapper/219:42:342
1089919250,1sleep20-21swapper/219:34:452
1062127250,1sleep00-21swapper/019:14:240
1056052250,3sleep21056056-21find19:10:142
1498824240,2sleep11498828-21ssh00:05:081
1460399240,1sleep00-21swapper/023:40:220
1360629241,1sleep20-21swapper/222:40:202
1271115240,2sleep00-21swapper/021:45:180
1213165241,1sleep20-21swapper/221:10:202
1197709241,1sleep20-21swapper/220:55:262
1154318240,1sleep20-21swapper/220:25:142
1543765230,1sleep00-21swapper/000:30:190
1184723230,1sleep20-21swapper/220:45:212
1184723230,1sleep20-21swapper/220:45:202
1177227230,1sleep20-21swapper/220:40:212
1160836230,1sleep20-21swapper/220:30:152
1142606230,1sleep00-21swapper/020:15:180
1135238230,1sleep00-21swapper/020:10:090
1117447230,1sleep20-21swapper/219:55:202
1077502230,1sleep20-21swapper/219:25:162
1062429230,1sleep20-21swapper/219:15:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional