You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-29 - 05:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Mon Dec 29, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1131015214446,93sleep30-21swapper/319:09:443
113074328655,9sleep20-21swapper/219:06:182
113086728367,10sleep10-21swapper/119:07:501
113074627354,13sleep00-21swapper/019:06:200
1131265992926,1cyclictest0-21swapper/323:19:593
1131265992623,1cyclictest0-21swapper/322:30:143
1131265992623,1cyclictest0-21swapper/322:30:143
1131265992623,1cyclictest0-21swapper/319:24:593
1131265992522,1cyclictest0-21swapper/319:39:593
1131265992321,1cyclictest0-21swapper/323:45:013
1131265992320,1cyclictest0-21swapper/319:25:213
1131265992320,1cyclictest0-21swapper/319:25:203
1131265992320,1cyclictest0-21swapper/300:00:013
13846832221,9sleep11384734-21/usr/sbin/munin23:05:221
113126599221,8cyclictest1212058-21taskset20:21:543
113126599221,15cyclictest0-21swapper/300:20:133
1131265992117,2cyclictest0-21swapper/321:45:003
12966352200,2sleep10-21swapper/121:45:011
1131265992017,1cyclictest0-21swapper/321:00:193
1131265992017,1cyclictest0-21swapper/320:00:143
14452582195,6sleep21445556-21kthreadcore00:05:162
13765872191,11sleep11378267-21cut23:00:211
12019872191,13sleep11202160-21expr20:15:161
1131265991916,1cyclictest0-21swapper/323:25:123
1131265991916,1cyclictest0-21swapper/322:09:593
1131265991916,1cyclictest0-21swapper/319:30:253
1131265991915,2cyclictest0-21swapper/323:25:013
1131265991915,2cyclictest0-21swapper/321:20:003
1131265991915,2cyclictest0-21swapper/319:50:003
113126599191,14cyclictest0-21swapper/322:10:143
113126599191,14cyclictest0-21swapper/319:40:143
113126599191,12cyclictest0-21swapper/321:20:213
113126599191,11cyclictest1175580-21taskset19:50:113
1131265991815,1cyclictest0-21swapper/322:25:153
1131265991815,1cyclictest0-21swapper/322:00:003
1131265991815,1cyclictest0-21swapper/300:10:163
1131265991815,1cyclictest0-21swapper/300:05:173
1131265991814,2cyclictest0-21swapper/322:40:013
113126599181,13cyclictest1306165-21taskset21:50:243
113126599181,13cyclictest0-21swapper/323:50:153
113126599181,13cyclictest0-21swapper/323:10:133
1131265991714,2cyclictest0-21swapper/321:10:193
1131265991713,2cyclictest0-21swapper/300:35:193
113126599171,13cyclictest0-21swapper/323:45:173
113126599171,13cyclictest0-21swapper/320:45:203
113126599171,12cyclictest1444487-21turbostat00:05:003
113126599171,12cyclictest1296814-21chrt21:45:143
113126599171,12cyclictest1217822-21chrt20:30:113
113126599171,11cyclictest1243944-21chrt20:55:003
113126599170,14cyclictest0-21swapper/323:35:143
113126599170,13cyclictest0-21swapper/322:55:163
113126599170,13cyclictest0-21swapper/319:10:123
11393742161,10sleep21139470-21cut19:15:162
113126599161,9cyclictest1380625-21taskset23:05:063
1131265991613,1cyclictest0-21swapper/322:50:113
1131265991613,1cyclictest0-21swapper/322:20:233
1131265991613,1cyclictest0-21swapper/320:35:183
113126599161,13cyclictest0-21swapper/300:25:143
113126599161,12cyclictest1185736-21turbostat20:00:003
113126599161,11cyclictest1249183-21sleep21:00:013
113126599161,10cyclictest1359324-21taskset22:43:553
113126599160,14cyclictest0-21swapper/321:25:273
113126599160,13cyclictest0-21swapper/323:00:193
113126599160,13cyclictest0-21swapper/321:35:203
113126599160,13cyclictest0-21swapper/321:30:133
113126599160,13cyclictest0-21swapper/320:12:323
113126599160,13cyclictest0-21swapper/319:15:153
113126599160,12cyclictest1328237-21chrt22:15:133
113126599160,10cyclictest1232833-21taskset20:40:413
14760372151,9sleep11476230-21cat00:35:011
13748922151,9sleep01374895-21sshd22:57:320
12439072151,5sleep21243932-21perf20:55:012
113126599151,9cyclictest1460127-21taskset00:17:383
113126599151,9cyclictest1412133-21chrt23:34:293
113126599151,2cyclictest0-21swapper/300:30:133
1131265991512,1cyclictest0-21swapper/320:05:163
113126599151,12cyclictest0-21swapper/322:00:203
113126599150,9cyclictest1360082-21taskset22:45:143
113126599150,12cyclictest0-21swapper/321:05:163
113126599150,12cyclictest0-21swapper/320:25:153
113126599150,10cyclictest1206934-21chrt20:19:293
14290932141,9sleep21429117-21/usr/sbin/munin23:50:162
13642652141,9sleep21364266-21sshd22:45:302
13069722141,7sleep21307020-21munin-run21:55:012
12708942147,3sleep11271837-21grep21:20:171
11795842142,7sleep11179638-21ntpq19:50:211
14405322131,8sleep01442169-21tr00:00:170
14725832121,7sleep01473101-21kthreadcore00:30:190
13961362121,7sleep01396215-21cat23:20:000
13348192121,7sleep11336521-21cut22:20:161
13235012121,7sleep01323779-21kthreadcore22:10:170
12394602121,7sleep11239537-21kthreadcore20:50:151
12375792121,6sleep01237667-21ntpq20:45:200
12128542121,6sleep21213063-21hddtemp_smartct20:25:152
12078482121,7sleep01207871-21kthreadcore20:20:190
12068332122,7sleep0800-21snmpd20:18:110
11774992124,4sleep01179260-21cut19:50:160
14714632110,1sleep20-21swapper/200:30:172
14532222111,6sleep11453979-21/usr/sbin/munin00:10:201
13430712110,7sleep21338663-21/usr/sbin/munin22:25:272
13163912111,5sleep01316480-21ntpq22:00:190
13112192112,5sleep21311265-21ntpq21:55:202
12848232111,4sleep01284873-21ntpq21:30:220
12760772111,7sleep10-21swapper/121:25:171
12604042113,5sleep01191026-21kworker/0:0-events21:10:180
11967372111,6sleep21196842-21grep20:10:152
11856042110,8sleep01185605-21sshd19:58:310
14593432101,4sleep11459387-21ntpq00:15:211
14328502101,4sleep11432898-21ntpq23:50:231
14288172102,4sleep01428836-21apt-get23:50:140
14018892101,3sleep11401395-21apt-get23:25:021
13426732101,4sleep01342719-21ntpq22:25:210
13220292101,5sleep11322066-21switchtime22:05:261
13111662101,3sleep11311217-21ntpq21:55:201
13060232101,4sleep11306079-21ntpq21:50:231
12799462101,4sleep01280164-21users21:25:280
12608452100,2sleep10-21swapper/121:10:191
12549442101,4sleep11255047-21perl21:05:171
12505812106,1sleep00-21swapper/021:00:200
12377252101,4sleep21237779-21ntpq20:45:232
12271462100,1sleep20-21swapper/220:35:252
12264182101,5sleep11226876-21grep20:35:201
12165232101,4sleep01216569-21ntpq20:25:210
11964772101,4sleep11196736-21df_inode20:10:141
11960882100,1sleep00-21swapper/020:08:410
11755582101,4sleep21175577-21apt-get19:50:112
11691482101,5sleep11169199-21open_inodes19:40:241
11481772101,4sleep01148215-21ntpq19:20:190
11468442101,4sleep21148035-21memory19:20:162
11320612101,4sleep21132177-21irqstats19:10:142
1464587290,1sleep20-21swapper/200:20:232
1460600291,3sleep10-21swapper/100:20:141
1455915291,4sleep01456383-21grep00:15:150
1423868291,3sleep01424078-21hddtemp_smartct23:45:180
1418559291,4sleep01418887-21cut23:40:160
1413820291,4sleep11415473-21kthreadcore23:35:171
1408352291,4sleep21410027-21cut23:30:202
1407169291,3sleep11407188-21apt-get23:30:141
1395972290,1sleep20-21swapper/223:17:582
1380653291,3sleep01380906-21df23:05:130
1364951291,3sleep11365165-21df_inode22:50:111
1360238291,4sleep01361150-21cut22:45:170
1333168291,4sleep01333220-21sendmail22:20:000
1328765291,4sleep21329398-21grep22:15:172
1313919291,4sleep21315610-21cut22:00:182
1311619291,4sleep01311835-21users21:55:270
1296792291,2sleep20-21swapper/221:45:142
1292133291,4sleep01293326-21grep21:40:160
1286030291,3sleep21286049-21apt-get21:35:122
1280794291,3sleep21280813-21apt-get21:30:122
1274347291,3sleep21274403-21ntpq21:20:222
1253245291,4sleep21253385-21ntp_192.168.11421:00:222
1197016291,4sleep01197753-21cut20:10:170
1165944291,4sleep01167571-21cut19:40:180
1480604281,3sleep21480762-21timerwakeupswit00:35:302
1454244280,1sleep20-21swapper/200:10:242
1423574280,3sleep11423593-21apt-get23:45:151
1412712281,2sleep00-21swapper/023:35:110
1375381281,2sleep00-21swapper/023:00:150
1370692281,3sleep21371834-21kthreadcore22:55:202
1361151281,4sleep11362846-21kthreadcore22:45:171
1355030281,3sleep01355065-21kthreadcore22:40:160
1347939281,3sleep21347986-21ntpq22:30:202
1347939281,3sleep21347986-21ntpq22:30:192
1343921280,3sleep10-21swapper/122:30:111
1343921280,3sleep10-21swapper/122:30:101
1338112280,1sleep20-21swapper/222:21:212
1317769281,3sleep01317871-21cpuspeed_turbos22:05:130
1297076281,3sleep01297200-21perl21:45:170
1296635280,1sleep10-21swapper/121:45:021
1295641281,3sleep21295750-21processes21:40:222
1290996280,1sleep00-21swapper/021:39:560
1274202281,3sleep01274293-21gcc21:20:200
1268881281,4sleep10-21swapper/121:15:191
1248013281,3sleep21248126-21ntpq20:55:212
1245465281,3sleep01247155-21kthreadcore20:55:180
1239234280,4sleep01239459-21cat20:50:140
1233225281,3sleep11233684-21packagekitd20:45:021
1206619280,1sleep20-21swapper/220:15:332
1195986280,1sleep20-21swapper/220:07:272
1192484281,4sleep11194167-21cut20:05:181
1186568281,3sleep10-21swapper/120:00:151
1170907281,3sleep10-21swapper/119:45:171
1160448281,3sleep11160483-21kthreadcore19:35:151
1133238281,3sleep00-21swapper/019:10:160
1481094273,1sleep00-21swapper/000:37:310
1467444271,2sleep01469177-21grep00:25:170
1460257270,1sleep20-21swapper/200:19:192
1444316270,1sleep20-21swapper/200:03:062
1439006270,1sleep10-21swapper/123:59:291
1424176271,2sleep20-21swapper/223:45:202
1398300271,2sleep20-21swapper/223:20:182
1396993271,2sleep11398299-21cut23:20:171
1396420271,2sleep01396439-21apt-get23:20:110
1390832270,1sleep10-21swapper/123:14:241
1370121270,2sleep10-21swapper/122:55:161
1369811270,1sleep20-21swapper/222:54:292
1364897271,3sleep00-21swapper/022:50:100
1350212271,2sleep20-21swapper/222:35:162
1349180270,2sleep10-21swapper/122:35:111
1334013270,4sleep00-21swapper/022:20:150
1327835270,1sleep10-21swapper/122:13:311
1321832271,2sleep20-21swapper/222:05:232
1313058271,2sleep10-21swapper/122:00:161
1305974271,2sleep00-21swapper/021:50:210
1276275270,3sleep21277893-21cut21:25:182
1269101270,4sleep00-21swapper/021:15:220
1265638271,2sleep20-21swapper/221:15:162
1259545270,1sleep20-21swapper/221:09:322
1229115271,2sleep10-21swapper/120:40:171
1224677271,3sleep01226415-21kthreadcore20:35:190
1222577270,1sleep10-21swapper/120:32:461
1222545270,1sleep00-21swapper/020:32:260
1217800270,2sleep20-21swapper/220:30:112
1214445271,2sleep10-21swapper/120:25:181
1211080270,2sleep21211256-21strings20:20:212
1187370271,3sleep21188040-21kthreadcore20:00:172
1185440270,1sleep10-21swapper/119:56:281
1159450270,1sleep20-21swapper/219:32:522
1159373270,4sleep10-21swapper/119:31:551
1156804271,3sleep01158424-21latency19:30:160
1153690271,2sleep10-21swapper/119:25:251
1153690271,2sleep10-21swapper/119:25:241
1150024271,2sleep00-21swapper/019:25:160
1150024271,2sleep00-21swapper/019:25:160
1149447270,4sleep20-21swapper/219:25:062
1149447270,4sleep20-21swapper/219:25:052
1136296270,1sleep10-21swapper/119:10:291
1470661260,4sleep20-21swapper/200:28:242
1464404261,2sleep00-21swapper/000:20:210
1451499261,2sleep00-21swapper/000:10:190
1448843260,2sleep00-21swapper/000:05:220
1439642261,2sleep10-21swapper/100:00:131
1435197261,2sleep20-21swapper/223:55:182
1422828260,4sleep10-21swapper/123:42:251
1422799260,1sleep20-21swapper/223:42:022
1416823260,2sleep20-21swapper/223:35:222
1406553260,1sleep20-21swapper/223:25:362
1403100261,2sleep00-21swapper/023:25:170
1396032260,4sleep10-21swapper/123:18:411
1390694261,2sleep21390680-21fwupd23:13:012
1359297260,1sleep10-21swapper/122:43:341
1353307261,2sleep00-21swapper/022:35:230
1343603260,4sleep10-21swapper/122:29:191
1327034261,2sleep20-21swapper/222:10:242
1291004261,2sleep10-21swapper/121:40:001
1264782260,1sleep20-21swapper/221:14:292
1254305260,4sleep10-21swapper/121:03:491
1248238261,2sleep10-21swapper/120:55:231
1232968260,4sleep00-21swapper/020:42:280
1212061260,4sleep10-21swapper/120:21:561
1190659260,1sleep00-21swapper/020:01:150
1170325260,2sleep00-21swapper/019:45:120
1164642260,1sleep00-21swapper/019:37:350
1161585261,2sleep20-21swapper/219:35:152
1149086260,1sleep10-21swapper/119:23:361
1481128250,1sleep10-21swapper/100:37:561
1448608250,1sleep11448753-21strings00:05:191
1390687250,1sleep00-21swapper/023:12:580
1384735250,1sleep20-21swapper/223:05:222
1375661250,1sleep20-21swapper/223:00:182
1348351250,1sleep00-21swapper/022:30:260
1348351250,1sleep00-21swapper/022:30:250
1332008250,1sleep10-21swapper/122:15:201
1285474252,1sleep10-21swapper/121:31:191
1255874251,1sleep00-21swapper/021:05:200
1230758250,1sleep20-21swapper/220:40:172
1175053250,3sleep20-21swapper/219:46:542
1139160250,1sleep10-21swapper/119:15:151
1137068250,1sleep00-21swapper/019:15:020
1466112240,1sleep10-21swapper/100:25:151
1434434240,1sleep00-21swapper/023:55:160
1407555240,1sleep00-21swapper/023:30:170
1354748240,1sleep20-21swapper/222:40:142
1181093240,1sleep20-21swapper/219:55:162
1164832240,1sleep20-21swapper/219:40:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional