You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-19 - 20:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Fri Dec 19, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1219200212354,64sleep30-21swapper/307:06:053
121492728654,20sleep00-21swapper/007:05:150
121926928252,9sleep20-21swapper/207:06:572
121943228165,10sleep10-21swapper/107:09:011
1219791992420,2cyclictest0-21swapper/310:10:183
1219791992319,2cyclictest0-21swapper/311:55:013
13905042220,1sleep11390508-21unixbench-2d09:45:271
121979199222,15cyclictest14961942chrt11:30:143
121979199222,15cyclictest14961942chrt11:30:133
1219791992219,1cyclictest0-21swapper/309:40:243
121979199221,8cyclictest1237318-21taskset07:20:403
121979199215,8cyclictest1570171-21turbostat12:40:003
121979199212,12cyclictest1385985-21taskset09:45:123
1219791992118,1cyclictest0-21swapper/307:39:563
1219791992117,2cyclictest0-21swapper/310:50:223
14443722201,9sleep21445044-21cut10:40:162
1219791992017,1cyclictest0-21swapper/311:50:003
1219791992017,1cyclictest0-21swapper/310:00:243
121979199201,12cyclictest0-21swapper/310:20:203
14650062191,9sleep11465112-21/usr/sbin/munin11:00:141
1219791991916,1cyclictest0-21swapper/309:35:163
1219791991916,1cyclictest0-21swapper/309:05:003
1219791991915,2cyclictest0-21swapper/308:40:003
1219791991915,1cyclictest0-21swapper/308:55:063
1219791991912,4cyclictest0-21swapper/309:05:123
121979199191,14cyclictest0-21swapper/312:15:113
121979199191,13cyclictest0-21swapper/311:25:143
121979199188,7cyclictest1511326-21chrt11:41:123
1219791991815,1cyclictest0-21swapper/312:10:013
121979199181,14cyclictest0-21swapper/311:35:213
121979199181,13cyclictest0-21swapper/312:30:103
15530472171,10sleep21553098-21ntpq12:20:192
1219791991713,2cyclictest0-21swapper/309:55:103
1219791991713,2cyclictest0-21swapper/309:25:153
1219791991713,2cyclictest0-21swapper/308:40:163
1219791991713,2cyclictest0-21swapper/307:25:223
121979199171,13cyclictest0-21swapper/310:35:103
121979199171,13cyclictest0-21swapper/309:20:203
121979199171,13cyclictest0-21swapper/309:20:203
121979199171,13cyclictest0-21swapper/309:15:113
121979199171,13cyclictest0-21swapper/308:30:203
121979199171,13cyclictest0-21swapper/307:40:283
121979199171,12cyclictest1225137-21turbostat07:15:003
121979199171,10cyclictest1555003-21taskset12:25:133
121979199171,10cyclictest1259102-21taskset07:45:093
1219791991710,2cyclictest0-21swapper/311:10:213
121979199170,7cyclictest12854071turbostat08:10:003
121979199170,14cyclictest0-21swapper/307:55:133
121979199170,13cyclictest0-21swapper/309:50:193
121979199170,11cyclictest1243146-21chrt07:30:013
14797542160,3sleep11479755-21unixbench_singl11:10:271
12240882161,10sleep01224139-21ntpq07:10:190
121979199164,8cyclictest1411126-21taskset10:05:213
1219791991613,1cyclictest0-21swapper/307:15:023
1219791991612,2cyclictest0-21swapper/311:15:253
1219791991612,2cyclictest0-21swapper/311:00:013
1219791991611,3cyclictest0-21swapper/312:20:113
1219791991611,3cyclictest0-21swapper/308:50:133
121979199161,11cyclictest1490658-21turbostat11:24:593
121979199161,11cyclictest1432070-21chrt10:25:213
121979199161,11cyclictest1432070-21chrt10:25:203
121979199161,11cyclictest1306431-21sleep08:29:593
121979199161,10cyclictest1453618-21taskset10:45:323
121979199161,10cyclictest1443830-21chrt10:40:123
121979199161,10cyclictest1288834-21chrt08:10:173
121979199160,13cyclictest0-21swapper/311:55:213
121979199160,13cyclictest0-21swapper/309:30:153
121979199160,13cyclictest0-21swapper/308:00:153
121979199160,10cyclictest1269296-21taskset07:53:373
14634962151,4sleep11463554-21ntp_states10:55:221
12443282151,9sleep01246048-21grep07:30:150
121979199151,9cyclictest1469485-21taskset11:02:203
121979199151,9cyclictest1353581-21taskset09:10:503
121979199151,9cyclictest1291125-21taskset08:15:113
1219791991512,1cyclictest0-21swapper/310:30:123
1219791991512,1cyclictest0-21swapper/308:20:243
1219791991511,2cyclictest0-21swapper/310:15:243
121979199151,10cyclictest1532828-21chrt12:01:043
121979199150,12cyclictest0-21swapper/312:10:133
15398192141,8sleep11541541-21cut12:10:161
15132342141,8sleep21514946-21cut11:45:172
15040082141,8sleep11505241-21memory11:35:181
14705602141,8sleep11472109-21cut11:05:161
13225952144,5sleep244-21ksoftirqd/208:45:122
121979199140,12cyclictest0-21swapper/311:05:163
13937522131,8sleep11395065-21memory09:50:161
13847132131,9sleep00-21swapper/009:40:210
13290052131,7sleep21330714-21kthreadcore08:50:182
12908902132,5sleep11290909-21apt-get08:15:091
12539652131,7sleep11253782-21/usr/sbin/munin07:40:141
121979199130,9cyclictest1322422-21chrt08:45:043
14959052121,7sleep11495982-21latency_hist11:30:011
14128372121,7sleep01413122-21cat10:10:150
13480552121,7sleep01348326-21awk09:05:270
12895402121,5sleep01289629-21ntpq08:10:200
12683642120,2sleep00-21swapper/007:50:200
12543592121,7sleep01255705-21cut07:40:180
15648932111,5sleep01564941-21munin-run12:35:010
15171312111,5sleep11517154-21munin-run11:50:011
15106552111,5sleep01510702-21ntpq11:40:210
15023102111,6sleep01504004-21kthreadcore11:35:170
14909322111,5sleep01491191-21df11:25:150
14856542111,5sleep00-21swapper/011:20:140
14581912111,5sleep01458237-21ntpq10:50:190
14539702111,6sleep11453988-21ls10:50:011
14332932111,4sleep01433312-21apt-get10:30:120
14319312110,2sleep10-21swapper/110:25:201
14319312110,2sleep10-21swapper/110:25:191
13634532111,5sleep01363503-21ntpq09:20:220
13634532111,5sleep01363503-21ntpq09:20:210
13536732111,7sleep1800-21snmpd09:12:001
13158872112,4sleep01315922-21ntpq08:35:210
12842212111,5sleep01284362-21ntpq08:05:180
12537612111,5sleep21253444-31find07:40:032
15692022101,4sleep01569256-21ntpq12:35:200
15478992101,5sleep01547959-21ld12:15:210
15320982101,4sleep21532146-21ntpq12:00:192
14856792101,4sleep11485935-21df11:20:141
14387092101,4sleep01438839-21diskstats10:35:130
14017952101,5sleep11401842-21grep10:00:101
13753152101,4sleep11375441-21diskstats09:35:141
13597092101,4sleep11359812-21perl09:20:151
13597092101,4sleep11359812-21perl09:20:141
13384912101,4sleep20-21swapper/209:00:122
13279792101,4sleep11327998-21apt-get08:50:131
13011702101,4sleep21301219-21munin-run08:25:012
13001792101,5sleep01300235-21ntpq08:20:220
12896792102,3sleep21289724-21ntpq08:10:222
12788462101,5sleep21278893-21ntpq08:00:192
12524992101,6sleep21252613-21/usr/sbin/munin07:35:232
12484642101,4sleep11248683-21diskstats07:35:141
12379212101,4sleep11237940-21apt-get07:25:121
12343032101,5sleep20-21swapper/207:20:152
12327122101,5sleep11232719-21cpu07:20:101
1569497291,4sleep11569771-21grep12:35:261
1565604291,4sleep21565723-21irqstats12:35:152
1561078291,4sleep11562809-21grep12:30:171
1557592291,4sleep21558483-21expr12:25:172
1523864291,4sleep11525560-21tr11:55:181
1523439293,2sleep01523862-21tr11:55:170
1516257291,4sleep01516374-21sendmail_mailqu11:45:230
1511628291,4sleep21511637-21apt-get11:45:012
1491213291,4sleep21491276-21grep11:25:172
1490647291,4sleep21490773-21cut11:25:002
1480705290,1sleep10-21swapper/111:15:141
1469953291,3sleep01469972-21apt-get11:05:110
1466126291,4sleep01467867-21tr11:00:170
1442480291,3sleep21442528-21ntpq10:35:182
1434560291,4sleep11436295-21grep10:30:171
1426742291,4sleep21426780-21ntpq10:20:192
1365023291,4sleep01365220-21if_enp1s009:25:150
1364411291,5sleep20-21swapper/209:25:012
1364411291,5sleep20-21swapper/209:25:002
1333220291,3sleep01333239-21apt-get08:55:130
1322429291,3sleep01322448-21apt-get08:45:110
1315930291,4sleep11315977-21ld08:35:211
1314692291,3sleep21315743-21memory08:35:182
1301451291,4sleep11301633-21cat08:25:111
1300879290,3sleep11300835-21fwupd08:21:191
1294707291,4sleep01294841-21netstat08:15:160
1280387291,2sleep10-21swapper/108:05:111
1269664291,4sleep21269683-21apt-get07:55:112
1259669294,2sleep11225242-21kworker/1:2-events07:45:161
1241749291,3sleep01241881-21ntpq07:25:190
1237601290,1sleep00-21swapper/007:24:140
1564202281,3sleep21564356-21sed12:30:252
1554619280,3sleep00-21swapper/012:25:100
1547960280,1sleep10-21swapper/112:15:221
1543871281,4sleep21543878-21basename12:15:112
1537257281,3sleep11537346-21ntpq12:05:201
1536794281,3sleep01533401-21/usr/sbin/munin12:05:190
1533972281,3sleep21535012-21kthreadcore12:05:172
1517171280,4sleep21517144-21apt-get11:50:022
1485190280,1sleep20-21swapper/211:17:472
1475728281,3sleep21475159-21/usr/sbin/munin11:10:172
1460038281,3sleep01461273-21kthreadcore10:55:170
1459136281,3sleep11459137-21sshd10:54:121
1428602281,3sleep21428636-21kthreadcore10:25:172
1428602281,3sleep21428636-21kthreadcore10:25:162
1427760280,1sleep10-21swapper/110:24:551
1413123281,3sleep11414804-21cut10:10:161
1407073280,2sleep10-21swapper/110:05:121
1401843281,3sleep21402063-21df_inode10:00:122
1400613281,3sleep11396483-21/usr/sbin/munin09:55:221
1390287281,3sleep01390502-21grep09:45:270
1386516281,3sleep21387075-21kthreadcore09:45:162
1380653280,1sleep10-21swapper/109:40:031
1364670281,2sleep10-21swapper/109:25:111
1355898281,3sleep01357695-21cut09:15:170
1347702280,4sleep21347749-21ntpq09:05:212
1338185280,1sleep20-21swapper/208:59:432
1310620281,2sleep01310667-21ntpq08:30:200
1306333281,3sleep21306334-21sshd08:29:132
1291572281,3sleep21293237-21kthreadcore08:15:142
1271608280,1sleep00-21swapper/007:55:160
1268420281,3sleep11268469-21ntpq07:50:211
1263021281,3sleep01263142-21ntpq07:45:190
1253280280,1sleep00-21swapper/007:38:450
1225265281,3sleep11226624-21cut07:15:021
1224823280,1sleep20-21swapper/207:11:142
1224141281,3sleep11224192-21ld07:10:201
1558671271,2sleep10-21swapper/112:25:201
1554074270,1sleep00-21swapper/012:24:550
1521585270,3sleep01521642-21ntpq11:50:190
1511411270,1sleep10-21swapper/111:42:141
1496169271,2sleep20-21swapper/211:30:142
1496169271,2sleep20-21swapper/211:30:132
1464255270,1sleep20-21swapper/210:57:432
1449593271,2sleep21450840-21cut10:45:172
1437363271,3sleep20-21swapper/210:30:222
1431605270,1sleep00-21swapper/010:25:190
1431605270,1sleep00-21swapper/010:25:190
1417850271,3sleep20-21swapper/210:15:152
1417081270,1sleep20-21swapper/210:12:412
1411885270,1sleep20-21swapper/210:08:052
1401476270,1sleep00-21swapper/009:59:460
1396482270,2sleep20-21swapper/209:55:112
1380971270,3sleep21381184-21hddtemp_smartct09:40:162
1369897270,2sleep20-21swapper/209:30:102
1354459271,3sleep11354663-21head09:15:141
1354169270,2sleep20-21swapper/209:15:112
1353884270,1sleep00-21swapper/009:14:400
1337212271,1sleep10-21swapper/108:55:221
1326457271,3sleep10-21swapper/108:45:201
1318806271,3sleep01320592-21kthreadcore08:40:170
1311326270,1sleep20-21swapper/208:31:142
1301429270,2sleep00-21swapper/008:25:100
1280383270,1sleep20-21swapper/208:05:092
1279573270,1sleep00-21swapper/008:01:300
1279043270,4sleep10-21swapper/108:00:231
1274429270,1sleep10-21swapper/107:57:381
1248128270,1sleep10-21swapper/107:34:571
1241936271,3sleep21241990-21ld07:25:212
1228819271,2sleep00-21swapper/007:15:170
1539165261,2sleep00-21swapper/012:10:160
1538893261,2sleep20-21swapper/212:10:132
1533071260,1sleep10-21swapper/112:04:101
1532790260,1sleep00-21swapper/012:00:330
1501390260,4sleep20-21swapper/211:35:032
1500951260,1sleep00-21swapper/011:32:360
1500951260,1sleep00-21swapper/011:32:350
1479265261,2sleep00-21swapper/011:10:230
1474700260,1sleep20-21swapper/211:07:332
1448466260,3sleep00-21swapper/010:41:170
1448449260,4sleep10-21swapper/110:41:071
1421526261,1sleep00-21swapper/010:15:210
1417801261,2sleep10-21swapper/110:15:141
1405720261,1sleep00-21swapper/010:00:170
1395247260,1sleep00-21swapper/009:50:190
1391534261,2sleep20-21swapper/209:50:142
1365761261,2sleep20-21swapper/209:25:162
1350285261,2sleep20-21swapper/209:10:172
1348473260,4sleep10-21swapper/109:07:041
1342590261,2sleep10-21swapper/109:00:221
1332406261,2sleep00-21swapper/008:50:270
1322170261,2sleep20-21swapper/208:45:012
1320595261,2sleep10-21swapper/108:40:191
1308077261,2sleep10-21swapper/108:30:171
1286207261,2sleep10-21swapper/108:10:171
1247790260,1sleep20-21swapper/207:30:412
1553954250,3sleep10-21swapper/112:23:231
1526974250,2sleep20-21swapper/211:55:232
1521644250,2sleep11521692-21ldconfig11:50:201
1496163251,1sleep10-21swapper/111:30:091
1496163251,1sleep10-21swapper/111:30:091
1468602250,1sleep20-21swapper/211:00:192
1457110250,1sleep20-21swapper/210:50:182
1452670250,1sleep01452920-21strings10:45:190
1442301250,1sleep10-21swapper/110:35:181
1426831250,1sleep00-21swapper/010:20:210
1379131250,1sleep00-21swapper/009:35:210
1378892250,1sleep20-21swapper/209:35:182
1374890250,1sleep00-21swapper/009:35:010
1264402250,3sleep20-21swapper/207:50:052
1260448250,2sleep20-21swapper/207:45:172
1226626250,1sleep20-21swapper/207:15:032
1480153240,1sleep00-21swapper/011:15:010
1407377240,1sleep00-21swapper/010:05:150
1370196240,1sleep10-21swapper/109:30:131
1338727240,1sleep00-21swapper/009:00:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional