You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 05:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Nov 19, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1125973212159,58sleep30-21swapper/319:05:093
140950821200,2sleep10-21swapper/122:36:581
113088628553,10sleep10-21swapper/119:09:321
113073028252,9sleep20-21swapper/219:07:342
113078628156,19sleep00-21swapper/019:08:150
113121399351,30cyclictest1378487-21turbostat22:20:003
1131213992723,2cyclictest0-21swapper/323:50:003
113121399272,10cyclictest1444077-21taskset22:58:533
14591482260,2sleep1321rcuc/123:07:011
1131213992623,1cyclictest0-21swapper/319:50:003
113121399252,16cyclictest1596300-21taskset00:30:223
113121399237,12cyclictest1386052-21chrt22:23:543
113121399232,16cyclictest1261693-21turbostat21:10:003
113121399231,18cyclictest1369934-21taskset22:14:353
113121399222,14cyclictest1503358-21taskset23:35:183
113121399222,13cyclictest1591062-21taskset00:28:413
113121399221,8cyclictest1285967-21taskset21:23:403
1131213992217,3cyclictest0-21swapper/322:35:203
1131213992213,4cyclictest12766842sleep321:17:123
113121399221,17cyclictest1606059-21taskset00:37:193
113121399221,17cyclictest1342356-21taskset21:55:553
113121399221,13cyclictest1345708-21taskset22:00:083
15739852211,7sleep11261686-21diskmemload00:16:181
13097942211,8sleep01261686-21diskmemload21:37:050
113121399215,13cyclictest66550irq/131-enp1s0-TxRx-323:03:223
113121399212,14cyclictest1469403-21sleep23:15:003
113121399212,13cyclictest1584045-21taskset00:24:573
113121399212,13cyclictest1494191-21taskset23:29:563
1131213992119,1cyclictest0-21swapper/322:50:003
1131213992117,2cyclictest0-21swapper/322:50:183
1131213992117,2cyclictest0-21swapper/322:50:183
1131213992116,3cyclictest0-21swapper/321:45:233
1131213992115,1cyclictest0-21swapper/300:00:003
113121399211,16cyclictest1359384-21taskset22:06:143
113121399211,15cyclictest1548421-21chrt00:00:193
14368782201,13sleep21261686-21diskmemload22:55:092
13283942202,11sleep01328453-21munin-run21:50:000
113121399202,12cyclictest1334598-21taskset21:51:583
1131213992017,1cyclictest0-21swapper/323:22:133
1131213992017,1cyclictest0-21swapper/319:15:183
1131213992016,2cyclictest0-21swapper/323:17:003
1131213992016,2cyclictest0-21swapper/320:00:003
1131213992016,2cyclictest0-21swapper/300:20:013
1131213992014,3cyclictest0-21swapper/322:40:193
113121399201,3cyclictest0-21swapper/321:32:113
113121399201,15cyclictest1402702-21chrt22:34:083
113121399201,13cyclictest1495235-21chrt23:30:163
13684202191,2sleep10-21swapper/122:12:221
13664722191,5sleep21366531-21ntpq22:10:232
12976062191,13sleep11299272-21grep21:30:211
113121399192,13cyclictest1535553-21taskset23:54:463
1131213991916,1cyclictest0-21swapper/319:45:003
1131213991916,1cyclictest0-21swapper/300:05:183
113121399191,14cyclictest1319556-21chrt21:44:133
113121399191,14cyclictest1293771-21chrt21:27:573
113121399191,14cyclictest0-21swapper/322:29:023
113121399191,14cyclictest0-21swapper/300:10:283
113121399191,13cyclictest1458522-21chrt23:05:593
15674932181,10sleep21261686-21diskmemload00:13:382
12858122181,3sleep10-21swapper/121:23:271
113121399188,7cyclictest1256131-21taskset21:02:283
113121399182,10cyclictest1269574-21taskset21:13:583
1131213991814,2cyclictest0-21swapper/319:25:173
113121399181,14cyclictest0-21swapper/321:35:303
15994792171,3sleep20-21swapper/200:34:512
15505302171,3sleep20-21swapper/200:02:202
14187842171,2sleep10-21swapper/122:43:291
14028102171,3sleep10-21swapper/122:34:191
14025092171,10sleep21261686-21diskmemload22:33:522
13509572171,12sleep11350969-21ssh22:00:591
12959062171,4sleep21296128-21hddtemp_smartct21:30:182
12459852172,4sleep11246004-21apt-get20:55:131
113121399172,10cyclictest1187331-21taskset20:00:153
1131213991714,1cyclictest0-21swapper/320:30:153
1131213991714,1cyclictest0-21swapper/320:10:203
113121399171,3cyclictest1159744-21turbostat19:35:003
1131213991713,2cyclictest0-21swapper/323:40:193
1131213991711,4cyclictest0-21swapper/320:20:133
113121399171,13cyclictest0-21swapper/320:55:223
113121399171,10cyclictest1240711-21chrt20:50:053
113121399170,14cyclictest0-21swapper/320:35:173
113121399170,10cyclictest1214070-21taskset20:25:083
15895842161,3sleep00-21swapper/000:26:030
15095692160,2sleep10-21swapper/123:38:211
14921382161,9sleep01261686-21diskmemload23:26:540
14272172161,2sleep20-21swapper/222:48:192
13101182161,8sleep11261686-21diskmemload21:37:341
12779892161,2sleep20-21swapper/221:19:072
113121399164,6cyclictest1147334-21taskset19:20:193
1131213991613,1cyclictest0-21swapper/320:40:183
113121399161,11cyclictest1176470-21chrt19:50:133
113121399160,13cyclictest0-21swapper/320:45:223
113121399160,13cyclictest0-21swapper/320:15:203
16076192151,9sleep21-21systemd00:40:012
15747712151,8sleep21261686-21diskmemload00:17:362
15608362151,2sleep01560846-21ssh00:10:110
15418862150,9sleep272-21kcompactd023:56:502
15416922150,10sleep11541703-21ssh23:56:341
15070962151,8sleep21507155-21ntp_states23:35:262
14835112151,8sleep11261686-21diskmemload23:21:251
14215272150,9sleep01423111-21cut22:45:200
13843072151,8sleep21261686-21diskmemload22:21:212
13609692150,6sleep11261686-21diskmemload22:08:371
13274632153,7sleep11261686-21diskmemload21:48:381
13187002150,1sleep10-21swapper/121:42:551
12702452152,7sleep01270279-21munin-run21:15:000
12630702154,6sleep21264623-21cut21:10:172
113121399151,13cyclictest0-21swapper/320:05:223
113121399151,10cyclictest1131484-21chrt19:10:013
15909482140,1sleep20-21swapper/200:28:302
15265102141,6sleep21261686-21diskmemload23:48:572
15168502141,2sleep21261686-21diskmemload23:41:552
14751082141,7sleep11261686-21diskmemload23:16:161
14670092141,9sleep01467020-21sh23:11:300
14423132141,6sleep00-21swapper/022:56:160
14170742141,7sleep21261686-21diskmemload22:40:532
13876472141,8sleep01387766-21/usr/sbin/munin22:25:160
13755672141,6sleep01261686-21diskmemload22:15:420
13511502141,7sleep20-21swapper/222:01:172
13293732141,9sleep21329768-21grep21:50:182
12831212141,7sleep01283179-21ntpq21:20:210
12706382142,6sleep11270651-21apt-get21:15:131
12104342141,8sleep11212217-21cut20:20:181
12090752141,8sleep01210433-21cut20:20:170
11916432141,9sleep01191723-21cat20:00:300
11667032141,8sleep01168416-21kthreadcore19:40:170
113121399141,3cyclictest0-21swapper/319:35:153
16074022131,6sleep01261686-21diskmemload00:39:380
15921202132,5sleep11592140-21apt-get00:30:131
15652952131,8sleep11565384-21timerandwakeup00:10:281
15595802131,1sleep20-21swapper/200:08:302
15587232130,1sleep00-21swapper/000:07:110
15521042131,8sleep01552117-21sh00:04:420
15345662131,7sleep01261686-21diskmemload23:53:160
15204922131,4sleep01522112-21grep23:45:190
14946162130,2sleep20-21swapper/223:30:122
13921232131,3sleep21392134-21ssh22:25:392
13847022131,8sleep11384711-21ssh22:21:561
13500912131,6sleep01261686-21diskmemload22:00:260
13105742131,8sleep21310576-21ssh21:38:152
12871582131,5sleep21261686-21diskmemload21:25:082
12759312130,3sleep01261686-21diskmemload21:16:080
12313592131,8sleep01233086-21tr20:40:210
15974522121,4sleep01261686-21diskmemload00:31:200
15834022121,5sleep20-21swapper/200:23:402
15731552122,6sleep00-21swapper/000:15:280
15355402121,4sleep21261686-21diskmemload23:54:462
15258372121,4sleep11261686-21diskmemload23:47:551
15020072121,4sleep01261686-21diskmemload23:34:230
15005402121,3sleep10-21swapper/123:32:081
14913792121,5sleep21261686-21diskmemload23:25:472
14838912121,4sleep01261686-21diskmemload23:22:010
14602222121,7sleep21460231-21ssh23:08:402
14517282122,6sleep21451739-21ssh23:03:062
14342862121,6sleep01434296-21ssh22:51:340
14342862121,6sleep01434296-21ssh22:51:340
14247472121,6sleep11424810-21ntp_states22:45:251
14176602121,4sleep00-21swapper/022:41:480
14031882121,4sleep01261686-21diskmemload22:34:540
13853822121,3sleep00-21swapper/022:22:550
13717622120,7sleep11373461-21kthreadcore22:15:191
13446542121,5sleep21261686-21diskmemload21:59:242
12571662121,6sleep01258279-21kthreadcore21:05:180
12553842121,5sleep11255438-21ntpq21:00:221
11805072122,4sleep01180556-21ntpq19:50:230
11447382121,7sleep11145596-21grep19:20:181
16001122110,4sleep11599585-21apt-get00:35:021
15886752113,4sleep11588722-21open_inodes00:25:241
15818942111,4sleep01261686-21diskmemload00:20:500
15810712111,5sleep11581131-21ntpq00:20:231
15503992111,3sleep10-21swapper/100:02:071
15378402110,7sleep01539521-21cut23:55:200
15323072110,3sleep10-21swapper/123:50:241
15179562111,4sleep11261686-21diskmemload23:43:341
15140682111,5sleep01515057-21memory23:40:210
15094302111,2sleep00-21swapper/023:38:090
14920042111,4sleep11261686-21diskmemload23:26:431
14828972111,4sleep21261686-21diskmemload23:20:302
14668102110,1sleep20-21swapper/223:11:122
14516682111,4sleep11261686-21diskmemload23:03:011
14427002111,2sleep10-21swapper/122:56:481
14364822111,3sleep10-21swapper/122:54:551
14364822111,3sleep10-21swapper/122:54:551
14322892111,4sleep21261686-21diskmemload22:50:192
14322892111,4sleep21261686-21diskmemload22:50:192
14109442111,3sleep20-21swapper/222:39:092
14077582111,5sleep01407811-21ntpq22:35:220
13923652111,3sleep10-21swapper/122:26:011
13536902111,4sleep21261686-21diskmemload22:05:022
13434442111,4sleep01261686-21diskmemload21:57:340
13370822111,4sleep11336558-21apt-get21:55:021
13358912111,3sleep00-21swapper/021:54:010
13348372111,4sleep10-21swapper/121:52:221
13253652111,4sleep21261686-21diskmemload21:45:332
13180792111,6sleep21261686-21diskmemload21:42:002
13175032111,3sleep00-21swapper/021:41:100
12955132110,5sleep00-21swapper/021:30:080
12872212111,4sleep01261686-21diskmemload21:25:150
12032082111,5sleep01203463-21df20:15:160
11844692111,4sleep21185587-21memory19:55:182
11314972111,4sleep01131516-21apt-get19:10:110
15530002101,5sleep11553215-21tail00:05:151
14755402100,4sleep01261686-21diskmemload23:16:550
14697422101,3sleep21261686-21diskmemload23:15:132
14672382100,3sleep10-21swapper/123:11:511
14514192101,3sleep01261686-21diskmemload23:02:360
13760242100,1sleep20-21swapper/222:16:242
13614032100,3sleep00-21swapper/022:09:160
12933782100,1sleep10-21swapper/121:27:241
12832422100,3sleep21261686-21diskmemload21:20:232
12676012101,2sleep10-21swapper/121:11:001
12498612101,4sleep01250002-21ntpq20:55:200
12445662101,4sleep21244706-21ntp_192.168.11420:50:222
12280272100,1sleep00-21swapper/020:35:200
12140742101,4sleep21214093-21apt-get20:25:122
12004982101,4sleep21201763-21memory20:10:212
11749342101,4sleep11174980-21ntpq19:45:231
11602952101,5sleep01159996-21/usr/sbin/munin19:35:160
11522162101,4sleep11153300-21memory19:25:191
1453352291,3sleep01453374-21apt-get23:05:130
1362272291,4sleep01362275-21ssh22:10:130
1261673291,3sleep11261688-21apt-get21:10:001
1237058291,4sleep11238828-21grep20:45:191
1233821291,3sleep21233910-21ntpq20:40:242
1223349291,5sleep20-21swapper/220:30:222
1198823291,4sleep11200497-21grep20:10:201
1170895291,3sleep01170914-21apt-get19:45:140
1158795291,4sleep21158848-21ntp_states19:30:252
1145597291,4sleep21147332-21cut19:20:192
1131519291,4sleep11131771-21/usr/sbin/munin19:10:121
1254724281,3sleep21255238-21needreboot21:00:192
1250325280,1sleep20-21swapper/220:55:262
1240744281,3sleep01240999-21df20:50:150
1224141280,1sleep00-21swapper/020:32:240
1197890280,3sleep01197909-21apt-get20:10:150
1187309280,3sleep20-21swapper/220:00:152
1185768281,3sleep11185822-21modprobe19:55:211
1181713281,3sleep01181732-21apt-get19:55:120
1181303280,1sleep10-21swapper/119:53:091
1176256281,3sleep21175931-21apt-get19:50:012
1155304281,3sleep11155327-21kthreadcore19:30:201
1148027281,3sleep01148162-21netstat19:20:210
1143852280,1sleep00-21swapper/019:19:330
1229420270,1sleep10-21swapper/120:37:301
1224111270,1sleep10-21swapper/120:32:011
1219091270,1sleep10-21swapper/120:29:561
1217140270,3sleep00-21swapper/020:25:180
1212218270,2sleep21212484-21strings20:20:192
1196722270,3sleep11196800-21grep20:05:231
1170356271,3sleep21170791-21latency_hist19:45:002
1170088270,1sleep10-21swapper/119:41:391
1164855270,1sleep20-21swapper/219:37:512
1159995270,2sleep10-21swapper/119:35:131
1154157270,1sleep00-21swapper/019:26:140
1261356260,4sleep20-21swapper/221:06:002
1251869260,2sleep01252951-21kthreadcore21:00:170
1239947260,1sleep20-21swapper/220:47:012
1234789261,2sleep10-21swapper/120:43:271
1229406260,1sleep20-21swapper/220:37:192
1207962260,1sleep10-21swapper/120:16:501
1197586260,4sleep20-21swapper/220:09:332
1194022261,2sleep00-21swapper/020:05:180
1191451261,2sleep10-21swapper/120:00:261
1171060260,2sleep20-21swapper/219:45:162
1158688261,2sleep00-21swapper/019:30:230
1136965261,2sleep20-21swapper/219:15:022
1136622261,2sleep10-21swapper/119:15:011
1243503250,1sleep10-21swapper/120:50:201
1235198251,1sleep00-21swapper/020:45:140
1208102250,1sleep20-21swapper/220:18:352
1149790250,1sleep20-21swapper/219:25:162
1131024250,1sleep20-21swapper/219:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional