You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 01:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot3s.osadl.org (updated Tue Jul 15, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1924992180157,15sleep30-21swapper/319:08:353
1923752159134,19sleep10-21swapper/119:07:021
1925082145100,15sleep20-21swapper/219:08:422
192468214398,16sleep00-21swapper/019:08:130
23358521290,2sleep0221rcuc/019:45:190
27114821240,2sleep20-21swapper/220:25:172
192732993631,3cyclictest1-21systemd00:00:003
2356652340,3sleep2411rcuc/219:50:182
192732993328,3cyclictest521735-21sendmail23:19:593
192732993126,3cyclictest425946-21gltestperf22:20:153
192732993121,7cyclictest327364-21grep21:15:263
192732993026,2cyclictest0-21swapper/300:20:193
192732992919,7cyclictest422031-21grep22:15:313
192732992919,7cyclictest349796-21ntpq21:30:213
60348122813,10sleep1604679-21cut00:10:161
192732992824,2cyclictest728-21rs:main3
192732992824,2cyclictest468009-21kthreadcore22:45:173
192732992623,2cyclictest279544-21munin-plugin-st20:35:013
192732992621,3cyclictest361147-21munin-run21:40:003
19273299261,13cyclictest369147-21awk21:45:003
192732992521,2cyclictest0-21swapper/323:35:243
192732992520,3cyclictest334320-21missed_timers21:20:183
192732992516,6cyclictest542668-21cat23:30:253
192732992516,6cyclictest542668-21cat23:30:253
192732992420,2cyclictest264529-21cron20:19:593
192732992419,3cyclictest642032-21cpuspeed_turbos00:35:153
192732992419,3cyclictest559023-21unixbench_singl23:40:273
192732992417,5cyclictest342075-21/usr/sbin/munin21:25:243
192732992415,5cyclictest533953-21grep23:25:193
192732992413,7cyclictest945-21snmpd00:04:063
192732992412,8cyclictest603063-21awk00:10:133
5860252231,10sleep2314766-21diskmemload23:59:552
192732992320,2cyclictest0-21swapper/321:55:143
192732992319,2cyclictest284516-21idleruntime-cro20:40:013
192732992318,3cyclictest209586-21idleruntime-cro19:25:013
192732992318,2cyclictest0-21swapper/322:50:133
192732992316,4cyclictest453742-21/usr/sbin/munin22:35:203
19273299230,8cyclictest409385-21runrttasks22:10:053
19273299221,7cyclictest578324-21latency_hist23:55:013
192732992217,3cyclictest627048-21gltestperf00:25:193
192732992217,3cyclictest561140-21kworker/u16:2-rpciod00:10:013
192732992217,3cyclictest262-21systemd-journal22:34:593
192732992213,6cyclictest502501-21grep23:05:273
19273299221,17cyclictest408405-21ssh22:08:593
19273299220,19cyclictest507993-21kthreadcore23:10:193
192732992117,2cyclictest397981-21grep22:00:273
192732992113,5cyclictest611153-21sort00:15:123
192732992112,6cyclictest458025-21apt-get22:40:133
192732992111,7cyclictest496314-21ssh23:02:553
19273299211,17cyclictest369514-21sh21:45:133
3348232201,13sleep0334925-21cat21:20:270
19273299205,3cyclictest511ktimers/323:24:353
192732992016,2cyclictest0-21swapper/323:49:233
192732992015,3cyclictest304849-21idleruntime-cro21:00:003
192732992012,5cyclictest310390-21grep21:05:143
192732992011,7cyclictest245024-21tune2fs20:00:173
192732992011,7cyclictest0-21swapper/322:28:223
192732992011,7cyclictest0-21swapper/322:28:223
192732992011,6cyclictest197209-21grep19:10:273
19273299201,15cyclictest486029-21ntpq22:55:233
3694802191,13sleep1369515-21ssh21:45:131
19273299196,8cyclictest213631-21ntpq19:25:213
19273299191,7cyclictest233710-21/usr/sbin/munin19:45:223
192732991915,2cyclictest720-21tpm2-abrmd19:50:263
192732991915,2cyclictest289544-21apt-get20:45:013
192732991912,5cyclictest319872-21ssh21:10:513
19273299191,15cyclictest943-21runrttasks21:51:313
19273299190,7cyclictest200020-21/usr/sbin/munin19:15:153
19273299180,15cyclictest249780-21/usr/sbin/munin20:05:263
2885452171,4sleep0288594-21ntpq20:40:230
19273299178,6cyclictest308664-21memory21:00:193
19273299178,6cyclictest219975-21diskstats19:35:133
19273299171,6cyclictest2703132kthreadcore20:25:173
192732991712,3cyclictest254796-21/usr/sbin/munin20:10:143
192732991712,3cyclictest254796-21/usr/sbin/munin20:10:133
192732991712,3cyclictest229592-21cron19:45:003
192732991710,5cyclictest244530-21cron20:00:003
19273299170,15cyclictest215903-21kthreadcore19:30:183
5443352160,3sleep00-21swapper/023:32:310
5443352160,3sleep00-21swapper/023:32:300
5257992161,2sleep00-21swapper/023:20:200
3845292160,2sleep00-21swapper/021:54:100
3206502161,3sleep20-21swapper/221:12:112
19273299167,6cyclictest290226-21df_inode20:45:153
19273299161,13cyclictest266129-21tr20:20:173
192732991611,3cyclictest0-21swapper/320:50:293
6476422151,4sleep2314766-21diskmemload00:38:002
6321042151,2sleep10-21swapper/100:26:561
6085492150,2sleep00-21swapper/000:11:470
4952652150,1sleep10-21swapper/123:01:221
4059352151,3sleep00-21swapper/022:05:280
3294422153,2sleep2191rcu_preempt21:18:452
5779992141,9sleep1578013-21ssh23:54:491
5692342140,1sleep20-21swapper/223:48:402
4397752140,1sleep10-21swapper/122:27:551
4397752140,1sleep10-21swapper/122:27:541
3943472141,9sleep2395931-21cut22:00:182
3602042140,2sleep20-21swapper/221:38:412
2312532141,9sleep10-21swapper/119:45:171
6393382131,8sleep1639348-21rm00:31:241
6230392130,2sleep10-21swapper/100:20:211
6150712131,6sleep2615126-21/usr/sbin/munin00:15:232
5999692131,6sleep20-21swapper/200:06:332
5125182130,7sleep2314766-21diskmemload23:13:172
4080482130,5sleep243-21ksoftirqd/222:08:272
3421522131,8sleep2342207-21awk21:25:262
3368212131,5sleep10-21swapper/121:23:131
6333992121,6sleep0314766-21diskmemload00:29:200
6328332120,1sleep20-21swapper/200:28:262
5790602120,8sleep1579975-21grep23:55:221
4988822121,7sleep1500516-21kthreadcore23:05:191
4985282124,5sleep0498881-21cut23:05:180
4858722121,7sleep2485924-21modprobe22:55:212
4390552121,6sleep2314766-21diskmemload22:26:512
4390552121,6sleep2314766-21diskmemload22:26:502
3306232122,7sleep2314766-21diskmemload21:20:132
2988232121,7sleep2298871-21modprobe20:50:222
2885962121,7sleep1284784-21/usr/sbin/munin20:40:231
6457532111,5sleep1645804-21ntpq00:35:221
6153102111,3sleep10-21swapper/100:15:261
5775902111,3sleep00-21swapper/023:54:120
5598302111,4sleep2314766-21diskmemload23:41:362
5199772111,3sleep20-21swapper/223:17:262
4874602111,4sleep0314766-21diskmemload22:56:440
4810962111,3sleep20-21swapper/222:54:132
4716632111,3sleep10-21swapper/122:46:581
4477962110,7sleep1447799-21ssh22:32:561
4302812111,3sleep10-21swapper/122:20:451
4246322111,4sleep10-21swapper/122:19:161
4228242111,2sleep20-21swapper/222:16:322
4222872111,3sleep00-21swapper/022:15:430
4170602111,4sleep2314766-21diskmemload22:14:532
4019492111,6sleep1402839-21tr22:05:171
3918572110,6sleep1314766-21diskmemload21:58:071
3747312111,4sleep20-21swapper/221:46:272
3665582111,3sleep10-21swapper/121:41:121
3543592111,6sleep0355930-21cut21:35:170
3422562111,3sleep00-21swapper/021:25:270
3288502111,7sleep1328860-21ssh21:17:471
2352722111,6sleep1235662-21kthreadcore19:50:171
6394752101,2sleep20-21swapper/200:31:372
6250702101,6sleep2625080-21ssh00:22:512
6250142101,4sleep0314766-21diskmemload00:22:460
6163452101,4sleep0314766-21diskmemload00:16:250
6015242101,2sleep00-21swapper/000:08:510
5996282101,3sleep10-21swapper/100:06:041
5923052101,3sleep20-21swapper/200:02:122
5913492101,3sleep10-21swapper/100:00:471
5823342101,4sleep0314766-21diskmemload23:55:250
5600442101,3sleep10-21swapper/123:41:551
5536032100,2sleep1191rcu_preempt23:39:211
5439022100,3sleep20-21swapper/223:31:532
5439022100,3sleep20-21swapper/223:31:532
5390902101,6sleep1540682-21cut23:30:161
5390902101,6sleep1540682-21cut23:30:151
5356932101,2sleep10-21swapper/123:26:511
5196002101,3sleep00-21swapper/023:16:520
5189422101,2sleep10-21swapper/123:15:531
5127652101,3sleep00-21swapper/023:13:390
5031102101,3sleep20-21swapper/223:06:112
4947082100,3sleep00-21swapper/023:00:330
4878682100,5sleep1314766-21diskmemload22:57:201
4799202101,3sleep0314766-21diskmemload22:52:270
4790702101,2sleep10-21swapper/122:51:121
4725322101,5sleep2472546-21ssh22:48:162
4653422100,3sleep20-21swapper/222:44:312
4592532102,4sleep1460950-21kthreadcore22:40:171
4575102101,4sleep0314766-21diskmemload22:39:510
4540742101,3sleep20-21swapper/222:35:252
4470862100,3sleep20-21swapper/222:31:522
4416762101,2sleep00-21swapper/022:30:140
4305252101,6sleep2314766-21diskmemload22:21:062
4292272101,2sleep00-21swapper/022:20:200
4164162101,3sleep10-21swapper/122:13:561
3984542100,1sleep00-21swapper/022:01:040
3983702103,1sleep10-21swapper/122:00:571
3904602101,3sleep0314766-21diskmemload21:56:020
3743952101,4sleep0314766-21diskmemload21:45:560
3661742101,3sleep2314766-21diskmemload21:40:372
3577372101,2sleep10-21swapper/121:35:271
3525062101,3sleep2314766-21diskmemload21:33:562
3459132101,4sleep0314766-21diskmemload21:30:130
3382712101,3sleep10-21swapper/121:25:171
3215152101,3sleep0314766-21diskmemload21:13:330
3201922101,2sleep10-21swapper/121:11:261
3037962101,4sleep2303842-21ntpq20:55:202
2734482101,5sleep0273521-21ntpq20:25:200
2469842102,5sleep1248377-21cut20:00:191
2234922106,1sleep10-21swapper/119:35:201
2148242101,4sleep1214843-21apt-get19:30:141
606932291,3sleep2606982-21ntpq00:10:222
591342290,2sleep00-21swapper/000:00:450
576274291,4sleep2576286-21ssh23:52:142
566511291,2sleep10-21swapper/123:45:231
560623290,2sleep00-21swapper/023:42:460
553484291,2sleep20-21swapper/223:39:102
552693291,2sleep00-21swapper/023:38:010
532467290,3sleep00-21swapper/023:25:180
530145291,3sleep2530173-21apt-get23:25:122
526848290,3sleep2314766-21diskmemload23:20:432
526751291,4sleep1526761-21ssh23:20:341
507716290,4sleep1509416-21grep23:10:201
497048291,3sleep2314766-21diskmemload23:04:032
466018290,3sleep00-21swapper/022:45:080
455271293,3sleep10-21swapper/122:36:371
438845291,3sleep0314766-21diskmemload22:26:330
438845291,3sleep0314766-21diskmemload22:26:330
409642291,4sleep0409776-21perl22:10:140
392570290,3sleep20-21swapper/221:59:132
382917291,2sleep20-21swapper/221:51:482
381510290,2sleep10-21swapper/121:50:261
365339291,4sleep0365397-21ntpq21:40:190
345926291,2sleep10-21swapper/121:30:141
327601291,3sleep0314766-21diskmemload21:15:450
310087291,4sleep1310233-21df21:05:131
308870291,4sleep1308923-21ntp_states21:00:241
274815291,4sleep1274970-21diskstats20:30:131
273528291,4sleep1273577-21ntpq20:25:211
264204290,1sleep00-21swapper/020:15:580
260134291,4sleep2260250-21irqstats20:15:152
224859291,3sleep2224878-21apt-get19:40:132
218558291,3sleep2218603-21ntpq19:30:242
215166291,4sleep0215282-21irqstats19:30:170
646819280,2sleep00-21swapper/000:36:130
639497280,3sleep0314766-21diskmemload00:31:400
567222280,3sleep0314766-21diskmemload23:45:420
462960281,2sleep0314766-21diskmemload22:41:000
269377280,1sleep00-21swapper/020:22:430
250242281,3sleep0251522-21cat20:05:160
244486280,1sleep10-21swapper/119:59:321
220246281,3sleep2221068-21grep19:35:172
203555281,3sleep0203601-21ntpq19:15:230
312704271,3sleep2313636-21sed21:05:182
310065271,2sleep0310084-21apt-get21:05:120
299768270,1sleep10-21swapper/120:54:061
299153271,3sleep0299332-21cat20:50:290
294824271,3sleep0294871-21systemctl20:50:010
290053270,2sleep20-21swapper/220:45:142
275255271,3sleep0276549-21cut20:30:160
265414271,3sleep2267074-21kthreadcore20:20:172
263169270,2sleep1263473-21strings20:15:191
254779271,2sleep2254797-21apt-get20:10:152
254779271,2sleep2254797-21apt-get20:10:142
249960270,3sleep2250169-21hddtemp_smartct20:05:142
219971271,3sleep0220094-21perl19:35:150
204855270,2sleep20-21swapper/219:20:142
309666260,4sleep00-21swapper/021:03:150
304667260,1sleep00-21swapper/020:58:100
279812260,3sleep2279931-21cpuspeed_turbos20:35:162
279790260,2sleep10-21swapper/120:35:151
264812260,2sleep10-21swapper/120:20:121
258488261,2sleep10-21swapper/120:10:221
258488261,2sleep10-21swapper/120:10:211
244835262,1sleep00-21swapper/020:00:140
240288261,2sleep20-21swapper/219:55:162
238904261,2sleep00-21swapper/019:50:280
229501260,1sleep10-21swapper/119:43:521
213867261,2sleep20-21swapper/219:25:272
209834261,2sleep10-21swapper/119:25:141
209564260,2sleep0209589-21perf19:25:000
200683261,2sleep10-21swapper/119:15:201
305278250,1sleep20-21swapper/221:00:162
300317250,2sleep10-21swapper/120:55:151
293682250,1sleep10-21swapper/120:45:211
253575250,1sleep10-21swapper/120:05:211
243941250,1sleep00-21swapper/019:55:260
213520250,1sleep00-21swapper/019:25:210
196608250,1sleep20-21swapper/219:10:202
193080250,1sleep10-21swapper/119:10:151
288373240,1sleep20-21swapper/220:40:202
279720240,1sleep00-21swapper/020:35:020
274971240,1sleep20-21swapper/220:30:142
254553240,1sleep00-21swapper/020:10:010
254553240,1sleep00-21swapper/020:10:010
244598240,1sleep20-21swapper/220:00:002
233022240,1sleep20-21swapper/219:45:182
225032240,1sleep00-21swapper/019:40:140
205046240,1sleep10-21swapper/119:20:161
200006240,1sleep20-21swapper/219:15:172
192676240,1sleep00-21swapper/019:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional