You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-11 - 22:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Jul 11, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6591229810,272sleep16657-21unin-run07:24:011
4342228083,129sleep20-21swapper/208:59:222
16772227487,35sleep20-21swapper/207:54:312
16280225118,212sleep016360-21tune2fs11:19:580
16280225118,212sleep016360-21tune2fs11:19:580
28799224820,211sleep028852-21ls11:59:190
13204223914,200sleep213269-21fschecks_time07:44:212
6113222914,203sleep26183-21cat09:04:262
2906222633,139sleep23069-21grep07:09:322
15572220914,149sleep215593-21latency_hist11:18:592
16094220433,133sleep215591-21apt-get11:19:112
16094220433,133sleep215591-21apt-get11:19:112
2289220116,35sleep2227-21systemd-journal12:18:592
2289220116,35sleep2227-21systemd-journal12:18:592
23700220013,169sleep223701-21cron08:18:592
23700220013,169sleep223701-21cron08:18:592
29348219911,116sleep229396-21awk06:54:172
4748218311,32sleep24840-21latency_hist10:44:012
23825218214,147sleep223852-21systemctl11:43:582
31223217916,25sleep20-21swapper/210:24:192
9300217513,18sleep20-21swapper/209:14:242
2999621728,124sleep230146-21cat08:39:022
7852217023,116sleep10-21swapper/110:54:021
3185521689,90sleep231927-21sort06:59:302
26784216753,18sleep20-21swapper/210:09:282
20814216620,124sleep320960-21unixbench_singl09:49:353
807421610,30sleep18079-21timerandwakeup09:09:331
1768021465,11sleep20-21swapper/211:24:162
3276521459,78sleep20-21swapper/210:29:152
302982144100,19sleep20-21swapper/210:19:342
24230214421,96sleep324270-21fschecks_count11:44:203
5014214116,21sleep20-21swapper/207:18:532
668214028,91sleep1796-21latency_hist08:49:031
12872213256,62sleep212969-21cat09:24:302
9907213014,93sleep19972-21grep10:59:231
9907213014,93sleep19972-21grep10:59:231
3188213011,106sleep23280-21timerandwakeup12:19:312
16937212827,75sleep316963-21cpu09:39:173
9700212528,25sleep29740-21modprobe07:33:592
7285212315,87sleep07438-21/usr/sbin/munin10:49:330
2667212312,97sleep32728-21fschecks_time12:19:203
2493021224,109sleep00-21swapper/008:19:500
2493021224,109sleep00-21swapper/008:19:500
21551212164,40sleep20-21swapper/208:09:342
21551212164,40sleep20-21swapper/208:09:342
29017212018,81sleep129056-21idleruntime-cro06:54:001
9495211924,65sleep30-21swapper/307:31:223
19329211826,24sleep20-21swapper/211:29:292
19329211826,24sleep20-21swapper/211:29:292
14314211817,79sleep20-21swapper/209:29:302
1282211810,96sleep21299-21netstat08:49:282
1933211710,90sleep21946-21modprobe07:08:592
1933211710,90sleep21946-21modprobe07:08:592
32267211516,83sleep332303-21wc08:44:283
1000211518,76sleep11013-21fschecks_count08:49:231
24512211218,78sleep124662-21latency_hist10:04:021
7275211112,17sleep10-21swapper/107:24:291
13549211124,65sleep3420-21cron09:28:593
23164210929,64sleep20-21swapper/209:59:152
16936210954,17sleep229113-21cyclictest09:39:162
9637210874,16sleep20-21swapper/210:59:162
9637210874,16sleep20-21swapper/210:59:162
616210812,82sleep2625-21modprobe08:48:592
2940221078,79sleep229541-21latency_hist10:19:022
5325210613,15sleep20-21swapper/207:19:172
11543210647,39sleep20-21swapper/207:39:172
11543210647,39sleep20-21swapper/207:39:172
8174210520,67sleep20-21swapper/210:54:192
68021056,41sleep20-21swapper/212:13:092
378821059,79sleep23790-21munin-run08:59:012
26246210553,23sleep20-21swapper/211:49:322
10730210511,77sleep20-21swapper/207:34:362
797210418,72sleep1927-21latency_hist12:14:011
28843210421,55sleep028991-21grep10:14:480
614421015,55sleep20-21swapper/210:47:372
1226621016,58sleep20-21swapper/211:06:152
1226621016,58sleep20-21swapper/211:06:152
32569210029,47sleep332590-21apt-get07:03:593
1205429910,73sleep212145-21latency_hist09:24:012
66172988,15sleep20-21swapper/210:49:182
596329814,66sleep36018-21fschecks_time09:04:233
315782986,53sleep20-21swapper/212:08:052
259429816,65sleep32199-21/usr/sbin/munin07:09:253
1932529814,16sleep10-21swapper/111:29:281
1932529814,16sleep10-21swapper/111:29:281
158852988,71sleep20-21swapper/209:34:312
438529711,68sleep14387-21sshd10:40:211
238692970,11sleep30-21swapper/311:43:593
2202029712,13sleep20-21swapper/209:54:262
1973529732,34sleep20-21swapper/209:48:242
3065529633,15sleep20-21swapper/208:39:262
2785529613,69sleep327936-21timerandwakeup11:54:333
2546029613,61sleep025593-21latency_hist11:49:010
1018129523,55sleep00-21swapper/010:59:300
1018129523,55sleep00-21swapper/010:59:300
939929418,57sleep09507-21latency_hist10:59:020
252202946,50sleep20-21swapper/211:46:332
1786229423,53sleep017954-21perl07:59:220
1786229423,53sleep017954-21perl07:59:220
385729320,47sleep13898-21fschecks_count07:14:191
36232939,20sleep00-21swapper/010:39:230
2007329310,16sleep20-21swapper/209:49:172
39129216,61sleep1450-21grep10:29:181
3155129211,65sleep031561-21munin-run08:44:010
2557829215,38sleep00-21swapper/008:24:180
247252927,14sleep20-21swapper/210:04:112
1590529228,36sleep20-21swapper/207:53:422
1124929222,55sleep111277-21modprobe07:38:591
155702916,13sleep30-21swapper/307:49:363
1131029021,51sleep111376-21fschecks_time11:04:201
1131029021,51sleep111376-21fschecks_time11:04:201
29902897,12sleep20-21swapper/210:37:192
2909028911,62sleep329178-21cat06:54:023
2821928911,62sleep028234-21open_inodes06:49:270
775828814,54sleep20-21swapper/209:09:272
285862880,19sleep10-21swapper/110:14:441
255392885,11sleep20-21swapper/208:24:082
203392888,12sleep20-21swapper/208:05:302
203392888,12sleep20-21swapper/208:05:302
189572886,11sleep20-21swapper/208:03:042
189572886,11sleep20-21swapper/208:03:042
139312886,11sleep20-21swapper/211:12:452
139312886,11sleep20-21swapper/211:12:452
1106328816,50sleep010682-21/usr/sbin/munin09:20:030
629928722,19sleep10-21swapper/110:49:011
3098028731,33sleep10-21swapper/110:24:011
2778528730,13sleep20-21swapper/211:54:322
299322867,10sleep20-21swapper/212:02:112
81312855,10sleep20-21swapper/207:28:422
702428528,33sleep00-21swapper/007:24:240
2983428515,17sleep30-21swapper/306:54:303
220212856,11sleep20-21swapper/211:38:502
220212856,11sleep20-21swapper/211:38:502
281022845,11sleep20-21swapper/208:30:262
2656228428,35sleep30-21swapper/310:09:213
1539328421,38sleep015386-21apt-get09:34:170
112428434,21sleep30-21swapper/312:14:203
112428434,21sleep30-21swapper/312:14:203
2866728318,36sleep328662-21apt-get08:34:143
251522835,11sleep20-21swapper/208:22:422
251522835,11sleep20-21swapper/208:22:422
162628310,58sleep01649-21latency_hist10:34:020
130328319,48sleep11415-21cat07:04:421
130328319,48sleep11415-21cat07:04:421
1134628314,53sleep311415-21switchtime09:20:103
1032928324,27sleep00-21swapper/009:19:020
52128115,48sleep3579-21sshd10:29:203
122892817,14sleep10-21swapper/109:24:121
80892809,53sleep10-21swapper/110:54:141
2830328032,24sleep20-21swapper/206:49:292
3252627926,25sleep00-21swapper/010:29:010
355327816,39sleep00-21swapper/007:14:020
1859127815,45sleep30-21swapper/309:44:173
1407927815,47sleep114188-21cut11:14:011
1407927815,47sleep114188-21cut11:14:011
559727715,40sleep30-21swapper/307:19:263
402927618,27sleep30-21swapper/310:39:343
2837427622,35sleep028394-21modprobe08:33:590
2644427616,38sleep026594-21grep11:49:340
194627621,31sleep00-21swapper/010:34:180
2733927518,28sleep00-21swapper/011:54:220
845827411,36sleep30-21swapper/310:54:243
756827419,42sleep37590-21grep07:24:393
199792740,11sleep0165-21hwrng09:49:020
1903427425,23sleep10-21swapper/111:29:111
3086927329,26sleep30-21swapper/308:39:323
2584627323,24sleep30-21swapper/308:24:253
1576727316,24sleep10-21swapper/109:34:261
383827222,23sleep10-21swapper/112:23:581
2316527218,22sleep30-21swapper/309:59:163
1659327213,20sleep10-21swapper/107:54:261
1468427218,27sleep30-21swapper/311:14:233
1366427218,24sleep00-21swapper/007:44:320
508227115,14sleep30-21swapper/307:19:023
1912327119,24sleep30-21swapper/311:29:133
3113427017,32sleep031279-21unixbench_singl12:04:320
308832705,12sleep30-21swapper/310:23:473
220752700,8sleep00-21swapper/011:38:590
220752700,8sleep00-21swapper/011:38:590
1158827013,42sleep30-21swapper/307:39:183
1158827013,42sleep30-21swapper/307:39:183
1155127014,37sleep111673-21/usr/sbin/munin09:20:121
2992226918,39sleep030002-21switchtime06:54:310
236526922,29sleep30-21swapper/310:34:293
2170426917,28sleep021721-21fschecks_count09:54:200
1000626915,37sleep310052-21/usr/sbin/munin10:59:253
1000626915,37sleep310052-21/usr/sbin/munin10:59:253
2695626819,31sleep1227-21systemd-journal11:53:591
2578426814,41sleep325820-21fschecks_count11:49:203
2268826821,31sleep122703-21/usr/sbin/munin11:39:461
2267026811,45sleep322671-21sshd09:56:213
1842526822,25sleep30-21swapper/307:59:333
1842526822,25sleep30-21swapper/307:59:333
3257726716,22sleep30-21swapper/312:09:313
3156226717,21sleep10-21swapper/108:44:021
2771926718,34sleep127779-21processes11:54:311
2748926715,22sleep10-21swapper/108:29:241
2483326721,26sleep10-21swapper/110:04:191
2400126722,25sleep10-21swapper/108:19:181
2400126722,25sleep10-21swapper/108:19:181
2088026715,27sleep00-21swapper/008:09:120
2088026715,27sleep00-21swapper/008:09:120
1831526714,28sleep00-21swapper/009:44:020
1372826715,32sleep113829-21/usr/sbin/munin07:44:331
2878826617,26sleep00-21swapper/008:34:200
2626926614,37sleep026288-21apt-get10:09:100
1744726613,37sleep117541-21mailstats09:39:311
448626517,30sleep14527-21/usr/sbin/munin08:59:411
3140626518,23sleep30-21swapper/310:24:243
117626516,29sleep30-21swapper/307:04:403
117626516,29sleep30-21swapper/307:04:403
1046026518,22sleep10-21swapper/109:19:031
3148426417,26sleep10-21swapper/106:59:211
1415826413,34sleep114215-21memory09:29:251
747226315,29sleep07489-21cut09:09:210
658026314,31sleep06589-21modprobe07:24:000
51382637,12sleep00-21swapper/007:19:030
1943826320,25sleep10-21swapper/108:04:201
1943826320,25sleep10-21swapper/108:04:201
1752526320,21sleep00-21swapper/007:59:020
1595926317,28sleep00-21swapper/007:54:010
257392626,11sleep00-21swapper/010:05:310
1627526213,24sleep30-21swapper/311:19:573
1627526213,24sleep30-21swapper/311:19:573
138222626,12sleep30-21swapper/311:11:203
138222626,12sleep30-21swapper/311:11:203
506226114,31sleep35069-21diskstats10:44:193
2980326117,28sleep129809-21ntp_states06:54:281
2247726112,28sleep10-21swapper/111:39:131
2247726112,28sleep10-21swapper/111:39:131
83672605,12sleep10-21swapper/107:29:061
3962606,11sleep00-21swapper/012:09:360
2888126013,24sleep30-21swapper/311:59:223
2594726016,28sleep125978-21nfs_client08:24:271
222726018,19sleep30-21swapper/308:54:013
2067126017,24sleep00-21swapper/011:34:110
2934925910,34sleep329350-21cron10:18:593
284525914,23sleep10-21swapper/108:54:281
2300425918,22sleep30-21swapper/308:14:323
2300425918,22sleep30-21swapper/308:14:323
218412599,13sleep30-21swapper/311:36:283
218412599,13sleep30-21swapper/311:36:283
1172725924,18sleep30-21swapper/311:04:303
1172725924,18sleep30-21swapper/311:04:303
1130025914,29sleep311265-21latency_hist07:39:013
583925814,27sleep05831-21apt-get09:04:160
464725814,19sleep00-21swapper/008:59:440
387325819,21sleep30-21swapper/308:59:023
35042586,11sleep00-21swapper/012:19:420
2631525815,27sleep126366-21df10:09:151
2495525818,20sleep30-21swapper/310:04:213
1471325817,23sleep00-21swapper/011:14:240
1405625816,27sleep314115-21irqstats09:29:233
1359625819,20sleep30-21swapper/307:44:313
120925817,24sleep01246-21idleruntime12:14:210
120925817,24sleep01246-21idleruntime12:14:210
2172225711,19sleep10-21swapper/109:54:211
2132925723,20sleep00-21swapper/009:54:000
2009525715,26sleep320142-21users11:29:423
2009525715,26sleep320142-21users11:29:423
1247825715,23sleep00-21swapper/011:08:590
1247825715,23sleep00-21swapper/011:08:590
999225615,24sleep09988-21apt-get07:34:140
3262825616,22sleep032671-21munin-run07:04:000
256325613,26sleep00-21swapper/008:54:230
2375225615,23sleep00-21swapper/008:19:010
2375225615,23sleep00-21swapper/008:19:010
2127025620,17sleep30-21swapper/308:09:273
2127025620,17sleep30-21swapper/308:09:273
2998225515,19sleep10-21swapper/108:39:011
25382556,13sleep10-21swapper/110:34:311
1942225513,26sleep319428-21df_inode08:04:183
1942225513,26sleep319428-21df_inode08:04:183
173342557,12sleep10-21swapper/111:23:381
173342557,12sleep10-21swapper/111:23:381
148925514,22sleep00-21swapper/008:49:340
1439225513,26sleep114399-21diskstats11:14:171
138312557,36sleep00-21swapper/011:11:250
138312557,36sleep00-21swapper/011:11:250
592525413,23sleep15926-21diskstats09:04:211
3038625414,25sleep130383-21apt-get12:04:121
2389525417,22sleep023912-21latency_hist11:44:010
2377725418,20sleep00-21swapper/009:59:330
2147825316,23sleep10-21swapper/108:09:331
2147825316,23sleep10-21swapper/108:09:331
204232535,36sleep00-21swapper/011:33:100
198525314,19sleep00-21swapper/007:09:010
198525314,19sleep00-21swapper/007:09:010
1943025313,23sleep019437-21diskstats08:04:190
1943025313,23sleep019437-21diskstats08:04:190
1684425317,21sleep30-21swapper/307:54:323
1597125317,16sleep10-21swapper/107:54:021
975625214,23sleep09862-21latency_hist07:34:010
43872526,31sleep30-21swapper/308:59:233
3149325214,23sleep10-21swapper/110:24:261
2104725212,24sleep121048-21sshd09:50:211
1888225214,23sleep118487-21/usr/sbin/munin09:44:241
1279825214,17sleep10-21swapper/107:44:001
1279825214,17sleep10-21swapper/107:44:001
2963625112,24sleep029632-21apt-get10:19:120
1668625113,24sleep316696-21munin-run09:39:003
1384125115,18sleep00-21swapper/009:29:160
1309425112,22sleep013197-21users09:24:320
2931425016,16sleep10-21swapper/111:59:321
111225012,24sleep11118-21diskstats12:14:171
111225012,24sleep11118-21diskstats12:14:171
911324912,17sleep30-21swapper/309:14:213
2842624912,15sleep30-21swapper/308:34:023
267724914,20sleep12679-21ntp_states07:09:271
2532624916,18sleep30-21swapper/308:24:013
2532624916,18sleep30-21swapper/308:24:013
251142497,29sleep10-21swapper/111:45:111
1779924910,15sleep10-21swapper/107:59:191
1779924910,15sleep10-21swapper/107:59:191
863924813,20sleep30-21swapper/309:14:003
498724812,18sleep00-21swapper/010:44:160
1819724812,18sleep00-21swapper/011:24:380
322012479,17sleep00-21swapper/008:44:250
1178524711,17sleep00-21swapper/007:39:230
1178524711,17sleep00-21swapper/007:39:230
654124512,16sleep30-21swapper/310:49:153
3100524415,15sleep30-21swapper/312:04:303
225702439,19sleep122659-21sshd08:14:221
225702439,19sleep122659-21sshd08:14:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional