You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-03 - 11:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Mon Nov 03, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21883210207,1001sleep021924-21ntpq21:00:170
23731259747,523sleep023919-21latency_hist22:49:480
23731259747,523sleep023919-21latency_hist22:49:480
139724008,381sleep01437-21ntpq19:55:120
139724008,381sleep01437-21ntpq19:55:120
2179823498,328sleep021888-21apt-config22:44:470
683723186,300sleep06873-21ntpq21:55:130
21272230737,257sleep021312-21latency19:15:140
25730230024,263sleep00-21swapper/019:30:040
29298228620,246sleep229292-21apt-get21:25:022
17900226715,157sleep017969-21latency_hist00:14:470
15449226569,181sleep215450-21ntp_states00:05:172
11967225562,125sleep012022-21fschecks_count23:55:050
11967225562,125sleep012022-21fschecks_count23:55:050
18626224856,132sleep00-21swapper/020:50:160
18626224856,132sleep00-21swapper/020:50:160
2589622479,198sleep025926-21kernelversion22:55:090
6919221925,92sleep36938-21idleruntime-cro20:14:453
20949221767,132sleep020985-21munin-plugin-st00:24:450
1400922149,115sleep014156-21latency_hist22:19:480
532022138,190sleep35326-21ntpq21:50:153
31670221270,53sleep00-21swapper/023:14:470
29368220822,150sleep00-21swapper/021:25:040
19411220012,175sleep219413-21grep19:10:032
2150321904,173sleep00-21swapper/000:25:090
2142218978,93sleep02271-21timerwakeupswit21:40:190
5613218717,48sleep00-21swapper/023:34:500
14088218149,50sleep00-21swapper/000:00:180
16544217031,103sleep00-21swapper/019:00:250
29105216720,129sleep229108-21/usr/sbin/munin23:05:122
1990721656,94sleep00-21swapper/020:54:560
6119216124,117sleep16135-21modprobe21:54:451
30960216157,89sleep00-21swapper/023:10:190
19879216144,21sleep00-21swapper/019:10:180
1066621599,137sleep010702-21ntp_kernel_err23:50:120
1337215710,64sleep01351-21needreboot23:20:120
5340214728,81sleep01-21systemd23:34:460
890821436,12sleep10-21swapper/122:00:271
23169214350,54sleep023225-21fschecks_time21:05:140
2411821429,120sleep024147-21latency_hist00:34:460
31916214023,78sleep00-21swapper/019:50:070
28139213921,54sleep00-21swapper/019:35:250
28139213921,54sleep00-21swapper/019:35:250
17735213851,76sleep01714-1kworker/0:1H20:48:300
28446213742,77sleep00-21swapper/023:04:460
10100213610,112sleep010245-21latency_hist20:24:480
2439721358,110sleep00-21swapper/022:50:250
22632213519,44sleep322662-21awk22:45:153
22632213519,44sleep322662-21awk22:45:153
9254213110,80sleep09326-21mailstats23:45:160
18726212924,82sleep118760-21munin-plugin-st19:09:461
18223212918,90sleep118279-21fschecks_count00:15:051
23157212823,87sleep323166-21forks21:05:133
1630421288,61sleep016408-21date00:09:470
477721248,75sleep04891-21sed20:05:170
1587121208,14sleep20-21swapper/222:25:112
25408211917,82sleep00-21swapper/019:29:480
25408211917,82sleep00-21swapper/019:29:480
1155921190,8sleep10-21swapper/123:54:451
27509211730,19sleep00-21swapper/021:19:480
2927211616,82sleep13082-21cut21:44:481
1629121159,48sleep01-21systemd20:44:460
7707211213,81sleep17748-21systemd-run21:59:461
25020211118,70sleep325063-21ntp_offset21:10:183
17134211143,45sleep117154-21sh19:04:451
1432121118,92sleep014356-21fschecks_count22:20:080
15100211019,76sleep215117-21uniq20:40:082
8680210714,74sleep38791-21cat22:00:253
1899721079,23sleep20-21swapper/219:09:512
31525210620,62sleep131670-21sed21:30:181
28118210643,46sleep00-21swapper/021:20:120
27835210627,64sleep327914-21fschecks_time21:20:073
16681210629,55sleep00-21swapper/022:25:250
18599210530,63sleep018667-21modprobe22:34:460
26384210419,64sleep126521-21head19:30:171
11557210441,21sleep00-21swapper/022:10:150
29763210319,69sleep329800-21wc21:25:163
18406210311,75sleep018444-21latency00:15:080
13540210312,17sleep30-21swapper/320:35:043
8509210112,77sleep08597-21wc23:44:470
8509210112,77sleep08597-21wc23:44:470
5328210127,62sleep05366-21processes21:50:160
10803210118,61sleep110315-21/usr/sbin/munin20:25:171
7771210021,23sleep30-21swapper/321:59:483
32546210048,15sleep032630-21perl21:35:060
18488210014,71sleep318519-21nfs_client20:50:113
18488210014,71sleep318519-21nfs_client20:50:113
1275921007,61sleep10-21swapper/122:15:041
1195129916,66sleep211960-21df_inode23:55:032
1195129916,66sleep211960-21df_inode23:55:032
3003229825,60sleep030081-21systemd-run19:44:460
2149929719,38sleep10-21swapper/121:00:021
1919029726,52sleep00-21swapper/022:35:100
3187729615,68sleep331913-21cpu19:50:063
1472429620,14sleep00-21swapper/020:39:460
186522945,33sleep00-21swapper/019:08:480
1846129313,67sleep318462-21sshd19:06:253
1791129328,29sleep10-21swapper/120:49:481
25242928,31sleep00-21swapper/023:25:010
1754729012,61sleep217603-21/usr/sbin/munin22:30:072
435728919,13sleep00-21swapper/021:47:050
3065428915,53sleep130262-21/usr/sbin/munin23:10:121
3065428915,53sleep130262-21/usr/sbin/munin23:10:121
2107628916,43sleep10-21swapper/122:40:161
199642897,16sleep20-21swapper/200:20:102
319662889,64sleep032056-21perl23:15:080
319662889,64sleep032056-21perl23:15:080
2355728836,30sleep20-21swapper/222:49:462
2355728836,30sleep20-21swapper/222:49:462
1025728821,48sleep110399-21unixbench_singl22:05:191
225472877,16sleep00-21swapper/019:20:030
2023228723,31sleep10-21swapper/122:39:471
1746028731,28sleep10-21swapper/122:30:061
95462867,15sleep00-21swapper/022:05:020
837328613,23sleep00-21swapper/022:00:180
774928612,60sleep27770-21latency_hist21:59:472
602228620,49sleep30-21swapper/323:35:093
60172867,38sleep00-21swapper/020:10:100
3141628615,60sleep00-21swapper/019:47:370
287382868,15sleep00-21swapper/023:05:020
1481628616,44sleep314211-21/usr/sbin/munin22:20:193
1637028529,29sleep20-21swapper/219:00:232
732628410,62sleep17332-21cat20:15:081
67728419,46sleep2682-21modprobe19:54:452
255128421,16sleep00-21swapper/020:00:000
255128421,16sleep00-21swapper/020:00:000
613828315,54sleep26197-21munin-run21:54:462
2538928322,30sleep30-21swapper/322:54:483
115142835,12sleep00-21swapper/020:28:390
1828228216,18sleep20-21swapper/200:15:062
883228124,38sleep28852-21fschecks_count23:45:062
2768328024,26sleep20-21swapper/219:35:152
2768328024,26sleep20-21swapper/219:35:152
246152806,15sleep00-21swapper/021:10:020
2788927913,50sleep127994-21wc19:35:181
2788927913,50sleep127994-21wc19:35:181
2288027825,31sleep20-21swapper/200:30:082
1926127824,33sleep119292-21cut19:09:541
108327822,29sleep20-21swapper/219:55:062
108327822,29sleep20-21swapper/219:55:062
45442777,16sleep30-21swapper/320:05:123
2652327718,42sleep226546-21users19:30:182
2177727715,17sleep20-21swapper/221:00:152
1233427720,15sleep00-21swapper/020:30:130
197352760,10sleep20-21swapper/222:35:192
1398227621,35sleep30-21swapper/322:19:473
1322127624,26sleep20-21swapper/220:34:472
858727519,34sleep27978-21/usr/sbin/munin22:00:242
767027522,37sleep00-21swapper/020:15:150
312922758,18sleep00-21swapper/021:30:130
2934627521,27sleep10-21swapper/123:05:201
2886227520,34sleep228922-21fschecks_time19:40:072
2436827519,23sleep10-21swapper/119:25:091
2436827519,23sleep10-21swapper/119:25:091
604627426,24sleep10-21swapper/120:10:111
1396627424,23sleep20-21swapper/220:35:162
625727319,30sleep10-21swapper/123:35:161
390727316,42sleep23986-21sort21:45:342
2047027318,39sleep220463-21apt-get22:40:012
1793927320,38sleep318013-21find22:30:173
1392727316,14sleep20-21swapper/200:00:162
1234227316,41sleep112348-21sh20:30:141
845727219,40sleep28483-21munin-run23:44:452
845727219,40sleep28483-21munin-run23:44:452
309902726,13sleep20-21swapper/221:30:072
2018027223,27sleep30-21swapper/322:39:453
158727221,29sleep30-21swapper/323:20:203
1214927218,42sleep312195-21sed23:55:083
1214927218,42sleep312195-21sed23:55:083
378027119,35sleep33781-21cron23:29:443
3244827120,22sleep30-21swapper/323:15:193
3244827120,22sleep30-21swapper/323:15:193
2956627115,39sleep129607-21latency21:25:101
255327120,32sleep12547-21apt-get20:00:011
255327120,32sleep12547-21apt-get20:00:011
2173027124,30sleep321796-21apt-config22:44:463
1714727125,26sleep10-21swapper/100:10:161
1615127122,21sleep30-21swapper/322:25:163
1539927120,21sleep10-21swapper/100:05:161
1136727120,36sleep311459-21load22:10:103
882527013,18sleep10-21swapper/123:45:051
2921527013,29sleep30-21swapper/323:05:173
261127026,26sleep20-21swapper/223:25:072
2697226914,35sleep227100-21/usr/sbin/munin21:15:212
2102026916,37sleep321023-21df_inode19:15:053
533026818,33sleep35331-21cron23:34:453
3091826818,32sleep230996-21threads19:45:162
1609326814,39sleep116095-21sshd20:42:251
149612686,51sleep30-21swapper/300:04:533
1244426822,18sleep10-21swapper/122:14:481
1085726823,20sleep20-21swapper/222:09:482
541126719,25sleep20-21swapper/220:09:472
2925826727,26sleep329324-21cat19:40:173
2900726721,29sleep21-21systemd21:24:462
2767126715,27sleep10-21swapper/123:00:191
185672670,7sleep10-21swapper/120:50:121
185672670,7sleep10-21swapper/120:50:121
1401426720,24sleep30-21swapper/300:00:173
540126614,35sleep15410-21munin-run20:09:461
3055426614,21sleep30-21swapper/323:10:103
3055426614,21sleep30-21swapper/323:10:103
2133626619,18sleep10-21swapper/120:59:501
2126126621,23sleep30-21swapper/320:59:483
742426526,22sleep30-21swapper/320:15:103
466926515,31sleep24746-21/usr/sbin/munin23:30:162
2301426511,35sleep223029-21/usr/sbin/munin19:20:212
2144926510,39sleep321501-21/usr/sbin/munin00:25:083
1356026522,27sleep10-21swapper/120:35:061
2124226419,28sleep121236-21apt-get00:25:021
1654226412,30sleep20-21swapper/200:10:012
1645726424,26sleep316541-21sed19:00:243
2826926321,27sleep128348-21grep21:20:171
2414826326,19sleep10-21swapper/100:34:471
136382637,12sleep20-21swapper/222:15:542
11842639,11sleep30-21swapper/321:39:093
770226222,22sleep20-21swapper/220:15:172
622326216,32sleep26295-21switchtime20:10:162
3264126218,27sleep1303-21uptime23:15:211
3264126218,27sleep1303-21uptime23:15:211
2456426219,26sleep20-21swapper/222:50:312
1080526213,35sleep310829-21awk22:09:453
266292618,12sleep30-21swapper/319:31:163
2418526129,18sleep20-21swapper/219:25:052
2418526129,18sleep20-21swapper/219:25:052
2414226115,24sleep124143-21sshd21:07:251
2152626120,22sleep30-21swapper/319:15:213
2137126118,18sleep20-21swapper/219:15:162
136426116,29sleep31395-21/usr/sbin/munin19:55:113
136426116,29sleep31395-21/usr/sbin/munin19:55:113
128426121,19sleep10-21swapper/121:39:471
115362617,10sleep20-21swapper/223:54:302
739326020,17sleep30-21swapper/323:40:103
739326020,17sleep30-21swapper/323:40:103
385926025,18sleep20-21swapper/220:04:472
385926025,18sleep20-21swapper/220:04:472
2446726019,25sleep30-21swapper/319:25:113
2446726019,25sleep30-21swapper/319:25:113
115362606,12sleep20-21swapper/220:28:582
905825918,24sleep29141-21sed20:20:092
53225920,19sleep20-21swapper/221:35:162
2694925920,20sleep30-21swapper/322:59:483
2563025916,28sleep225700-21df22:55:032
2263925915,18sleep10-21swapper/119:20:121
1990925918,25sleep219928-21apt-get20:54:582
126525917,23sleep10-21swapper/119:55:091
126525917,23sleep10-21swapper/119:55:091
1195625917,21sleep30-21swapper/320:30:043
564725816,16sleep20-21swapper/223:35:002
3124425821,16sleep30-21swapper/321:30:123
3040625821,22sleep30-21swapper/319:45:053
278125815,19sleep30-21swapper/321:44:463
2761825815,27sleep327657-21grep19:35:123
2761825815,27sleep327657-21grep19:35:123
312825713,28sleep13202-21find23:25:181
3083025717,21sleep10-21swapper/119:45:151
2877625715,25sleep128769-21apt-get19:40:021
2422425716,19sleep10-21swapper/122:50:221
2380525717,25sleep323836-21munin-run19:24:463
2307125716,25sleep223055-21apt-get21:05:082
2124325715,22sleep221310-21df00:25:032
1977225715,26sleep119777-21diskstats00:20:051
104125712,28sleep11052-21df_inode23:20:051
2690425617,25sleep126945-21systemd-run22:59:461
2347025613,28sleep323591-21cat00:30:213
2114725618,20sleep10-21swapper/119:15:111
208522568,35sleep30-21swapper/300:23:313
1938125614,24sleep219406-21munin-run20:54:462
1938125614,24sleep219406-21munin-run20:54:462
1720925617,22sleep317341-21sed20:45:183
1460325617,24sleep114634-21nfs_client22:20:131
1075825619,22sleep310759-21ntp_states23:50:153
720125514,25sleep17194-21apt-get23:40:041
720125514,25sleep17194-21apt-get23:40:041
4172557,37sleep20-21swapper/223:16:272
4172557,37sleep20-21swapper/223:16:272
1627425520,22sleep316287-21cron20:44:453
1586425512,28sleep115870-21diskstats22:25:101
540925416,19sleep10-21swapper/123:34:471
2749825412,26sleep327507-21munin-run21:19:473
2665725415,23sleep126663-21ntp_states21:15:161
1701425418,21sleep317073-21sendmail_mailqu00:10:143
1353525412,27sleep113624-21/usr/sbin/munin00:00:071
2719025313,23sleep327185-21apt-get23:00:053
2523025314,23sleep225324-21sed21:10:252
2515725315,14sleep10-21swapper/121:10:241
2351925315,15sleep10-21swapper/122:49:451
2351925315,15sleep10-21swapper/122:49:451
2223825317,17sleep30-21swapper/321:00:243
1664025315,21sleep216647-21diskstats20:45:052
121725312,15sleep20-21swapper/223:20:102
347225219,16sleep30-21swapper/321:45:233
281132526,35sleep20-21swapper/223:00:322
1008825215,22sleep310098-21/usr/sbin/munin20:24:473
3242125114,21sleep10-21swapper/119:50:201
3068625116,18sleep20-21swapper/223:10:132
3068625116,18sleep20-21swapper/223:10:132
423425012,21sleep14236-21users21:45:371
1813525012,22sleep318132-21apt-get00:14:593
1152425014,20sleep211548-21ntp_states22:10:132
282724911,18sleep30-21swapper/320:00:113
282724911,18sleep30-21swapper/320:00:113
191924913,18sleep20-21swapper/221:40:132
116482488,25sleep311621-21latency_hist20:29:473
95562479,15sleep30-21swapper/323:45:193
902224710,22sleep19057-21irqstats20:20:081
1473624718,16sleep20-21swapper/222:20:182
1188324511,15sleep10-21swapper/123:55:021
1188324511,15sleep10-21swapper/123:55:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional