You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 06:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Nov 17, 2025 00:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
97672551617,5486sleep39661-21kworker/3:021:47:443
97672551617,5486sleep39661-21kworker/3:021:47:443
606028537,835sleep227147-1kworker/2:2H23:19:342
606028537,835sleep227147-1kworker/2:2H23:19:342
7521265714,628sleep27581-21/usr/sbin/munin21:39:282
26169264416,610sleep326222-21/usr/sbin/munin19:14:213
27850249131,444sleep327856-21df_inode19:19:173
27850249131,444sleep327856-21df_inode19:19:173
22993241410,359sleep023138-21latency_hist00:14:000
12385239217,317sleep012508-21wc21:54:310
12385239217,317sleep012508-21wc21:54:310
1260234916,312sleep01277-21modprobe19:38:560
253623337,293sleep0362-21snmpd19:40:300
3111232422,281sleep33251-21head21:24:313
3391228945,98sleep03526-21unixbench_singl23:09:310
24162282116,116sleep00-21swapper/023:08:590
346922735,50sleep00-21swapper/021:26:460
2699922728,217sleep027033-21ps00:24:300
8950226411,238sleep18951-21sshd19:59:441
7785225317,213sleep00-21swapper/019:58:590
23282224967,77sleep00-21swapper/000:14:200
20649223872,102sleep020730-21xargs00:04:310
22561223641,176sleep022658-21fschecks_time19:04:210
22561223641,176sleep022658-21fschecks_time19:04:210
5379223343,133sleep0362-21snmpd19:49:460
18699222423,142sleep018767-21latency_hist20:33:580
16098220410,181sleep016130-21awk22:08:560
23971219717,156sleep324026-21latency_hist22:33:583
1311219318,161sleep21274-21latency_hist19:38:592
24744218552,21sleep00-21swapper/019:09:210
18253218511,107sleep00-21swapper/023:58:580
10886218510,156sleep010975-21latency_hist20:09:000
3932218012,132sleep03983-21cpuspeed21:29:170
14535217242,82sleep014571-21latency20:19:250
14535217242,82sleep014571-21latency20:19:250
26537217061,95sleep026606-21switchtime19:14:300
1465217010,147sleep31542-21grep21:19:553
9334216840,86sleep09355-21latency_hist20:03:590
24950216813,139sleep324462-21apt-get00:19:083
24335216815,88sleep024484-21grep20:49:280
24335216815,88sleep024484-21grep20:49:280
28493216767,27sleep00-21swapper/019:19:470
28493216767,27sleep00-21swapper/019:19:470
3124521658,77sleep031281-21ntpq19:29:240
1584721648,144sleep1362-21snmpd22:05:471
1911421628,58sleep00-21swapper/023:59:300
6557215310,69sleep00-21swapper/023:20:090
6557215310,69sleep00-21swapper/023:20:090
5721215048,86sleep05904-21sed23:19:000
32008215017,119sleep332009-21cpu22:59:173
2106321498,125sleep021131-21switchtime20:39:270
2106321498,125sleep021131-21switchtime20:39:270
9741214717,112sleep10-21swapper/123:29:311
18525214416,115sleep318526-21sshd20:32:443
2926621416,10sleep10-21swapper/121:04:351
2926621416,10sleep10-21swapper/121:04:351
25565213965,18sleep00-21swapper/022:39:000
32492213832,91sleep032572-21grep22:59:300
6845213513,106sleep36889-21ntpq19:54:263
205921306,11sleep00-21swapper/021:23:350
17846213016,96sleep217911-21awk20:29:282
14435213032,47sleep014549-21timerwakeupswit23:44:280
10426213019,90sleep10-21swapper/121:49:241
9134212945,47sleep09127-21apt-get23:29:140
19896212829,63sleep019935-21ntpq22:19:220
25762212711,63sleep00-21swapper/022:39:120
18942212547,16sleep00-21swapper/020:34:150
28700212439,24sleep00-21swapper/000:29:310
14019212118,79sleep20-21swapper/220:18:592
16025212015,88sleep216135-21ls23:49:312
13656212011,95sleep013681-21awk21:59:240
18368211653,41sleep00-21swapper/022:14:210
437221149,65sleep04425-21apt-config19:48:580
31494211416,82sleep331564-21/usr/sbin/munin21:14:183
12714211432,55sleep312706-21apt-get20:14:163
31753211334,37sleep00-21swapper/022:58:580
24791211333,38sleep00-21swapper/022:34:280
26935211217,76sleep126972-21grep00:24:271
470211110,82sleep0677-21grep21:18:590
2675921106,74sleep00-21swapper/020:59:090
2675921106,74sleep00-21swapper/020:59:090
23272211012,60sleep023401-21timerwakeupswit22:29:310
2834221085,71sleep00-21swapper/022:45:580
1679421085,43sleep00-21swapper/020:25:270
18338210715,74sleep318366-21/usr/sbin/munin22:14:203
907021069,82sleep09117-21processes21:44:260
907021069,82sleep09117-21processes21:44:260
806721065,65sleep00-21swapper/021:40:510
2993210614,79sleep23110-21cat21:24:302
28842210622,49sleep10-21swapper/122:49:101
9474210513,76sleep29107-21/usr/sbin/munin23:29:242
7029210417,57sleep17048-21apt-get21:39:101
6553210424,19sleep10-21swapper/119:54:201
24581210416,72sleep224738-21latency_hist00:18:592
31336210217,72sleep331348-21open_files19:29:273
11394210225,59sleep111460-21wc20:09:211
19403210124,54sleep118917-21/usr/sbin/munin20:34:281
1864210010,12sleep10-21swapper/123:04:321
666629959,21sleep00-21swapper/021:37:440
3012729910,50sleep030204-21unin-run22:53:570
1544229920,61sleep115483-21fschecks_count23:49:181
2736429714,66sleep127515-21latency_hist19:18:591
2026529714,66sleep320407-21unixbench_singl22:19:293
2499129624,53sleep224971-21apt-get00:19:192
1070729512,23sleep20-21swapper/223:34:122
2649529345,36sleep00-21swapper/020:58:570
1024329323,49sleep210305-21grep21:49:212
771629222,56sleep27717-21sshd19:58:442
54562928,72sleep3362-21snmpd23:18:223
485729217,56sleep24371-21apt-get19:49:082
2098229219,55sleep321061-21/usr/sbin/munin20:39:263
2098229219,55sleep321061-21/usr/sbin/munin20:39:263
163622928,16sleep00-21swapper/022:09:090
1294129217,54sleep213069-21/usr/sbin/munin23:39:332
1180629121,48sleep211963-21sed20:09:302
455429017,52sleep24722-21apt-config19:49:002
44762875,13sleep20-21swapper/221:29:312
2832028732,45sleep00-21swapper/021:04:060
2832028732,45sleep00-21swapper/021:04:060
2729128718,48sleep127292-21/usr/sbin/munin22:44:091
2598828718,44sleep226032-21/usr/sbin/munin19:14:162
1712128718,53sleep117265-21cat20:28:591
166628616,55sleep31729-21switchtime23:04:303
1196528620,51sleep111980-21idleruntime-cro23:38:561
218332858,66sleep0362-21snmpd19:00:090
1490628510,43sleep30-21swapper/322:04:153
2414728416,18sleep30-21swapper/319:09:043
1614928322,40sleep216248-21latency_hist22:08:582
607428227,27sleep30-21swapper/323:19:443
607428227,27sleep30-21swapper/323:19:443
451228210,51sleep10-21swapper/123:14:211
2960828230,25sleep20-21swapper/221:08:562
2960828230,25sleep20-21swapper/221:08:562
251782818,57sleep025247-21grep00:20:140
2377128124,36sleep223783-21munin-node19:08:592
2377128124,36sleep223783-21munin-node19:08:592
163628123,27sleep10-21swapper/119:39:181
122612818,17sleep00-21swapper/023:39:150
614727915,50sleep36183-21munin-plugin-st19:53:573
1159327917,47sleep311655-21ps20:09:273
51027822,24sleep30-21swapper/319:34:313
314912789,54sleep10-21swapper/121:14:161
154362786,14sleep00-21swapper/023:49:170
128172787,18sleep012896-21perl20:14:210
221852777,60sleep0362-21snmpd22:26:570
1904527719,29sleep30-21swapper/323:59:293
2574527620,18sleep20-21swapper/219:14:002
2126527616,42sleep221301-21/usr/sbin/munin18:59:232
2061127622,24sleep20-21swapper/220:39:172
2061127622,24sleep20-21swapper/220:39:172
113442767,14sleep00-21swapper/020:09:200
103682766,30sleep00-21swapper/021:49:230
88832756,18sleep08868-21/usr/sbin/munin23:28:570
88832756,18sleep08868-21/usr/sbin/munin23:28:570
312492758,53sleep031387-21latency_hist21:13:590
294252757,19sleep029510-21perl19:24:190
1923727522,32sleep119370-21grep23:59:311
564827415,12sleep20-21swapper/221:34:192
1458827422,30sleep20-21swapper/222:03:582
1062427426,30sleep30-21swapper/321:49:303
2883427315,25sleep30-21swapper/321:04:293
2883427315,25sleep30-21swapper/321:04:293
2123927320,33sleep121264-21awk18:59:221
107052737,12sleep00-21swapper/023:34:100
83112729,15sleep30-21swapper/321:43:573
433827212,38sleep24435-21fschecks_time23:14:182
225602727,29sleep00-21swapper/020:44:280
3113127117,32sleep131290-21grep22:54:311
2987627116,36sleep129877-21/usr/sbin/munin21:09:111
2981227121,25sleep10-21swapper/119:24:281
290722716,53sleep30-21swapper/300:30:313
277742717,13sleep20-21swapper/219:19:122
277742717,13sleep20-21swapper/219:19:122
2622227123,21sleep10-21swapper/122:39:251
3097227027,26sleep330419-21/usr/sbin/munin22:54:293
1742427022,27sleep20-21swapper/223:54:322
1518027023,23sleep30-21swapper/323:48:593
2959926923,28sleep30-21swapper/319:24:223
2680226928,22sleep20-21swapper/200:24:242
167126910,14sleep20-21swapper/219:39:192
1356726912,45sleep313670-21cat23:43:593
442626829,21sleep10-21swapper/119:48:591
2254726824,23sleep30-21swapper/319:04:203
2254726824,23sleep30-21swapper/319:04:203
2169326812,25sleep30-21swapper/300:09:193
620926710,40sleep16351-21/usr/sbin/munin21:34:301
2337326719,36sleep123429-21cat00:14:211
2135626719,23sleep30-21swapper/300:08:583
411326612,25sleep10-21swapper/121:29:221
3274826628,25sleep2351-21apt-config21:18:572
311462666,10sleep20-21swapper/221:13:402
2471126612,28sleep30-21swapper/322:34:273
2471126612,28sleep30-21swapper/322:34:273
1097826622,21sleep30-21swapper/323:34:213
926326519,20sleep20-21swapper/220:03:452
2777826520,30sleep127822-21/usr/sbin/munin19:19:151
2777826520,30sleep127822-21/usr/sbin/munin19:19:151
2336026511,39sleep223427-21latency_hist20:48:582
1557126520,19sleep20-21swapper/220:23:592
1557126520,19sleep20-21swapper/220:23:592
1299726517,34sleep113018-21munin-run21:58:561
1299726517,34sleep113018-21munin-run21:58:561
600226417,30sleep16020-21apt23:19:091
600226417,30sleep16020-21apt23:19:091
435626418,28sleep34368-21sh19:48:573
2715626413,25sleep20-21swapper/220:59:232
2715626413,25sleep20-21swapper/220:59:232
2467126415,32sleep224709-21proc_pri22:34:252
2467126415,32sleep224709-21proc_pri22:34:252
2421226413,23sleep30-21swapper/320:49:273
2421226413,23sleep30-21swapper/320:49:273
1684926421,27sleep116885-21wc22:09:261
326926319,26sleep30-21swapper/323:09:303
244872636,12sleep10-21swapper/120:49:291
244872636,12sleep10-21swapper/120:49:291
1718426322,17sleep30-21swapper/323:54:243
142662638,39sleep113780-21/usr/sbin/munin23:44:251
2687526222,25sleep126965-21grep20:59:181
2687526222,25sleep126965-21grep20:59:181
2174826212,38sleep121749-21cron20:43:561
2174826212,38sleep121749-21cron20:43:561
1696226224,18sleep30-21swapper/322:09:283
1290226218,25sleep212903-21sshd21:57:442
1290226218,25sleep212903-21sshd21:57:442
540226113,30sleep15469-21systemctl19:49:471
320326110,27sleep20-21swapper/223:09:292
133732616,40sleep30-21swapper/321:59:193
968626015,25sleep30-21swapper/320:04:193
853226018,25sleep38627-21mailstats19:59:303
3264826019,19sleep20-21swapper/222:59:322
2014026021,23sleep20-21swapper/222:19:282
984525916,21sleep20-21swapper/220:04:222
889125920,24sleep10-21swapper/123:28:581
889125920,24sleep10-21swapper/123:28:581
3114525925,18sleep20-21swapper/219:29:222
1898125911,17sleep20-21swapper/223:59:282
758525816,26sleep27579-21apt-get23:24:152
758525816,26sleep27579-21apt-get23:24:152
2862725817,25sleep228758-21cat22:48:592
2653725820,23sleep30-21swapper/300:24:173
2549725816,18sleep10-21swapper/120:54:221
2128625814,19sleep30-21swapper/322:24:213
93925714,23sleep10-21swapper/121:19:371
2933025723,19sleep30-21swapper/322:49:283
2904825717,22sleep10-21swapper/100:30:121
2007025718,23sleep10-21swapper/122:19:271
1736425711,20sleep10-21swapper/120:29:151
1575425714,29sleep315796-21awk23:49:243
1400025714,22sleep20-21swapper/223:44:182
118132575,37sleep30-21swapper/321:54:183
118132575,37sleep30-21swapper/321:54:183
313902566,11sleep20-21swapper/222:54:562
3038825616,18sleep30-21swapper/321:09:303
2596225618,20sleep20-21swapper/220:54:312
2429625620,22sleep10-21swapper/122:34:161
2429625620,22sleep10-21swapper/122:34:161
2010325616,19sleep20-21swapper/200:04:172
1694025612,27sleep116937-21apt-get23:54:101
1286425614,26sleep112938-21find23:39:321
393025513,25sleep33924-21apt-get21:29:163
366825514,26sleep13686-21munin-run21:28:571
2988125518,17sleep20-21swapper/219:24:292
2804525513,25sleep328071-21munin-run21:03:563
2804525513,25sleep328071-21munin-run21:03:563
2633325515,20sleep30-21swapper/322:39:273
2204025514,24sleep122033-21apt-get20:44:131
1893925514,24sleep218935-21apt-get20:34:132
1397625514,26sleep114018-21unin-run20:18:581
110525513,25sleep21102-21apt-get23:04:122
3225425413,25sleep132277-21irqstats22:59:231
3108725415,24sleep131144-21irqstats19:29:211
2476525416,22sleep124372-21/usr/sbin/munin19:09:221
162025412,18sleep30-21swapper/319:39:163
1617925417,18sleep20-21swapper/220:24:242
867525314,22sleep18682-21df_inode21:44:151
867525314,22sleep18682-21df_inode21:44:151
571325314,22sleep35770-21irqstats21:34:203
3020625314,18sleep20-21swapper/222:53:592
2382925312,24sleep223869-21irqstats20:49:182
2382925312,24sleep223869-21irqstats20:49:182
2365025314,17sleep30-21swapper/300:14:283
2135125314,24sleep121445-21memory22:24:231
2005325312,24sleep120102-21/usr/sbin/munin00:04:161
1937025311,16sleep30-21swapper/320:34:263
1563925321,15sleep30-21swapper/320:24:003
1563925321,15sleep30-21swapper/320:24:003
960825213,16sleep30-21swapper/323:29:293
2312225216,20sleep20-21swapper/222:29:292
2225925213,17sleep30-21swapper/319:03:593
1773825213,18sleep20-21swapper/222:13:592
1581125216,19sleep30-21swapper/320:24:133
887125117,17sleep20-21swapper/221:44:202
887125117,17sleep20-21swapper/221:44:202
2522425113,23sleep325221-21apt-get20:54:123
1455325116,21sleep114586-21systemd-run22:03:571
68525015,20sleep1821-21awk19:34:341
292122507,31sleep20-21swapper/200:32:192
267052506,32sleep20-21swapper/222:39:542
220232509,27sleep122052-21ntp_states00:09:251
1457425014,21sleep114633-21memory20:19:261
1457425014,21sleep114633-21memory20:19:261
314024914,18sleep10-21swapper/123:09:281
1804924914,20sleep118055-21diskstats22:14:141
100424917,14sleep20-21swapper/221:19:422
192072486,9sleep20-21swapper/222:18:352
2216524713,16sleep30-21swapper/320:44:193
250272458,18sleep10-21swapper/100:19:471
2252824511,17sleep10-21swapper/119:04:181
2252824511,17sleep10-21swapper/119:04:181
223452458,15sleep20-21swapper/222:28:562
2781024410,20sleep327889-21sendmail_mailtr22:44:283
2273524417,14sleep10-21swapper/122:29:201
2159924310,16sleep20-21swapper/200:09:132
770424212,14sleep30-21swapper/323:24:203
770424212,14sleep30-21swapper/323:24:203
95752406,17sleep10-21swapper/120:04:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional