You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 07:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri May 09, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27541248646,424sleep127542-21ntp_states00:18:151
12257241955,307sleep0370-21snmpd23:28:540
15867236110,312sleep015908-21ntpq21:58:120
25332235612,328sleep025426-21fschecks_time19:03:080
8720230625,271sleep0370-21snmpd21:34:380
8720230625,271sleep0370-21snmpd21:34:380
19505226726,226sleep019656-21cat20:27:500
1430022648,16sleep00-21swapper/021:53:150
5459224345,20sleep00-21swapper/019:43:020
2499224247,172sleep00-21swapper/019:33:130
2499224247,172sleep00-21swapper/019:33:130
15017223725,197sleep00-21swapper/020:12:590
28378223539,139sleep00-21swapper/022:38:180
18689222957,120sleep018797-21fschecks_time22:08:050
18689222957,120sleep018797-21fschecks_time22:08:050
10966221830,150sleep011009-21idleruntime21:43:060
21668219927,137sleep021711-21sshd20:33:150
1153621946,156sleep0370-21snmpd20:01:360
10561219312,136sleep010687-21latency_hist21:42:490
6911219128,24sleep00-21swapper/021:28:200
19394219012,159sleep00-21swapper/023:53:060
15587218721,155sleep00-21swapper/023:42:030
10412218324,139sleep210561-21unixbench_multi23:23:192
31197218055,92sleep031244-21ntpq21:03:120
24356218043,97sleep00-21swapper/020:43:040
29368217145,20sleep029386-21fschecks_count20:58:070
6395217018,135sleep16357-21latency_hist23:12:481
6395217018,135sleep16357-21latency_hist23:12:481
5183216718,37sleep05197-21fschecks_count23:08:100
5183216718,37sleep05197-21fschecks_count23:08:100
20944216617,135sleep120986-21fschecks_count23:58:081
516321639,14sleep20-21swapper/223:08:082
516321639,14sleep20-21swapper/223:08:082
28366216327,15sleep00-21swapper/020:53:260
26171216011,110sleep026310-21latency_hist22:32:490
682421596,13sleep00-21swapper/023:13:110
682421596,13sleep00-21swapper/023:13:110
13495215516,98sleep10-21swapper/120:07:591
13057215328,110sleep213189-21sed21:48:212
1026521538,25sleep00-21swapper/023:23:170
30365215242,72sleep00-21swapper/019:18:140
12071215118,120sleep112132-21unin-run21:47:471
24023215017,23sleep00-21swapper/018:58:190
24023215017,23sleep00-21swapper/018:58:190
23023214728,103sleep223045-21latencyplot18:57:482
7715214328,91sleep37860-21sed19:48:183
10982214319,101sleep111010-21ls23:27:461
25254214220,106sleep225263-21vmstat20:43:232
1282821427,12sleep30-21swapper/320:03:213
20888213630,81sleep221038-21grep22:13:202
20888213630,81sleep221038-21grep22:13:202
4553213374,44sleep04652-21timerandwakeup19:38:420
10341213220,92sleep19856-21apt-get19:57:581
32722213015,97sleep032764-21munin-run19:27:470
22157213019,12sleep00-21swapper/022:18:150
22157213019,12sleep00-21swapper/022:18:150
21220212427,87sleep0370-21snmpd22:15:280
21220212427,87sleep0370-21snmpd22:15:280
1798321245,84sleep00-21swapper/022:04:050
1624521227,18sleep016253-21df23:43:080
30600212024,71sleep130733-21sed19:18:191
27687212066,37sleep00-21swapper/000:18:180
1846212013,87sleep31448-21apt-get22:57:513
949021189,62sleep00-21swapper/023:22:490
2684821185,83sleep00-21swapper/020:48:250
13763211810,95sleep00-21swapper/020:08:070
12575211610,53sleep00-21swapper/020:03:180
5236211518,77sleep35387-21latency_hist19:42:493
1256211512,62sleep00-21swapper/021:12:490
26024211417,76sleep326120-21fschecks_time20:48:083
3553211214,73sleep30-21swapper/321:18:173
8905211017,64sleep29048-21grep23:18:222
29763211017,77sleep329479-21/usr/sbin/munin22:43:103
2672521105,10sleep00-21swapper/000:16:570
2672521105,10sleep00-21swapper/000:16:570
149321106,29sleep00-21swapper/019:28:200
28630210819,17sleep00-21swapper/000:23:020
3981210657,34sleep00-21swapper/023:03:180
2971621059,81sleep029872-21date19:17:500
9233210420,27sleep10-21swapper/121:38:021
2958721049,54sleep00-21swapper/022:43:070
1789321048,25sleep00-21swapper/020:22:490
8317210351,40sleep0286-21dbus-daemon19:52:480
254621037,63sleep02616-21mailstats22:58:580
4677210246,44sleep04746-21fschecks_time21:23:070
24817210219,14sleep30-21swapper/322:28:033
12603210249,30sleep20-21swapper/223:32:482
4301210114,46sleep00-21swapper/021:22:460
2553721015,14sleep30-21swapper/300:13:043
2553721015,14sleep30-21swapper/300:13:043
2112721007,12sleep00-21swapper/023:58:110
18277210020,65sleep2227-21systemd-journal22:07:472
974029914,67sleep29802-21/usr/sbin/munin21:38:192
1776229910,71sleep10-21swapper/123:48:061
2709229724,60sleep00-21swapper/019:08:080
2211329733,31sleep10-21swapper/118:53:121
168329727,14sleep00-21swapper/021:13:130
3137829521,56sleep131446-21switchtime21:03:171
241729517,64sleep12425-21grep22:58:501
2042129545,18sleep00-21swapper/020:28:180
2649529417,61sleep226549-21fschecks_count22:33:072
2534829433,39sleep00-21swapper/000:12:490
1284429410,39sleep00-21swapper/023:33:020
2063029317,59sleep320695-21awk22:13:173
2063029317,59sleep320695-21awk22:13:173
227972926,47sleep00-21swapper/020:37:590
1820729115,16sleep20-21swapper/220:23:032
3181128821,44sleep331953-21unixbench_singl22:48:183
2860528814,35sleep10-21swapper/100:22:591
687728613,50sleep16615-21/usr/sbin/munin23:13:121
687728613,50sleep16615-21/usr/sbin/munin23:13:121
268742867,16sleep00-21swapper/022:33:170
2299728623,29sleep10-21swapper/122:22:481
2299728623,29sleep10-21swapper/122:22:481
2974728534,30sleep10-21swapper/120:58:161
239828517,17sleep20-21swapper/219:33:112
239828517,17sleep20-21swapper/219:33:112
1212228515,20sleep20-21swapper/220:03:082
320472845,69sleep00-21swapper/022:49:000
2715528413,51sleep327164-21/usr/sbin/munin00:18:053
235572846,13sleep00-21swapper/022:23:120
121928423,39sleep10-21swapper/119:28:171
2375228319,46sleep123837-21latency_hist00:07:491
1784128325,38sleep20-21swapper/220:22:472
1034428220,37sleep310345-21/usr/sbin/munin19:58:003
750328125,27sleep20-21swapper/221:32:502
55828124,28sleep20-21swapper/219:27:582
2909928121,37sleep329232-21grep19:13:213
186692816,59sleep20-21swapper/222:08:032
186692816,59sleep20-21swapper/222:08:032
866628018,47sleep38725-21sendmail_mailqu23:18:193
182372808,14sleep00-21swapper/023:48:250
228452797,15sleep022858-21ntp_states00:03:150
1837227814,49sleep218449-21grep23:48:272
2304727716,43sleep323199-21latency_hist18:57:493
3162327616,41sleep10-21swapper/122:48:161
2425827614,46sleep124289-21/usr/sbin/munin00:08:091
5482757,12sleep00-21swapper/022:53:120
2007727516,45sleep120122-21ntp_kernel_pll_20:28:111
2722527415,29sleep30-21swapper/319:08:123
2904227319,24sleep20-21swapper/220:57:492
2850027316,42sleep228569-21/usr/sbin/munin22:38:242
2847027319,23sleep10-21swapper/120:53:271
2723527331,25sleep20-21swapper/220:52:482
193527315,43sleep21448-21apt-get22:57:582
3128627217,19sleep30-21swapper/321:03:153
1572127213,24sleep10-21swapper/123:42:491
133422725,11sleep00-21swapper/021:48:590
1330727221,30sleep10-21swapper/123:33:161
2381927116,33sleep323904-21switchtime22:23:193
2390627025,28sleep324016-21ls18:58:183
2390627025,28sleep324016-21ls18:58:183
2373927018,28sleep20-21swapper/222:23:182
201327023,27sleep10-21swapper/121:13:221
1772327020,30sleep316988-21/usr/sbin/munin22:03:203
1422727018,33sleep314298-21/usr/sbin/munin21:53:143
136392708,12sleep20-21swapper/220:08:052
95362696,10sleep00-21swapper/019:53:490
36526919,30sleep3512-21cat19:27:503
3235726914,39sleep232371-21/usr/sbin/munin19:23:212
2996326923,21sleep20-21swapper/200:27:492
2986126926,26sleep30-21swapper/320:58:183
288412697,11sleep30-21swapper/300:23:103
1088226921,30sleep310962-21fschecks_time21:43:053
676026822,28sleep30-21swapper/323:13:103
676026822,28sleep30-21swapper/323:13:103
313326823,24sleep13188-21fschecks_count21:18:071
2412126821,24sleep30-21swapper/300:08:073
1269826814,21sleep10-21swapper/120:03:191
1169226811,13sleep30-21swapper/323:28:133
1037226814,30sleep210365-21apt-get19:58:032
635326715,38sleep16411-21if_eth021:28:081
3115926714,40sleep231164-21entropy22:48:052
2992126720,29sleep229988-21df19:18:042
2022926722,25sleep10-21swapper/122:13:071
2022926722,25sleep10-21swapper/122:13:071
1977426711,41sleep119813-21ps23:53:151
1668726719,26sleep10-21swapper/123:43:381
1374226714,22sleep20-21swapper/223:33:212
984126619,25sleep39901-21systemctl19:57:483
3084026620,22sleep20-21swapper/222:47:492
2754826622,22sleep20-21swapper/200:18:162
1056326612,39sleep310632-21users23:23:203
600126517,31sleep36004-21/usr/sbin/munin23:08:263
600126517,31sleep36004-21/usr/sbin/munin23:08:263
2813326525,21sleep10-21swapper/119:12:501
2719126520,29sleep127230-21http20:52:471
2256026514,34sleep222694-21sed20:37:492
2025226514,29sleep20-21swapper/220:28:162
2921126412,36sleep129261-21unin-run22:42:471
1864326418,29sleep118668-21/usr/sbin/munin22:08:021
1864326418,29sleep118668-21/usr/sbin/munin22:08:021
1863526417,27sleep10-21swapper/120:23:141
128726416,27sleep10-21swapper/121:12:501
440726317,33sleep14421-21awk19:38:391
3042526312,35sleep330517-21mailstats19:18:173
2860926310,39sleep128643-21irqstats19:13:101
2677726320,22sleep30-21swapper/322:33:123
2061826320,19sleep20-21swapper/223:57:482
1760126315,31sleep317497-40mandb23:47:523
150326316,22sleep10-21swapper/122:57:491
63326217,32sleep3559-21/usr/sbin/munin19:28:033
496026214,32sleep14994-21ntp_kernel_err21:23:121
3145026216,21sleep20-21swapper/221:03:182
2683126219,28sleep226841-21unin-run00:17:472
2683126219,28sleep226841-21unin-run00:17:472
2307726211,22sleep10-21swapper/120:38:091
2136426222,23sleep321460-21mailstats23:58:183
1663126213,34sleep216655-21ntp_states23:43:352
3268226114,33sleep332763-21sed21:08:113
3220926117,24sleep10-21swapper/119:23:201
2650626119,26sleep326542-21munin-run19:07:473
2383326119,24sleep20-21swapper/218:58:172
2383326119,24sleep20-21swapper/218:58:172
2288426116,28sleep222948-21sort20:38:052
1919126111,36sleep319193-21sshd20:25:143
1555326115,33sleep215576-21cut21:58:062
1390026115,25sleep30-21swapper/320:08:103
79752607,12sleep20-21swapper/221:33:102
79752607,12sleep20-21swapper/221:33:102
2950426014,23sleep10-21swapper/122:43:031
2396426020,26sleep223980-21awk00:07:582
2320026012,28sleep323314-21/usr/sbin/munin00:03:233
218526014,31sleep31483-21/usr/sbin/munin21:13:243
2031626013,22sleep30-21swapper/320:28:173
1907026011,26sleep30-21swapper/323:52:483
1672226021,21sleep30-21swapper/322:02:473
467125913,30sleep34675-21diskstats21:23:063
2766225920,20sleep10-21swapper/122:37:461
220025916,27sleep12209-21df_inode19:33:061
220025916,27sleep12209-21df_inode19:33:061
1926025914,22sleep30-21swapper/322:08:163
1926025914,22sleep30-21swapper/322:08:163
1572925913,29sleep115444-21/usr/sbin/munin21:58:091
1339625921,22sleep30-21swapper/323:33:183
836125816,20sleep30-21swapper/321:33:193
836125816,20sleep30-21swapper/321:33:193
570625815,27sleep25729-21irqstats19:43:082
44512585,11sleep20-21swapper/223:03:442
2152625812,24sleep10-21swapper/120:33:121
949325718,17sleep30-21swapper/321:38:123
845725717,23sleep10-21swapper/123:18:131
643525719,19sleep30-21swapper/321:28:103
2898825723,18sleep20-21swapper/219:13:202
2168125713,17sleep20-21swapper/223:58:212
1551125717,18sleep10-21swapper/120:13:171
1106625715,28sleep211112-21sed21:43:082
998925619,19sleep10-21swapper/119:57:501
789025616,19sleep20-21swapper/223:17:462
789025616,19sleep20-21swapper/223:17:462
351025616,25sleep13502-21apt-get23:03:021
257832565,41sleep20-21swapper/222:29:032
2489325614,26sleep124899-21diskstats22:28:051
2037725611,10sleep30-21swapper/323:55:193
1525125620,20sleep20-21swapper/220:13:092
903325518,22sleep20-21swapper/219:53:142
902325514,17sleep10-21swapper/121:37:501
902325514,17sleep10-21swapper/121:37:501
834725518,17sleep10-21swapper/119:52:491
448925514,26sleep34490-21sshd23:04:143
2529425513,20sleep10-21swapper/119:03:051
2195725514,28sleep222004-21cat22:18:092
2195725514,28sleep222004-21cat22:18:092
2169025518,20sleep00-21swapper/018:52:580
1504125512,26sleep315037-21apt-get20:13:023
545525412,26sleep15452-21apt-get19:42:591
3271625419,21sleep332747-21grep22:53:073
2524225414,24sleep225237-21apt-get19:03:022
2370325413,22sleep10-21swapper/118:58:121
2370325413,22sleep10-21swapper/118:58:121
2125225413,24sleep321250-21apt-get20:33:023
1297925416,21sleep10-21swapper/121:48:201
1138725412,23sleep10-21swapper/123:28:071
63932535,37sleep30-21swapper/319:43:383
2828625311,19sleep30-21swapper/320:53:253
280425316,21sleep32926-21sed19:33:203
280425316,21sleep32926-21sed19:33:203
2682325311,17sleep20-21swapper/219:08:022
247752539,17sleep30-21swapper/320:43:153
2435725319,19sleep10-21swapper/120:43:051
2406325313,19sleep30-21swapper/320:42:473
2236925319,16sleep30-21swapper/322:18:183
2236925319,16sleep30-21swapper/322:18:183
1742825314,21sleep117465-21ntp_states22:03:141
703025218,19sleep20-21swapper/219:48:012
64625210,22sleep20-21swapper/222:53:182
512625215,19sleep20-21swapper/221:23:172
435225213,17sleep20-21swapper/221:22:482
1785025218,18sleep31-21systemd20:22:483
1523225218,17sleep10-21swapper/121:57:481
150725211,24sleep21501-21apt-get21:13:072
1468725213,22sleep214754-21memory23:38:092
2934825112,23sleep229359-21df_inode20:58:052
2810925116,16sleep30-21swapper/322:38:133
2404025111,23sleep124145-21users22:23:211
143892509,16sleep30-21swapper/323:38:023
411924917,19sleep20-21swapper/219:38:322
3248424911,22sleep232493-21diskstats21:08:062
2307724917,15sleep20-21swapper/200:03:222
1437724913,21sleep20-21swapper/221:53:172
1237124912,21sleep312369-21apt-get21:48:013
2561324710,21sleep125621-21diskstats00:13:061
2561324710,21sleep125621-21diskstats00:13:061
1679624716,17sleep10-21swapper/120:18:091
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional