You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 03:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Sun Dec 07, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
728527808,759sleep27321-21ntpq23:43:132
27364271119,674sleep127384-21/usr/sbin/munin19:38:171
20406264015,608sleep020399-21apt-get22:43:030
13473255674,413sleep213535-21fschecks_time00:03:082
13473255674,413sleep213535-21fschecks_time00:03:082
23013250415,476sleep223077-21cat21:08:112
11438244018,404sleep111447-21/usr/sbin/munin22:13:241
11438244018,404sleep111447-21/usr/sbin/munin22:13:241
29052243923,402sleep129055-21df21:28:051
24857240116,371sleep124858-21sshd19:30:331
9941236113,331sleep39989-40ogrotate23:52:473
9941236113,331sleep39989-40ogrotate23:52:473
1993234357,103sleep20-21swapper/220:02:492
2718523348,314sleep227170-21latency_hist21:22:482
32394230715,276sleep332459-21ls21:38:363
30239228012,91sleep20-21swapper/223:13:172
2957022678,248sleep05666-21snmpd19:45:590
16898225712,231sleep116943-21ntpq00:13:121
16898225712,231sleep116943-21ntpq00:13:121
31260225489,79sleep231274-21modprobe19:52:472
22027225344,131sleep20-21swapper/222:48:052
1058224954,111sleep2225-21systemd-journal21:41:302
28986224132,198sleep228428-21/usr/sbin/munin19:43:182
27703222863,90sleep20-21swapper/221:23:132
27703222863,90sleep20-21swapper/221:23:132
3790221916,183sleep03792-21cpu20:08:080
12104221118,176sleep212169-21grep23:58:112
12104221118,176sleep212169-21grep23:58:112
27906220118,166sleep027965-21/usr/sbin/munin21:23:190
27906220118,166sleep027965-21/usr/sbin/munin21:23:190
2253021808,156sleep222679-21sed21:07:502
25801217647,111sleep225067-21/usr/sbin/munin22:58:192
14844216526,89sleep20-21swapper/222:23:302
14844216526,89sleep20-21swapper/222:23:302
18208216318,134sleep221461-1kworker/2:1H20:53:212
5443216115,131sleep15571-21cat21:53:211
20001216152,46sleep20-21swapper/220:58:132
10612215840,21sleep20-21swapper/222:11:062
24044215616,56sleep20-21swapper/219:28:122
436421559,110sleep24451-21sed20:08:222
29559215014,17sleep00-21swapper/023:12:510
1392821506,131sleep213995-21date22:22:492
4497214824,101sleep04521-21apt21:52:470
21174214851,77sleep20-21swapper/219:18:192
15860214810,56sleep20-21swapper/219:03:122
19021214712,119sleep219071-21basename22:38:082
32681214247,56sleep2371-21latency_hist23:22:502
17829214279,49sleep20-21swapper/222:33:322
16383214122,57sleep20-21swapper/222:28:312
4530213816,81sleep24679-21grep23:33:202
27902213867,34sleep20-21swapper/223:07:482
20435213722,59sleep20-21swapper/219:17:502
20435213722,59sleep20-21swapper/219:17:502
24216213621,32sleep00-21swapper/022:53:200
1504821358,109sleep20-21swapper/200:08:092
1106521348,72sleep211084-21grep22:13:042
1106521348,72sleep211084-21grep22:13:042
6098212919,86sleep36245-21unixbench_singl20:13:203
28564212910,106sleep20-21swapper/223:08:162
21284212910,64sleep221360-21grep00:28:072
11867212826,84sleep211950-21latency20:33:142
894212614,37sleep3951-21/usr/sbin/munin19:58:103
1808621268,78sleep218150-21switchtime19:08:172
6895212511,54sleep20-21swapper/220:17:592
16747212221,12sleep20-21swapper/200:13:092
16747212221,12sleep20-21swapper/200:13:092
10010212273,34sleep010000-40mandb23:52:480
10010212273,34sleep010000-40mandb23:52:480
31133212116,46sleep20-21swapper/221:33:192
22832212120,84sleep122918-21timerandwakeup19:23:231
18658211914,89sleep018808-21latency_hist22:37:500
1689211918,54sleep21679-21apt-get21:43:042
1689211918,54sleep21679-21apt-get21:43:042
7637211719,83sleep37672-21basename22:02:473
7637211719,83sleep37672-21basename22:02:473
18085211745,16sleep20-21swapper/200:18:062
12619211720,45sleep10-21swapper/122:18:041
10494211710,55sleep210555-21grep20:28:172
13803211514,81sleep013840-21ntp_states00:03:140
13803211514,81sleep013840-21ntp_states00:03:140
3741211310,82sleep20-21swapper/221:48:172
2194211314,83sleep32255-21sendmail_mailst21:43:333
2194211314,83sleep32255-21sendmail_mailst21:43:333
10277211310,83sleep20-21swapper/223:53:022
6651211150,16sleep20-21swapper/223:42:492
10745211112,80sleep110780-21idleruntime-cro22:12:471
1007211116,77sleep31100-21/usr/sbin/munin23:23:203
5052210817,72sleep35054-21cron20:12:473
21716210847,43sleep20-21swapper/222:47:492
23704210712,76sleep10-21swapper/122:53:091
8650210520,19sleep20-21swapper/223:47:582
8650210520,19sleep20-21swapper/223:47:582
3272121048,61sleep2910-21sendmail-mta19:56:112
298421049,43sleep20-21swapper/223:28:352
18043210421,62sleep017537-21apt-get20:52:580
1264221048,13sleep00-21swapper/023:58:210
1264221048,13sleep00-21swapper/023:58:210
10728210424,56sleep110877-21unixbench_singl20:28:201
7505210312,72sleep07632-21/usr/sbin/munin23:43:190
20677210218,68sleep120740-21cpuspeed19:18:051
14671210214,73sleep014691-21apt-get20:43:010
8232210014,38sleep00-21swapper/020:22:500
25592210029,17sleep20-21swapper/221:17:482
25592210029,17sleep20-21swapper/221:17:482
92929941,18sleep21001-21mailstats23:23:182
61372998,69sleep20-21swapper/221:57:502
1517929823,57sleep20-21swapper/220:43:192
773029742,13sleep20-21swapper/222:02:502
773029742,13sleep20-21swapper/222:02:502
1271129711,71sleep312715-21entropy22:18:143
1754529610,45sleep21-21systemd20:52:482
1861329517,21sleep30-21swapper/322:37:493
1774629535,44sleep017944-21sed20:52:500
1361529532,17sleep20-21swapper/220:38:182
1400029413,22sleep00-21swapper/018:58:020
65032936,47sleep20-21swapper/220:16:102
2821029224,56sleep228247-21date19:42:502
223122927,46sleep20-21swapper/219:23:112
1159629211,67sleep111694-21cat23:57:501
2933729113,63sleep129338-21sshd23:11:141
246492908,46sleep25666-21snmpd22:55:532
854928921,13sleep20-21swapper/220:23:082
1567628820,42sleep315741-21sort19:03:093
1971928718,47sleep119775-21tune2fs20:58:081
1751728720,36sleep117543-21modprobe20:52:471
206422866,47sleep20-21swapper/200:24:402
193162866,12sleep10-21swapper/122:38:151
149992860,8sleep30-21swapper/300:08:053
914028516,49sleep39263-21timerwakeupswit20:23:213
2754528515,46sleep327628-21fschecks_time21:23:103
2754528515,46sleep327628-21fschecks_time21:23:103
253542858,61sleep20-21swapper/219:33:052
1464228517,22sleep314666-21proc_pri22:23:273
1464228517,22sleep314666-21proc_pri22:23:273
2100728418,30sleep30-21swapper/322:43:183
560828310,59sleep05639-21grep23:38:100
2895628316,50sleep328964-21/usr/sbin/munin21:27:583
1963228311,52sleep019695-21/usr/sbin/munin00:23:050
1610828315,52sleep316131-21netstat22:28:243
312928211,55sleep33142-21/usr/sbin/munin23:28:363
2353428219,47sleep023578-21latency_hist19:27:500
3263428126,28sleep10-21swapper/123:22:491
2038328119,45sleep320387-21modprobe19:17:473
2038328119,45sleep320387-21modprobe19:17:473
935428012,19sleep20-21swapper/222:07:512
2283028013,51sleep322922-21fschecks_time21:08:083
2195828022,35sleep021950-21apt-get22:48:030
1446228020,46sleep214489-21proc_pri18:58:162
2517527915,28sleep10-21swapper/122:58:061
1131327916,48sleep311040-21/usr/sbin/munin22:13:223
1131327916,48sleep311040-21/usr/sbin/munin22:13:223
738227816,30sleep30-21swapper/320:18:173
555227812,39sleep05345-21/usr/sbin/munin20:13:090
310212786,12sleep00-21swapper/021:33:160
260912787,16sleep00-21swapper/022:59:300
2500927821,34sleep125153-21grep21:13:191
2500927821,34sleep125153-21grep21:13:191
181827815,46sleep11908-21latency_hist23:27:501
1314927810,52sleep00-21swapper/020:38:040
2459627616,29sleep00-21swapper/021:13:090
2459627616,29sleep00-21swapper/021:13:090
196962760,7sleep10-21swapper/100:23:051
1883627622,36sleep118975-21grep20:53:441
22372757,17sleep22234-21apt-get20:03:052
1829027514,28sleep00-21swapper/019:08:190
978927413,43sleep39855-21memory22:08:213
808327414,47sleep18111-21cut22:03:181
2720927417,29sleep30-21swapper/321:22:503
1714227416,41sleep317263-21sed00:13:193
1714227416,41sleep317263-21sed00:13:193
157262748,50sleep10-21swapper/122:28:011
571627317,41sleep35728-21vmstat21:53:233
2642627319,34sleep10-21swapper/121:18:191
136727320,34sleep11512-21sed19:58:201
1251327320,31sleep30-21swapper/323:58:203
1251327320,31sleep30-21swapper/323:58:203
2077327216,33sleep10-21swapper/122:43:121
1496527225,26sleep30-21swapper/320:43:123
316502719,49sleep131657-21df_inode19:53:071
2016627121,28sleep320137-21latency_hist22:42:493
513227019,27sleep10-21swapper/120:12:491
1815127022,28sleep30-21swapper/319:08:183
982526917,29sleep00-21swapper/020:27:510
2976226921,25sleep10-21swapper/119:47:501
2115026914,22sleep10-21swapper/100:27:581
2023126915,22sleep00-21swapper/020:58:190
79842686,50sleep10-21swapper/123:45:421
3194926818,28sleep10-21swapper/121:37:501
194682688,46sleep019589-21timerandwakeup22:38:180
24482670,7sleep00-21swapper/020:03:140
139932677,12sleep30-21swapper/318:58:013
985726615,36sleep09888-21/usr/sbin/munin22:08:220
2871126622,26sleep30-21swapper/323:08:223
1663926613,37sleep016652-21/usr/sbin/munin22:28:330
1467626622,19sleep10-21swapper/118:58:191
536926523,18sleep00-21swapper/021:53:200
2700226522,24sleep30-21swapper/323:03:133
1778626514,35sleep117826-21ps22:33:311
554926421,22sleep30-21swapper/323:38:093
355126413,35sleep03647-21cat23:32:490
350526421,23sleep30-21swapper/321:48:103
3039326413,24sleep10-21swapper/123:13:201
2905826419,27sleep329166-21grep19:43:193
2666626416,23sleep30-21swapper/319:37:503
2321926411,29sleep00-21swapper/021:08:170
172826417,27sleep01811-21systemd-run23:27:490
264422634,49sleep00-21swapper/019:35:370
2588126316,28sleep025875-21apt-get21:18:020
1175726318,21sleep00-21swapper/020:33:120
90212625,19sleep00-21swapper/022:05:430
9002625,10sleep10-21swapper/121:39:311
3134126220,23sleep10-21swapper/119:52:501
1761726217,22sleep30-21swapper/320:52:493
1404426218,21sleep00-21swapper/022:22:510
1230826214,20sleep30-21swapper/320:33:233
37032617,12sleep10-21swapper/121:48:161
3032426114,18sleep00-21swapper/023:13:190
2990326121,29sleep20-21swapper/221:28:302
2432826117,28sleep324325-21apt-get21:13:003
2432826117,28sleep324325-21apt-get21:13:003
2414426117,28sleep024178-21/usr/sbin/munin19:28:140
2157326120,21sleep30-21swapper/300:28:123
1984226112,33sleep319898-21/usr/sbin/munin20:58:103
1701126122,23sleep00-21swapper/019:07:510
1365626115,18sleep10-21swapper/100:03:111
1365626115,18sleep10-21swapper/100:03:111
1312426113,24sleep10-21swapper/120:38:011
67242606,13sleep30-21swapper/323:42:503
584826016,22sleep10-21swapper/123:38:201
387526015,26sleep13911-21fschecks_count23:33:071
2751726010,35sleep027641-21sed19:38:200
2329326013,28sleep10-21swapper/122:52:491
364925916,19sleep10-21swapper/123:32:501
32172596,10sleep10-21swapper/120:04:171
3064425918,23sleep30-21swapper/321:33:073
3000125914,28sleep329993-21apt-get19:48:043
2423725918,19sleep30-21swapper/319:28:173
2076025918,23sleep00-21swapper/019:18:090
2744725813,19sleep30-21swapper/319:38:193
2281825814,21sleep10-21swapper/121:08:061
2270125810,34sleep322201-21/usr/sbin/munin19:23:203
1386325813,26sleep313925-21wc00:03:173
1386325813,26sleep313925-21wc00:03:173
1025925811,33sleep310288-21irqstats20:28:103
434225715,18sleep00-21swapper/023:33:180
347225715,26sleep30-21swapper/320:07:303
2252925718,18sleep30-21swapper/322:48:193
1990225716,24sleep319995-21memory00:23:123
3008825619,21sleep00-21swapper/019:48:090
1002725617,21sleep10-21swapper/123:52:491
1002725617,21sleep10-21swapper/123:52:491
226125511,26sleep02339-21sed21:43:340
226125511,26sleep02339-21sed21:43:340
1808425513,25sleep118077-21apt-get00:18:051
1802025514,21sleep10-21swapper/119:08:161
149925516,20sleep10-21swapper/121:42:501
149925516,20sleep10-21swapper/121:42:501
1234125522,19sleep10-21swapper/123:58:181
1234125522,19sleep10-21swapper/123:58:181
112332555,15sleep00-21swapper/020:32:150
1073925514,13sleep00-21swapper/023:53:210
97082546,37sleep10-21swapper/120:27:311
3209225422,16sleep30-21swapper/323:18:193
2255725415,17sleep00-21swapper/000:32:500
1921025416,17sleep00-21swapper/019:13:080
1921025416,17sleep00-21swapper/019:13:080
1835725413,25sleep318375-21irqstats20:53:343
1314725414,24sleep313144-21apt-get20:38:033
1027825415,22sleep310253-21/usr/sbin/munin23:53:033
3204225315,23sleep00-21swapper/019:53:180
3166525312,25sleep331701-21fschecks_count19:53:093
216792538,20sleep30-21swapper/321:03:193
1661525313,18sleep00-21swapper/000:13:070
1661525313,18sleep00-21swapper/000:13:070
1561925318,17sleep10-21swapper/100:08:201
815725213,23sleep38206-21irqstats22:03:203
76525212,23sleep0770-21diskstats19:58:060
544325212,24sleep15452-21diskstats20:13:061
2896625212,22sleep028977-21apt-get21:27:590
1816025213,23sleep018167-21diskstats00:18:080
1506525212,26sleep115109-21ntpq20:43:141
637025114,22sleep16372-21apt-get21:58:011
637025114,22sleep16372-21apt-get21:58:011
3139725111,16sleep00-21swapper/023:18:040
2795325113,17sleep00-21swapper/023:07:500
2688225115,21sleep126911-21irqstats23:03:111
216172518,16sleep00-21swapper/021:03:160
1580225114,18sleep10-21swapper/119:03:111
1201425113,15sleep10-21swapper/120:33:171
2586025013,22sleep125937-21mailstats19:33:201
1519525014,19sleep00-21swapper/000:08:110
1416525013,21sleep114162-21apt-get22:23:011
1416525013,21sleep114162-21apt-get22:23:011
73512499,16sleep10-21swapper/120:18:151
767424812,20sleep07688-21munin-run22:02:480
767424812,20sleep07688-21munin-run22:02:480
510024813,19sleep30-21swapper/323:37:493
2380324815,18sleep30-21swapper/322:53:113
2249924816,15sleep10-21swapper/121:07:491
1992624811,19sleep10-21swapper/119:13:211
1992624811,19sleep10-21swapper/119:13:211
3231124711,20sleep031832-21apt-get21:37:530
251362478,15sleep30-21swapper/322:58:043
1444124713,16sleep00-21swapper/022:23:200
1444124713,16sleep00-21swapper/022:23:200
1153124712,20sleep011537-21ntp_states22:13:270
1153124712,20sleep011537-21ntp_states22:13:270
286132465,29sleep10-21swapper/121:26:471
286132465,29sleep10-21swapper/121:26:471
1933824613,17sleep30-21swapper/300:22:473
300142446,15sleep30-21swapper/323:13:103
2111524111,15sleep30-21swapper/319:18:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional