You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 12:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Nov 21, 2025 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27860263536,565sleep327892-21unin-run19:18:433
30053259126,527sleep330089-21ntpq19:24:093
1623252414,495sleep21630-21ntpq21:19:102
9405247433,357sleep39408-21cron20:03:423
13712242913,398sleep313818-21timerwakeupswit20:14:173
13712242913,398sleep313818-21timerwakeupswit20:14:173
22781242416,392sleep322920-21latency_hist19:03:453
18524241913,382sleep024582-1kworker/0:0H22:13:560
1947123430,93sleep319478-21unixbench_singl22:14:553
21116234019,264sleep321134-21uname20:39:093
20870231819,284sleep320931-21wc00:04:213
2470230913,281sleep02480-21unin-run21:23:440
14990230220,160sleep30-21swapper/320:19:093
14990230220,160sleep30-21swapper/320:19:093
2980227268,22sleep30-21swapper/323:08:443
11429226623,206sleep311469-21ntpq23:34:083
24961225720,218sleep025102-21head20:49:400
25895225422,214sleep125898-21/usr/sbin/munin20:54:091
876224514,212sleep3879-21cron21:18:423
12677223449,101sleep312765-21fschecks_time23:39:053
5607223372,121sleep35759-21latency_hist21:33:453
28121222322,184sleep0362-21snmpd00:28:420
28121222322,184sleep0362-21snmpd00:28:420
4568221796,42sleep34672-21latency_hist23:13:453
17992221285,106sleep31-21systemd22:13:433
5379220733,131sleep35406-21awk19:49:083
2287322069,185sleep1362-21snmpd00:09:361
2287322069,185sleep1362-21snmpd00:09:361
19384220311,179sleep019395-21awk20:34:050
9551220069,32sleep30-21swapper/321:44:153
22058219918,54sleep30-21swapper/300:09:103
22058219918,54sleep30-21swapper/300:09:103
8730219215,163sleep18828-21cat19:59:201
2694021879,163sleep127015-21find22:39:141
17352217634,101sleep317441-21grep23:54:043
18903217218,35sleep30-21swapper/323:59:003
18903217218,35sleep30-21swapper/323:59:003
15025216921,128sleep00-21swapper/020:19:100
15025216921,128sleep00-21swapper/020:19:100
12112216712,144sleep311462-21/usr/sbin/munin20:09:323
3591216516,135sleep03618-21awk23:09:090
6712216013,130sleep36787-21fschecks_time19:54:053
29479215912,129sleep029618-21sed19:23:450
14703215946,19sleep30-21swapper/323:44:143
32572215735,85sleep30-21swapper/319:33:463
452215519,119sleep1522-21tune2fs19:34:001
8132214933,21sleep30-21swapper/321:39:183
25682214938,97sleep325718-21ntpq00:19:083
24790214768,39sleep30-21swapper/322:33:593
1671214417,47sleep21736-21ls23:03:582
19948214316,100sleep20-21swapper/220:34:232
27648214113,94sleep327707-21date22:43:443
25774213910,96sleep30-21swapper/320:54:033
25115213830,87sleep30-21swapper/300:18:473
10472213610,83sleep30-21swapper/320:04:193
27062213458,40sleep327118-21if_eth000:24:043
27062213458,40sleep327118-21if_eth000:24:043
16581213137,41sleep30-21swapper/320:24:143
16581213137,41sleep30-21swapper/320:24:143
25676213020,85sleep025809-21unixbench_multi19:09:150
11438213018,28sleep30-21swapper/321:49:443
3273212635,73sleep30-21swapper/321:24:163
7612212513,93sleep37626-21apt23:23:423
25588212411,93sleep325673-21threads19:09:143
21570212426,78sleep221625-21tune2fs18:59:042
19572212255,22sleep30-21swapper/320:34:123
13386212221,17sleep30-21swapper/321:58:453
551121199,88sleep00-21swapper/019:49:140
2149521175,99sleep015521-1kworker/0:1H00:06:570
19799211717,86sleep00-21swapper/000:00:060
19799211717,86sleep00-21swapper/000:00:060
10292211416,77sleep210358-21/usr/sbin/munin23:29:192
8410211316,80sleep20-21swapper/223:24:142
17469211010,26sleep017479-21munin-run20:28:440
17469211010,26sleep017479-21munin-run20:28:440
24161210913,46sleep30-21swapper/320:48:543
18980210921,64sleep218997-21modprobe20:33:422
23503210829,38sleep30-21swapper/319:04:133
15766210716,73sleep015807-21fschecks_count23:49:040
13859210712,77sleep213893-21/usr/sbin/munin21:59:042
13859210712,77sleep213893-21/usr/sbin/munin21:59:042
659210618,62sleep00-21swapper/022:59:160
27327210616,72sleep30-21swapper/320:59:003
5490210530,61sleep35647-21tail23:14:173
4190210521,62sleep24334-21grep19:44:152
29442210421,18sleep30-21swapper/322:49:013
23732210423,62sleep223227-21apt-get00:13:532
15780210316,71sleep015857-21threads22:04:190
15780210316,71sleep015857-21threads22:04:190
15687210310,80sleep115689-21sshd20:21:171
15687210310,80sleep115689-21sshd20:21:171
7155210215,69sleep27271-21latency_hist21:38:452
6066210217,70sleep06074-21modprobe23:18:430
18626210217,64sleep018779-21latency_hist23:58:450
29711210117,69sleep229742-21latency22:49:082
8173210010,52sleep38204-21grep23:24:073
15583210026,36sleep30-21swapper/322:04:143
15583210026,36sleep30-21swapper/322:04:143
14244210030,32sleep314374-21sort21:59:133
14244210030,32sleep314374-21sort21:59:133
2662629925,55sleep00-21swapper/000:23:450
987129820,35sleep30-21swapper/323:29:123
1413129819,35sleep00-21swapper/023:43:590
1266729813,13sleep20-21swapper/223:39:042
1615029719,63sleep116167-21/usr/sbin/munin20:24:001
1615029719,63sleep116167-21/usr/sbin/munin20:24:001
627129624,34sleep10-21swapper/119:53:431
2797629613,40sleep30-21swapper/322:44:043
253642967,12sleep10-21swapper/119:09:081
2286229613,69sleep122869-21ntpq20:44:111
2378629513,67sleep223884-21date20:48:452
1845829512,68sleep318607-21grep20:29:153
364829426,35sleep30-21swapper/319:44:043
3098929410,38sleep30-21swapper/322:53:543
3018929411,37sleep30-21swapper/321:08:453
50192935,12sleep10-21swapper/123:14:071
1325629312,70sleep013309-21if_err_eth020:14:060
1325629312,70sleep013309-21if_err_eth020:14:060
1084529317,56sleep024582-1kworker/0:0H23:33:450
641829228,26sleep36515-21cat21:34:133
2848229223,33sleep30-21swapper/300:29:013
870429110,66sleep18798-21latency_hist21:43:441
2813429112,39sleep30-21swapper/319:19:003
19312919,69sleep01946-21kernelversion23:04:050
63192900,8sleep20-21swapper/219:53:432
96928912,62sleep01093-21timerandwakeup19:34:110
2857328919,42sleep028729-21grep22:44:180
163128910,43sleep31669-21grep21:19:113
3187628817,20sleep10-21swapper/119:29:151
2609128816,57sleep00-21swapper/022:38:440
1548628813,62sleep215558-21latency_hist23:48:452
2906628719,44sleep228860-21/usr/sbin/munin21:04:032
1687028711,59sleep116936-21fw_packets22:09:051
3180228611,26sleep00-21swapper/019:29:140
2981128619,28sleep00-21swapper/022:49:100
2270928614,19sleep10-21swapper/119:03:431
169792868,40sleep312-21rcu_sched23:53:433
640528523,33sleep20-21swapper/223:19:012
3068028510,35sleep30-21swapper/321:09:053
3169528417,52sleep231718-21pluginstate19:29:122
2943528413,54sleep029506-21wc21:04:140
1492828419,50sleep114957-21sed22:03:441
1492828419,50sleep114957-21sed22:03:441
776428321,29sleep20-21swapper/221:39:092
2363728311,58sleep123716-21find19:04:151
2320828314,53sleep023285-21/usr/sbin/munin00:13:430
2320828314,53sleep023285-21/usr/sbin/munin00:13:430
19032837,34sleep30-21swapper/319:38:553
14132839,34sleep30-21swapper/323:03:433
2454728223,37sleep124597-21latency_hist22:33:441
2279828228,37sleep30-21swapper/320:44:093
89882816,21sleep10-21swapper/121:44:011
232882817,62sleep323187-21/usr/sbin/munin22:29:023
2297228120,41sleep223062-21sort22:28:452
175428125,29sleep10-21swapper/123:04:021
1680028112,50sleep20-21swapper/222:09:022
894628021,30sleep00-21swapper/021:44:000
2085028010,47sleep20-21swapper/200:04:202
2055128021,44sleep30-21swapper/322:19:123
1966228018,45sleep119724-21/usr/sbin/munin23:59:151
1966228018,45sleep119724-21/usr/sbin/munin23:59:151
1861228015,52sleep018625-21vmstat20:29:160
1648628018,26sleep10-21swapper/122:08:441
1648628018,26sleep10-21swapper/122:08:441
1027328012,52sleep110356-21sed20:04:171
4292798,33sleep30-21swapper/319:33:583
2857627913,54sleep00-21swapper/021:03:370
1145227914,46sleep110971-21apt-get20:08:531
3170427818,44sleep031753-21modprobe21:13:430
1275327823,28sleep20-21swapper/220:13:432
220742767,36sleep30-21swapper/322:24:093
1654327622,33sleep20-21swapper/220:24:132
1654327622,33sleep20-21swapper/220:24:132
1437727616,32sleep00-21swapper/021:59:140
1437727616,32sleep00-21swapper/021:59:140
1315827613,31sleep20-21swapper/220:14:032
1315827613,31sleep20-21swapper/220:14:032
978527520,26sleep20-21swapper/220:04:062
945627518,44sleep19487-21cpu23:29:021
2888527516,44sleep128908-21cpu21:03:581
2634627515,44sleep226288-21latency_hist19:13:462
2838727418,30sleep10-21swapper/100:28:551
2376227415,38sleep023827-21cpuspeed00:14:000
199427420,27sleep20-21swapper/219:39:022
2599427320,32sleep226134-21sed00:19:152
58527221,32sleep20-21swapper/222:59:142
38927212,44sleep1467-21sed22:59:091
3202327117,38sleep232049-21cpu21:14:012
1693827123,30sleep216963-21munin-run23:53:422
1006827122,29sleep010142-21timerandwakeup23:29:170
2912927011,41sleep229266-21/usr/sbin/munin00:29:162
281322707,47sleep10-21swapper/119:18:581
1283427018,24sleep00-21swapper/020:13:450
1933726916,27sleep20-21swapper/222:14:552
882926814,25sleep20-21swapper/219:59:212
781126812,45sleep0362-21snmpd19:58:060
2919026820,21sleep20-21swapper/222:48:442
2443026822,23sleep10-21swapper/120:49:281
2429726817,24sleep00-21swapper/019:08:430
2165226811,12sleep10-21swapper/122:23:591
1672826722,26sleep00-21swapper/022:09:000
1564826719,28sleep20-21swapper/222:04:172
1564826719,28sleep20-21swapper/222:04:172
1333526722,21sleep10-21swapper/121:58:431
1101326720,21sleep00-21swapper/020:08:440
1086126719,18sleep00-21swapper/021:49:090
3103026614,29sleep00-21swapper/019:28:460
2337226620,21sleep20-21swapper/200:13:452
2337226620,21sleep20-21swapper/200:13:452
2292026614,31sleep022947-21munin-run22:28:430
2266226618,22sleep00-21swapper/020:44:060
88126520,26sleep00-21swapper/021:18:430
630126518,20sleep00-21swapper/021:34:100
3055326513,26sleep10-21swapper/121:09:031
2730026518,29sleep227318-21/usr/sbin/munin20:58:542
2667826517,32sleep226779-21fschecks_time19:14:022
2019126513,21sleep00-21swapper/022:19:050
1070126513,31sleep10-21swapper/121:49:061
94882646,12sleep20-21swapper/221:44:142
2789526414,34sleep127983-21latency_hist19:18:451
278292646,13sleep10-21swapper/100:24:591
278292646,13sleep10-21swapper/100:24:591
3178226321,21sleep20-21swapper/221:13:452
2864926312,34sleep128796-21sed21:03:461
2698326312,35sleep227061-21fschecks_time00:24:032
2698326312,35sleep227061-21fschecks_time00:24:032
2610126310,23sleep126200-21latency_hist22:38:451
2539826317,27sleep10-21swapper/100:19:011
2491226315,29sleep224975-21fschecks_time22:34:062
2304026318,23sleep20-21swapper/219:04:002
2249826216,24sleep00-21swapper/019:01:040
1277526216,20sleep00-21swapper/023:39:060
43932618,40sleep00-21swapper/019:44:330
1753626117,20sleep20-21swapper/223:54:072
72526020,21sleep20-21swapper/219:34:052
630826012,19sleep10-21swapper/121:34:111
2994126015,29sleep130020-21awk22:49:151
2349826014,18sleep00-21swapper/022:29:070
1854226018,24sleep10-21swapper/122:13:571
507825917,28sleep20-21swapper/223:14:082
370925915,30sleep13785-21tr23:09:151
288222596,40sleep10-21swapper/122:44:351
2759825916,27sleep227647-21unin-run22:43:432
269625910,21sleep10-21swapper/121:23:571
2622325918,19sleep00-21swapper/020:54:160
236952599,25sleep20-21swapper/222:29:132
2130525915,20sleep00-21swapper/020:39:140
1355825920,22sleep10-21swapper/120:14:151
1355825920,22sleep10-21swapper/120:14:151
3029625815,21sleep10-21swapper/119:24:151
2029225815,20sleep20-21swapper/200:03:482
1207325814,26sleep212140-21df21:54:032
704625713,19sleep20-21swapper/219:54:122
59725712,30sleep1601-21apt-config21:15:331
306952575,12sleep20-21swapper/219:25:332
2624225714,24sleep20-21swapper/220:54:182
1103125715,25sleep111025-21apt-get23:33:581
868525611,30sleep08703-21munin-run21:43:430
293572564,13sleep00-21swapper/000:29:430
2684525617,21sleep00-21swapper/022:39:130
231025618,18sleep00-21swapper/019:39:080
2084625614,26sleep220842-21apt-get20:38:592
1935525614,24sleep219424-21sensors_temp23:59:112
1749925615,22sleep10-21swapper/123:54:061
1214925620,19sleep00-21swapper/021:54:050
456925513,24sleep24636-21memory21:29:072
3108625514,25sleep131095-21df_inode22:54:001
941425415,21sleep00-21swapper/020:03:430
870725414,24sleep18805-21users23:24:171
391325314,23sleep13919-21ntp_states19:44:101
217525315,19sleep10-21swapper/119:39:051
2084325313,23sleep120849-21ntp_states00:04:191
1057125313,25sleep210577-21diskstats21:49:022
915325217,21sleep00-21swapper/023:28:360
332725214,17sleep00-21swapper/021:24:170
321772529,32sleep2362-21snmpd22:57:202
3138425215,17sleep00-21swapper/022:54:070
269862529,18sleep00-21swapper/019:14:080
2375825212,24sleep123755-21apt-get00:13:571
2165625212,22sleep121748-21users20:39:191
1796525211,25sleep117988-21irqstats20:29:051
174925213,16sleep10-21swapper/121:19:131
1649425213,23sleep116580-21users23:49:171
701325120,17sleep00-21swapper/023:19:150
659125112,23sleep16585-21apt-get19:53:591
52242516,34sleep00-21swapper/021:29:500
409425115,18sleep10-21swapper/121:28:461
211132515,33sleep10-21swapper/122:20:361
203192516,32sleep10-21swapper/120:35:431
271825011,23sleep22715-21apt-get21:23:582
645224912,15sleep10-21swapper/123:19:041
34792498,16sleep20-21swapper/223:09:072
1181124911,17sleep20-21swapper/223:34:162
2017824811,21sleep220184-21df_inode22:19:032
1462324811,17sleep20-21swapper/220:19:002
1462324811,17sleep20-21swapper/220:19:002
1544024714,21sleep115414-21latency_hist23:48:441
246682449,20sleep224675-21ntp_states20:49:342
2939224010,15sleep20-21swapper/219:23:432
134952409,13sleep10-21swapper/123:39:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional