You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-18 - 05:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Fri Sep 18, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16582196818,1933sleep11660-21ntp_states23:48:171
16799268114,648sleep216801-21sshd00:35:412
2448026287,609sleep024520-21ntpq19:53:100
2065425298,505sleep020861-21apt-config19:42:480
9303246021,424sleep29310-21/usr/sbin/munin19:08:022
3225624600,8sleep00-21swapper/020:17:480
842233929,294sleep0928-21fschecks_time22:03:050
30403227516,243sleep230411-21/usr/sbin/munin21:53:102
24308227053,130sleep024318-21find23:17:560
2120922538,234sleep021270-21if_eth019:43:090
16613225274,86sleep016709-21fschecks_time22:53:060
5328224729,50sleep35347-21sh22:17:463
31049224312,215sleep331056-21ntp_states20:13:243
31301221622,176sleep131311-21modprobe21:57:451
1740220620,148sleep00-21swapper/023:48:200
2166122037,186sleep0367-21snmpd21:23:330
2166122037,186sleep0367-21snmpd21:23:330
1940521999,56sleep00-21swapper/023:02:470
1212219513,119sleep01223-21modprobe20:22:450
10903219430,60sleep011061-21latency_hist20:52:480
8975219217,101sleep09119-21latency_hist19:07:480
16023217327,128sleep316037-21acpi21:07:573
973216910,96sleep01044-21latency_hist23:47:470
973216910,96sleep01044-21latency_hist23:47:470
2909521648,16sleep20-21swapper/223:33:042
19919216151,96sleep020008-21cat23:03:070
515121596,141sleep00-21swapper/022:15:290
16098215827,55sleep00-21swapper/019:27:590
16098215827,55sleep00-21swapper/019:27:590
10787215646,19sleep010852-21awk22:33:160
15026215417,111sleep015172-21grep19:23:190
13888215111,121sleep013906-21apt-get00:27:560
31808215021,35sleep20-21swapper/223:42:452
934421458,55sleep09495-21latency_hist20:47:480
3017214512,115sleep03081-21latency23:53:080
29803214535,93sleep00-21swapper/021:52:480
22650214510,116sleep022695-21ntp_kernel_pll_21:28:090
12671214514,46sleep00-21swapper/020:57:570
850621448,68sleep08477-21latency_hist22:27:470
1882721445,15sleep00-21swapper/021:17:150
1882721445,15sleep00-21swapper/021:17:150
12855213959,30sleep012898-21fschecks_count19:18:060
28869213832,87sleep00-21swapper/020:08:040
696721326,108sleep00-21swapper/020:38:150
4887213014,94sleep34954-21/usr/sbin/munin23:58:203
14619212716,88sleep00-21swapper/021:03:100
6068212422,81sleep10-21swapper/122:18:121
407321240,11sleep24046-21munin-run23:57:452
21139211614,81sleep321204-21fschecks_time19:43:083
12766211531,66sleep00-21swapper/000:23:120
12644211514,82sleep211855-21/usr/sbin/munin22:38:192
11372211213,79sleep211424-21/usr/sbin/munin19:13:072
8473211114,76sleep20-21swapper/220:43:142
27430211111,86sleep0367-21snmpd21:43:150
14408211116,78sleep314487-21/usr/sbin/munin00:28:153
13791211012,34sleep113820-21/usr/sbin/munin22:43:101
9911210917,19sleep00-21swapper/019:08:170
9658210925,61sleep19713-21fschecks_count20:48:051
7147210917,80sleep07178-21munin-run00:07:450
3092121099,88sleep030965-21ntpq23:38:110
29653210929,49sleep129808-21grep23:33:151
23372210931,58sleep00-21swapper/019:48:180
24376210720,69sleep124443-21grep21:33:151
20442210724,64sleep120464-21idleruntime-cro21:22:451
20442210724,64sleep120464-21idleruntime-cro21:22:451
18075210618,75sleep318133-21sendmail_mailqu21:13:143
18075210618,75sleep318133-21sendmail_mailqu21:13:143
22542210513,73sleep00-21swapper/023:12:480
10998210512,22sleep10-21swapper/119:12:541
6230210314,73sleep16289-21latency_hist20:37:481
6230210314,73sleep16289-21latency_hist20:37:481
2009621037,24sleep0367-21snmpd21:18:200
2009621037,24sleep0367-21snmpd21:18:200
16231210322,15sleep00-21swapper/022:52:460
7836210257,29sleep07872-21ntpq00:08:140
2489210220,54sleep31997-21apt-get22:07:573
15706210270,15sleep00-21swapper/000:33:050
460421008,78sleep01-21systemd20:32:470
25691210018,55sleep30-21swapper/323:22:483
21750210018,66sleep321832-21switchtime23:08:153
2676729918,28sleep00-21swapper/020:00:210
3183129822,60sleep00-21swapper/021:58:090
3030329810,18sleep00-21swapper/023:37:460
1079329814,68sleep310810-21/usr/sbin/munin00:17:583
489429721,22sleep00-21swapper/020:33:030
489429721,22sleep00-21swapper/020:33:030
2721929714,67sleep227307-21/usr/sbin/munin21:43:092
2713829719,13sleep00-21swapper/023:27:450
219129621,58sleep31997-21apt-get22:07:493
253772958,16sleep00-21swapper/021:37:580
3011529423,54sleep330196-21apt-key20:12:473
280732947,13sleep10-21swapper/123:28:191
2489229416,12sleep00-21swapper/021:35:200
433829316,65sleep14385-21grep23:58:021
129129216,58sleep21304-21fschecks_count23:48:102
90012918,15sleep00-21swapper/022:28:120
90012918,15sleep00-21swapper/022:28:120
667829110,60sleep20-21swapper/200:03:192
43372917,16sleep00-21swapper/023:58:010
3229929113,56sleep231595-21/usr/sbin/munin21:58:192
2798829112,17sleep00-21swapper/023:28:180
227202917,17sleep00-21swapper/023:12:560
101362916,12sleep00-21swapper/000:13:230
95029019,54sleep3932-21latency_hist23:47:463
95029019,54sleep3932-21latency_hist23:47:463
1644629017,55sleep316480-21/usr/sbin/munin19:29:273
1644629017,55sleep316480-21/usr/sbin/munin19:29:273
25212897,16sleep00-21swapper/022:08:060
2128928922,49sleep10-21swapper/123:08:051
1736028957,18sleep00-21swapper/021:12:470
1613028913,34sleep216136-21diskstats21:08:042
108792897,14sleep00-21swapper/019:12:530
3007228717,53sleep230111-21unin-run20:12:462
117882879,66sleep00-21swapper/000:19:040
2567228619,45sleep325729-21latency21:38:093
2157428613,58sleep221624-21awk23:08:102
1517028619,45sleep115184-21apt-get00:32:451
118962869,58sleep00-21swapper/020:53:170
515928518,50sleep25174-21uname20:33:092
515928518,50sleep25174-21uname20:33:092
175028556,17sleep01729-21apt-get20:23:060
175028556,17sleep01729-21apt-get20:23:060
932428422,39sleep39340-21latency_hist20:47:473
1527528410,61sleep32-21kthreadd19:23:583
151762830,6sleep10-21swapper/119:23:191
1401228325,29sleep20-21swapper/221:02:482
1243228321,42sleep00-21swapper/022:38:170
883428221,45sleep18841-21forks22:28:091
883428221,45sleep18841-21forks22:28:091
801028232,24sleep10-21swapper/119:03:091
2544728227,28sleep10-21swapper/119:57:471
1955228218,48sleep219585-21ntp_kernel_err21:18:102
1955228218,48sleep219585-21ntp_kernel_err21:18:102
1899328217,18sleep00-21swapper/019:37:480
181072829,58sleep10-21swapper/119:33:131
1039328214,48sleep30-21swapper/322:33:073
935228125,35sleep29359-21fschecks_count00:13:082
30628114,48sleep332374-21/usr/sbin/munin20:19:253
283428129,31sleep10-21swapper/123:53:051
1303728023,27sleep30-21swapper/319:18:093
930027916,45sleep19306-21df_inode00:13:031
876227930,24sleep10-21swapper/100:12:471
2947227914,49sleep129622-21sed20:08:171
2566127911,55sleep325682-21dpkg19:57:573
2264127919,42sleep222648-21diskstats19:48:032
1524227911,55sleep1231-21systemd-journal21:04:201
1083427926,28sleep30-21swapper/319:12:523
2705927826,29sleep20-21swapper/220:02:492
1710427815,44sleep217194-21/usr/sbin/munin22:53:162
99402779,18sleep20-21swapper/220:48:102
2282027715,27sleep10-21swapper/121:28:141
2097127717,17sleep20-21swapper/221:23:062
2097127717,17sleep20-21swapper/221:23:062
2949927611,52sleep329574-21/usr/sbin/munin23:33:133
2729327645,15sleep00-21swapper/020:03:030
250827518,26sleep30-21swapper/323:52:473
179292750,9sleep20-21swapper/221:13:082
179292750,9sleep20-21swapper/221:13:082
1572627519,34sleep215931-21gpgv19:27:492
1557327514,19sleep30-21swapper/322:48:173
128127521,24sleep20-21swapper/220:22:482
780727421,37sleep37415-21/usr/sbin/munin00:08:133
912127312,45sleep38705-21apt-get00:12:493
573227313,28sleep20-21swapper/222:18:052
3057427318,35sleep130567-21apt-get20:13:011
2470327322,26sleep20-21swapper/223:18:112
66362725,11sleep00-21swapper/022:19:200
312312727,53sleep00-21swapper/021:56:520
2253827224,25sleep30-21swapper/319:47:563
1786927213,26sleep10-21swapper/121:13:081
1786927213,26sleep10-21swapper/121:13:081
880027121,33sleep38823-21sh22:28:073
880027121,33sleep38823-21sh22:28:073
497727115,32sleep35021-21fschecks_count20:33:063
497727115,32sleep35021-21fschecks_count20:33:063
3193327115,43sleep232036-21date20:17:472
2645127124,28sleep20-21swapper/223:23:152
2091627113,21sleep10-21swapper/121:23:051
2091627113,21sleep10-21swapper/121:23:051
1468327111,46sleep014686-21cron22:47:450
815327024,19sleep20-21swapper/219:03:142
2906527023,22sleep30-21swapper/321:48:153
2489327014,37sleep324953-21wc23:18:163
2090927018,39sleep220932-21awk23:07:452
846226915,41sleep18472-21ntp_states20:43:131
3188926925,24sleep10-21swapper/121:58:101
2704626915,24sleep30-21swapper/321:43:063
1283026912,38sleep30-21swapper/300:23:153
540926817,32sleep10-21swapper/122:17:481
2319626822,23sleep30-21swapper/323:13:113
1741526819,36sleep117487-21cat19:32:481
1741526819,36sleep117487-21cat19:32:481
1387326812,12sleep20-21swapper/219:20:462
78712676,14sleep30-21swapper/322:23:153
2300026723,26sleep323143-21sed21:28:163
1961126715,36sleep119659-21ntpq19:38:131
1777626723,28sleep317801-21awk22:57:453
1176626719,25sleep30-21swapper/320:53:163
1816526615,35sleep118171-21df_inode22:58:041
3227126519,23sleep20-21swapper/223:43:052
3227126519,23sleep20-21swapper/223:43:052
121992650,9sleep10-21swapper/100:22:471
381226433,14sleep10-21swapper/122:12:471
36852647,12sleep30-21swapper/320:28:303
1578526420,19sleep30-21swapper/321:07:493
1351126426,17sleep10-21swapper/119:18:191
1188126420,22sleep30-21swapper/322:38:043
389426318,22sleep10-21swapper/120:28:361
253352637,12sleep20-21swapper/219:57:202
2054726317,29sleep320649-21sed19:42:473
1400126319,24sleep314098-21/usr/sbin/munin22:43:163
1238026315,30sleep112439-21modprobe20:57:461
1092126329,14sleep20-21swapper/222:33:182
2779926213,32sleep227833-21/usr/sbin/munin23:28:122
1839426212,32sleep318460-21memory22:58:103
170326213,35sleep11227-21apt-get20:22:511
170326213,35sleep11227-21apt-get20:22:511
1382126218,23sleep20-21swapper/222:43:112
132722628,15sleep30-21swapper/320:58:163
129942627,43sleep10-21swapper/122:40:561
810726121,21sleep20-21swapper/200:08:202
48226118,22sleep30-21swapper/322:02:483
2578326120,20sleep10-21swapper/119:58:081
2408626114,32sleep30-21swapper/319:52:573
2227726119,25sleep122317-21modprobe19:47:461
215426113,29sleep32199-21/usr/sbin/munin20:23:213
215426113,29sleep32199-21/usr/sbin/munin20:23:213
1671226119,16sleep10-21swapper/122:53:071
1207526119,29sleep212082-21modprobe00:22:452
2577026014,29sleep10-21swapper/121:38:111
2401526017,28sleep124056-21systemctl23:17:451
2169326016,30sleep221814-21cat19:43:192
1645126017,25sleep116492-21ntpq21:08:121
1547726017,26sleep215572-21cut22:48:162
597725916,23sleep10-21swapper/100:03:061
36812596,11sleep30-21swapper/323:53:363
3210025919,22sleep10-21swapper/123:43:001
3210025919,22sleep10-21swapper/123:43:001
2749325911,20sleep10-21swapper/121:43:161
207732595,16sleep30-21swapper/323:06:013
2037925918,17sleep10-21swapper/123:03:161
1853725912,26sleep20-21swapper/222:58:132
1471425917,23sleep214814-21users00:28:182
65532588,12sleep20-21swapper/220:38:052
315142586,10sleep30-21swapper/323:38:593
2693725812,33sleep1367-21snmpd23:25:121
2207425818,18sleep30-21swapper/321:27:483
1765025815,26sleep317647-21apt-get19:32:583
1612625819,22sleep216093-21apt-get19:28:172
1612625819,22sleep216093-21apt-get19:28:172
1602425817,23sleep116117-21memory00:33:461
824525720,18sleep30-21swapper/319:03:153
776625721,20sleep30-21swapper/320:42:463
76862576,11sleep10-21swapper/120:42:161
325525717,17sleep20-21swapper/220:27:562
325525717,17sleep20-21swapper/220:27:562
723725614,24sleep10-21swapper/122:23:001
3248025617,21sleep10-21swapper/120:18:591
2854425615,16sleep20-21swapper/221:48:012
2540025614,24sleep225396-21apt-get21:38:002
1774325615,23sleep217834-21perl19:33:062
1580725615,22sleep10-21swapper/122:48:191
1573125616,24sleep315690-21apt-get00:33:243
1236625614,22sleep20-21swapper/200:23:012
588925515,23sleep35886-21apt-get00:03:003
398925524,17sleep20-21swapper/220:28:372
2724325518,22sleep127238-21apt-get20:03:001
2358825517,22sleep223640-21latency_hist21:32:472
997525415,22sleep310023-21ntp_states20:48:113
967625419,18sleep30-21swapper/319:08:123
845325418,23sleep28455-21cron22:27:452
600925416,16sleep30-21swapper/322:18:103
3096725412,16sleep10-21swapper/123:38:121
2516325415,22sleep20-21swapper/221:37:482
209725412,24sleep22190-21gpgv22:07:482
2049225416,16sleep30-21swapper/321:22:473
2049225416,16sleep30-21swapper/321:22:473
1111825413,24sleep211114-21apt-get20:52:592
88422538,33sleep28938-21cut22:28:102
88422538,33sleep28938-21cut22:28:102
3138025313,23sleep30-21swapper/321:57:473
2724525318,17sleep30-21swapper/320:03:023
2195425321,16sleep30-21swapper/321:27:193
200625317,23sleep12096-21cp22:07:471
1281325316,18sleep10-21swapper/100:23:131
463525219,18sleep20-21swapper/222:13:172
382925215,16sleep20-21swapper/222:12:482
2391825118,19sleep30-21swapper/321:33:053
239172517,35sleep20-21swapper/223:16:292
1948025113,22sleep219503-21irqstats19:38:102
53792506,32sleep20-21swapper/223:59:562
2442525017,17sleep10-21swapper/123:18:061
148262509,17sleep20-21swapper/221:03:162
1048825015,20sleep110492-21modprobe19:12:461
1278924915,16sleep10-21swapper/120:58:061
1085524915,19sleep10-21swapper/122:33:171
2885924711,22sleep128469-21/usr/sbin/munin21:48:091
229632479,17sleep10-21swapper/119:48:111
274292377,14sleep30-21swapper/323:28:013
257342367,14sleep20-21swapper/219:58:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional