You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 12:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Tue Mar 24, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23935293788,9353sleep0191-21jbd2/mmcblk2p2-23:57:370
278282664120,6609sleep227765-21kworker/2:122:27:012
11056267270,204sleep011180-21latency_hist23:17:390
20702242542,326sleep020726-21awk20:22:360
20702242542,326sleep020726-21awk20:22:360
16471241519,246sleep016501-21fschecks_time20:08:270
1320240216,373sleep21410-21perl21:02:572
31528239613,368sleep231565-21ntpq19:13:012
183512381179,188sleep00-21swapper/023:38:260
306382371161,29sleep00-21swapper/020:52:570
306382371161,29sleep00-21swapper/020:52:570
3313229811,274sleep13373-21awk21:08:091
14325226526,226sleep014361-21ntpq21:43:050
601922616,218sleep04861-1kworker/0:1H22:58:410
4976225770,173sleep05084-21cat21:13:080
198592253126,21sleep00-21swapper/022:02:350
19391223122,189sleep119480-21fschecks_time23:42:561
20355222719,191sleep219873-21apt-get22:02:452
25893222232,32sleep00-21swapper/020:37:470
25893222232,32sleep00-21swapper/020:37:470
24827222046,51sleep00-21swapper/022:17:380
27285220814,179sleep027414-21timerwakeupswit22:23:070
2748220612,141sleep02852-21fschecks_time00:32:580
21709219555,21sleep00-21swapper/023:48:160
13014219511,29sleep00-21swapper/019:58:010
29432219153,97sleep029455-21munin-run22:32:360
29432219153,97sleep029455-21munin-run22:32:360
10558218752,24sleep00-21swapper/021:32:510
32506218619,102sleep00-21swapper/020:58:020
2039721856,171sleep310481-1kworker/3:1H22:02:593
8242217610,127sleep08276-21cut23:07:540
150521739,127sleep01548-21ntpq00:28:010
395721729,143sleep04028-21switchtime19:28:050
1622721720,6sleep20-21swapper/220:07:432
3262217014,58sleep03300-21processes21:08:080
18357216954,20sleep00-21swapper/021:57:390
11244216743,20sleep00-21swapper/019:52:560
27818216614,137sleep127843-21awk20:43:051
25472216643,75sleep025519-21ntpq22:18:020
32095215729,22sleep00-21swapper/000:22:560
2256721559,124sleep00-21swapper/022:08:090
2745021539,39sleep00-21swapper/020:42:560
31767215210,93sleep00-21swapper/019:13:070
13730215214,115sleep00-21swapper/023:23:080
7817214910,127sleep07893-21sed19:42:380
8673214710,16sleep00-21swapper/019:43:110
221721475,105sleep00-21swapper/021:04:080
31058214412,116sleep231139-21threads00:18:062
31058214412,116sleep231139-21threads00:18:062
28999214411,24sleep00-21swapper/020:47:520
591121419,120sleep05913-21apt-get21:17:490
29694214016,112sleep329710-21awk22:32:473
23422213717,103sleep323548-21sed20:28:083
23422213717,103sleep323548-21sed20:28:083
96962136102,17sleep00-21swapper/019:47:550
15840212910,100sleep015861-21munin-run20:07:380
13633212932,64sleep013662-21munin-run21:42:360
13633212932,64sleep013662-21munin-run21:42:360
4740212811,100sleep0281-21dbus-daemon22:57:360
27835212818,92sleep327893-21awk19:02:583
22553212716,88sleep222702-21latency_hist20:27:392
753321269,79sleep00-21swapper/021:22:530
31161212316,31sleep10-21swapper/119:12:521
17813212315,26sleep10-21swapper/121:53:071
25313212146,65sleep013441-21snmpd00:00:270
2324221198,70sleep00-21swapper/020:28:050
2324221198,70sleep00-21swapper/020:28:050
4169211622,24sleep10-21swapper/122:53:041
25043211617,28sleep225062-21apt-get22:17:482
21573211620,80sleep221638-21sendmail_mailqu23:48:142
11692211413,86sleep311693-21sshd21:36:013
29722211235,61sleep00-21swapper/022:32:520
29720211210,67sleep029811-21cut19:08:030
29720211210,67sleep029811-21cut19:08:030
954521109,16sleep00-21swapper/021:28:060
20555211042,21sleep00-21swapper/023:47:350
1629121108,16sleep00-21swapper/021:48:100
28886210942,42sleep228924-21munin-run19:07:362
2038021097,15sleep00-21swapper/022:02:470
2207210740,45sleep00-21swapper/022:47:590
1470210615,71sleep31503-21/usr/sbin/munin00:28:003
30142210436,49sleep00-21swapper/000:17:350
2650210318,67sleep12797-21grep22:48:081
24403210315,67sleep224301-21/usr/sbin/munin20:32:562
24403210315,67sleep224301-21/usr/sbin/munin20:32:562
6498210222,59sleep26573-21grep21:18:112
21200210218,65sleep320715-21apt-get20:22:473
20218210213,11sleep20-21swapper/220:18:102
20218210213,11sleep20-21swapper/220:18:102
15180210022,62sleep315200-21munin-run21:47:363
308302998,38sleep00-21swapper/000:18:000
308302998,38sleep00-21swapper/000:18:000
257752998,33sleep00-21swapper/000:02:530
2402129915,68sleep124093-21sensors22:13:101
2248929932,56sleep022531-21systemd-run20:27:370
1732029920,58sleep017355-21apt-key23:37:380
2816229821,44sleep10-21swapper/122:27:481
2816229821,44sleep10-21swapper/122:27:481
1719129828,49sleep217216-21munin-run23:37:362
1451529816,62sleep314008-21apt-get20:02:483
68322978,34sleep00-21swapper/023:03:010
68322978,34sleep00-21swapper/023:03:010
1634929628,42sleep116302-21apt-get20:08:191
3169529514,24sleep30-21swapper/319:13:063
309629537,19sleep01-21systemd19:27:370
2181829519,57sleep10-21swapper/123:48:171
347529424,48sleep33620-21sed21:08:113
2921029427,40sleep229203-21apt-get19:07:482
2921029427,40sleep229203-21apt-get19:07:482
164629420,42sleep10-21swapper/122:47:351
2895829334,46sleep029063-21latency_hist19:07:380
2599129327,49sleep326082-21uniq20:37:543
2599129327,49sleep326082-21uniq20:37:543
13482916,30sleep00-21swapper/022:43:470
1912828917,49sleep119151-21idleruntime-cro20:17:361
1507228945,18sleep00-21swapper/023:28:090
125412897,62sleep30-21swapper/321:39:003
125412897,62sleep30-21swapper/321:39:003
3076028815,57sleep230811-21cat20:52:592
3076028815,57sleep230811-21cat20:52:592
253452885,30sleep00-21swapper/020:34:360
253452885,30sleep00-21swapper/020:34:360
567328716,17sleep10-21swapper/121:17:371
2451828712,61sleep124573-21if_eth020:32:591
2451828712,61sleep124573-21if_eth020:32:591
2323428718,53sleep223308-21/usr/sbin/munin23:53:062
1283828714,56sleep312882-21cpu23:22:513
1255028712,30sleep012568-21awk23:22:350
27662859,29sleep20-21swapper/221:07:492
159728526,28sleep20-21swapper/219:22:382
111428537,12sleep00-21swapper/019:18:080
2771228440,24sleep10-21swapper/119:02:561
171428412,54sleep31775-21sendmail_mailst21:03:063
1097428426,37sleep110984-21modprobe23:17:361
2330928315,49sleep323440-21/usr/sbin/munin23:53:083
2738628234,14sleep00-21swapper/000:07:540
2738628234,14sleep00-21swapper/000:07:540
776528126,28sleep20-21swapper/219:42:362
2567228013,51sleep113441-21snmpd22:18:071
2802327924,28sleep30-21swapper/320:43:113
320627814,14sleep20-21swapper/219:27:392
2170027811,52sleep121725-21proc_pri20:23:101
493327720,42sleep34981-21awk19:32:533
872027622,33sleep28735-21apt21:27:362
386927621,18sleep30-21swapper/319:28:043
229727613,33sleep30-21swapper/319:23:043
1456027616,41sleep114513-21apt-get20:03:181
2857627524,35sleep1222-21systemd-journal00:12:361
2857627524,35sleep1222-21systemd-journal00:12:361
2397727521,22sleep10-21swapper/123:57:381
469527423,34sleep14823-21ls19:32:381
3260927414,15sleep10-21swapper/122:42:381
320092747,14sleep30-21swapper/300:22:513
3180927417,39sleep331875-21modprobe20:57:363
3180927417,39sleep331875-21modprobe20:57:363
2798527429,22sleep20-21swapper/200:08:082
2798527429,22sleep20-21swapper/200:08:082
2040127422,31sleep120376-21apt-get22:03:011
1994327421,26sleep20-21swapper/222:02:372
1898527419,43sleep00-21swapper/020:15:460
186152740,6sleep30-21swapper/320:13:093
2447727315,26sleep10-21swapper/123:57:581
212027311,46sleep22156-21/usr/sbin/munin22:47:572
1987127321,30sleep30-21swapper/323:43:063
1248927317,40sleep212566-21latency_hist19:57:382
586527219,42sleep00-21swapper/019:33:260
3235227231,26sleep232353-21sshd22:40:012
3022327219,27sleep20-21swapper/200:17:372
2985227215,37sleep129695-21/usr/sbin/munin22:32:571
274162720,7sleep10-21swapper/122:23:071
2556627219,36sleep225713-21latency_hist00:02:392
1675127215,29sleep20-21swapper/220:08:332
864527121,29sleep20-21swapper/223:08:042
58752719,14sleep10-21swapper/122:58:091
3251027119,41sleep00-21swapper/022:42:010
3176027121,36sleep331784-21proc_pri22:38:043
239372718,14sleep20-21swapper/222:13:072
1358627024,22sleep10-21swapper/121:42:011
1358627024,22sleep10-21swapper/121:42:011
1293627018,26sleep20-21swapper/219:57:592
47326916,19sleep20-21swapper/222:43:002
3101626924,19sleep10-21swapper/120:53:061
3101626924,19sleep10-21swapper/120:53:061
2784426922,28sleep20-21swapper/220:43:062
2357426927,22sleep30-21swapper/322:13:003
1759826919,24sleep30-21swapper/321:53:053
313826819,26sleep10-21swapper/100:33:071
2565826817,20sleep30-21swapper/320:37:373
2565826817,20sleep30-21swapper/320:37:373
247026812,41sleep22597-21cat19:23:072
2334826822,24sleep20-21swapper/220:28:072
2334826822,24sleep20-21swapper/220:28:072
2079426716,25sleep30-21swapper/323:47:383
2326626623,20sleep10-21swapper/120:28:061
2326626623,20sleep10-21swapper/120:28:061
1934426616,26sleep20-21swapper/223:42:532
1849626618,26sleep20-21swapper/220:13:092
107626620,25sleep10-21swapper/100:27:461
961626514,32sleep19693-21/usr/sbin/munin21:28:071
80226514,28sleep10-21swapper/119:18:051
3030626516,28sleep10-21swapper/120:52:381
2073126518,23sleep10-21swapper/120:22:371
2073126518,23sleep10-21swapper/120:22:371
190826515,22sleep10-21swapper/119:22:531
1724426519,24sleep30-21swapper/323:37:373
1555126512,32sleep115444-21/usr/sbin/munin21:47:541
86426422,25sleep10-21swapper/100:27:381
85726413,29sleep30-21swapper/322:43:093
852226417,20sleep10-21swapper/119:43:081
708726413,36sleep37148-21ps23:03:083
708726413,36sleep37148-21ps23:03:083
3034626415,30sleep230491-21latency_hist20:52:392
2661326415,29sleep326609-21apt-get22:22:503
1989226420,25sleep10-21swapper/122:02:361
1672826417,31sleep20-21swapper/221:52:362
1629126418,29sleep10-21swapper/123:32:581
123112647,45sleep30-21swapper/323:19:323
627126320,21sleep10-21swapper/119:37:391
3249726320,27sleep232583-21mailstats00:23:062
3239926314,35sleep232489-21load20:58:002
30526316,24sleep30-21swapper/320:58:093
1854726319,20sleep20-21swapper/223:38:292
941826216,24sleep19428-21modprobe23:12:351
489426225,17sleep30-21swapper/321:13:073
2957726218,30sleep329184-21/usr/sbin/munin19:07:583
2957726218,30sleep329184-21/usr/sbin/munin19:07:583
2837126211,24sleep30-21swapper/322:27:553
2837126211,24sleep30-21swapper/322:27:553
2521426214,25sleep30-21swapper/322:17:573
3109526119,25sleep20-21swapper/222:37:392
162832617,24sleep30-21swapper/320:07:493
1126926119,22sleep10-21swapper/123:17:481
778526020,23sleep3281-21dbus-daemon19:42:373
729826018,20sleep20-21swapper/223:03:112
729826018,20sleep20-21swapper/223:03:112
3096926015,22sleep10-21swapper/100:18:051
3096926015,22sleep10-21swapper/100:18:051
3043526016,27sleep30-21swapper/300:17:463
3043526016,27sleep30-21swapper/300:17:463
1309526016,18sleep10-21swapper/119:58:021
1260626017,21sleep20-21swapper/221:39:032
1260626017,21sleep20-21swapper/221:39:032
1188626013,31sleep211910-21munin-run21:37:372
1154426020,23sleep111585-21ntpq19:53:011
1138126013,25sleep20-21swapper/223:17:572
1017426018,19sleep20-21swapper/223:13:042
999525920,23sleep10-21swapper/119:48:001
613125918,25sleep16181-21if_eth021:18:021
1817125918,28sleep318208-21ntp_kernel_pll_23:38:213
1523225917,18sleep10-21swapper/121:47:381
136922599,10sleep30-21swapper/319:58:443
968325817,26sleep30-21swapper/319:47:543
52532585,10sleep20-21swapper/221:13:142
3236925816,23sleep132398-21irqstats20:57:591
2829225816,23sleep228370-21/usr/sbin/munin22:27:532
2829225816,23sleep228370-21/usr/sbin/munin22:27:532
959825716,24sleep29594-21apt-get19:47:492
2214725712,20sleep20-21swapper/222:07:582
1447325714,22sleep30-21swapper/323:27:553
1446125718,22sleep10-21swapper/123:27:531
2249325611,29sleep322562-21expr22:08:083
1182425616,19sleep30-21swapper/319:53:073
96925516,17sleep30-21swapper/319:18:073
901025517,20sleep20-21swapper/221:27:522
488525519,14sleep30-21swapper/322:57:393
2604125521,17sleep20-21swapper/200:03:002
1827625516,16sleep20-21swapper/221:57:362
1546525515,24sleep315462-21apt-get21:47:483
554125416,17sleep20-21swapper/219:33:082
505825413,24sleep25054-21apt-get22:57:502
394625417,17sleep20-21swapper/222:53:012
2936325417,23sleep30-21swapper/300:13:063
2589425414,24sleep125889-21apt-get20:37:481
2589425414,24sleep125889-21apt-get20:37:481
139512547,11sleep20-21swapper/223:25:442
115632549,36sleep10-21swapper/121:34:211
860025317,20sleep30-21swapper/319:43:103
823725316,21sleep38240-21diskstats23:07:533
637925314,23sleep36385-21ntp_states21:18:083
254772536,10sleep30-21swapper/300:02:323
248625313,24sleep12523-21munin-run21:07:361
1993425311,17sleep30-21swapper/320:18:073
1993425311,17sleep30-21swapper/320:18:073
1015625311,11sleep30-21swapper/321:31:083
970425213,22sleep39700-21apt-get23:12:493
782925214,17sleep30-21swapper/321:23:013
307925214,22sleep23105-21ntp_states00:33:042
2947825220,16sleep10-21swapper/119:07:561
2947825220,16sleep10-21swapper/119:07:561
262292528,18sleep30-21swapper/300:03:053
843025112,24sleep28469-21ntpq19:43:052
2324525117,19sleep123222-21latency_hist22:12:381
1283625111,23sleep112831-21apt-get23:22:491
110325112,15sleep20-21swapper/200:27:512
354925014,21sleep33455-21/usr/sbin/munin22:52:503
2862725015,16sleep30-21swapper/300:12:383
2862725015,16sleep30-21swapper/300:12:383
1859125012,20sleep318587-21apt-get21:57:503
1438725011,24sleep214381-21apt-get23:27:502
687324912,17sleep10-21swapper/119:37:591
193624910,22sleep31932-21apt-get22:47:493
1438324913,20sleep214389-21ntp_states21:43:072
310624810,15sleep30-21swapper/300:33:053
772824713,19sleep10-21swapper/121:22:591
707024710,23sleep17077-21ntp_states23:03:061
707024710,23sleep17077-21ntp_states23:03:061
301032478,30sleep10-21swapper/100:17:041
2617524715,18sleep10-21swapper/100:03:031
1775424711,21sleep117739-21apt-get23:37:551
2651024615,16sleep20-21swapper/220:38:052
2651024615,16sleep20-21swapper/220:38:052
2907524510,20sleep329084-21df_inode20:47:553
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional