You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-25 - 17:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Thu Jun 25, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9326212238,1200sleep09385-21sendmail-msp07:19:540
8576267627,632sleep08666-21cut10:40:120
3321251914,490sleep23322-21sshd08:41:012
283522444131,138sleep00-21swapper/008:19:550
31157241749,250sleep031278-21latency_hist10:09:550
14493240324,340sleep014545-21if_eth007:35:130
14493240324,340sleep014545-21if_eth007:35:130
3055323348,290sleep030598-21ntpq08:25:200
28985232321,280sleep229126-21grep10:00:282
1530229024,239sleep31663-21grep08:35:253
12486225422,222sleep0364-21snmpd10:50:260
6675224211,192sleep06802-21wc10:34:550
6675224211,192sleep06802-21wc10:34:550
7221222410,192sleep2364-21snmpd08:55:142
7221222410,192sleep2364-21snmpd08:55:142
5218220910,187sleep05260-21latency_hist08:49:550
1995122079,17sleep00-21swapper/009:34:540
1995122079,17sleep00-21swapper/009:34:540
304652206171,21sleep330467-21sshd11:50:013
304652206171,21sleep330467-21sshd11:50:013
3896220323,163sleep23932-21/usr/sbin/munin10:25:122
3896220323,163sleep23932-21/usr/sbin/munin10:25:122
350522019,178sleep3364-21snmpd08:43:243
26767220114,172sleep326773-21forks09:55:293
24939219010,161sleep024942-21cron08:09:530
24939219010,161sleep024942-21cron08:09:530
432218669,60sleep0533-21latency_hist10:14:560
16509218446,66sleep00-21swapper/009:24:570
652421815,141sleep00-21swapper/008:52:180
652421815,141sleep00-21swapper/008:52:180
4517218017,131sleep00-21swapper/007:05:090
11701217444,51sleep00-21swapper/012:31:010
19407217126,132sleep019444-21tr07:50:190
29204217037,65sleep00-21swapper/008:20:240
15540216824,126sleep115583-21systemctl07:39:541
15540216824,126sleep115583-21systemctl07:39:541
4707216532,122sleep04718-21vmstat10:25:260
4707216532,122sleep04718-21vmstat10:25:260
21856216210,138sleep021757-21/usr/sbin/munin09:40:110
2955216114,89sleep03109-21unixbench_multi10:20:250
2955216114,89sleep03109-21unixbench_multi10:20:250
6642216034,64sleep06079-21/usr/sbin/munin07:10:250
22239215827,107sleep00-21swapper/011:24:540
10013215531,58sleep00-21swapper/010:45:050
11196215013,123sleep211262-21sort07:25:092
16829214920,58sleep00-21swapper/009:25:010
1138021498,12sleep10-21swapper/109:06:281
25448214833,57sleep025460-21apt-get08:10:060
22741214714,63sleep022719-21apt-get11:25:150
19029214719,113sleep119054-21awk09:30:231
19029214719,113sleep119054-21awk09:30:231
1194521428,91sleep00-21swapper/009:10:070
2920821368,17sleep00-21swapper/011:45:140
2920821368,17sleep00-21swapper/011:45:140
1756213419,11sleep00-21swapper/008:36:060
10833213418,11sleep00-21swapper/007:24:020
3169621338,12sleep00-21swapper/008:30:090
28711213313,105sleep228814-21latency_hist11:44:552
27998213315,102sleep128051-21sort09:59:551
16347213320,17sleep30-21swapper/311:05:113
1632213111,100sleep11671-21processes12:00:211
1794121307,108sleep018008-21run-parts07:45:220
13012213011,72sleep00-21swapper/007:30:160
5978212913,96sleep00-21swapper/012:15:140
5978212913,96sleep00-21swapper/012:15:140
12635212915,98sleep212780-21unixbench_multi09:10:252
4905212815,17sleep20-21swapper/212:10:242
1295212712,54sleep00-21swapper/010:15:250
9226212625,55sleep00-21swapper/009:00:250
18196212522,83sleep318329-21unixbench_singl07:45:253
1250021259,63sleep012561-21latency_hist07:29:560
12264212218,85sleep212403-21grep10:50:242
22698212110,96sleep122213-21apt-get11:25:041
20555212110,55sleep00-21swapper/011:19:010
22322211722,76sleep122458-21latency_hist11:24:551
31668211212,84sleep331682-21acpi08:30:043
8063211117,18sleep30-21swapper/307:15:093
5922211016,78sleep16001-21sed10:30:241
5922211016,78sleep16001-21sed10:30:241
2562021109,58sleep00-21swapper/011:34:560
18625211016,79sleep318627-21munin-node09:30:053
18625211016,79sleep318627-21munin-node09:30:053
2879721098,49sleep028865-21wc10:00:260
13807210811,85sleep013933-21sed10:55:270
2776321075,90sleep00-21swapper/009:57:520
1789421078,48sleep017974-21tune2fs11:10:130
8565210614,32sleep30-21swapper/309:00:043
31599210522,64sleep131571-21apt-get10:10:211
12551210520,64sleep112633-21timerandwakeup09:10:241
66921035,68sleep00-21swapper/011:57:380
2215021036,42sleep00-21swapper/008:00:240
896121029,78sleep09012-21latency_hist12:24:550
29227210214,24sleep129313-21memory11:45:151
29227210214,24sleep129313-21memory11:45:151
5082210124,55sleep25156-21/usr/sbin/munin07:05:232
30516210020,64sleep230657-21awk10:05:252
23694210016,69sleep123720-21netstat09:45:171
23694210016,69sleep123720-21netstat09:45:171
2547829916,65sleep125460-21apt-get08:10:151
894229810,11sleep30-21swapper/310:40:193
2220029816,64sleep322225-21munin-run11:24:533
1721629819,56sleep217344-21latency_hist07:44:562
1674029814,69sleep016798-21sendmail_mailqu11:05:200
1286629814,70sleep212935-21/usr/sbin/munin12:35:232
3100229630,46sleep131086-21mailstats11:50:241
3100229630,46sleep131086-21mailstats11:50:241
1527829613,64sleep00-21swapper/011:00:230
1484129644,35sleep00-21swapper/009:19:560
2425529511,70sleep124274-21apt11:30:041
2028729514,60sleep220177-21/usr/sbin/munin09:35:122
1737429519,59sleep317454-21/usr/sbin/munin09:25:323
1713729429,24sleep017178-21systemd-run07:44:540
727429321,57sleep37292-21uname08:55:153
727429321,57sleep37292-21uname08:55:153
2748829320,59sleep327546-21cut11:40:123
1996129214,61sleep220053-21cat11:15:242
1491129213,60sleep314964-21/usr/sbin/munin11:00:143
2188829129,51sleep021976-21grep07:59:560
2188829129,51sleep021976-21grep07:59:560
205982919,62sleep00-21swapper/009:35:180
998529024,49sleep210117-21apt-key09:04:562
1094229022,44sleep311009-21wc09:06:223
280728953,19sleep00-21swapper/012:05:130
2613328925,44sleep00-21swapper/009:54:530
1012128915,19sleep30-21swapper/310:45:123
940528813,58sleep29531-21wc07:19:562
266642880,19sleep226665-21unixbench_multi11:35:262
2634028814,50sleep326507-21sort09:54:563
2509828815,50sleep325223-21latency_hist08:09:573
2509828815,50sleep325223-21latency_hist08:09:573
2200328816,54sleep222079-21latency_hist07:59:582
2200328816,54sleep222079-21latency_hist07:59:582
1507128820,32sleep20-21swapper/211:00:172
782128717,54sleep17885-21/usr/sbin/munin12:20:321
988128618,46sleep110023-21grep12:25:251
96482869,39sleep09659-21ntpq12:25:190
817128626,39sleep18197-21munin-run10:39:531
3049528622,47sleep00-21swapper/011:50:060
3049528622,47sleep00-21swapper/011:50:060
1044328622,47sleep00-21swapper/009:05:500
751728518,48sleep37586-21sed10:35:233
737428535,35sleep00-21swapper/010:35:190
577028532,31sleep15805-21munin-run07:09:541
53528514,52sleep2557-21sort08:34:552
47532857,66sleep04830-21basename12:10:220
158632859,14sleep30-21swapper/307:40:083
125728520,52sleep00-21swapper/012:00:120
297922837,15sleep30-21swapper/310:05:063
2615528321,45sleep00-21swapper/011:35:160
1043328323,42sleep210454-21ntp_kernel_pll_10:45:192
919828210,50sleep20-21swapper/212:25:052
823528227,44sleep00-21swapper/008:59:240
823528227,44sleep00-21swapper/008:59:240
762128217,38sleep27123-21apt-get12:20:052
584228224,38sleep35949-21sort07:09:563
275482828,17sleep00-21swapper/011:40:130
964728119,48sleep29648-21cpu07:20:082
30582818,52sleep00-21swapper/008:40:270
1278728115,22sleep10-21swapper/112:35:221
828128017,48sleep38372-21sed12:20:423
582928015,31sleep20-21swapper/208:50:182
582928015,31sleep20-21swapper/208:50:182
1266128019,12sleep00-21swapper/012:35:170
234122798,58sleep023426-21grep09:45:120
234122798,58sleep023426-21grep09:45:120
1817527918,28sleep10-21swapper/111:10:181
608227722,37sleep16191-21sed08:50:251
608227722,37sleep16191-21sed08:50:251
1476127711,50sleep20-21swapper/209:19:542
740727521,30sleep2278-21dbus-daemon07:14:552
247527514,47sleep22506-21cat12:04:542
1640127523,36sleep316502-21apt-config09:24:563
122727519,38sleep31290-21switchtime10:15:243
775227426,29sleep17892-21sed08:55:261
775227426,29sleep17892-21sed08:55:261
2159627417,39sleep121736-21latency_hist09:39:561
115852745,12sleep30-21swapper/310:50:093
1090027429,25sleep30-21swapper/307:24:543
746627315,35sleep37489-21munin-node07:14:563
2934527318,37sleep329380-21ntp_kernel_pll_11:45:173
2934527318,37sleep329380-21ntp_kernel_pll_11:45:173
289692735,53sleep20-21swapper/208:20:172
84727217,36sleep1886-21sh11:59:531
2779827217,40sleep127834-21ntpq11:40:181
1025727215,33sleep110360-21grep07:20:241
356927118,37sleep23644-21latency_hist10:24:562
356927118,37sleep23644-21latency_hist10:24:562
3014927118,38sleep130143-21apt-get08:25:091
283827114,24sleep10-21swapper/108:40:231
1340227113,42sleep313475-21sed10:55:173
2449027020,31sleep324547-21/usr/sbin/munin11:30:153
1544627019,27sleep10-21swapper/109:20:171
116027021,32sleep21225-21awk10:15:232
612626922,29sleep36262-21/usr/sbin/munin10:30:263
612626922,29sleep36262-21/usr/sbin/munin10:30:263
285526916,34sleep12930-21/usr/sbin/munin12:05:141
2500726915,26sleep20-21swapper/208:09:552
2500726915,26sleep20-21swapper/208:09:552
2389126920,28sleep10-21swapper/108:05:141
2389126920,28sleep10-21swapper/108:05:141
1900226914,12sleep20-21swapper/207:50:092
1892026914,41sleep218921-21sshd11:13:012
2844626818,34sleep328485-21if_err_eth010:00:173
1495926811,43sleep215084-21cat07:35:232
1495926811,43sleep215084-21cat07:35:232
1137726812,37sleep311435-21/usr/sbin/munin07:25:143
361626719,23sleep20-21swapper/212:05:272
3128026717,32sleep131472-21cut10:09:561
297952677,18sleep10-21swapper/108:24:041
2600926716,18sleep10-21swapper/111:35:131
1395626715,21sleep30-21swapper/309:15:233
483126622,22sleep10-21swapper/112:10:231
452226613,27sleep10-21swapper/108:45:241
3031026623,26sleep20-21swapper/211:49:562
2549226611,33sleep30-21swapper/308:10:253
2353426616,36sleep223574-21if_eth009:45:142
2353426616,36sleep223574-21if_eth009:45:142
1745526614,27sleep20-21swapper/207:45:082
1095326613,35sleep111003-21/usr/sbin/munin12:30:131
970526512,35sleep39797-21mailstats12:25:233
2307626514,30sleep30-21swapper/309:44:543
103026511,37sleep11059-21irqstats08:35:151
273626415,24sleep10-21swapper/110:20:221
273626415,24sleep10-21swapper/110:20:221
236822647,44sleep20-21swapper/211:25:582
2235726413,22sleep10-21swapper/109:40:241
2036326421,19sleep10-21swapper/107:54:561
2036326421,19sleep10-21swapper/107:54:561
1876326422,19sleep10-21swapper/107:49:561
860526315,32sleep18668-21/usr/sbin/munin07:15:231
3237226316,28sleep132498-21/usr/sbin/munin08:30:261
286626318,26sleep30-21swapper/310:20:243
286626318,26sleep30-21swapper/310:20:243
2030026312,38sleep320404-21fschecks_time09:35:133
632726220,23sleep10-21swapper/112:15:231
632726220,23sleep10-21swapper/112:15:231
2537626214,23sleep10-21swapper/109:50:221
1273626212,26sleep20-21swapper/207:30:062
92626116,20sleep30-21swapper/311:59:553
596126115,23sleep30-21swapper/308:50:233
596126115,23sleep30-21swapper/308:50:233
2955926117,19sleep10-21swapper/110:04:551
2339426117,29sleep123424-21munin-run08:04:541
1885126113,30sleep218909-21/usr/sbin/munin09:30:202
1885126113,30sleep218909-21/usr/sbin/munin09:30:202
3229926011,27sleep20-21swapper/210:10:502
2070026011,19sleep10-21swapper/111:19:551
194226020,25sleep11-21systemd10:19:541
1633726020,25sleep21-21systemd09:24:542
1359626016,21sleep10-21swapper/110:55:241
31725917,19sleep20-21swapper/211:55:252
2805225919,19sleep20-21swapper/209:59:562
271642599,36sleep327183-21date11:39:563
2026025917,26sleep30-21swapper/307:54:543
1680025918,20sleep10-21swapper/111:05:211
1137625916,16sleep30-21swapper/312:30:233
607825813,30sleep16079-21munin-node07:10:061
463025816,19sleep20-21swapper/208:45:252
2102825820,22sleep20-21swapper/211:20:122
49182575,14sleep10-21swapper/110:27:571
49182575,14sleep10-21swapper/110:27:571
463425717,24sleep34660-21netstat12:10:163
2787225714,23sleep127919-21users08:15:281
2710225710,22sleep30-21swapper/308:15:143
2728625619,17sleep20-21swapper/208:15:172
1604625617,22sleep10-21swapper/111:04:571
1345725615,25sleep112714-21/usr/sbin/munin07:30:251
1022025617,23sleep110281-21cat10:45:141
69112555,12sleep20-21swapper/212:17:342
69112555,12sleep20-21swapper/212:17:342
3157525515,24sleep331067-21apt-get10:10:063
3140625511,31sleep231454-21systemd-run08:29:542
2403225513,21sleep30-21swapper/311:29:543
2375725513,26sleep323780-21ntp_states09:45:193
2375725513,26sleep323780-21ntp_states09:45:193
2344425516,18sleep30-21swapper/308:04:563
2075725517,17sleep30-21swapper/307:55:153
2075725517,17sleep30-21swapper/307:55:153
176325518,18sleep30-21swapper/312:00:233
1586625519,18sleep10-21swapper/107:40:101
1562025515,20sleep30-21swapper/307:39:563
1562025515,20sleep30-21swapper/307:39:563
1403525519,20sleep30-21swapper/307:34:553
125172556,38sleep10-21swapper/110:50:431
646725417,14sleep30-21swapper/312:15:253
646725417,14sleep30-21swapper/312:15:253
2859825414,23sleep328594-21apt-get08:20:063
2441625412,23sleep20-21swapper/211:30:142
2175625412,24sleep221775-21apt-get09:40:052
168382548,18sleep10-21swapper/109:25:061
1652825416,17sleep20-21swapper/211:05:142
1039125414,21sleep20-21swapper/209:05:122
854025322,19sleep18466-21/usr/sbin/munin10:40:091
544325314,17sleep20-21swapper/210:30:132
544325314,17sleep20-21swapper/210:30:132
516725312,17sleep30-21swapper/308:49:543
2513425312,17sleep20-21swapper/209:50:152
1392525312,24sleep113948-21ntp_states09:15:211
1180225319,14sleep10-21swapper/107:25:241
78925211,25sleep2782-21apt-get08:35:072
743425212,15sleep20-21swapper/210:35:212
404725212,15sleep30-21swapper/312:09:563
312982527,32sleep30-21swapper/308:28:303
2894025215,22sleep228942-21apt-get11:45:042
2214725212,24sleep222123-21apt-get08:00:222
1958025114,24sleep119647-21switchtime07:50:241
3094425020,18sleep20-21swapper/211:50:212
3094425020,18sleep20-21swapper/211:50:212
1788825014,21sleep317892-21diskstats11:10:123
1682525012,21sleep216368-21apt-get09:24:592
1321025011,23sleep213216-21diskstats10:55:122
117602506,32sleep20-21swapper/212:31:432
85652499,16sleep20-21swapper/210:40:102
2547424912,22sleep225460-21apt-get08:10:122
1967724911,22sleep319721-21memory11:15:163
1367624713,17sleep30-21swapper/312:39:563
506624416,14sleep30-21swapper/310:29:533
506624416,14sleep30-21swapper/310:29:533
98952417,13sleep10-21swapper/109:04:551
1320324111,15sleep30-21swapper/309:14:543
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional