You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-08 - 20:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Apr 08, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20955263918,605sleep121015-21/usr/sbin/munin09:12:001
7892256655,442sleep28040-21latency_hist11:56:432
1657524868,462sleep016619-21ntpq12:22:500
1752024559,434sleep217528-21ntpq07:17:092
138862298119,29sleep20-21swapper/210:32:042
1726622809,259sleep013441-21snmpd08:59:030
18994227015,240sleep319039-21awk07:22:073
5017225438,37sleep20-21swapper/211:47:142
5017225438,37sleep20-21swapper/211:47:142
2624622450,6sleep30-21swapper/309:27:113
13568224451,177sleep213627-21latency07:07:042
13568224451,177sleep213627-21latency07:07:042
29737223716,187sleep20-21swapper/207:56:542
15476223550,132sleep215470-21apt-get07:11:562
1727223271,144sleep20-21swapper/208:11:442
20503222820,185sleep220506-21cron12:36:412
12541222518,188sleep212568-21modprobe08:46:412
12541222518,188sleep212568-21modprobe08:46:412
5155222471,25sleep20-21swapper/208:22:022
29986222115,187sleep130073-21latency_hist09:41:441
25080221519,47sleep20-21swapper/207:42:042
21177219035,19sleep20-21swapper/210:56:432
4299218417,144sleep20-21swapper/208:17:142
933021839,162sleep013441-21snmpd08:35:330
26158218214,145sleep20-21swapper/209:27:112
13235218212,157sleep213270-21ntp_kernel_pll_08:47:062
31482217816,143sleep231525-21true09:46:412
26763217516,139sleep20-21swapper/207:47:012
2382821739,143sleep20-21swapper/207:37:122
9500216913,142sleep29536-21latency_hist08:36:442
2274621658,142sleep222892-21date11:01:442
572216411,141sleep2620-21systemd-run09:51:422
22637216319,26sleep20-21swapper/209:17:022
26070216217,55sleep20-21swapper/211:11:572
21677216013,126sleep20-21swapper/207:31:512
21677216013,126sleep20-21swapper/207:31:512
30752215721,54sleep20-21swapper/211:26:432
30752215721,54sleep20-21swapper/211:26:432
440215618,19sleep20-21swapper/211:32:042
10008215418,54sleep20-21swapper/208:37:042
10008215418,54sleep20-21swapper/208:37:042
18369215112,16sleep20-21swapper/209:02:122
2162421497,13sleep20-21swapper/209:12:132
16160214911,117sleep20-21swapper/212:22:072
23338214722,109sleep20-21swapper/211:02:042
31463214513,116sleep031508-21basename08:02:010
2000621439,119sleep220156-21unixbench_multi09:07:122
32234214111,116sleep332269-21grep09:47:113
1556421405,14sleep30-21swapper/308:52:323
5011213510,111sleep34978-21apt-get11:47:113
5011213510,111sleep34978-21apt-get11:47:113
3661213416,22sleep00-21swapper/011:42:080
15600213217,101sleep315607-21sh07:12:343
4093212914,102sleep04172-21sort08:17:110
17153212819,93sleep317243-21fschecks_time07:17:023
14965212812,60sleep20-21swapper/212:17:102
11069212812,101sleep011143-21find10:22:130
31848212514,41sleep231926-21switchtime08:02:112
2121212513,59sleep21630-21/usr/sbin/munin11:37:092
2203212422,85sleep32328-21latency_hist09:56:443
14898212310,54sleep20-21swapper/208:52:182
5252212117,84sleep35254-21cron10:06:413
5252212117,84sleep35254-21cron10:06:413
3999212118,80sleep00-21swapper/010:01:580
3999212118,80sleep00-21swapper/010:01:580
7462211313,88sleep27507-21ntpq10:12:052
20667211311,42sleep220747-21mailstats07:27:112
29416211116,76sleep229565-21grep09:37:132
8595211022,17sleep20-21swapper/211:57:082
18977211062,24sleep20-21swapper/207:22:062
9382210915,74sleep09419-21ps10:17:120
11302210911,12sleep30-21swapper/312:07:023
1046210712,78sleep1840-21/usr/sbin/munin09:52:011
31549210611,82sleep131622-21sed11:27:111
1053321068,86sleep210539-21df_inode10:21:592
16813210510,38sleep20-21swapper/210:41:572
16813210510,38sleep20-21swapper/210:41:572
11146210517,16sleep10-21swapper/110:22:141
3322210413,71sleep13176-21/usr/sbin/munin11:42:011
30294210414,78sleep330354-21switchtime07:57:123
17537210413,68sleep017576-21processes07:17:110
31623210318,24sleep20-21swapper/211:27:122
3107210234,11sleep20-21swapper/208:15:282
46621015,11sleep30-21swapper/308:07:013
22207210111,75sleep022302-21mailstats07:32:090
22207210111,75sleep022302-21mailstats07:32:090
8157210031,43sleep18154-21apt-get08:31:541
4280210033,11sleep20-21swapper/211:44:242
20549210017,69sleep320648-21unixbench_multi10:52:123
344029935,10sleep20-21swapper/209:58:162
2897729959,23sleep229064-21apt-config11:21:422
203029832,13sleep20-21swapper/209:55:342
1519229835,48sleep20-21swapper/210:36:532
1242329827,46sleep212596-21grep07:02:122
2449729731,50sleep20-21swapper/211:06:532
1561029712,23sleep00-21swapper/007:12:350
966629667,12sleep20-21swapper/212:01:562
654329634,10sleep20-21swapper/210:08:222
3153429615,62sleep031617-21wc09:46:430
2993429626,46sleep329963-21munin-plugin-st09:41:423
1746629632,10sleep20-21swapper/209:01:352
785329532,9sleep20-21swapper/208:31:272
709129513,63sleep17156-21grep08:27:101
2815229532,10sleep20-21swapper/209:34:062
2655229515,64sleep326614-21sendmail_mailst11:12:113
2513929531,11sleep20-21swapper/209:25:092
2058329513,30sleep00-21swapper/012:36:430
1934929537,43sleep20-21swapper/210:49:082
1934929537,43sleep20-21swapper/210:49:082
1218629536,11sleep20-21swapper/212:08:022
482929413,58sleep14952-21latency_hist08:21:451
1307829432,10sleep20-21swapper/210:28:332
2909729331,10sleep20-21swapper/207:52:302
1884829228,12sleep20-21swapper/212:30:192
1375829232,11sleep20-21swapper/212:13:112
1004729234,11sleep20-21swapper/210:20:052
663529115,58sleep00-21swapper/011:52:010
246572919,65sleep00-21swapper/009:22:130
869229012,63sleep18825-21apt-key10:16:451
530829036,31sleep20-21swapper/210:06:442
530829036,31sleep20-21swapper/210:06:442
3045928912,61sleep130493-21irqstats09:42:041
1731828915,50sleep316719-21/usr/sbin/munin10:42:113
1731828915,50sleep316719-21/usr/sbin/munin10:42:113
1506828926,38sleep315213-21unixbench_singl12:17:113
1370328925,47sleep313773-21fschecks_time10:32:013
1491028825,31sleep00-21swapper/007:11:460
1477928825,43sleep314879-21memory07:11:453
1477928825,43sleep314879-21memory07:11:453
2681328712,61sleep126857-21systemd-run09:31:421
8522865,63sleep10-21swapper/108:07:111
1259428625,33sleep00-21swapper/008:46:430
1259428625,33sleep00-21swapper/008:46:430
759728512,58sleep37677-21mailstats10:12:103
2109928520,47sleep021163-21modprobe10:56:410
1097528524,32sleep00-21swapper/012:06:430
358928312,56sleep33596-21df_inode08:16:583
838128218,27sleep00-21swapper/011:57:020
1415428223,35sleep30-21swapper/308:51:433
1861128121,37sleep018606-21apt-get07:21:560
1332928123,39sleep013354-21munin-plugin-st10:31:420
1171228121,45sleep311743-21grep07:01:563
2784427915,43sleep127951-21/usr/sbin/munin09:32:151
2722727914,52sleep127322-21timerandwakeup07:47:121
2245627915,51sleep122457-21cpu09:16:571
1455927917,42sleep014573-21apt-get07:11:420
1301027914,48sleep113016-21apt-config10:28:031
2668427824,26sleep10-21swapper/111:12:131
2188427817,27sleep30-21swapper/310:57:093
1846927817,43sleep318611-21/usr/sbin/munin09:02:133
1689027811,51sleep117043-21awk08:57:141
953927719,42sleep39683-21cat08:36:453
2063327723,28sleep10-21swapper/109:11:441
1362327712,45sleep10-21swapper/108:47:131
2792227615,47sleep328011-21cat11:17:033
2461127617,39sleep10-21swapper/107:41:431
2142427618,40sleep021500-21sed09:12:110
1029227613,42sleep110365-21switchtime08:37:111
1029227613,42sleep110365-21switchtime08:37:111
2824927514,42sleep028285-21/usr/sbin/munin07:52:000
2062527512,49sleep120664-21ps07:27:101
706427411,49sleep07069-21sort10:11:510
1833327420,35sleep10-21swapper/112:27:111
3227227311,42sleep032346-21/usr/sbin/munin09:47:120
2186627214,41sleep121873-21/usr/sbin/munin10:57:071
1477027220,35sleep014854-21tune2fs08:52:160
73227122,26sleep10-21swapper/111:32:111
2710127111,46sleep027149-21grep09:31:580
2517127113,45sleep125176-21/usr/sbin/munin07:42:121
2457927112,44sleep324648-21switchtime09:22:123
1826427124,24sleep30-21swapper/310:46:513
1826427124,24sleep30-21swapper/310:46:513
1297427119,31sleep113009-21munin-run07:06:421
1211127124,32sleep012156-21ntpq07:02:060
1013527113,32sleep30-21swapper/312:02:093
973827024,23sleep30-21swapper/308:36:573
973827024,23sleep30-21swapper/308:36:573
80862706,14sleep10-21swapper/111:56:501
56127017,14sleep30-21swapper/311:32:093
2736927027,28sleep30-21swapper/309:32:053
2459827013,38sleep00-21swapper/011:06:590
1614927017,27sleep116137-21apt-get12:21:581
1284627018,31sleep00-21swapper/012:12:000
669426915,37sleep36698-21diskstats08:27:003
272812696,50sleep00-21swapper/011:15:590
1888426915,22sleep10-21swapper/107:22:051
937826820,32sleep39406-21munin-run12:01:413
590626813,38sleep35968-21users08:22:153
320226822,24sleep30-21swapper/308:16:423
270726815,22sleep00-21swapper/008:12:150
1875226814,39sleep018789-21grep10:47:080
1875226814,39sleep018789-21grep10:47:080
1837626822,24sleep10-21swapper/107:21:441
2317826715,25sleep30-21swapper/311:02:013
2058226722,24sleep30-21swapper/309:11:423
1529126711,40sleep115400-21fschecks_time10:37:001
448526626,24sleep10-21swapper/110:02:111
448526626,24sleep10-21swapper/110:02:111
1996126617,25sleep10-21swapper/112:32:161
188732666,11sleep00-21swapper/009:04:480
1494726611,40sleep014957-21munin-run10:36:420
1102626620,20sleep10-21swapper/112:06:441
632426521,30sleep16341-21ls08:26:411
2074826516,25sleep30-21swapper/307:27:123
3135126417,30sleep331363-21/usr/sbin/munin11:27:053
2983626416,24sleep10-21swapper/107:57:021
1786126418,22sleep30-21swapper/312:27:013
1650526419,30sleep016481-21latency_hist10:41:430
1589026419,23sleep10-21swapper/108:56:421
1097726421,18sleep30-21swapper/310:22:123
786726320,24sleep10-21swapper/111:56:421
2437126313,32sleep124395-21/usr/sbin/munin09:22:051
2213926320,23sleep30-21swapper/309:16:423
1459626318,22sleep10-21swapper/110:33:081
560026210,39sleep05671-21sensors_temp08:22:120
376026217,20sleep30-21swapper/310:01:453
30522629,12sleep10-21swapper/109:57:141
1425326217,20sleep00-21swapper/008:51:440
2272626115,34sleep022817-21load09:17:040
18882615,44sleep00-21swapper/009:53:420
3030926013,25sleep330404-21perl09:42:023
251972607,12sleep10-21swapper/111:07:121
1895826016,29sleep019004-21sh12:31:420
1507926021,17sleep10-21swapper/107:11:471
718625915,21sleep10-21swapper/110:12:001
3234725920,19sleep10-21swapper/109:47:131
275862597,11sleep30-21swapper/307:47:593
1684225911,22sleep10-21swapper/110:42:001
1684225911,22sleep10-21swapper/110:42:001
116712597,13sleep30-21swapper/308:42:063
116712597,13sleep30-21swapper/308:42:063
3253325814,27sleep032528-21apt-get11:31:540
2895925810,30sleep129065-21users07:52:131
2891125815,23sleep10-21swapper/111:21:411
1303325814,30sleep313130-21/usr/sbin/munin12:12:033
1201125820,19sleep30-21swapper/310:26:563
670025718,22sleep00-21swapper/008:27:010
651125710,19sleep30-21swapper/311:51:523
328225713,19sleep10-21swapper/108:16:441
2297225715,24sleep122966-21apt-get11:01:551
210912577,38sleep00-21swapper/007:27:290
174125718,25sleep10-21swapper/111:37:001
562825618,24sleep10-21swapper/110:07:001
2964425615,16sleep00-21swapper/011:22:170
2964425615,16sleep00-21swapper/011:22:170
2472725619,16sleep30-21swapper/311:07:023
2352025612,27sleep023587-21/usr/sbin/munin11:02:100
2304425619,21sleep322985-21latency_hist07:36:443
2304425619,21sleep322985-21latency_hist07:36:443
2287425614,25sleep322906-21ntp_states09:17:073
1693025617,23sleep317070-21grep12:22:573
1476425614,26sleep114809-21ntp_states12:17:051
1472325614,20sleep00-21swapper/012:17:040
978525517,19sleep00-21swapper/008:36:580
978525517,19sleep00-21swapper/008:36:580
898625514,20sleep30-21swapper/310:16:593
498125520,17sleep10-21swapper/111:46:531
498125520,17sleep10-21swapper/111:46:531
296525515,19sleep00-21swapper/011:41:430
1985825512,23sleep00-21swapper/009:07:100
1519325514,24sleep315189-21apt-get10:36:543
1161925517,18sleep10-21swapper/112:07:081
1121925511,18sleep00-21swapper/012:06:590
65292545,10sleep30-21swapper/310:08:113
42592545,19sleep30-21swapper/311:44:093
3147425413,19sleep00-21swapper/011:27:100
2762725412,18sleep00-21swapper/011:16:520
2661725414,23sleep026611-21apt-get07:46:550
2508225414,24sleep325059-21apt-get07:42:063
1647425415,20sleep30-21swapper/308:57:043
3129325313,24sleep331285-21apt-get08:01:563
2637125313,23sleep00-21swapper/007:46:430
2197125314,18sleep10-21swapper/107:32:021
2197125314,18sleep10-21swapper/107:32:021
1805425317,15sleep00-21swapper/010:46:440
1805425317,15sleep00-21swapper/010:46:440
1128225312,25sleep111276-21apt-get08:41:561
1128225312,25sleep111276-21apt-get08:41:561
307092525,11sleep00-21swapper/007:57:530
2969425212,25sleep129730-21irqstats11:22:181
2969425212,25sleep129730-21irqstats11:22:181
294072529,18sleep30-21swapper/311:21:523
294072529,18sleep30-21swapper/311:21:523
1829725212,14sleep10-21swapper/109:02:111
165225214,22sleep31648-21apt-get11:36:543
1275625214,22sleep112754-21apt-get12:11:521
298392515,37sleep00-21swapper/009:40:300
1945925111,25sleep319516-21irqstats12:32:053
538625012,19sleep00-21swapper/011:47:250
538625012,19sleep00-21swapper/011:47:250
2824025013,22sleep328247-21diskstats07:51:593
2268425011,19sleep00-21swapper/011:01:420
104925015,18sleep00-21swapper/008:07:140
685324912,21sleep06928-21latency_hist10:11:430
255482499,15sleep00-21swapper/009:26:530
1337124913,20sleep113378-21df_inode07:06:591
1337124913,20sleep113378-21df_inode07:06:591
3136624812,19sleep10-21swapper/108:01:581
2325824811,22sleep023276-21apt-get07:36:540
2336524716,15sleep30-21swapper/307:37:013
1983824313,17sleep10-21swapper/110:51:541
1955024311,19sleep1281-21dbus-daemon10:51:411
1955024311,19sleep1281-21dbus-daemon10:51:411
244224111,17sleep00-21swapper/009:56:570
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional