You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-11 - 06:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon May 11, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
234152209112,2066sleep29237-21kworker/2:220:42:052
15294288319,845sleep215413-21/usr/sbin/munin23:40:122
15294288319,845sleep215413-21/usr/sbin/munin23:40:122
27509284017,802sleep027598-21/usr/sbin/munin00:20:000
1039281014,777sleep21117-21/usr/sbin/munin19:30:082
3073625958,573sleep130772-21ntpq00:30:031
18460255848,461sleep118603-21latency_hist22:09:401
1381250246,62sleep10-21swapper/121:15:001
1381250246,62sleep10-21swapper/121:15:001
32675236150,52sleep10-21swapper/121:10:071
5005230798,147sleep15081-21fschecks_time23:09:581
26998229348,205sleep127069-21cut00:19:411
27337228915,135sleep127375-21fschecks_count20:54:561
2671322699,58sleep10-21swapper/122:35:001
27964226431,212sleep128122-21cut19:14:441
27964226431,212sleep128122-21cut19:14:441
65532254137,75sleep10-21swapper/123:14:581
8026225169,137sleep18055-21munin-run19:54:391
3427224239,85sleep13476-21latency_hist19:39:401
3937223939,185sleep14011-21switchtime23:05:091
7220223617,204sleep37228-21ntpq19:50:043
28861221922,173sleep028934-21grep19:15:080
28861221922,173sleep028934-21grep19:15:080
5409220525,160sleep35491-21/usr/sbin/munin23:10:083
166022059,185sleep0363-21snmpd19:32:490
29469219511,126sleep129616-21grep00:25:101
9772218761,58sleep10-21swapper/123:24:561
25266218711,138sleep125302-21ntpq22:30:041
16271218536,22sleep10-21swapper/122:00:111
20658216840,63sleep120710-21systemd23:59:371
20658216840,63sleep120710-21systemd23:59:371
10644216436,89sleep110727-21latency_hist21:44:401
5164216331,113sleep15315-21head21:25:091
13692216212,91sleep113706-21apt21:54:381
13692216212,91sleep113706-21apt21:54:381
24603216117,129sleep3309-21rs:main3
24603216117,129sleep3309-21rs:main3
3758215912,19sleep00-21swapper/023:05:040
15818215835,112sleep115821-21cron20:19:381
19024215510,123sleep119213-21cmp23:54:411
31816215311,125sleep331848-21awk19:25:043
278621517,107sleep1363-21snmpd23:01:011
1428421506,16sleep10-21swapper/121:55:021
1428421506,16sleep10-21swapper/121:55:021
11114214716,109sleep10-21swapper/123:29:491
4983214619,84sleep10-21swapper/119:44:411
32705214625,51sleep10-21swapper/119:29:391
28514214663,68sleep128578-21mailstats22:40:051
25403214523,102sleep025428-21/usr/sbin/munin20:49:380
25307214416,78sleep10-21swapper/100:14:031
2086321446,17sleep10-21swapper/120:34:571
2086321446,17sleep10-21swapper/120:34:571
32147214124,39sleep10-21swapper/100:34:591
32147214124,39sleep10-21swapper/100:34:591
2880321417,122sleep128813-21find20:59:491
23154214124,76sleep118714-1kworker/1:2H20:40:101
27915214029,62sleep10-21swapper/100:20:091
2765921378,88sleep127768-21latency_hist19:14:401
1563213711,83sleep11695-21latency_hist22:59:401
24656213627,72sleep124721-21date22:29:391
24656213627,72sleep124721-21date22:29:391
11490213657,22sleep10-21swapper/121:45:091
23853213010,76sleep10-21swapper/100:09:391
31013212818,17sleep20-21swapper/221:05:152
2571212819,87sleep02683-21unixbench_singl23:00:110
18456212113,57sleep10-21swapper/120:25:121
2440521208,92sleep10-21swapper/120:45:031
154421195,10sleep10-21swapper/119:31:201
3136211867,40sleep13145-21irqstats21:20:041
3136211867,40sleep13145-21irqstats21:20:041
6699211517,84sleep36733-21irqstats23:15:003
3213021145,10sleep10-21swapper/122:51:201
3213021145,10sleep10-21swapper/122:51:201
20027211015,73sleep220103-21/usr/sbin/munin23:55:262
20027211015,73sleep220103-21/usr/sbin/munin23:55:262
21480210914,79sleep121519-21systemctl22:19:371
21480210914,79sleep121519-21systemctl22:19:371
21081210940,54sleep121161-21fschecks_time23:59:581
21081210940,54sleep121161-21fschecks_time23:59:581
813210723,64sleep3824-21kernelversion19:30:013
2988821077,29sleep10-21swapper/119:19:491
26537210716,73sleep226566-21fschecks_count22:34:572
1101921075,68sleep10-21swapper/120:02:501
3456210621,67sleep33545-21fschecks_time23:04:593
29341210615,75sleep029482-21cut22:44:410
5218210528,59sleep10-21swapper/119:44:531
29157210315,71sleep229197-21grep00:25:032
13307210317,73sleep313334-21netstat23:35:083
9590210118,66sleep09617-21cpu23:24:510
31887210124,59sleep10-21swapper/121:09:381
25729210120,62sleep325819-21fschecks_time00:14:583
22251210125,64sleep122256-21ntp_states22:20:061
11989210117,71sleep112058-21swap20:05:091
8779210019,63sleep10-21swapper/119:55:071
942029913,68sleep39490-21fschecks_time21:39:583
287429816,67sleep22875-21http21:19:572
287429816,67sleep22875-21http21:19:572
965429712,69sleep39781-21latency_hist19:59:413
255429712,14sleep00-21swapper/021:19:390
255429712,14sleep00-21swapper/021:19:390
1947029721,13sleep10-21swapper/120:30:001
707329614,64sleep07175-21/usr/sbin/munin23:15:100
1787829618,65sleep217910-21latency23:50:012
307532957,31sleep1363-21snmpd22:48:331
307532957,31sleep1363-21snmpd22:48:331
3038529517,56sleep20-21swapper/219:20:072
2853229413,66sleep328590-21unin-run20:59:393
2634929416,59sleep326323-21/usr/sbin/munin19:09:543
621329321,38sleep10-21swapper/121:29:541
621329321,38sleep10-21swapper/121:29:541
2582129315,65sleep025878-21idleruntime00:14:590
1736829310,69sleep117473-21date23:49:401
1297229322,53sleep10-21swapper/120:09:491
591729219,58sleep06112-21modprobe21:29:410
591729219,58sleep06112-21modprobe21:29:410
307082926,12sleep00-21swapper/000:30:020
2569029220,13sleep10-21swapper/120:49:511
2540229123,51sleep225535-21grep19:05:092
151712917,70sleep115292-21ls23:40:111
151712917,70sleep115292-21ls23:40:111
2230229016,58sleep322422-21date00:04:403
2230229016,58sleep322422-21date00:04:403
1159429017,52sleep211743-21sed21:45:102
621028914,22sleep36207-21apt-get21:29:523
621028914,22sleep36207-21apt-get21:29:523
2851328826,40sleep228529-21modprobe20:59:382
2694928824,41sleep227030-21awk19:10:092
82292867,45sleep18269-21ps21:35:111
2476028625,42sleep024825-21cpuspeed19:04:540
2890028510,55sleep30-21swapper/320:59:553
1311828522,47sleep213268-21sed21:50:102
1311828522,47sleep213268-21sed21:50:102
1243728512,55sleep212505-21apt-key23:34:382
301692846,65sleep00-21swapper/019:20:000
1740228430,43sleep117422-21munin-run20:24:391
1739228419,35sleep0225-21systemd-journal20:24:380
1685328422,42sleep316990-21unixbench_singl20:20:093
3120828311,14sleep30-21swapper/319:24:403
1720228312,55sleep017278-21fschecks_time22:04:590
1720228312,55sleep017278-21fschecks_time22:04:590
1472828324,37sleep214870-21grep21:55:112
1472828324,37sleep214870-21grep21:55:112
61792828,14sleep20-21swapper/221:29:472
61792828,14sleep20-21swapper/221:29:472
1589628223,30sleep30-21swapper/320:19:403
1432628218,47sleep0225-21systemd-journal23:39:410
1432628218,47sleep0225-21systemd-journal23:39:410
804528120,15sleep10-21swapper/123:19:511
307128124,42sleep222-21ksoftirqd/223:04:382
2077928113,53sleep220829-21cpuspeed20:34:542
2077928113,53sleep220829-21cpuspeed20:34:542
1437928116,42sleep314400-21sshd21:55:043
1437928116,42sleep314400-21sshd21:55:043
128282817,18sleep112428-21apt-get23:34:411
813428017,44sleep38199-21fschecks_time23:19:573
1954228017,47sleep319567-21fschecks_count23:55:153
1954228017,47sleep319567-21fschecks_count23:55:153
1592928012,48sleep215965-21ntp_kernel_err22:00:042
1674527919,25sleep20-21swapper/220:20:082
2305727823,23sleep00-21swapper/022:24:370
1191027616,34sleep30-21swapper/320:05:073
81662758,51sleep20-21swapper/221:35:082
183127514,26sleep30-21swapper/321:15:103
183127514,26sleep30-21swapper/321:15:103
1333627518,22sleep00-21swapper/023:35:090
788527413,48sleep07894-21fschecks_count21:35:020
2780827419,40sleep027889-21switchtime20:55:080
2394027414,34sleep023643-21apt-get20:44:420
217527414,43sleep32182-21/usr/sbin/munin19:34:583
3253527314,26sleep20-21swapper/200:35:092
3253527314,26sleep20-21swapper/200:35:092
3234827335,22sleep30-21swapper/300:35:063
3234827335,22sleep30-21swapper/300:35:063
1276127317,16sleep00-21swapper/020:09:400
3009927224,25sleep20-21swapper/221:04:392
2770127220,31sleep20-21swapper/200:20:062
2456627217,34sleep00-21swapper/000:10:070
1931827212,47sleep219355-21fschecks_count20:29:572
1875227218,27sleep20-21swapper/222:09:572
1875227218,27sleep20-21swapper/222:09:572
1786627216,35sleep217655-21/usr/sbin/munin20:25:012
998727113,45sleep010075-21fschecks_time19:59:590
970327125,28sleep00-21swapper/021:40:030
911527119,24sleep30-21swapper/321:39:413
3213127116,33sleep232278-21unixbench_multi19:25:112
1894127119,36sleep318961-21modprobe20:29:383
1768227118,29sleep00-21swapper/020:24:550
1524327117,37sleep015261-21apt21:59:370
1524327117,37sleep015261-21apt21:59:370
1460027114,39sleep014622-21/usr/sbin/munin20:14:570
1115727121,31sleep30-21swapper/320:04:383
1797827019,24sleep00-21swapper/023:50:030
1758027016,39sleep217604-21/usr/sbin/munin22:05:092
1758027016,39sleep217604-21/usr/sbin/munin22:05:092
110632708,53sleep29237-21kworker/2:220:03:242
549426918,31sleep00-21swapper/023:10:090
2315626920,32sleep30-21swapper/300:05:093
2078126916,38sleep020921-21latency_hist23:59:400
2078126916,38sleep020921-21latency_hist23:59:400
570826812,29sleep20-21swapper/223:10:112
197552685,30sleep10-21swapper/122:12:271
197552685,30sleep10-21swapper/122:12:271
180482685,10sleep10-21swapper/122:05:281
1169726814,39sleep011765-21wc23:30:080
3196826721,22sleep00-21swapper/021:09:410
2597426712,38sleep325975-21sshd19:09:053
335026616,34sleep03426-21systemd-run19:39:390
2691526613,22sleep30-21swapper/322:35:063
2403026615,24sleep00-21swapper/022:25:100
2403026615,24sleep00-21swapper/022:25:100
894426518,24sleep00-21swapper/019:55:100
869326512,26sleep20-21swapper/219:55:042
2374826514,32sleep223795-21/usr/sbin/munin22:25:042
2374826514,32sleep223795-21/usr/sbin/munin22:25:042
2211826528,23sleep022057-21latency_hist20:39:410
2211826528,23sleep022057-21latency_hist20:39:410
2077526518,29sleep320772-21apt-get20:34:513
2077526518,29sleep320772-21apt-get20:34:513
572126416,21sleep20-21swapper/221:29:392
3165926421,18sleep20-21swapper/200:34:402
315262645,11sleep30-21swapper/300:33:443
3014626415,17sleep30-21swapper/300:29:403
2684826417,28sleep026883-21/usr/sbin/munin22:35:030
155372645,47sleep30-21swapper/323:41:423
155372645,47sleep30-21swapper/323:41:423
572426317,23sleep30-21swapper/319:45:073
258212635,11sleep10-21swapper/119:07:051
2409526314,36sleep324097-21cpu00:09:533
235942635,45sleep00-21swapper/000:07:000
48412620,7sleep30-21swapper/321:25:023
2607426217,20sleep00-21swapper/020:50:030
258882625,11sleep20-21swapper/222:31:142
1336726215,30sleep213412-21/usr/sbin/munin20:10:022
1299726215,28sleep012991-21apt-get20:09:520
1153626220,25sleep311563-21ntpq23:30:033
275562616,46sleep00-21swapper/019:13:590
2417926114,30sleep224184-21/usr/sbin/munin00:09:562
1888326117,25sleep30-21swapper/323:54:393
435226015,24sleep24495-21grep19:40:102
2826526017,18sleep20-21swapper/222:39:582
2811326019,23sleep30-21swapper/322:39:553
2382226012,31sleep323933-21cmp20:44:403
1711426021,22sleep317109-21apt-get22:04:533
1711426021,22sleep317109-21apt-get22:04:533
95225916,22sleep30-21swapper/322:55:103
324325916,19sleep30-21swapper/321:20:073
324325916,19sleep30-21swapper/321:20:073
3195025916,30sleep331966-21sed21:09:403
3083725916,27sleep00-21swapper/022:49:370
3083725916,27sleep00-21swapper/022:49:370
242182590,6sleep30-21swapper/320:44:563
1946925915,25sleep319567-21users22:10:113
1946925915,25sleep319567-21users22:10:113
1682425917,23sleep00-21swapper/022:04:380
12812599,24sleep20-21swapper/221:14:522
12812599,24sleep20-21swapper/221:14:522
865625814,24sleep20-21swapper/223:20:082
653025812,25sleep00-21swapper/019:49:410
413425818,19sleep30-21swapper/319:40:073
3267525817,26sleep032669-21apt-get22:54:500
3088925812,21sleep20-21swapper/222:49:392
3088925812,21sleep20-21swapper/222:49:392
1930325815,29sleep019311-21df20:29:550
1633225813,17sleep00-21swapper/023:45:020
1454825817,24sleep214541-21apt-get20:14:552
139372586,10sleep20-21swapper/223:36:242
1270825817,24sleep20-21swapper/220:09:382
45882578,11sleep20-21swapper/223:09:102
2591325720,24sleep225990-21sed00:15:012
2060225718,20sleep30-21swapper/322:15:033
2060225718,20sleep30-21swapper/322:15:033
1969525719,20sleep30-21swapper/320:30:073
129712579,14sleep00-21swapper/021:50:080
129712579,14sleep00-21swapper/021:50:080
675725614,19sleep00-21swapper/019:49:490
3249525612,28sleep232639-21modprobe22:54:402
3227925616,18sleep00-21swapper/021:09:580
2261825613,27sleep222625-21diskstats00:04:562
2225125616,16sleep00-21swapper/000:04:370
2225125616,16sleep00-21swapper/000:04:370
104525612,18sleep20-21swapper/221:14:402
676425515,23sleep26760-21apt-get19:49:512
3592556,38sleep00-21swapper/000:35:270
3592556,38sleep00-21swapper/000:35:270
2879725516,22sleep328859-21/usr/sbin/munin19:15:073
2879725516,22sleep328859-21/usr/sbin/munin19:15:073
2731425515,23sleep327320-21diskstats00:19:553
1135925517,20sleep30-21swapper/321:45:063
3015525413,15sleep30-21swapper/322:45:083
3015525413,15sleep30-21swapper/322:45:083
2930125421,18sleep00-21swapper/021:00:080
2776925417,19sleep20-21swapper/219:14:412
2776925417,19sleep20-21swapper/219:14:412
2489625412,26sleep324891-21apt-get22:29:533
1663425413,24sleep316703-21/usr/sbin/munin23:45:093
1129825413,25sleep011334-21ntp_states21:45:030
442025315,19sleep00-21swapper/021:24:520
3227425313,24sleep332275-21/usr/sbin/munin21:09:573
251425313,24sleep22543-21ntp_states19:35:052
1887625314,19sleep00-21swapper/022:09:590
1887625314,19sleep00-21swapper/022:09:590
1563725315,22sleep315643-21diskstats21:59:573
1307125316,21sleep313080-21diskstats20:09:553
1085125314,25sleep210877-21munin-run23:29:372
820125221,17sleep00-21swapper/023:19:580
55852529,30sleep20-21swapper/219:45:022
3234425214,19sleep20-21swapper/222:54:032
3234425214,19sleep20-21swapper/222:54:032
3140325216,18sleep00-21swapper/022:50:020
3140325216,18sleep00-21swapper/022:50:020
308325210,18sleep00-21swapper/021:20:030
308325210,18sleep00-21swapper/021:20:030
2878725212,23sleep028781-21apt-get00:24:530
244225214,22sleep32546-21timerwakeupswit23:00:103
2411025211,25sleep223643-21apt-get20:44:482
2226325214,16sleep30-21swapper/322:20:083
2156325215,23sleep021678-21sed22:19:410
2156325215,23sleep021678-21sed22:19:410
2098225213,23sleep220979-21apt-get23:59:512
2098225213,23sleep220979-21apt-get23:59:512
2569125111,22sleep225685-21apt-get20:49:522
645925013,22sleep26456-21apt-get23:14:512
2522225011,16sleep00-21swapper/022:30:030
165552509,17sleep20-21swapper/223:45:082
3112824913,20sleep331124-21apt-get22:49:523
3112824913,20sleep331124-21apt-get22:49:523
994524816,17sleep20-21swapper/221:40:092
2237924813,18sleep20-21swapper/222:20:112
1290424712,19sleep312968-21ps21:50:073
1290424712,19sleep312968-21ps21:50:073
3276124510,20sleep0442-21sed19:29:420
257862437,13sleep30-21swapper/320:49:583
2097924313,12sleep20-21swapper/222:15:102
2097924313,12sleep20-21swapper/222:15:102
95882428,17sleep30-21swapper/323:24:503
497024011,17sleep00-21swapper/019:44:400
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional