You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 10:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Jan 10, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25322687212,6848sleep231822-21kworker/2:021:56:322
24334240447,4001sleep124138-21apt-get19:41:201
8942210898,1037sleep115576-1kworker/1:2H00:01:041
1425226296,614sleep228313-1kworker/2:0H22:35:092
388125756,558sleep37694-1kworker/3:1H23:45:273
1547224556,439sleep019503-1kworker/0:2H20:56:290
1547224556,439sleep019503-1kworker/0:2H20:56:290
26881232349,185sleep10-21swapper/121:30:181
30153229618,260sleep130296-21latency_hist19:59:491
2725229234,244sleep12843-21cat23:40:161
2725229234,244sleep12843-21cat23:40:161
24336228214,246sleep224431-21fschecks_time19:41:212
577022748,249sleep15814-21sed23:50:091
4723226720,221sleep24863-21unixbench_singl23:46:592
13769224240,152sleep113865-21apt-config22:34:481
13769224240,152sleep113865-21apt-config22:34:481
14528223831,159sleep114565-21ntpq19:10:101
15043223740,124sleep115149-21timerwakeupswit22:36:581
1318622309,14sleep30-21swapper/320:49:463
78822214110,43sleep10-21swapper/122:15:051
2789221330,141sleep12846-21basename21:59:461
14880221339,71sleep10-21swapper/120:54:481
14880221339,71sleep10-21swapper/120:54:481
4928220319,121sleep10-21swapper/122:05:081
21040220217,171sleep221132-21sed22:55:162
28559220132,107sleep10-21swapper/119:54:481
31640220018,166sleep131643-21cron20:04:461
31640220018,166sleep131643-21cron20:04:461
13186219514,165sleep013288-21latency_hist00:14:480
21791218233,131sleep121835-21latency21:15:081
21791218233,131sleep121835-21latency21:15:081
19667218137,94sleep119719-21sort00:35:001
19667218137,94sleep119719-21sort00:35:001
22711217919,67sleep10-21swapper/119:35:101
16536217951,110sleep116686-21unixbench_singl22:40:171
24129217344,77sleep10-21swapper/121:21:421
17234217249,81sleep10-21swapper/100:25:151
981121699,68sleep19318-21/usr/sbin/munin22:20:141
14201216868,84sleep113398-21/usr/sbin/munin00:15:181
14201216868,84sleep113398-21/usr/sbin/munin00:15:181
3186921678,56sleep131947-21grep23:30:151
6254216513,138sleep26278-21df22:10:012
22141216534,65sleep122286-21latency_hist19:34:491
17454216146,95sleep117543-21fschecks_time22:45:061
4136216017,82sleep10-21swapper/120:19:561
18346216028,92sleep118468-21cat19:20:151
30720214215,83sleep10-21swapper/123:25:181
18878214017,22sleep00-21swapper/021:05:140
20379213910,114sleep120425-21ntpq21:10:111
2243221389,93sleep122468-21ntpq23:00:111
297921348,68sleep13026-21ntpq20:15:111
297921348,68sleep13026-21ntpq20:15:111
1952221346,14sleep20-21swapper/221:07:502
8250213259,56sleep18265-40mandb23:59:471
31346213231,43sleep131406-21/usr/sbin/munin21:45:171
24729213210,107sleep224737-21df21:25:022
32295213028,89sleep132320-21awk20:05:101
15897212544,25sleep10-21swapper/119:14:541
563212269,16sleep10-21swapper/123:35:031
2088021226,104sleep00-21swapper/000:39:110
2088021226,104sleep00-21swapper/000:39:110
20443212115,88sleep00-21swapper/021:10:140
18116212034,23sleep10-21swapper/100:30:001
18116212034,23sleep10-21swapper/100:30:001
10985211712,92sleep111064-21wc20:40:151
10985211712,92sleep111064-21wc20:40:151
6134211620,78sleep06275-21awk23:50:170
25552211659,44sleep125565-21awk21:25:181
7864211533,63sleep10-21swapper/120:30:181
23598211439,41sleep10-21swapper/123:05:001
23598211439,41sleep10-21swapper/123:05:001
20186211410,58sleep1286-21dbus-daemon22:54:461
28548211316,83sleep328563-21kernelversion23:20:083
28548211316,83sleep328563-21kernelversion23:20:083
2173321129,16sleep00-21swapper/021:15:070
2173321129,16sleep00-21swapper/021:15:070
23701210918,72sleep323782-21/usr/sbin/munin21:20:153
23735210833,44sleep00-21swapper/019:39:490
2339021087,48sleep00-21swapper/021:20:080
17834210811,83sleep217875-21cut22:45:152
10064210843,52sleep110099-21modprobe20:39:461
5382210723,64sleep25525-21unixbench_singl22:05:172
28394210611,79sleep328519-21timerwakeupswit21:35:173
14612210618,72sleep314619-21ntp_states19:10:123
5684210523,66sleep10-21swapper/120:24:561
32013210515,72sleep332130-21grep23:30:173
26778210516,33sleep20-21swapper/223:15:062
30572210213,75sleep030718-21unixbench_singl23:25:170
1722721028,16sleep10-21swapper/121:00:141
1722721028,16sleep10-21swapper/121:00:141
1721210216,64sleep31856-21grep20:10:173
15292210210,22sleep10-21swapper/120:55:131
15292210210,22sleep10-21swapper/120:55:131
4253210123,13sleep10-21swapper/123:46:501
32401210011,20sleep10-21swapper/121:49:571
13545210038,51sleep00-21swapper/022:32:450
13545210038,51sleep00-21swapper/022:32:450
11052210015,71sleep211198-21unixbench_singl00:05:172
1240329716,17sleep10-21swapper/120:45:151
36029415,63sleep2398-21/usr/sbin/munin21:50:102
2278529424,48sleep022926-21grep23:00:190
933429315,63sleep29402-21systemctl00:01:132
84629316,60sleep2869-21netstat23:35:082
2804529314,64sleep027992-21latency_hist23:19:480
2677129317,15sleep10-21swapper/123:15:051
2645629312,18sleep10-21swapper/123:14:491
2037629320,10sleep10-21swapper/119:28:111
2037629320,10sleep10-21swapper/119:28:111
1086729317,37sleep30-21swapper/322:24:573
2620529211,41sleep126294-21mailstats19:45:151
1556129212,65sleep315630-21/usr/sbin/munin00:20:163
857029127,41sleep08694-21latency_hist20:34:490
3064028912,63sleep130758-21latency_hist21:44:481
116272899,69sleep011635-21systemd-run00:09:470
384628821,46sleep314386-21systemd-journal20:19:463
384628821,46sleep314386-21systemd-journal20:19:463
3087828810,32sleep30-21swapper/321:45:023
1407428810,64sleep030095-21snmpd00:15:170
1407428810,64sleep030095-21snmpd00:15:170
2606528719,27sleep30-21swapper/319:45:093
1501328713,15sleep10-21swapper/100:19:581
1499128517,45sleep015010-21apt-get00:19:570
105728515,49sleep01147-21switchtime23:35:140
629028416,33sleep00-21swapper/022:10:030
3012828316,53sleep030152-21latency_hist19:59:480
2947728325,35sleep229632-21grep19:55:182
2896228317,42sleep029104-21grep23:20:170
2896228317,42sleep029104-21grep23:20:170
2667428312,58sleep226676-21sshd19:46:092
122032838,63sleep112202-21munin-run22:29:471
1059328311,44sleep10-21swapper/100:05:081
437028228,32sleep00-21swapper/022:04:460
2932728220,43sleep229302-21/usr/sbin/munin21:40:012
2096128247,16sleep10-21swapper/122:55:151
1302928232,25sleep30-21swapper/319:05:123
3083428111,57sleep230852-21open_inodes20:00:142
3083428111,57sleep230852-21open_inodes20:00:142
1424528124,29sleep00-21swapper/019:10:050
31002809,54sleep10-21swapper/121:59:591
1535328010,51sleep315423-21df20:56:003
1535328010,51sleep315423-21df20:56:003
2908127926,40sleep129099-21wc21:39:471
2783227912,50sleep227836-21diskstats21:35:042
2492627914,26sleep30-21swapper/321:25:083
129327912,51sleep01428-21cat21:54:490
618127825,35sleep10-21swapper/122:09:561
1744427812,46sleep20-21swapper/219:19:482
1481027824,23sleep30-21swapper/300:19:483
127922775,37sleep10-21swapper/100:10:251
242527616,41sleep22033-21/usr/sbin/munin23:40:092
242527616,41sleep22033-21/usr/sbin/munin23:40:092
1849627618,29sleep30-21swapper/321:05:043
3156027519,30sleep00-21swapper/021:45:200
156327524,34sleep10-21swapper/120:10:151
1384827517,31sleep30-21swapper/320:50:103
1384827517,31sleep30-21swapper/320:50:103
97627414,45sleep31055-21/usr/sbin/munin23:35:133
294352746,57sleep10-21swapper/123:23:251
294352746,57sleep10-21swapper/123:23:251
1746727420,29sleep30-21swapper/300:25:173
1558127419,42sleep015649-21latency_hist22:39:480
907527325,26sleep30-21swapper/320:35:083
2990127318,39sleep029997-21switchtime21:40:150
2087527318,38sleep220941-21fschecks_time19:30:062
2009227310,16sleep30-21swapper/300:35:113
2009227310,16sleep30-21swapper/300:35:113
196562737,10sleep10-21swapper/121:09:351
1208527319,30sleep20-21swapper/200:10:102
893427212,47sleep08941-21/usr/sbin/munin00:01:030
675327217,34sleep36825-21grep22:10:153
2004427221,24sleep20-21swapper/200:35:102
2004427221,24sleep20-21swapper/200:35:102
1586927218,32sleep20-21swapper/222:40:022
1214427218,27sleep30-21swapper/300:10:113
934427118,37sleep39338-21apt-get22:20:003
462727012,31sleep20-21swapper/220:20:132
2590727013,33sleep00-21swapper/019:45:060
1653927021,27sleep20-21swapper/220:59:462
1653927021,27sleep20-21swapper/220:59:462
233226922,28sleep20-21swapper/220:14:482
1725926922,27sleep20-21swapper/221:00:152
1725926922,27sleep20-21swapper/221:00:152
1305626912,43sleep313151-21timerandwakeup22:30:183
1305626912,43sleep313151-21timerandwakeup22:30:183
1283626923,31sleep212891-21latency19:05:072
1056126925,22sleep30-21swapper/320:40:053
1056126925,22sleep30-21swapper/320:40:053
404026816,38sleep314386-21systemd-journal22:00:413
2511626821,26sleep00-21swapper/021:25:130
232026812,16sleep00-21swapper/023:40:070
232026812,16sleep00-21swapper/023:40:070
2279226823,22sleep30-21swapper/321:19:463
2279226823,22sleep30-21swapper/321:19:463
2201526810,13sleep30-21swapper/322:59:573
2006226818,27sleep30-21swapper/321:10:053
1745426812,22sleep30-21swapper/321:00:203
1745426812,22sleep30-21swapper/321:00:203
1293826824,27sleep00-21swapper/019:05:090
703526720,27sleep30-21swapper/323:55:043
1713026725,23sleep00-21swapper/022:44:480
1628526716,36sleep016077-21/usr/sbin/munin19:15:060
566426612,22sleep30-21swapper/323:50:073
3010126624,27sleep30-21swapper/323:25:083
2361226620,21sleep20-21swapper/221:20:142
515926510,40sleep05290-21latency_hist23:49:480
2926326514,26sleep30-21swapper/319:55:153
2822426514,37sleep228225-21sshd19:51:102
2414526521,28sleep323614-21apt-get19:40:003
2260526520,24sleep30-21swapper/319:35:083
1069826515,21sleep30-21swapper/300:05:103
80822646,12sleep30-21swapper/320:30:203
77532648,43sleep20-21swapper/222:14:582
620526415,21sleep00-21swapper/020:25:160
2850826423,27sleep310202-21kworker/u8:119:54:463
1785526413,37sleep012-21rcu_sched21:01:220
1785526413,37sleep012-21rcu_sched21:01:220
152926415,21sleep00-21swapper/021:55:010
1169426417,32sleep211833-21latency_hist20:44:482
1169426417,32sleep211833-21latency_hist20:44:482
820626317,28sleep01-21systemd23:59:450
483226319,23sleep30-21swapper/322:05:063
460726322,26sleep00-21swapper/020:20:110
3054026313,35sleep330591-21/usr/sbin/munin20:00:073
3054026313,35sleep330591-21/usr/sbin/munin20:00:073
2958426315,33sleep20-21swapper/223:24:472
2958426315,33sleep20-21swapper/223:24:472
2716026318,24sleep30-21swapper/323:15:153
2132426310,29sleep00-21swapper/019:30:160
94032629,26sleep30-21swapper/300:01:143
466726222,19sleep30-21swapper/320:20:143
3114226217,21sleep20-21swapper/223:29:472
773026119,19sleep00-21swapper/022:14:560
2523326121,20sleep20-21swapper/223:10:052
2046726115,22sleep30-21swapper/322:55:013
1937726110,15sleep00-21swapper/022:50:130
139726116,21sleep00-21swapper/020:10:100
346726017,28sleep23491-21netstat22:00:082
3094926018,25sleep20-21swapper/221:45:052
2751426017,21sleep20-21swapper/221:34:482
1952226020,20sleep30-21swapper/322:50:163
1477526016,20sleep20-21swapper/200:19:472
1477526016,20sleep20-21swapper/200:19:472
802025915,30sleep38032-21cut22:15:073
317925912,34sleep23255-21awk20:15:162
317925912,34sleep23255-21awk20:15:162
245725915,18sleep30-21swapper/323:40:103
245725915,18sleep30-21swapper/323:40:103
2384525912,33sleep023868-21irqstats23:05:070
2384525912,33sleep023868-21irqstats23:05:070
1975025915,22sleep30-21swapper/319:25:183
1975025915,22sleep30-21swapper/319:25:183
1851925915,27sleep018568-21/usr/sbin/munin00:30:110
1851925915,27sleep018568-21/usr/sbin/munin00:30:110
1786125912,34sleep017872-21systemd-run00:29:460
1045325912,26sleep00-21swapper/020:40:020
1045325912,26sleep00-21swapper/020:40:020
721225814,19sleep20-21swapper/223:55:072
465825814,27sleep04653-21apt-get22:04:580
2083425814,20sleep00-21swapper/022:55:100
1781525814,28sleep017821-21ntp_states22:45:130
1271225811,31sleep212417-21/usr/sbin/munin22:30:092
1271225811,31sleep212417-21/usr/sbin/munin22:30:092
880625721,18sleep00-21swapper/020:35:010
57525713,24sleep00-21swapper/020:07:430
2288525714,23sleep20-21swapper/219:35:152
204672574,14sleep20-21swapper/219:29:212
204672574,14sleep20-21swapper/219:29:212
1866925718,17sleep30-21swapper/322:49:483
1820225715,26sleep20-21swapper/200:30:052
1820225715,26sleep20-21swapper/200:30:052
1778225718,17sleep20-21swapper/219:20:042
1666225714,26sleep216759-21fschecks_time00:25:042
150125713,29sleep31503-21/usr/sbin/munin21:54:563
143432577,12sleep30-21swapper/300:15:203
59402569,12sleep20-21swapper/223:50:142
4252565,12sleep20-21swapper/220:05:522
3200525615,25sleep332011-21diskstats20:05:043
2772225616,18sleep00-21swapper/019:50:150
1897025613,16sleep30-21swapper/300:30:183
1897025613,16sleep30-21swapper/300:30:183
1768725614,25sleep017682-21apt-get19:19:580
1610325617,17sleep30-21swapper/319:15:003
3250225516,24sleep332508-21diskstats21:50:033
3145925513,24sleep231466-21diskstats23:30:042
2932325514,27sleep329320-21apt-get21:39:583
2538325516,18sleep00-21swapper/023:10:070
235732556,17sleep20-21swapper/223:04:572
235732556,17sleep20-21swapper/223:04:572
2259625514,25sleep20-21swapper/223:00:172
1589725510,22sleep20-21swapper/200:20:192
1190525511,30sleep011924-21apt-get20:44:580
1137725518,23sleep011459-21mailstats22:25:140
92842546,12sleep20-21swapper/222:19:512
598425413,22sleep26077-21memory20:25:102
2852225411,20sleep00-21swapper/021:35:180
2364425412,26sleep323670-21df23:05:023
2364425412,26sleep323670-21df23:05:023
1421125414,18sleep20-21swapper/220:50:172
1421125414,18sleep20-21swapper/220:50:172
1002525414,22sleep00-21swapper/022:20:170
83952535,37sleep20-21swapper/220:33:132
242102528,13sleep00-21swapper/019:40:460
2161125217,19sleep20-21swapper/221:15:052
2161125217,19sleep20-21swapper/221:15:052
2024525217,16sleep20-21swapper/221:10:082
2022925213,14sleep20-21swapper/222:54:472
1808225216,16sleep30-21swapper/319:20:093
1256625217,18sleep10-21swapper/119:04:581
3264125114,24sleep01-21systemd23:34:460
144912515,35sleep00-21swapper/020:51:280
144912515,35sleep00-21swapper/020:51:280
580625013,22sleep35905-21fschecks_time20:25:073
2532025019,18sleep30-21swapper/323:10:063
1932725012,23sleep019328-21diskstats19:25:070
1932725012,23sleep019328-21diskstats19:25:070
1589525012,23sleep315898-21df_inode22:40:033
3174924916,18sleep00-21swapper/020:04:490
1324124916,18sleep20-21swapper/220:49:492
113452499,17sleep20-21swapper/222:25:122
904124811,19sleep20-21swapper/220:35:072
3090824811,21sleep030977-21sensors_temp20:00:170
1464224713,16sleep30-21swapper/322:36:483
304424611,15sleep00-21swapper/020:15:140
304424611,15sleep00-21swapper/020:15:140
2647624612,16sleep00-21swapper/021:30:090
2163824611,16sleep30-21swapper/319:30:193
1468624614,16sleep20-21swapper/219:10:152
2660524316,14sleep30-21swapper/321:30:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional