You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 22:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Fri Mar 20, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19741210168,992sleep019788-21ntpq09:33:250
6025280918,773sleep36088-21/usr/sbin/munin10:33:043
17508267512,604sleep010773-21kworker/u8:109:28:000
1883926207,601sleep018875-21ntpq11:13:140
2935225756,551sleep329364-21/usr/sbin/munin08:23:073
2032225539,530sleep120358-21ntpq07:53:211
12512241413,385sleep212586-21find07:28:182
23083241117,380sleep323111-21memory09:43:543
1000624016,357sleep010042-21ntpq12:28:110
8303235548,266sleep08394-21xargs08:58:200
23113232610,143sleep023158-21awk09:43:550
31185230523,48sleep00-21swapper/010:08:180
31185230523,48sleep00-21swapper/010:08:180
12473230029,235sleep013441-21snmpd12:35:220
25097228022,210sleep00-21swapper/008:08:180
9327226923,230sleep19357-21awk12:27:481
8713225684,112sleep00-21swapper/012:23:190
28191225453,152sleep028205-21fschecks_count11:43:420
13342225070,54sleep00-21swapper/009:13:180
3001122448,223sleep030060-21grep11:48:140
1179223244,176sleep01249-21unin-run08:37:490
24694222043,63sleep00-21swapper/009:48:150
14368221540,162sleep014446-21switchtime10:58:160
485522128,189sleep04901-21ntp_kernel_pll_10:28:120
27281220977,114sleep027296-21modprobe09:57:480
9511220533,112sleep00-21swapper/009:03:050
3712220515,166sleep03853-21unixbench_multi08:43:180
2100219577,99sleep00-21swapper/008:38:180
30529219318,162sleep130573-21/usr/sbin/munin08:27:491
21862219247,88sleep00-21swapper/007:58:120
21072219228,101sleep00-21swapper/009:38:100
21072219228,101sleep00-21swapper/009:38:100
22376219117,156sleep222181-21latency_hist09:42:522
22376219117,156sleep222181-21latency_hist09:42:522
9143218216,150sleep39181-21wc07:18:543
7620217743,122sleep07743-21cat07:13:180
6283217511,102sleep06406-21latency_hist12:17:500
6283217511,102sleep06406-21latency_hist12:17:500
27528216620,124sleep327550-21sh11:42:483
27528216620,124sleep327550-21sh11:42:483
13664216635,80sleep00-21swapper/007:33:090
6742216217,19sleep00-21swapper/008:53:200
14566216213,132sleep114610-21ntpq09:18:121
17910216115,132sleep217953-21tr09:28:182
2027621605,112sleep00-21swapper/011:18:100
9944215826,54sleep013441-21snmpd07:21:090
14284215845,53sleep00-21swapper/009:18:070
12880215813,128sleep312951-21latency09:13:093
1999221578,49sleep00-21swapper/007:53:080
1789421458,61sleep00-21swapper/007:47:500
890721418,119sleep08966-21cat10:42:500
4561213818,84sleep30-21swapper/310:28:073
1044921378,93sleep010505-21tune2fs07:23:070
18147213630,56sleep013441-21snmpd11:12:250
15745213633,45sleep00-21swapper/009:23:030
5811213515,54sleep00-21swapper/007:08:210
3541213531,67sleep30-21swapper/307:03:083
29270213469,52sleep00-21swapper/008:23:040
24862213121,91sleep324980-21wc09:48:183
5599212931,18sleep00-21swapper/008:49:360
2254921275,31sleep00-21swapper/011:23:480
8220212526,81sleep38299-21mailstats08:58:193
29535212558,29sleep029576-21ntp_kernel_pll_10:03:170
24847212310,58sleep024917-21idleruntime11:33:080
1138212315,95sleep31065-21kworker/u8:310:17:503
1138212315,95sleep31065-21kworker/u8:310:17:503
28320211929,73sleep013441-21snmpd09:58:200
18848211927,68sleep118934-21latency_hist09:32:491
7718211711,77sleep20-21swapper/208:58:062
3465211716,82sleep23537-21fschecks_time12:08:072
23947211430,27sleep20-21swapper/209:47:482
20738211423,68sleep220871-21sed11:18:212
360211314,23sleep3387-21proc_pri08:33:163
17135211311,86sleep317034-21latency_hist09:27:513
27408211124,67sleep227451-21systemctl08:17:482
26227210914,81sleep126263-21fschecks_count08:13:071
21148210911,82sleep321150-21sshd07:57:273
18275210910,18sleep00-21swapper/007:48:070
18275210910,18sleep00-21swapper/007:48:070
4667210817,73sleep14707-21fschecks_count08:48:081
10780210831,34sleep010857-21fschecks_time10:48:070
7802210714,79sleep17853-21fschecks_count08:58:091
8556210612,77sleep28642-21mailstats12:23:172
29897210316,46sleep330025-21unixbench_singl10:03:243
1157421039,37sleep00-21swapper/009:08:180
1157421039,37sleep00-21swapper/009:08:180
701210219,14sleep00-21swapper/010:13:220
701210219,14sleep00-21swapper/010:13:220
31345210220,65sleep231476-21timerwakeupswit10:08:202
31345210220,65sleep231476-21timerwakeupswit10:08:202
2687521027,45sleep026949-21find11:38:170
2687521027,45sleep026949-21find11:38:170
23238210248,39sleep00-21swapper/011:28:070
18739210216,71sleep218793-21latency11:13:122
24780210125,57sleep324843-21cut11:33:073
1827210034,18sleep00-21swapper/012:03:000
16160210013,34sleep216195-21ntp_states09:23:142
135021008,36sleep00-21swapper/010:17:590
2430829940,15sleep00-21swapper/008:07:490
2827429814,72sleep328345-21expr08:18:183
2058529814,63sleep320614-21munin-plugin-st09:37:483
91212977,28sleep00-21swapper/010:42:580
417929613,70sleep14229-21unin-run10:27:491
300429623,51sleep23036-21tune2fs10:23:052
1559029634,14sleep00-21swapper/011:03:080
154502956,34sleep00-21swapper/007:38:140
110692956,16sleep20-21swapper/209:08:032
110692956,16sleep20-21swapper/209:08:032
628929412,62sleep15999-21/usr/sbin/munin10:33:101
1592129315,64sleep115976-21idleruntime09:23:091
636229237,26sleep20-21swapper/207:12:522
39142925,14sleep00-21swapper/010:24:250
378629235,15sleep00-21swapper/012:08:130
2674629244,14sleep00-21swapper/008:13:180
3115629112,64sleep131226-21df11:53:041
1191129136,13sleep00-21swapper/007:28:040
636729012,62sleep36406-21uname08:53:113
1932329017,37sleep20-21swapper/209:33:032
954328914,60sleep29586-21ntp_kernel_pll_10:43:122
2430228917,49sleep124309-21/usr/sbin/munin09:48:041
2125228816,55sleep321270-21/usr/sbin/munin09:38:133
2125228816,55sleep321270-21/usr/sbin/munin09:38:133
1102928826,44sleep311085-21apt-key12:32:513
65512867,18sleep00-21swapper/010:33:170
3201128630,23sleep013441-21snmpd08:32:000
2267228613,57sleep122172-21apt-get09:42:591
1413728616,45sleep214270-21unixbench_multi07:33:192
1083228620,18sleep10-21swapper/109:07:501
778028520,48sleep1222-21systemd-journal12:22:471
468728521,42sleep14665-21latency_hist12:12:501
382628518,49sleep03431-21/usr/sbin/munin07:03:130
314762848,55sleep00-21swapper/011:53:110
180728311,17sleep20-21swapper/210:18:132
1258928328,30sleep312733-21grep07:28:193
843028220,48sleep18516-21mktemp07:17:511
2602328113,49sleep126035-21latency_hist11:37:491
2591228125,22sleep20-21swapper/208:12:522
2304828112,32sleep20-21swapper/208:03:032
130372817,15sleep00-21swapper/010:53:190
629728025,35sleep30-21swapper/307:12:493
2066528026,29sleep10-21swapper/109:37:501
2903427910,40sleep00-21swapper/008:22:510
314127818,28sleep20-21swapper/212:07:502
3021327821,40sleep130290-21timerandwakeup11:48:191
2961027813,46sleep329680-21/usr/sbin/munin11:48:053
18342780,6sleep10-21swapper/108:38:101
105627825,33sleep11071-21modprobe10:17:481
105627825,33sleep11071-21modprobe10:17:481
745027718,11sleep00-21swapper/012:18:300
1750127715,45sleep317505-21acpi09:27:593
63927614,44sleep1652-21/usr/sbin/munin11:58:131
3121627621,40sleep131261-21ntpq08:28:131
270162766,16sleep10-21swapper/109:54:261
2160427612,49sleep121625-21cpu11:23:041
1881027623,31sleep30-21swapper/307:48:183
1881027623,31sleep30-21swapper/307:48:183
1129827526,26sleep30-21swapper/310:48:183
851727326,28sleep20-21swapper/207:17:522
786427321,39sleep37877-21kernelversion10:38:133
3270227318,10sleep00-21swapper/011:57:590
2530327312,27sleep20-21swapper/211:33:182
1549527316,43sleep21-21systemd09:22:492
1166727311,46sleep111745-21find09:08:191
1166727311,46sleep111745-21find09:08:191
1062927320,26sleep30-21swapper/307:23:103
363227210,50sleep33710-21xargs08:43:173
2759927227,23sleep10-21swapper/111:42:501
2759927227,23sleep10-21swapper/111:42:501
2603627221,35sleep326071-21grep09:53:153
151322729,20sleep20-21swapper/211:02:502
1512127220,34sleep215181-21/usr/sbin/munin07:38:072
479327114,26sleep30-21swapper/308:48:103
2939427117,27sleep10-21swapper/111:47:511
2673927126,23sleep20-21swapper/211:38:152
2673927126,23sleep20-21swapper/211:38:152
2355827122,25sleep20-21swapper/211:28:142
1317127122,31sleep10-21swapper/107:32:491
1110027112,43sleep211145-21/usr/sbin/munin10:48:132
1825427022,26sleep10-21swapper/111:12:501
3212726920,18sleep20-21swapper/208:32:512
3106526928,21sleep30-21swapper/310:08:133
3106526928,21sleep30-21swapper/310:08:133
908426818,22sleep20-21swapper/207:18:532
2768026824,29sleep327687-21/usr/sbin/munin09:58:073
2209826816,19sleep20-21swapper/207:58:182
1855626820,26sleep30-21swapper/311:13:093
125126817,27sleep20-21swapper/208:37:512
1006526824,19sleep20-21swapper/209:03:232
937826718,19sleep30-21swapper/312:27:503
274526715,38sleep22752-21modprobe08:42:482
152132677,11sleep30-21swapper/311:02:513
992026615,22sleep10-21swapper/110:43:191
579726617,34sleep25940-21sed10:32:512
2585326621,28sleep325879-21munin-run08:12:493
1357726624,19sleep20-21swapper/210:57:502
47526518,24sleep10-21swapper/108:33:181
3149226520,25sleep30-21swapper/308:28:193
145826521,19sleep30-21swapper/310:18:063
429126422,23sleep20-21swapper/208:47:492
2891426416,25sleep20-21swapper/210:02:502
2435226425,25sleep10-21swapper/108:07:501
2052526412,35sleep320563-21/usr/sbin/munin11:18:183
47092635,12sleep20-21swapper/208:48:092
1564126323,23sleep10-21swapper/111:03:091
1268426316,28sleep213441-21snmpd09:13:042
1162126317,21sleep10-21swapper/107:27:491
631926214,25sleep20-21swapper/208:53:102
2831326214,23sleep20-21swapper/211:43:442
180572628,10sleep30-21swapper/311:11:153
162126221,19sleep30-21swapper/312:02:513
1444926220,24sleep10-21swapper/110:58:171
1343226213,18sleep30-21swapper/307:33:003
1221526218,27sleep312217-21/usr/sbin/munin10:52:583
1076526214,31sleep210810-21/usr/sbin/munin07:23:132
372526121,22sleep23804-21cat07:03:112
305352617,11sleep20-21swapper/211:48:332
2942726116,28sleep229519-21/usr/sbin/munin10:03:152
2003026115,27sleep10-21swapper/111:17:591
1713526126,20sleep20-21swapper/207:43:202
1480326115,20sleep30-21swapper/309:18:183
1404726117,28sleep10-21swapper/107:33:181
1027426124,18sleep20-21swapper/212:28:172
806926020,18sleep10-21swapper/110:38:191
76026019,19sleep20-21swapper/211:58:182
503926017,26sleep15034-21apt-get07:08:021
2735926021,17sleep20-21swapper/209:57:502
2178726022,23sleep20-21swapper/211:23:092
190026017,25sleep31937-21ntp_states08:38:133
1087526017,25sleep10-21swapper/112:32:491
826025918,24sleep38283-21munin-run07:17:493
2146825921,19sleep30-21swapper/307:57:593
1629425912,32sleep316324-21munin-run07:42:493
530025811,19sleep10-21swapper/112:13:101
530025811,19sleep10-21swapper/112:13:101
492925817,27sleep34925-21apt-get12:13:003
492925817,27sleep34925-21apt-get12:13:003
422825817,24sleep34281-21users12:08:203
2809125817,22sleep10-21swapper/108:18:121
2590525819,20sleep10-21swapper/108:12:511
2478425820,21sleep20-21swapper/209:48:172
2427825819,19sleep30-21swapper/308:07:283
1962325825,18sleep119817-21sed07:52:521
1962325825,18sleep119817-21sed07:52:521
1706725813,21sleep10-21swapper/107:43:191
156925820,17sleep10-21swapper/110:18:081
3249025715,25sleep332600-21latency_hist11:57:503
2805025715,25sleep327542-21apt-get11:43:003
2315325714,17sleep10-21swapper/111:28:041
1839725721,23sleep218451-21sed07:48:092
1839725721,23sleep218451-21sed07:48:092
174972576,10sleep30-21swapper/307:43:253
610825610,32sleep26136-21cut10:33:072
119482568,12sleep20-21swapper/212:34:072
1085925614,28sleep110918-21cat10:48:081
880725516,25sleep18279-21apt-get07:17:591
1534525517,22sleep215364-21apt-get11:02:582
1429825516,24sleep314365-21mailstats10:58:153
950925416,21sleep39504-21apt-get09:03:043
914625412,25sleep39142-21apt-get10:43:013
2796625413,21sleep10-21swapper/109:58:131
2276525419,18sleep10-21swapper/108:02:491
2144725412,23sleep121515-21/usr/sbin/munin09:38:191
2144725412,23sleep121515-21/usr/sbin/munin09:38:191
182525416,23sleep31821-21apt-get12:02:593
1530425421,16sleep10-21swapper/107:38:111
1143725414,16sleep30-21swapper/309:08:133
1143725414,16sleep30-21swapper/309:08:133
367525313,17sleep10-21swapper/107:03:101
303425316,23sleep13101-21df08:43:031
264625313,23sleep12729-21users12:03:171
1981825311,24sleep219402-21apt-get07:52:532
1137225310,25sleep30-21swapper/312:32:593
338525214,19sleep30-21swapper/310:23:153
1143625213,22sleep111390-21apt-get12:33:251
739725111,23sleep17420-21munin-run08:57:491
3200825114,15sleep10-21swapper/110:12:511
3200825114,15sleep10-21swapper/110:12:511
3057425113,14sleep20-21swapper/208:27:502
3046325116,17sleep10-21swapper/110:07:501
2772125114,21sleep227718-21apt-get08:18:012
194452515,34sleep10-21swapper/111:14:121
3114925010,23sleep231145-21apt-get11:53:012
1223925012,22sleep112236-21apt-get10:53:001
651924913,20sleep26516-21apt-get12:18:002
577624916,15sleep10-21swapper/110:32:491
2654224914,18sleep30-21swapper/308:13:133
2342024918,14sleep30-21swapper/311:28:113
2290924918,15sleep30-21swapper/311:27:503
1278124913,21sleep112838-21idleruntime09:13:071
107524913,17sleep20-21swapper/210:17:492
107524913,17sleep20-21swapper/210:17:492
543424815,16sleep20-21swapper/212:13:152
543424815,16sleep20-21swapper/212:13:152
1689624710,21sleep216915-21apt-get11:07:592
709624310,17sleep36498-21/usr/sbin/munin12:18:173
760223810,15sleep20-21swapper/210:38:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional