You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 16:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Nov 20, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
84132614620,6113sleep03702-21kworker/0:008:46:220
23424262436,574sleep123472-21unin-run11:18:451
20762253415,503sleep220820-21/usr/sbin/munin11:09:052
2198324088,387sleep122019-21ntpq09:29:061
2198324088,387sleep122019-21ntpq09:29:061
29496235513,324sleep029599-21/usr/sbin/munin08:09:320
19394234119,309sleep119432-21fschecks_count07:39:161
11651233836,289sleep111692-21ntpq07:14:091
76802313102,186sleep17786-21timerandwakeup12:09:161
51272279104,63sleep15346-21latency_hist12:03:471
25650225112,220sleep125688-21fschecks_time07:59:071
25650225112,220sleep125688-21fschecks_time07:59:071
32752224335,35sleep10-21swapper/110:03:591
19019224130,28sleep10-21swapper/107:38:481
10089223412,208sleep010127-21ntpq10:34:100
2412221591,27sleep10-21swapper/108:28:461
13401221313,179sleep113509-21timerandwakeup10:44:171
9778221017,178sleep09803-21fschecks_count07:09:070
9778221017,178sleep09803-21fschecks_count07:09:070
2927220242,26sleep12630-21/usr/sbin/munin08:29:111
11512220211,176sleep011562-21sed10:39:070
5191219964,27sleep10-21swapper/110:19:051
30406217718,99sleep10-21swapper/111:39:161
25799217110,146sleep025870-21sensors11:24:170
1307121709,115sleep113175-21latency_hist12:28:461
2845421689,102sleep10-21swapper/109:49:171
4811216610,144sleep14870-21latency_hist10:18:491
28148216511,138sleep128288-21latency_hist11:33:471
1156321617,32sleep10-21swapper/110:39:081
744821508,94sleep17464-21vmstat10:24:181
744821508,94sleep17464-21vmstat10:24:181
29802214941,89sleep10-21swapper/109:54:101
30133214711,28sleep130174-21basename08:13:441
22097214712,86sleep1284-21dbus-daemon07:48:451
31249214611,120sleep331210-21apt-get09:59:203
1185214518,108sleep21279-21cut08:24:032
11273214339,87sleep111348-21mailstats08:54:511
11273214339,87sleep111348-21mailstats08:54:511
1525321424,14sleep30-21swapper/310:49:193
10923214016,102sleep10-21swapper/110:38:231
7395213930,88sleep37435-21idleruntime-cro07:03:453
624021369,84sleep16276-21ps08:39:121
624021369,84sleep16276-21ps08:39:121
3266021349,114sleep1362-21snmpd11:47:361
7092213014,93sleep03919-21kworker/u8:008:43:470
7092213014,93sleep03919-21kworker/u8:008:43:470
31464212919,91sleep231483-21apt-get11:43:572
3169221276,10sleep10-21swapper/108:18:291
937421268,11sleep10-21swapper/110:33:431
2484821258,106sleep1362-21snmpd11:22:081
20665212417,80sleep120759-21fschecks_time11:09:041
365212313,95sleep2394-21sh10:04:042
17364212313,95sleep117407-21ntp_kernel_pll_09:14:091
9590211818,64sleep10-21swapper/112:14:331
19263211619,76sleep018773-21apt-get07:38:550
13301211639,61sleep113367-21sendmail_mailqu07:19:131
11694211611,79sleep211726-21ntp_states07:14:102
16289211216,76sleep30-21swapper/309:09:223
2110621117,29sleep10-21swapper/109:25:511
2110621117,29sleep10-21swapper/109:25:511
1691621106,12sleep10-21swapper/110:55:151
14989210910,81sleep10-21swapper/107:24:181
10831210918,65sleep00-21swapper/008:54:370
10831210918,65sleep00-21swapper/008:54:370
6593210818,68sleep10-21swapper/106:59:111
26568210815,41sleep10-21swapper/111:28:461
162621085,36sleep10-21swapper/111:50:511
9372210749,43sleep19437-21ps08:49:221
13785210711,75sleep10-21swapper/112:29:141
1069721075,29sleep10-21swapper/107:10:331
1069721075,29sleep10-21swapper/107:10:331
1593021057,50sleep115920-21apt-get07:29:031
23202210443,43sleep10-21swapper/109:34:031
19506210310,75sleep10-21swapper/111:04:111
19506210310,75sleep10-21swapper/111:04:111
27541210219,63sleep327048-21/usr/sbin/munin08:04:143
14799210211,79sleep014802-21ntp_states10:49:120
773821019,78sleep37782-21ntp_offset08:44:113
5915210130,55sleep10-21swapper/112:04:271
32749210120,65sleep032761-21modprobe11:48:440
29426210118,68sleep029479-21ls09:54:000
17733210111,39sleep10-21swapper/110:59:061
17733210111,39sleep10-21swapper/110:59:061
1459210014,66sleep30-21swapper/311:49:153
3179429811,68sleep331947-21latency_hist08:18:473
2773829815,58sleep326786-21/usr/sbin/munin11:29:413
840529621,50sleep28542-21unixbench_multi07:04:162
559929620,54sleep25612-21fschecks_count12:04:042
2838729620,46sleep328384-21apt-get11:33:573
236472967,18sleep00-21swapper/007:53:450
1380729621,46sleep213826-21apt-get09:03:562
270722958,14sleep10-21swapper/108:03:591
2015329515,62sleep320210-21fschecks_count09:24:043
2015329515,62sleep320210-21fschecks_count09:24:043
904529415,34sleep19163-21latency_hist07:08:481
843129428,55sleep10-21swapper/108:46:351
1912229414,55sleep219208-21ls09:19:142
431229311,25sleep30-21swapper/308:34:013
431229311,25sleep30-21swapper/308:34:013
3185829321,52sleep331465-21/usr/sbin/munin11:44:103
2965129328,42sleep229682-21idleruntime-cro11:38:442
216329338,43sleep12158-21apt-get11:53:581
1437729314,66sleep0362-21snmpd07:24:010
2925829213,61sleep228843-21/usr/sbin/munin08:09:252
267632929,14sleep10-21swapper/109:44:161
2306329226,23sleep10-21swapper/107:49:141
1226929211,31sleep10-21swapper/108:58:591
884829128,42sleep38999-21sed10:29:173
527929121,53sleep35346-21sort10:19:073
2847429127,26sleep10-21swapper/111:34:031
2528629135,42sleep125431-21sed07:58:461
1020729134,15sleep10-21swapper/112:18:571
3253429033,40sleep10-21swapper/110:03:471
2043629012,65sleep020479-21tr09:24:090
2043629012,65sleep020479-21tr09:24:090
978728915,59sleep39883-21grep10:34:053
674328923,23sleep00-21swapper/006:59:140
1511128818,52sleep215259-21grep07:24:192
128252887,19sleep30-21swapper/307:18:593
2380128724,18sleep30-21swapper/311:19:073
325932857,29sleep10-21swapper/108:19:181
2785228512,56sleep327922-21/usr/sbin/munin09:49:033
1978028512,58sleep119829-21unin-run09:23:451
2627228421,33sleep20-21swapper/209:43:572
2552428417,50sleep225578-21sort07:59:002
2552428417,50sleep225578-21sort07:59:002
158462846,32sleep10-21swapper/109:09:161
1177528415,19sleep30-21swapper/310:39:143
700128341,24sleep20-21swapper/206:59:162
2295828314,16sleep00-21swapper/009:33:460
2295828314,16sleep00-21swapper/009:33:460
429228227,14sleep10-21swapper/108:33:591
429228227,14sleep10-21swapper/108:33:591
206832820,7sleep20-21swapper/207:43:482
1590028212,52sleep215902-21/usr/sbin/munin07:28:552
151632828,22sleep10-21swapper/110:49:171
1410128112,55sleep314139-21latency_hist10:48:473
996028015,45sleep29988-21latency_hist12:18:462
2793628028,27sleep20-21swapper/209:49:062
1718128020,42sleep317205-21munin-run07:33:453
1279628023,42sleep212823-21grep10:44:012
503927922,38sleep05125-21grep12:03:460
214627928,13sleep10-21swapper/110:09:081
2660527824,31sleep20-21swapper/211:28:472
258352786,30sleep10-21swapper/109:41:301
149932786,33sleep10-21swapper/109:07:261
2225027720,38sleep222338-21fschecks_time11:14:052
1245027721,16sleep20-21swapper/208:59:042
1183827715,48sleep211842-21diskstats12:24:022
2526227619,43sleep225340-21mailstats09:39:212
54232759,13sleep20-21swapper/210:19:102
3083927519,42sleep230848-21vmstat11:39:212
2738827515,28sleep00-21swapper/011:29:380
3082527424,25sleep20-21swapper/209:58:472
3021327417,36sleep330272-21/usr/sbin/munin11:39:103
2758527421,34sleep227736-21head11:29:402
272827414,27sleep00-21swapper/008:29:060
2558227414,43sleep325602-21df_inode07:59:013
2558227414,43sleep325602-21df_inode07:59:013
945527318,39sleep09517-21sed10:33:460
549127319,40sleep25527-21basename08:38:452
549127319,40sleep25527-21basename08:38:452
1818527318,33sleep218332-21grep07:34:172
1407927319,36sleep014084-21modprobe07:23:450
617527218,34sleep26219-21ntp_kernel_pll_08:39:092
617527218,34sleep26219-21ntp_kernel_pll_08:39:092
2387127216,25sleep00-21swapper/011:19:080
11762727,48sleep00-21swapper/008:24:010
2057427111,45sleep00-21swapper/011:08:590
1927127114,23sleep30-21swapper/307:38:583
1828027118,33sleep318370-21ls09:18:473
1514427111,36sleep00-21swapper/009:08:470
143422717,52sleep20-21swapper/212:30:492
1135027115,37sleep211445-21/usr/sbin/munin08:54:522
1135027115,37sleep211445-21/usr/sbin/munin08:54:522
372427013,42sleep23726-21grep11:58:592
2567427012,45sleep3362-21snmpd09:39:263
264632695,11sleep30-21swapper/311:27:593
235026917,33sleep30-21swapper/311:54:083
2091226917,37sleep220918-21df_inode07:44:022
2033626910,13sleep20-21swapper/209:24:072
2033626910,13sleep20-21swapper/209:24:072
169432696,12sleep00-21swapper/007:30:390
1464626912,14sleep00-21swapper/012:33:450
112752695,12sleep00-21swapper/012:21:140
1088126911,39sleep310944-21/usr/sbin/munin08:54:403
1088126911,39sleep310944-21/usr/sbin/munin08:54:403
715026814,24sleep20-21swapper/212:09:052
406626823,23sleep00-21swapper/010:14:150
1319126822,20sleep00-21swapper/008:59:220
1154526815,33sleep311251-21/usr/sbin/munin07:14:073
626826715,25sleep00-21swapper/010:23:440
498426720,29sleep35029-21apt-key12:03:453
42292677,43sleep00-21swapper/008:33:580
42292677,43sleep00-21swapper/008:33:580
309132678,40sleep20-21swapper/208:14:142
2263126715,34sleep322691-21/usr/sbin/munin11:14:153
1619226719,28sleep00-21swapper/010:54:090
1433026718,33sleep314408-21mailstats09:04:133
1184326722,22sleep30-21swapper/312:24:033
3094526612,21sleep00-21swapper/008:14:160
287772669,44sleep028778-21sshd09:49:280
2484226618,30sleep024874-21tune2fs09:39:110
196626618,23sleep20-21swapper/211:53:462
1031526617,20sleep20-21swapper/208:53:472
871526511,39sleep08716-21sshd07:05:190
551326521,22sleep00-21swapper/012:03:580
265526519,27sleep20-21swapper/208:29:042
2398426522,23sleep20-21swapper/211:19:102
874826420,25sleep00-21swapper/012:14:030
682726420,22sleep30-21swapper/310:24:083
682726420,22sleep30-21swapper/310:24:083
2950826412,37sleep329510-21/usr/sbin/munin09:54:043
2848526412,36sleep028702-21cut08:08:470
2397026425,23sleep30-21swapper/307:54:003
2206726412,36sleep322073-21ntp_states09:29:083
2206726412,36sleep322073-21ntp_states09:29:083
1750726416,30sleep317561-21sort07:34:013
1599426414,31sleep30-21swapper/310:54:043
1888626314,18sleep00-21swapper/009:19:080
810026117,23sleep00-21swapper/012:10:440
6242615,14sleep20-21swapper/208:21:212
59726114,29sleep0691-21/usr/sbin/munin10:04:100
2372626110,26sleep20-21swapper/207:53:472
2084026120,25sleep320836-21apt-get07:43:593
1725826119,18sleep20-21swapper/210:58:472
142442618,11sleep30-21swapper/312:29:373
2761026024,17sleep30-21swapper/309:48:473
2192426019,19sleep30-21swapper/311:13:463
1059026018,22sleep20-21swapper/207:09:212
1059026018,22sleep20-21swapper/207:09:212
2833425920,25sleep228387-21apt-config08:08:452
255325913,22sleep20-21swapper/210:09:172
2377525916,20sleep30-21swapper/309:34:173
1665125917,26sleep20-21swapper/209:13:452
937525813,26sleep29380-21modprobe10:33:442
3196925816,25sleep032127-21sed10:00:020
236125815,25sleep3225-21systemd-journal08:28:443
1983125818,16sleep30-21swapper/309:23:483
1943925815,25sleep219475-21ntp_states11:04:082
1943925815,25sleep219475-21ntp_states11:04:082
1485025820,16sleep20-21swapper/210:49:142
1023925819,25sleep310306-21/usr/sbin/munin07:09:183
1023925819,25sleep310306-21/usr/sbin/munin07:09:183
30825717,24sleep3304-21apt-get10:04:013
1284125716,18sleep30-21swapper/308:59:133
1020425715,25sleep310201-21apt-get12:18:553
363125617,23sleep20-21swapper/210:14:052
2948025611,31sleep0362-21snmpd11:36:320
283625613,29sleep32891-21if_err_eth008:29:093
2412425622,19sleep00-21swapper/009:34:260
2338925615,18sleep20-21swapper/209:34:082
2278325622,18sleep30-21swapper/307:49:083
2168525616,25sleep221691-21df_inode09:28:592
2168525616,25sleep221691-21df_inode09:28:592
185025622,20sleep00-21swapper/011:53:440
1434825612,25sleep30-21swapper/307:23:563
1413225616,25sleep214203-21latency_hist07:23:472
2552225513,24sleep025517-21apt-get07:58:580
2552225513,24sleep025517-21apt-get07:58:580
2216325514,24sleep022158-21apt-get11:13:590
1965725521,18sleep30-21swapper/311:04:133
1965725521,18sleep30-21swapper/311:04:133
889025413,24sleep08884-21apt-get08:49:000
864925412,27sleep28716-21cat08:48:462
3207225417,17sleep30-21swapper/308:18:593
969325311,24sleep29687-21apt-get10:34:002
842025313,18sleep30-21swapper/312:13:463
81125313,21sleep0840-21munin-run08:23:450
679725312,18sleep20-21swapper/210:24:072
679725312,18sleep20-21swapper/210:24:072
238025313,23sleep02471-21mailstats10:09:150
1899925312,24sleep019018-21apt-get11:03:540
1899925312,24sleep019018-21apt-get11:03:540
577725213,23sleep35773-21apt-get08:38:583
577725213,23sleep35773-21apt-get08:38:583
1808725213,25sleep218201-21timerandwakeup10:59:162
1808725213,25sleep218201-21timerandwakeup10:59:162
1787225219,20sleep00-21swapper/010:59:090
1787225219,20sleep00-21swapper/010:59:090
1151925216,20sleep20-21swapper/212:23:462
1127625214,23sleep011269-21apt-get07:14:000
664325112,17sleep00-21swapper/010:24:050
664325112,17sleep00-21swapper/010:24:050
300262516,33sleep30-21swapper/308:12:213
208025111,18sleep30-21swapper/310:09:063
100172515,16sleep30-21swapper/308:52:023
998925011,17sleep30-21swapper/312:18:473
489525013,22sleep34933-21cat10:18:513
3118625012,23sleep230721-21apt-get09:58:512
258525015,21sleep02620-21ntp_states11:54:130
2442225014,21sleep224428-21ntp_states07:54:112
1747725011,16sleep30-21swapper/310:59:003
1747725011,16sleep30-21swapper/310:59:003
1565625014,22sleep015747-21memory09:09:070
225424915,20sleep22349-21perl11:54:072
580924812,16sleep30-21swapper/312:04:253
3003124810,21sleep030091-21fschecks_time11:39:070
2415224812,22sleep024211-21if_err_eth007:54:050
128842485,32sleep00-21swapper/012:26:590
1124024813,17sleep20-21swapper/210:38:572
78442478,15sleep30-21swapper/307:04:053
2081224610,22sleep0362-21snmpd07:43:560
177692469,13sleep00-21swapper/009:14:160
1414424614,16sleep00-21swapper/010:48:480
2657224311,12sleep00-21swapper/009:44:110
1891624311,17sleep019017-21dpkg07:38:470
174092438,12sleep20-21swapper/209:14:102
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional