You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 09:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Nov 20, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9311265325,609sleep39335-21df00:29:033
17296241923,370sleep017255-21latency_hist21:28:480
9349232646,268sleep29428-21fschecks_time00:29:072
24806232332,279sleep224842-21cat20:09:102
3612230412,209sleep23720-21sort20:44:202
7803230090,190sleep27896-21fschecks_time00:24:102
1555229632,247sleep21578-21fschecks_count00:04:072
250022276134,110sleep225046-21awk21:53:462
6853227113,243sleep06896-21unin-run22:38:470
9013224920,216sleep39022-21unin-run00:28:473
1799224566,87sleep21808-21sshd22:19:192
2472922420,7sleep00-21swapper/023:34:070
2472922420,7sleep00-21swapper/023:34:070
2040722357,213sleep220467-21latency_hist21:38:482
14090222631,152sleep214215-21timerwakeupswit22:59:172
20975222530,178sleep221101-21latency_hist23:23:482
29496221614,187sleep229544-21latency20:24:132
13694220824,166sleep212752-21/usr/sbin/munin21:15:052
13694220824,166sleep212752-21/usr/sbin/munin21:15:052
26199220212,176sleep126206-21sh20:14:101
2064421888,51sleep220707-21cpuspeed21:39:002
2635721839,162sleep226367-21awk23:39:192
7604217925,141sleep27642-21processes22:39:182
19632217414,143sleep219655-21latency_hist19:53:482
18065217313,141sleep018033-21latency_hist19:48:490
18065217313,141sleep018033-21latency_hist19:48:490
31272217218,55sleep20-21swapper/223:55:142
31272217218,55sleep20-21swapper/223:55:142
27711216989,24sleep20-21swapper/223:44:122
26734216519,54sleep226821-21sort20:14:232
13027216498,17sleep20-21swapper/222:57:192
10157216313,87sleep20-21swapper/221:04:392
9997216113,49sleep00-21swapper/022:48:490
615215788,23sleep2632-21modprobe22:18:462
15064215717,114sleep20-21swapper/223:04:052
15064215717,114sleep20-21swapper/223:04:052
2169221567,56sleep20-21swapper/219:59:092
28147215212,124sleep20-21swapper/220:19:162
5344215044,90sleep25449-21latency_hist22:33:482
6425214631,103sleep26484-21latency00:19:092
110621465,31sleep20-21swapper/200:02:492
15681214230,99sleep215684-21cron21:23:452
15681214230,99sleep215684-21cron21:23:452
161521418,55sleep21676-21cat20:39:072
3086121397,119sleep127165-1kworker/1:2H23:54:371
3086121397,119sleep127165-1kworker/1:2H23:54:371
6195213415,41sleep20-21swapper/222:34:212
1178821347,113sleep211925-21sed21:09:202
1178821347,113sleep211925-21sed21:09:202
5791213120,99sleep20-21swapper/220:53:322
8941213030,85sleep29002-21apt-config21:03:462
15685213020,93sleep315730-21unin-run21:23:463
15685213020,93sleep315730-21unin-run21:23:463
2460721299,16sleep20-21swapper/223:34:062
2460721299,16sleep20-21swapper/223:34:062
13832212912,25sleep312752-21/usr/sbin/munin21:15:063
13832212912,25sleep312752-21/usr/sbin/munin21:15:063
30707212824,82sleep20-21swapper/222:09:172
814212610,101sleep00-21swapper/023:59:200
3501212621,49sleep20-21swapper/222:26:012
3501212621,49sleep20-21swapper/222:26:012
22738212619,23sleep211280-21cyclictest23:28:572
22738212619,23sleep211280-21cyclictest23:28:572
1708521266,17sleep20-21swapper/221:26:532
404021257,16sleep20-21swapper/200:10:412
404021257,16sleep20-21swapper/200:10:412
1061212518,51sleep20-21swapper/220:37:292
9935212418,51sleep20-21swapper/222:48:392
15540212447,58sleep215648-21latency_hist19:33:492
26266212119,26sleep00-21swapper/021:54:410
22538212110,16sleep20-21swapper/223:28:492
22538212110,16sleep20-21swapper/223:28:492
14663212115,52sleep20-21swapper/219:26:542
17686212019,51sleep20-21swapper/219:46:082
17686212019,51sleep20-21swapper/219:46:082
2105121195,17sleep20-21swapper/219:57:312
27855211822,16sleep20-21swapper/221:59:192
10312211829,53sleep20-21swapper/222:49:072
1058221175,12sleep00-21swapper/021:06:390
4682211622,13sleep20-21swapper/200:14:042
14500211617,69sleep00-21swapper/021:19:070
14500211617,69sleep00-21swapper/021:19:070
16913211412,88sleep3362-21snmpd21:24:443
6242211327,67sleep36318-21fschecks_time00:19:063
19783211325,68sleep319938-21grep21:34:173
2924721128,15sleep20-21swapper/223:49:082
13543211017,15sleep20-21swapper/219:18:472
6858210825,60sleep16961-21unixbench_multi20:54:201
1595821077,15sleep20-21swapper/219:33:572
436521060,6sleep10-21swapper/100:13:461
436521060,6sleep10-21swapper/100:13:461
294321060,6sleep10-21swapper/120:43:491
9913210432,36sleep30-21swapper/318:59:023
3192021049,15sleep20-21swapper/220:32:202
29450210415,75sleep029488-21grep23:49:140
8539210311,12sleep00-21swapper/020:59:200
5220210316,70sleep15346-21unixbench_multi20:49:151
19561210314,71sleep119631-21systemd-run19:53:471
1800421027,83sleep21-21systemd21:29:152
2311321016,30sleep20-21swapper/221:44:252
9084210019,21sleep10-21swapper/100:28:491
3150329831,40sleep031642-21grep20:29:190
1652829836,17sleep20-21swapper/223:09:022
948829615,67sleep39500-21grep21:04:043
3082229520,58sleep030804-21apt-get23:54:070
3082229520,58sleep030804-21apt-get23:54:070
1828829513,67sleep118418-21/usr/sbin/munin21:29:181
2997929410,71sleep230023-21date22:08:482
1534629317,61sleep315389-21ntp_kernel_err23:04:103
1534629317,61sleep315389-21ntp_kernel_err23:04:103
2043029112,59sleep30-21swapper/323:19:213
192082916,30sleep20-21swapper/223:16:372
35129025,30sleep00-21swapper/020:34:160
167272905,31sleep20-21swapper/219:40:392
747428917,51sleep07522-21grep00:23:470
273228922,45sleep12765-21basename00:08:461
266152896,12sleep20-21swapper/221:56:002
1595928919,50sleep315454-21apt-get19:33:583
2274928815,22sleep10-21swapper/120:03:481
128422888,26sleep20-21swapper/219:13:142
239612876,27sleep20-21swapper/220:05:062
239612876,27sleep20-21swapper/220:05:062
1787028721,45sleep317225-21/usr/sbin/munin19:46:583
1787028721,45sleep317225-21/usr/sbin/munin19:46:583
2743428617,51sleep36765-1kworker/3:0H20:18:483
1838428613,53sleep318476-21/usr/sbin/munin23:14:083
139702866,27sleep20-21swapper/219:21:052
11752860,8sleep30-21swapper/320:38:463
115992865,29sleep20-21swapper/219:05:392
72252856,28sleep20-21swapper/220:56:352
588928523,33sleep30-21swapper/320:53:503
591928316,42sleep16003-21latency_hist00:18:481
1123828326,26sleep20-21swapper/219:03:492
3198828215,52sleep332024-21/usr/sbin/munin22:14:073
2490828211,53sleep124977-21/usr/sbin/munin23:34:131
2490828211,53sleep124977-21/usr/sbin/munin23:34:131
1196028220,18sleep30-21swapper/322:54:093
2429928122,32sleep10-21swapper/120:08:491
2429928122,32sleep10-21swapper/120:08:491
275328018,17sleep30-21swapper/320:43:483
2421727911,49sleep324276-21/usr/sbin/munin21:49:153
1670927913,40sleep316503-21/usr/sbin/munin23:09:073
131812799,46sleep00-21swapper/021:14:550
131812799,46sleep00-21swapper/021:14:550
1065527917,35sleep110809-21uptime18:59:201
2813327715,27sleep10-21swapper/120:19:151
2565127719,31sleep30-21swapper/323:38:473
2565127719,31sleep30-21swapper/323:38:473
1885527615,48sleep018968-21unixbench_singl23:14:170
765327416,38sleep37736-21grep22:39:193
354127410,20sleep00-21swapper/022:26:310
354127410,20sleep00-21swapper/022:26:310
2820827412,46sleep328277-21sensors20:19:173
2553727419,41sleep325554-21/usr/sbin/munin21:53:593
1997127417,39sleep020051-21tune2fs19:54:100
167827417,33sleep01445-21/usr/sbin/munin00:04:090
526527319,34sleep00-21swapper/022:33:460
2845627317,26sleep30-21swapper/322:03:493
2552927317,38sleep125026-21apt-get21:53:571
1637627310,18sleep00-21swapper/019:38:480
1295427318,42sleep012968-21wc19:13:480
117627310,13sleep00-21swapper/022:19:080
1131627324,28sleep10-21swapper/121:09:101
1131627324,28sleep10-21swapper/121:09:101
8992728,44sleep10-21swapper/122:18:571
3031127215,27sleep00-21swapper/022:09:070
299927218,29sleep00-21swapper/000:08:570
299927218,29sleep00-21swapper/000:08:570
1231327224,27sleep012333-21apt21:13:460
1231327224,27sleep012333-21apt21:13:460
2247227123,28sleep30-21swapper/323:28:463
2247227123,28sleep30-21swapper/323:28:463
2194527122,34sleep121955-21munin-run21:43:481
145827122,28sleep30-21swapper/322:19:163
141432714,10sleep30-21swapper/319:23:243
117927117,38sleep11197-21awk00:03:451
321522708,16sleep10-21swapper/120:33:491
30082709,18sleep00-21swapper/020:44:050
3002427022,25sleep30-21swapper/322:08:493
2511927020,21sleep00-21swapper/021:53:480
2500427016,14sleep00-21swapper/020:09:160
1141527016,37sleep311461-21/usr/sbin/munin21:09:123
1141527016,37sleep311461-21/usr/sbin/munin21:09:123
38026916,33sleep1464-21mailstats20:34:171
3074726912,43sleep330766-21apt20:28:583
302122696,51sleep00-21swapper/020:25:160
2494026917,37sleep124989-21/usr/sbin/munin20:09:131
2170826918,32sleep30-21swapper/319:59:103
1565026919,25sleep30-21swapper/319:33:503
278526624,20sleep30-21swapper/300:08:483
302826518,24sleep30-21swapper/300:09:043
302826518,24sleep30-21swapper/300:09:043
2401326523,23sleep10-21swapper/121:49:091
2154426516,22sleep00-21swapper/019:59:070
58682647,12sleep30-21swapper/300:18:463
2720226419,25sleep30-21swapper/323:43:483
850326316,31sleep18644-21date22:43:491
740526320,20sleep00-21swapper/022:39:120
245726310,26sleep30-21swapper/322:24:043
245726310,26sleep30-21swapper/322:24:043
1849326322,24sleep10-21swapper/123:14:101
1359426323,15sleep00-21swapper/019:18:490
2325526214,20sleep30-21swapper/323:29:143
2325526214,20sleep30-21swapper/323:29:143
1965726219,22sleep30-21swapper/319:53:493
1838126223,24sleep00-21swapper/019:49:010
1760626214,32sleep017613-21/usr/sbin/munin21:29:050
1701526215,30sleep016978-21latency_hist19:43:480
1539426211,36sleep015431-21ntp_states23:04:110
1539426211,36sleep015431-21ntp_states23:04:110
824426122,21sleep00-21swapper/000:24:200
532526114,33sleep15343-21munin-run22:33:471
324226112,35sleep13271-21irqstats20:44:111
3237926117,27sleep10-21swapper/122:14:171
297626117,27sleep32995-21apt-get20:44:003
1188626117,21sleep10-21swapper/119:08:481
88842606,11sleep10-21swapper/100:27:361
2944126019,25sleep329447-21ntp_states23:49:133
252726017,22sleep10-21swapper/122:24:061
252726017,22sleep10-21swapper/122:24:061
2306726014,31sleep023074-21diskstats20:04:070
2306726014,31sleep023074-21diskstats20:04:070
2259226014,24sleep10-21swapper/121:44:121
2256226017,23sleep00-21swapper/021:44:110
2016026020,17sleep30-21swapper/319:54:173
105262608,30sleep00-21swapper/018:59:190
559025918,27sleep30-21swapper/322:34:063
2767325916,26sleep027668-21apt-get20:19:020
2711925915,26sleep027113-21apt-get21:59:010
2625725913,27sleep026338-21/usr/sbin/munin20:14:130
2251525916,22sleep10-21swapper/123:28:481
2251525916,22sleep10-21swapper/123:28:481
1946325925,19sleep00-21swapper/021:34:090
1363525913,21sleep00-21swapper/022:59:070
4762586,10sleep00-21swapper/022:16:580
2922825815,26sleep329221-21apt-get20:24:053
2773525814,20sleep10-21swapper/121:59:181
1752025817,25sleep317516-21apt-get21:29:003
160622586,12sleep10-21swapper/121:24:051
868125713,25sleep38677-21apt-get22:43:593
206225721,18sleep00-21swapper/020:39:170
1988725711,26sleep10-21swapper/119:54:071
1930425713,27sleep119362-21/usr/sbin/munin21:34:061
1497825716,25sleep114972-21apt-get23:04:001
1497825716,25sleep114972-21apt-get23:04:001
1427425719,17sleep00-21swapper/019:23:500
1247125716,18sleep10-21swapper/119:08:571
3156925611,28sleep131686-21cat22:13:481
2949025616,24sleep129563-21/usr/sbin/munin23:49:151
2785925614,25sleep127895-21ntp_states23:44:151
2350825616,18sleep30-21swapper/320:04:193
2350825616,18sleep30-21swapper/320:04:193
1713525618,19sleep10-21swapper/123:09:171
1641025616,18sleep10-21swapper/119:38:491
1357925615,22sleep10-21swapper/121:15:041
1357925615,22sleep10-21swapper/121:15:041
659125512,20sleep00-21swapper/020:54:150
530125520,18sleep00-21swapper/000:14:190
3098025515,24sleep330981-21diskstats23:55:033
3098025515,24sleep330981-21diskstats23:55:033
2597825515,24sleep125981-21diskstats23:39:101
2276225514,25sleep022757-21apt-get23:28:590
2276225514,25sleep022757-21apt-get23:28:590
2117725516,19sleep00-21swapper/021:39:140
1547625518,21sleep115539-21apt-config19:33:481
1485525516,17sleep30-21swapper/319:28:493
145162555,10sleep30-21swapper/323:01:323
1224325513,25sleep012271-21/usr/sbin/munin22:54:140
994525415,24sleep19968-21ntp_states21:04:341
90472546,11sleep10-21swapper/122:44:091
615525415,22sleep16150-21apt-get00:19:001
2284725411,28sleep122857-21/usr/sbin/munin23:29:041
2284725411,28sleep122857-21/usr/sbin/munin23:29:041
2111625415,24sleep321175-21/usr/sbin/munin21:39:133
2006825417,20sleep00-21swapper/023:19:140
1725425415,22sleep117250-21apt-get19:44:011
1725425415,22sleep117250-21apt-get19:44:011
926125315,22sleep09256-21apt-get00:29:000
3022125312,24sleep330216-21apt-get22:09:023
2762525318,18sleep00-21swapper/023:44:100
2121425317,18sleep00-21swapper/023:24:020
2121425317,18sleep00-21swapper/023:24:020
2022225313,19sleep10-21swapper/123:19:191
1543325314,23sleep015475-21idleruntime-cro19:33:470
146312535,35sleep00-21swapper/019:26:280
128962536,37sleep10-21swapper/122:55:361
1071425316,17sleep10-21swapper/122:49:181
822825213,18sleep10-21swapper/120:59:171
777125216,20sleep30-21swapper/320:59:073
609925211,28sleep36112-21acpi20:53:583
2597025214,22sleep025976-21df_inode23:39:090
125092528,33sleep30-21swapper/319:08:593
113842525,38sleep00-21swapper/022:51:350
2462325118,20sleep324667-21grep20:09:073
58072508,17sleep10-21swapper/122:34:121
2912725016,16sleep10-21swapper/122:04:111
1652625013,22sleep016521-21apt-get23:09:000
1021125012,17sleep30-21swapper/322:49:003
462424920,16sleep30-21swapper/322:29:213
3234524912,22sleep332364-21apt-get23:58:593
2943824910,21sleep129495-21irqstats20:24:121
115432494,11sleep30-21swapper/319:04:553
2874824816,16sleep00-21swapper/022:04:020
1469724813,20sleep114402-21/usr/sbin/munin21:19:121
1469724813,20sleep114402-21/usr/sbin/munin21:19:121
1350524812,22sleep113511-21diskstats22:59:031
498224711,22sleep04998-21ntp_states20:49:100
2812424714,16sleep328257-21sed23:44:213
1146824710,25sleep0362-21snmpd19:03:590
2219524615,17sleep30-21swapper/321:44:023
132242464,30sleep30-21swapper/319:14:363
1700424511,20sleep317013-21munin-run19:43:473
148052356,14sleep10-21swapper/119:28:471
136962354,13sleep10-21swapper/119:18:501
1675623210,12sleep10-21swapper/119:41:031
131702286,12sleep10-21swapper/119:13:531
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional