You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 14:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Nov 25, 2025 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12038213609,1339sleep120581-21kworker/1:100:06:521
12038213609,1339sleep120581-21kworker/1:100:06:521
14767244816,417sleep014774-21ntp_states00:14:140
26790242763,345sleep326877-21fschecks_time21:28:543
351323867,367sleep3362-21snmpd21:57:533
351323867,367sleep3362-21snmpd21:57:533
29212229910,276sleep329223-21uname23:18:533
11839229115,255sleep111913-21latency_hist19:03:311
161252263107,117sleep30-21swapper/319:13:593
12187224864,160sleep30-21swapper/320:43:313
31079224132,71sleep331209-21unixbench_multi23:24:023
31079224132,71sleep331209-21unixbench_multi23:24:023
21138222245,52sleep30-21swapper/322:53:503
21138222245,52sleep30-21swapper/322:53:503
24771221857,54sleep30-21swapper/323:04:033
16944221459,47sleep30-21swapper/322:39:003
16944221459,47sleep30-21swapper/322:39:003
18904221311,27sleep30-21swapper/300:28:393
1054322116,194sleep3362-21snmpd20:38:023
14545221037,64sleep314609-21mailstats19:08:583
4497220358,119sleep30-21swapper/323:43:493
712219432,104sleep30-21swapper/321:48:443
22035218551,42sleep322089-21http19:33:293
8587218354,27sleep30-21swapper/322:13:443
10223218160,105sleep310307-21fschecks_time22:18:493
2606218019,52sleep30-21swapper/323:38:323
20571216728,86sleep320711-21sed19:28:323
1538216655,62sleep30-21swapper/323:33:593
914921627,18sleep20-21swapper/220:33:332
16135216130,117sleep316225-21sort22:38:313
16135216130,117sleep316225-21sort22:38:313
9434215621,33sleep20-21swapper/223:58:502
9434215621,33sleep20-21swapper/223:58:502
18674215411,82sleep318725-21cpuspeed21:03:463
8948214920,105sleep16568-1kworker/1:1H20:33:321
3471214849,58sleep33608-21timerwakeupswit20:14:033
3471214849,58sleep33608-21timerwakeupswit20:14:033
384621477,17sleep30-21swapper/321:58:423
384621477,17sleep30-21swapper/321:58:423
19474214232,59sleep319475-21cpu22:48:453
26464214016,65sleep30-21swapper/319:44:033
26464214016,65sleep30-21swapper/319:44:033
11034213919,104sleep111043-21/usr/sbin/munin22:19:031
6173213314,106sleep36193-21apt-get23:48:413
3111821326,18sleep30-21swapper/321:40:533
25791212913,61sleep30-21swapper/321:24:013
747721287,39sleep30-21swapper/320:28:413
12267212832,80sleep312339-21switchtime22:23:583
29828212710,105sleep329797-21latency_hist21:38:313
23076212538,29sleep30-21swapper/322:58:593
11450212564,15sleep30-21swapper/300:04:003
11450212564,15sleep30-21swapper/300:04:003
1721821238,64sleep317312-21tune2fs20:58:493
12588212217,43sleep30-21swapper/319:04:013
3192121219,85sleep30-21swapper/323:28:423
3192121219,85sleep30-21swapper/323:28:423
4988212038,29sleep30-21swapper/320:19:003
21234211911,90sleep121241-21sh19:28:591
2259421178,71sleep30-21swapper/319:33:453
30506211638,25sleep30-21swapper/319:58:493
10655211625,76sleep310682-21fschecks_time18:59:093
7758211510,90sleep17533-40mandb23:53:361
7393211315,75sleep27430-21/usr/sbin/munin22:08:542
7393211315,75sleep27430-21/usr/sbin/munin22:08:542
23866211323,52sleep30-21swapper/321:18:523
20878211233,59sleep30-21swapper/321:09:113
20878211233,59sleep30-21swapper/321:09:113
4113211113,83sleep04134-21basename20:18:280
4113211113,83sleep04134-21basename20:18:280
20430211117,76sleep019950-21apt-get21:08:400
20430211117,76sleep019950-21apt-get21:08:400
29283211010,85sleep129316-21proc_pri19:53:581
28554211012,45sleep30-21swapper/323:18:293
1654421096,53sleep30-21swapper/320:54:583
1654421096,53sleep30-21swapper/320:54:583
7213210731,63sleep37251-21modprobe20:28:293
19368210718,71sleep119441-21fschecks_time19:24:061
3192921068,53sleep30-21swapper/320:03:463
3192921068,53sleep30-21swapper/320:03:463
15204210618,67sleep20-21swapper/222:33:542
15204210618,67sleep20-21swapper/222:33:542
10206210612,23sleep010253-21users23:59:040
10206210612,23sleep010253-21users23:59:040
8259210441,45sleep30-21swapper/323:54:023
803721048,48sleep30-21swapper/322:10:333
803721048,48sleep30-21swapper/322:10:333
14814210425,26sleep20-21swapper/200:14:162
2664921025,50sleep30-21swapper/323:09:243
10684210113,72sleep010755-21idleruntime18:59:100
246052997,42sleep324639-21ps19:38:583
246052997,42sleep324639-21ps19:38:583
1778629912,73sleep217898-21timerwakeupswit20:59:012
151772996,46sleep30-21swapper/320:52:443
1173129911,73sleep211758-21df22:23:442
620929817,64sleep26225-21uname20:23:522
81942978,12sleep00-21swapper/023:53:590
1918229628,57sleep319224-21systemd-run22:48:293
2569029512,16sleep00-21swapper/019:43:440
2569029512,16sleep00-21swapper/019:43:440
162032958,13sleep30-21swapper/300:18:543
1168629513,64sleep111730-21cpu22:23:431
1334729243,12sleep30-21swapper/320:44:123
2949229111,66sleep029494-21sshd21:34:530
2769629143,13sleep30-21swapper/319:48:563
2769629143,13sleep30-21swapper/319:48:563
1553629124,43sleep115686-21unixbench_singl22:34:011
1553629124,43sleep115686-21unixbench_singl22:34:011
1458629126,19sleep30-21swapper/322:33:313
78929014,58sleep2796-21diskstats21:48:482
2173129024,51sleep121775-21awk21:13:291
2173129024,51sleep121775-21awk21:13:291
304002898,66sleep030470-21df23:23:470
304002898,66sleep030470-21df23:23:470
1877528911,62sleep218883-21apt-config19:23:312
1870828920,55sleep118774-21find19:23:301
384928820,34sleep10-21swapper/121:58:441
384928820,34sleep10-21swapper/121:58:441
1461328823,47sleep014682-21grep19:08:590
3158228743,11sleep30-21swapper/321:43:363
1120928726,26sleep20-21swapper/218:59:202
617228623,43sleep25687-21apt-get23:48:402
2325328613,45sleep20-21swapper/222:59:022
885328522,42sleep08946-21apt-config20:33:310
2514928516,47sleep025122-21/usr/sbin/munin21:23:460
1792028426,37sleep117915-21apt-get22:43:421
1781228421,45sleep30-21swapper/300:23:583
135622849,13sleep20-21swapper/220:46:532
1334128420,40sleep213432-21cut22:28:492
1293428414,51sleep212433-21/usr/sbin/munin00:08:592
1242628421,30sleep10-21swapper/120:43:451
289362837,13sleep10-21swapper/121:34:331
1088128326,36sleep011033-21unixbench_singl22:19:020
2022728222,46sleep1302-21systemd-logind00:33:291
125412829,12sleep00-21swapper/000:08:490
2456728118,35sleep20-21swapper/223:03:582
1437028131,36sleep30-21swapper/300:13:533
1039828124,11sleep30-21swapper/300:00:533
1039828124,11sleep30-21swapper/300:00:533
2847928010,57sleep2362-21snmpd21:33:372
2490928010,48sleep10-21swapper/121:23:311
226482809,58sleep322728-21awk21:14:003
226482809,58sleep322728-21awk21:14:003
2048627812,50sleep020513-21sh19:28:290
1922427814,26sleep20-21swapper/219:23:472
1386527814,46sleep113992-21/usr/sbin/munin22:29:001
1378227822,43sleep213908-21if_eth019:08:372
764227718,42sleep07761-21timerwakeupswit22:09:000
764227718,42sleep07761-21timerwakeupswit22:09:000
664627718,11sleep30-21swapper/322:07:383
2213527717,27sleep00-21swapper/021:13:490
2213527717,27sleep00-21swapper/021:13:490
2019327723,37sleep020226-21sendmail00:33:280
1840527719,10sleep30-21swapper/319:19:483
3099827620,37sleep231078-21awk23:24:012
3099827620,37sleep231078-21awk23:24:012
223927631,11sleep30-21swapper/320:09:513
223927631,11sleep30-21swapper/320:09:513
3235427521,28sleep20-21swapper/221:43:592
2761227417,28sleep10-21swapper/119:48:541
2761227417,28sleep10-21swapper/119:48:541
2726827414,46sleep227282-21acpi23:13:392
2196927411,12sleep00-21swapper/019:32:370
2858027318,27sleep20-21swapper/219:53:312
2858027318,27sleep20-21swapper/219:53:312
1703927322,27sleep00-21swapper/019:18:320
1690827315,43sleep216951-21gpgv19:18:302
1491027314,43sleep014940-21fschecks_count22:33:480
1491027314,43sleep014940-21fschecks_count22:33:480
3202727212,15sleep00-21swapper/021:43:530
2974927228,11sleep30-21swapper/319:54:173
2795727218,29sleep20-21swapper/221:33:292
2540127214,32sleep10-21swapper/119:43:281
2540127214,32sleep10-21swapper/119:43:281
228832728,47sleep00-21swapper/022:58:510
1736727215,41sleep117380-21irqstats20:58:521
1340627217,10sleep30-21swapper/300:09:333
39162717,52sleep00-21swapper/023:40:510
924027021,32sleep29363-21wc22:13:592
340627012,27sleep20-21swapper/220:14:022
340627012,27sleep20-21swapper/220:14:022
2940127015,40sleep129475-21ls23:18:591
1138427018,24sleep20-21swapper/200:03:592
1138427018,24sleep20-21swapper/200:03:592
1108227023,25sleep00-21swapper/020:38:560
109727016,35sleep2751-21apt-get20:08:332
109727016,35sleep2751-21apt-get20:08:332
736926915,34sleep16977-21/usr/sbin/munin22:08:531
736926915,34sleep16977-21/usr/sbin/munin22:08:531
675926918,24sleep20-21swapper/222:08:302
602526917,27sleep00-21swapper/022:04:000
595426920,36sleep06006-21cpuspeed20:23:450
3169126920,34sleep231667-21latency_hist20:03:322
2428826915,28sleep10-21swapper/123:03:531
38502688,14sleep20-21swapper/221:58:452
38502688,14sleep20-21swapper/221:58:452
1869526823,25sleep00-21swapper/019:23:290
1750726817,27sleep00-21swapper/020:58:550
2933526725,26sleep00-21swapper/023:18:580
2128626713,36sleep221383-21users21:09:182
2128626713,36sleep221383-21users21:09:182
187322670,9sleep00-21swapper/021:03:460
1687126715,28sleep00-21swapper/020:58:300
1687126715,28sleep00-21swapper/020:58:300
1409826716,37sleep114143-21fschecks_time20:48:501
1341126721,18sleep10-21swapper/119:08:321
996526616,20sleep00-21swapper/020:34:230
3064426621,28sleep130684-21sshd19:58:511
2699626612,40sleep227027-21latency_hist19:48:302
2699626612,40sleep227027-21latency_hist19:48:302
21542669,48sleep00-21swapper/023:36:140
111426623,24sleep10-21swapper/121:48:551
989826521,24sleep10-21swapper/122:18:301
537626515,33sleep15377-21sshd20:19:511
2707326512,36sleep027119-21/usr/sbin/munin21:28:590
2563426518,21sleep10-21swapper/121:23:591
2272226515,37sleep122806-21fschecks_time22:58:481
1245926514,36sleep112474-21grep00:08:461
124426514,37sleep01263-21apt-get20:08:430
124426514,37sleep01263-21apt-get20:08:430
123626516,31sleep2751-21apt-get20:08:412
123626516,31sleep2751-21apt-get20:08:412
3232826419,23sleep00-21swapper/020:03:550
3232826419,23sleep00-21swapper/020:03:550
322726413,35sleep13273-21/usr/sbin/munin20:13:571
322726413,35sleep13273-21/usr/sbin/munin20:13:571
2419526424,21sleep00-21swapper/019:38:470
2419526424,21sleep00-21swapper/019:38:470
172026417,29sleep21846-21/usr/sbin/munin23:34:022
1145026413,29sleep10-21swapper/120:39:051
3017526320,20sleep20-21swapper/223:23:332
2766426313,38sleep027675-21netstat23:13:540
1768226312,34sleep117739-21cut22:43:301
1768226312,34sleep117739-21cut22:43:301
141326311,24sleep10-21swapper/120:09:091
141326311,24sleep10-21swapper/120:09:091
480326210,13sleep20-21swapper/220:18:552
454826213,23sleep00-21swapper/023:43:500
1864326221,24sleep10-21swapper/100:28:291
1414426218,23sleep20-21swapper/220:48:512
897626114,29sleep09016-21/usr/sbin/munin22:13:530
588326119,26sleep26067-21cat23:48:322
319726111,36sleep23281-21memory23:39:322
255626118,24sleep00-21swapper/021:53:500
255626118,24sleep00-21swapper/021:53:500
1406326119,23sleep00-21swapper/020:48:490
1339826117,25sleep01-21systemd19:08:300
3200526018,26sleep131903-21/usr/sbin/munin20:03:481
3200526018,26sleep131903-21/usr/sbin/munin20:03:481
2335126017,26sleep10-21swapper/121:18:301
2335126017,26sleep10-21swapper/121:18:301
21702606,12sleep10-21swapper/123:36:281
1802026013,30sleep00-21swapper/000:24:030
1757126011,19sleep20-21swapper/200:23:532
961825915,23sleep10-21swapper/123:58:531
961825915,23sleep10-21swapper/123:58:531
75225915,28sleep0753-21cron23:33:280
75225915,28sleep0753-21cron23:33:280
2619325918,17sleep20-21swapper/223:08:592
2579725915,27sleep125887-21cut23:08:501
222782596,12sleep00-21swapper/022:57:550
222782596,12sleep00-21swapper/022:57:550
2211925915,20sleep20-21swapper/221:13:472
2211925915,20sleep20-21swapper/221:13:472
2016925916,20sleep10-21swapper/121:08:331
3250325812,33sleep132579-21sed23:29:011
3250325812,33sleep132579-21sed23:29:011
2410425818,16sleep20-21swapper/221:19:002
2390525816,27sleep10-21swapper/119:38:321
2081325818,15sleep10-21swapper/122:53:311
1827525818,20sleep00-21swapper/022:43:530
1644125820,17sleep10-21swapper/100:19:001
1592025818,20sleep20-21swapper/200:18:482
1592025817,20sleep10-21swapper/119:13:531
1468425820,20sleep10-21swapper/119:09:001
579525714,23sleep15882-21gpgv23:48:311
2860825716,23sleep128699-21latency_hist23:18:311
2693525714,30sleep126972-21uname21:28:561
266912576,12sleep10-21swapper/119:45:101
266912576,12sleep10-21swapper/119:45:101
1426225714,18sleep10-21swapper/100:13:431
1300725711,28sleep013148-21sed22:28:310
1009125717,18sleep10-21swapper/120:34:241
846725612,25sleep20-21swapper/223:54:052
3258025613,28sleep232707-21cat23:29:022
3258025613,28sleep232707-21cat23:29:022
3133325622,19sleep10-21swapper/121:43:301
3132525614,23sleep00-21swapper/021:43:290
263022568,37sleep2362-21snmpd21:27:072
2266525614,23sleep222671-21df_inode19:33:482
1993325618,25sleep219981-21dpkg21:08:302
1152325617,20sleep00-21swapper/000:04:010
1152325617,20sleep00-21swapper/000:04:010
68612558,12sleep10-21swapper/123:49:121
417325520,16sleep10-21swapper/123:43:301
2414325514,24sleep224136-21apt-get19:38:452
2414325514,24sleep224136-21apt-get19:38:452
1789525516,24sleep217896-21munin-node22:43:392
1714625516,23sleep017286-21latency_hist00:23:310
1324625514,24sleep013241-21apt-get22:28:430
1239925518,17sleep00-21swapper/020:43:400
1219125511,17sleep20-21swapper/219:03:522
3168025412,16sleep00-21swapper/020:03:300
312812545,10sleep20-21swapper/221:42:542
2389425412,25sleep023904-21munin-run19:38:310
2112325415,24sleep121130-21df_inode22:53:481
2112325415,24sleep121130-21df_inode22:53:481
1637125411,22sleep20-21swapper/222:38:422
1637125411,22sleep20-21swapper/222:38:422
1555125414,19sleep20-21swapper/220:53:442
1555125414,19sleep20-21swapper/220:53:442
1095925415,16sleep20-21swapper/220:38:542
590625315,23sleep15912-21ntp_states22:03:571
384825313,23sleep03843-21apt-get21:58:430
384825313,23sleep03843-21apt-get21:58:430
3206725311,15sleep10-21swapper/121:43:541
252682535,14sleep00-21swapper/023:06:350
620525217,18sleep00-21swapper/023:48:460
229025212,23sleep12284-21apt-get21:53:431
229025212,23sleep12284-21apt-get21:53:431
3039525112,23sleep130390-21apt-get23:23:441
3039525112,23sleep130390-21apt-get23:23:441
2927525112,22sleep029282-21ntp_states19:53:570
2579225112,24sleep025796-21diskstats23:08:490
2489925113,22sleep024908-21munin-run21:23:300
83425010,18sleep00-21swapper/021:48:500
1929425012,23sleep219436-21modprobe22:48:322
1581225020,16sleep10-21swapper/120:53:511
1581225020,16sleep10-21swapper/120:53:511
437824915,21sleep00-21swapper/020:18:410
2027224915,20sleep20-21swapper/200:33:302
176332499,23sleep017681-21systemd-run22:43:290
176332499,23sleep017681-21systemd-run22:43:290
286724810,26sleep22936-21switchtime21:54:002
286724810,26sleep22936-21switchtime21:54:002
2757024811,22sleep027611-21memory19:48:530
2757024811,22sleep027611-21memory19:48:530
2149824814,18sleep221509-21ntp_states22:53:572
2149824814,18sleep221509-21ntp_states22:53:572
77752479,17sleep20-21swapper/220:28:522
2886924710,22sleep228894-21df_inode19:53:472
2075424712,20sleep220772-21apt-get19:28:442
1068324716,15sleep20-21swapper/222:19:002
705924516,16sleep10-21swapper/120:26:331
1949124310,19sleep019546-21df22:48:460
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional