You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-12 - 14:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Nov 12, 2025 00:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1016623880,65sleep27839-21cyclictest19:09:352
4112236235,266sleep222225-21kworker/u8:123:59:152
4112236235,266sleep222225-21kworker/u8:123:59:152
9821235913,329sleep19895-21find20:49:491
2320226815,203sleep22334-21fschecks_count22:09:332
5095224480,148sleep27839-21cyclictest23:59:482
5095224480,148sleep27839-21cyclictest23:59:482
29476224012,191sleep229502-21proc_pri20:10:412
26098223854,168sleep225613-21/usr/sbin/munin23:24:432
132202225129,39sleep213234-21fschecks_count19:19:332
7467221810,196sleep0362-21snmpd19:01:020
2818022159,188sleep228306-21latency_hist21:49:182
2480622140,8sleep00-21swapper/021:39:260
2480622140,8sleep00-21swapper/021:39:260
3430221215,185sleep13506-21sed23:55:431
3430221215,185sleep13506-21sed23:55:431
18714221215,183sleep118723-21vmstat19:34:511
1903220714,179sleep01978-21mailstats20:24:450
15609220137,122sleep214692-21/usr/sbin/munin22:49:472
19668220025,141sleep219722-21sed23:04:372
30117219713,127sleep230265-21latency_hist23:39:172
24729219464,64sleep224861-21timerwakeupswit23:19:472
10469219233,57sleep210493-21modprobe20:54:142
32313219122,81sleep331815-21apt-get20:19:313
16706219010,64sleep20-21swapper/219:29:472
900621830,6sleep20-21swapper/200:14:162
12875218310,155sleep212888-21sh22:44:142
30584217513,145sleep330657-21sed23:39:383
9025217410,152sleep29116-21date20:49:182
9025217410,152sleep29116-21date20:49:182
2102221739,15sleep00-21swapper/023:09:310
30713217114,137sleep330854-21grep21:54:483
8572216814,85sleep28464-21/usr/sbin/munin22:29:362
8572216814,85sleep28464-21/usr/sbin/munin22:29:362
1683021569,16sleep20-21swapper/221:14:172
1683021569,16sleep20-21swapper/221:14:172
18367215533,57sleep20-21swapper/219:34:482
1477121538,130sleep214826-21fschecks_count19:24:342
19400214920,108sleep319436-21cpu23:04:303
27784214712,117sleep327862-21find23:29:453
2404821478,11sleep30-21swapper/319:54:353
25037214514,79sleep20-21swapper/219:55:472
15038214555,78sleep215137-21latency_hist00:34:162
1239821439,122sleep212425-21df20:59:352
1239821439,122sleep212425-21df20:59:352
26860214210,115sleep21-21systemd20:04:152
3651213910,88sleep22946-21/usr/sbin/munin20:29:512
17924213931,94sleep217933-21df22:59:322
22217213615,107sleep222220-21cron23:14:142
1106221369,112sleep20-21swapper/220:54:432
1106221369,112sleep20-21swapper/220:54:432
607213426,91sleep2558-21apt-get22:04:332
607213426,91sleep2558-21apt-get22:04:332
5087213426,92sleep21-21systemd22:19:142
2458521338,109sleep224713-21ls21:39:172
26368213227,89sleep226459-21ls21:44:172
26368213227,89sleep226459-21ls21:44:172
23035213033,60sleep223139-21latency_hist21:34:162
9232212819,90sleep39354-21unixbench_singl22:29:493
9232212819,90sleep39354-21unixbench_singl22:29:493
3266021266,62sleep232728-21ls20:20:352
8244212513,76sleep28297-21latency_hist22:29:162
20672212529,61sleep220675-21munin-run19:44:162
6598212429,53sleep20-21swapper/222:23:472
742421227,48sleep20-21swapper/200:09:252
1345921229,16sleep20-21swapper/200:29:152
1345921229,16sleep20-21swapper/200:29:152
18357211818,85sleep018441-21latency_hist21:19:160
23071211617,83sleep323172-21sed19:49:473
32760211414,83sleep0330-21/usr/sbin/munin20:20:370
18279211413,79sleep018301-21/usr/sbin/munin19:34:460
7783211354,34sleep30-21swapper/319:04:163
247221135,26sleep20-21swapper/220:26:402
22956211323,76sleep023001-21readlink21:34:140
30730211220,50sleep20-21swapper/223:39:412
17578211232,58sleep017654-21latency_hist19:34:170
5461211123,64sleep15499-21grep22:19:341
30254210932,15sleep20-21swapper/221:54:392
12642210912,88sleep00-21swapper/022:41:170
28525210826,19sleep20-21swapper/223:34:162
13166210720,47sleep20-21swapper/222:44:302
270921057,60sleep20-21swapper/223:54:232
1749221055,27sleep20-21swapper/222:58:192
4437210314,73sleep14467-21fschecks_count23:59:351
4437210314,73sleep14467-21fschecks_count23:59:351
1235121039,18sleep00-21swapper/020:59:330
1235121039,18sleep00-21swapper/020:59:330
15829210117,69sleep115872-21tr21:09:401
15829210117,69sleep115872-21tr21:09:401
106832989,70sleep110828-21grep19:09:471
217862977,13sleep30-21swapper/321:29:373
518229610,71sleep05325-21cut22:19:160
3101229643,14sleep20-21swapper/220:14:452
3020729625,48sleep330209-21cron20:14:143
50102957,49sleep20-21swapper/220:34:562
50102957,49sleep20-21swapper/220:34:562
90152945,29sleep20-21swapper/219:05:322
276472949,43sleep00-21swapper/023:29:420
1334329317,55sleep00-21swapper/019:19:350
2145629223,52sleep20-21swapper/221:29:152
1238629223,15sleep20-21swapper/219:14:492
116792929,50sleep20-21swapper/200:22:002
116792929,50sleep20-21swapper/200:22:002
816829117,53sleep38316-21unixbench_singl00:09:463
389829110,16sleep10-21swapper/122:14:331
2302229111,62sleep20-21swapper/223:14:462
2261329112,63sleep322622-21diskstats23:14:353
3186429026,16sleep20-21swapper/223:44:302
3186429026,16sleep20-21swapper/223:44:302
198028916,58sleep12046-21sed20:24:461
1656628916,55sleep016655-21/usr/sbin/munin22:54:380
1068428916,57sleep310775-21fschecks_time00:19:353
1068428916,57sleep310775-21fschecks_time00:19:353
15032879,16sleep20-21swapper/222:04:572
15032879,16sleep20-21swapper/222:04:572
687328630,34sleep16928-21awk19:00:111
2837128620,17sleep20-21swapper/221:49:262
104802868,16sleep20-21swapper/222:34:412
981728513,59sleep19827-21pluginstate22:34:161
981728513,59sleep19827-21pluginstate22:34:161
157528518,40sleep00-21swapper/023:49:440
1370828517,45sleep313640-21latency_hist21:04:173
1370828517,45sleep313640-21latency_hist21:04:173
358628419,43sleep03680-21latency_hist22:14:160
1053028414,52sleep39979-21/usr/sbin/munin19:09:453
2628128323,40sleep326426-21sed19:59:483
226842838,31sleep20-21swapper/219:49:372
2007628214,21sleep10-21swapper/119:39:511
173142828,13sleep20-21swapper/221:14:382
1409028212,19sleep30-21swapper/300:29:403
1167528226,27sleep10-21swapper/119:14:361
430828116,51sleep34378-21cut20:34:173
271392817,13sleep20-21swapper/220:04:272
2642828110,54sleep026502-21/usr/sbin/munin19:59:490
1905728111,55sleep019063-21ntpq21:19:440
1905728111,55sleep019063-21ntpq21:19:440
503028021,40sleep05062-21proc_pri20:34:580
503028021,40sleep05062-21proc_pri20:34:580
1027228013,53sleep010331-21latency19:09:380
619027927,29sleep00-21swapper/020:39:390
619027927,29sleep00-21swapper/020:39:390
2899727915,49sleep129031-21irqstats23:34:351
2854027921,36sleep328718-21cmp20:09:173
2317427917,26sleep00-21swapper/019:49:480
1442827917,42sleep114429-21cron22:49:141
1130127914,51sleep111373-21mailstats20:54:501
1130127914,51sleep111373-21mailstats20:54:501
905127811,17sleep10-21swapper/100:14:321
2141827822,26sleep10-21swapper/119:44:441
271027716,41sleep32243-21apt-get23:54:243
271027716,41sleep32243-21apt-get23:54:243
1990327720,32sleep30-21swapper/319:39:493
345427516,19sleep10-21swapper/120:29:461
2533027517,43sleep125423-21cut21:39:441
2533027517,43sleep125423-21cut21:39:441
215142758,31sleep20-21swapper/219:44:452
1054927522,22sleep00-21swapper/020:54:160
31872745,12sleep10-21swapper/122:10:171
3036627414,42sleep130391-21/usr/sbin/munin23:39:321
2915527418,36sleep029295-21unixbench_singl21:50:010
2572327415,43sleep125805-21fschecks_time23:24:341
1015627410,48sleep010235-21fschecks_time22:34:350
3253927314,40sleep3369-21sshd23:44:473
3253927314,40sleep3369-21sshd23:44:473
2685627312,42sleep126858-21cron20:04:141
1912427323,29sleep30-21swapper/319:39:173
1035927311,46sleep010494-21cat00:19:160
59827221,30sleep10-21swapper/122:04:321
59827221,30sleep10-21swapper/122:04:321
3178627224,29sleep331796-21modprobe20:19:143
178527227,24sleep30-21swapper/320:24:403
1526727227,24sleep30-21swapper/322:49:443
2839727117,25sleep30-21swapper/321:49:273
1703527126,21sleep10-21swapper/122:54:471
1606627118,21sleep30-21swapper/321:09:463
1606627118,21sleep30-21swapper/321:09:463
1251827112,41sleep312552-21/usr/sbin/munin00:24:403
1251827112,41sleep312552-21/usr/sbin/munin00:24:403
940727021,29sleep00-21swapper/019:09:180
880527013,42sleep08861-21latency_hist00:14:150
721227021,29sleep10-21swapper/100:09:161
721227021,29sleep10-21swapper/100:09:161
60712705,29sleep20-21swapper/220:39:242
60712705,29sleep20-21swapper/220:39:242
2898427016,34sleep028989-21cpu20:10:030
2672327015,26sleep30-21swapper/321:44:513
2534627016,40sleep125361-21apt23:24:141
1886727016,32sleep318889-21kernelversion21:19:393
1886727016,32sleep318889-21kernelversion21:19:393
188092707,28sleep20-21swapper/221:19:382
188092707,28sleep20-21swapper/221:19:382
1785227011,42sleep017853-21/usr/sbin/munin22:59:300
145422709,14sleep00-21swapper/022:49:170
931526915,30sleep30-21swapper/300:14:393
614326921,33sleep20-21swapper/200:04:362
614326921,33sleep20-21swapper/200:04:362
2420626917,19sleep30-21swapper/323:19:363
707526821,21sleep20-21swapper/219:00:162
513426826,21sleep30-21swapper/322:19:153
297326815,25sleep00-21swapper/020:29:330
2550226819,21sleep30-21swapper/321:39:463
2550226819,21sleep30-21swapper/321:39:463
2300026818,25sleep10-21swapper/123:14:451
143992687,20sleep214459-21find21:04:522
1070026815,39sleep310749-21cpuspeed20:54:323
886326721,27sleep10-21swapper/100:14:161
815926724,23sleep10-21swapper/122:28:461
314352677,48sleep00-21swapper/023:42:300
1453026721,25sleep00-21swapper/021:04:540
3170026618,19sleep00-21swapper/021:59:460
3170026618,19sleep00-21swapper/021:59:460
3050326512,27sleep10-21swapper/120:14:281
235462656,12sleep00-21swapper/021:34:380
1835126519,28sleep318430-21awk22:59:463
1444726510,42sleep10-21swapper/119:24:171
1341526520,30sleep1284-21dbus-daemon00:29:141
1341526520,30sleep1284-21dbus-daemon00:29:141
119072657,16sleep00-21swapper/020:57:190
119072657,16sleep00-21swapper/020:57:190
109912659,44sleep30-21swapper/322:34:583
53726423,23sleep30-21swapper/322:04:253
53726423,23sleep30-21swapper/322:04:253
2610526421,25sleep30-21swapper/323:24:443
2453226420,21sleep30-21swapper/321:39:143
1663926413,32sleep316216-21/usr/sbin/munin19:29:443
1493226415,40sleep00-21swapper/000:33:530
33126324,21sleep10-21swapper/120:20:381
2327526313,31sleep123319-21/usr/sbin/munin21:34:311
214152639,13sleep30-21swapper/323:09:433
2099826314,32sleep321069-21fschecks_time19:44:343
1632626315,29sleep116362-21/usr/sbin/munin19:29:381
1529126318,33sleep115365-21timerandwakeup19:24:451
146872630,47sleep30-21swapper/319:24:313
1447826320,23sleep30-21swapper/322:49:163
793826220,22sleep10-21swapper/120:44:441
793826220,22sleep10-21swapper/120:44:441
315732627,11sleep00-21swapper/020:16:330
2068026222,20sleep00-21swapper/021:24:450
1292826216,29sleep10-21swapper/122:44:171
668626119,16sleep30-21swapper/322:24:163
564426115,20sleep00-21swapper/022:19:370
2885126117,23sleep30-21swapper/323:34:333
2840726122,22sleep10-21swapper/120:09:151
227322615,12sleep10-21swapper/121:31:241
222426115,30sleep12250-21munin-run23:54:141
1771926116,30sleep317837-21cat21:14:473
116212617,12sleep10-21swapper/100:21:151
116212617,12sleep10-21swapper/100:21:151
71792607,40sleep30-21swapper/320:42:003
71792607,40sleep30-21swapper/320:42:003
69426021,22sleep10-21swapper/122:04:361
69426021,22sleep10-21swapper/122:04:361
598526013,27sleep30-21swapper/300:04:333
598526013,27sleep30-21swapper/300:04:333
2879026015,24sleep10-21swapper/121:49:541
2809926015,29sleep127887-21apt-get21:49:171
1397226014,23sleep10-21swapper/100:29:381
886325913,29sleep08906-21ntpq22:29:420
886325913,29sleep08906-21ntpq22:29:420
2172225914,26sleep021814-21sshd19:44:470
2148625916,20sleep30-21swapper/321:29:163
1985425918,26sleep119872-21munin-run21:24:131
1985425918,26sleep119872-21munin-run21:24:131
194725917,26sleep3284-21dbus-daemon22:09:143
194725917,26sleep3284-21dbus-daemon22:09:143
1844225916,24sleep10-21swapper/121:19:171
2894025811,35sleep028995-21basename23:34:340
2465325812,21sleep10-21swapper/119:54:491
2075225815,25sleep10-21swapper/121:24:461
1392125816,22sleep30-21swapper/319:19:463
859225713,26sleep18659-21/usr/sbin/munin19:04:461
661525720,17sleep10-21swapper/120:39:521
661525720,17sleep10-21swapper/120:39:521
1156325711,20sleep00-21swapper/019:14:260
456525614,25sleep14550-21apt-get20:34:431
456525614,25sleep14550-21apt-get20:34:431
3028625613,25sleep10-21swapper/120:14:161
1391825614,25sleep113853-21apt-get21:04:351
1096825615,24sleep011013-21ntp_kernel_pll_00:19:400
1096825615,24sleep011013-21ntp_kernel_pll_00:19:400
96125512,27sleep3957-21apt-get23:49:273
765725513,26sleep37650-21apt-get20:44:343
765725513,26sleep37650-21apt-get20:44:343
2718925513,23sleep127259-21df23:29:311
2458025515,18sleep00-21swapper/023:19:450
1509225514,23sleep015137-21ntp_states19:24:400
1232625513,24sleep112345-21apt-get20:59:301
1232625513,24sleep112345-21apt-get20:59:301
732825413,23sleep37377-21/usr/sbin/munin22:24:413
3055525412,25sleep10-21swapper/121:54:461
3018925411,22sleep00-21swapper/021:54:370
260225416,18sleep30-21swapper/322:09:383
1916125414,18sleep10-21swapper/123:04:161
1554725422,17sleep00-21swapper/021:09:350
1554725422,17sleep00-21swapper/021:09:350
1371925412,20sleep00-21swapper/022:44:440
1210625414,27sleep312152-21latency_hist20:59:163
1210625414,27sleep312152-21latency_hist20:59:163
1517625317,20sleep10-21swapper/122:49:421
1214925314,22sleep012145-21apt-get00:24:290
1214925314,22sleep012145-21apt-get00:24:290
1188625316,16sleep30-21swapper/322:39:393
765925211,22sleep07727-21df20:44:350
765925211,22sleep07727-21df20:44:350
692525213,23sleep06919-21apt-get22:24:290
589425215,15sleep30-21swapper/320:39:173
589425215,15sleep30-21swapper/320:39:173
3186325211,25sleep131858-21apt-get23:44:291
3186325211,25sleep131858-21apt-get23:44:291
2411925212,24sleep024126-21df_inode19:54:360
2382425211,20sleep10-21swapper/119:54:171
2122025215,23sleep121311-21load23:09:371
1434925213,24sleep314375-21ntp_states21:04:493
657525110,18sleep00-21swapper/000:04:470
657525110,18sleep00-21swapper/000:04:470
2718725111,21sleep00-21swapper/021:45:040
2697225113,22sleep00-21swapper/020:04:170
2616925113,17sleep00-21swapper/023:24:450
1980225115,16sleep10-21swapper/123:04:401
850825014,19sleep30-21swapper/319:04:443
326692509,16sleep00-21swapper/023:44:480
326692509,16sleep00-21swapper/023:44:480
100025013,21sleep01068-21memory22:04:470
100025013,21sleep01068-21memory22:04:470
523224912,20sleep35255-21users23:59:493
523224912,20sleep35255-21users23:59:493
1166124910,22sleep311668-21df_inode19:14:343
435124811,21sleep04345-21apt-get23:59:300
435124811,21sleep04345-21apt-get23:59:300
2251424819,16sleep00-21swapper/023:14:280
194732487,15sleep00-21swapper/019:39:380
1653024810,13sleep30-21swapper/322:54:373
276224711,14sleep00-21swapper/023:54:250
276224711,14sleep00-21swapper/023:54:250
1023624713,21sleep110285-21idleruntime22:34:361
202472408,16sleep00-21swapper/023:04:470
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional