You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 20:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Dec 10, 2025 12:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
203792134116,1312sleep317321-21kworker/3:212:28:533
203792134116,1312sleep317321-21kworker/3:212:28:533
26291247761,389sleep026428-21unixbench_multi09:23:070
26291247761,389sleep026428-21unixbench_multi09:23:070
7193245856,342sleep07228-21latency11:47:580
29220232922,264sleep029366-21grep11:13:090
134872295160,98sleep013526-21ntp_kernel_err08:43:000
134872295160,98sleep013526-21ntp_kernel_err08:43:000
23977227114,242sleep323985-21ntpq07:33:043
31148226229,161sleep00-21swapper/009:38:100
15034224998,91sleep016192-1kworker/0:0H10:32:380
14320223078,116sleep014396-21switchtime07:03:080
14320223078,116sleep014396-21switchtime07:03:080
18046222310,163sleep018084-21idleruntime12:22:560
18046222310,163sleep018084-21idleruntime12:22:560
13201222314,193sleep313261-21fschecks_time08:42:553
13201222314,193sleep313261-21fschecks_time08:42:553
106422218,198sleep01110-21ntpq08:03:140
11270221942,129sleep011294-21awk08:37:350
11270221942,129sleep011294-21awk08:37:350
21705221576,61sleep00-21swapper/010:52:370
21705221576,61sleep00-21swapper/010:52:370
27796221322,174sleep227799-21sshd07:46:362
26380221217,180sleep326345-21latency_hist07:42:373
12483220816,176sleep312528-21ntp_kernel_pll_10:23:043
24346220213,98sleep023718-21/usr/sbin/munin10:58:470
32711219927,65sleep00-21swapper/008:02:380
75412198119,36sleep00-21swapper/010:07:580
1332821970,52sleep013534-21cyclictest12:07:570
23359219448,24sleep023513-21date10:57:380
11694219220,155sleep111740-21fschecks_count12:02:541
267321906,100sleep00-21swapper/008:08:070
267321906,100sleep00-21swapper/008:08:070
21012219051,124sleep00-21swapper/009:07:560
20577219057,53sleep00-21swapper/007:22:590
18628219027,59sleep00-21swapper/007:17:380
17779218814,153sleep317911-21head10:38:103
19846218038,55sleep00-21swapper/009:03:080
19846218038,55sleep00-21swapper/009:03:080
10066217947,48sleep00-21swapper/011:57:520
25401217031,97sleep025549-21latency_hist09:22:390
5003216713,90sleep00-21swapper/009:58:130
5003216713,90sleep00-21swapper/009:58:130
1484621619,18sleep00-21swapper/012:12:540
1484621619,18sleep00-21swapper/012:12:540
2825121589,129sleep00-21swapper/011:12:360
8194215728,47sleep00-21swapper/008:27:380
2345215741,63sleep00-21swapper/011:32:520
2345215741,63sleep00-21swapper/011:32:520
9773215530,76sleep09846-21sshd08:32:380
31015215521,108sleep331016-21cron07:57:363
19025215218,121sleep319058-21basename07:17:573
6483214729,80sleep06594-21ls10:03:140
15793214525,56sleep00-21swapper/007:07:570
12204214510,42sleep00-21swapper/012:03:050
32589214233,88sleep032684-21cut09:43:060
25711214010,114sleep025232-21/usr/sbin/munin11:03:030
25711214010,114sleep025232-21/usr/sbin/munin11:03:030
23203214026,92sleep323206-21cron07:32:363
23203214026,92sleep323206-21cron07:32:363
14289213729,69sleep014366-21expr10:28:140
16741213632,90sleep016822-21mailstats08:53:060
16741213632,90sleep016822-21mailstats08:53:060
25718213111,61sleep00-21swapper/007:38:100
877521298,74sleep00-21swapper/011:52:590
17654212820,89sleep217777-21timerandwakeup10:38:092
4238212533,75sleep00-21swapper/008:13:040
405621226,105sleep04075-21unin-run09:57:360
22189212120,82sleep222234-21modprobe09:12:352
191721215,81sleep00-21swapper/011:32:110
18268212116,90sleep118274-21ntpq08:58:041
15919212121,86sleep315961-21awk10:33:303
13423211942,47sleep013448-21munin-plugin-st07:02:350
29448211711,44sleep00-21swapper/009:33:050
13295211519,35sleep00-21swapper/010:26:440
5092211215,81sleep15176-21latency_hist08:17:371
15930211220,71sleep115947-21apt08:52:351
1865621117,15sleep10-21swapper/112:23:091
1865621117,15sleep10-21swapper/112:23:091
8892211013,79sleep28898-21ntpq08:28:042
8652211013,80sleep28716-21/usr/sbin/munin11:52:572
6702210931,29sleep00-21swapper/011:47:380
20143210911,82sleep220152-21pluginstate07:22:372
1882321095,11sleep00-21swapper/008:59:280
19981210720,73sleep220058-21grep09:03:102
19981210720,73sleep220058-21grep09:03:102
10554210714,17sleep20-21swapper/210:17:502
7495210651,34sleep17599-21timerandwakeup08:23:061
22039210613,68sleep121919-21/usr/sbin/munin07:27:571
22039210613,68sleep121919-21/usr/sbin/munin07:27:571
25536210515,72sleep125568-21/usr/sbin/munin07:38:071
21944210511,29sleep00-21swapper/007:27:520
21944210511,29sleep00-21swapper/007:27:520
1278721055,11sleep00-21swapper/008:42:080
10149210511,78sleep310196-21fschecks_count11:57:553
3274921046,12sleep00-21swapper/011:26:180
27403210419,70sleep00-21swapper/009:27:520
20341210416,69sleep219859-21apt-get10:47:462
20341210416,69sleep219859-21apt-get10:47:462
31872210318,65sleep131811-21latency_hist09:42:391
10519210315,75sleep00-21swapper/010:17:400
47221028,78sleep132578-21apt-get08:02:391
4925210131,51sleep35001-21awk09:58:123
4925210131,51sleep35001-21awk09:58:123
28172210110,48sleep00-21swapper/007:47:510
2081210115,69sleep32093-21vmstat09:48:103
3216621000,7sleep30-21swapper/307:58:073
1252429913,73sleep312526-21sshd08:38:473
971929817,57sleep28975-21/usr/sbin/munin10:13:092
971929817,57sleep28975-21/usr/sbin/munin10:13:092
246452986,10sleep00-21swapper/007:35:560
792829720,63sleep28013-21mailstats10:08:092
2297129717,21sleep322972-21ntp_states09:13:043
1506029725,54sleep115023-21latency_hist07:07:371
1506029725,54sleep115023-21latency_hist07:07:371
2587829419,56sleep325943-21latency09:22:583
2587829419,56sleep325943-21latency09:22:583
2378929418,58sleep223736-21apt-get10:58:212
3107829213,57sleep330503-21/usr/sbin/munin09:38:093
275729217,59sleep32808-21cpuspeed09:52:513
1988929216,59sleep019943-21latency_hist10:47:360
43292919,44sleep04408-21systemctl11:38:090
1750929110,61sleep217566-21/usr/sbin/munin07:13:042
353929018,59sleep23580-21cat08:12:372
353929018,59sleep23580-21cat08:12:372
1671828825,32sleep00-21swapper/010:37:140
541828729,29sleep30-21swapper/308:17:553
227128728,31sleep20-21swapper/211:32:502
227128728,31sleep20-21swapper/211:32:502
1891828716,30sleep00-21swapper/007:17:540
1511428728,32sleep20-21swapper/207:07:382
1511428728,32sleep20-21swapper/207:07:382
954428619,29sleep00-21swapper/010:13:070
954428619,29sleep00-21swapper/010:13:070
358228614,54sleep33482-21latency_hist08:12:383
358228614,54sleep33482-21latency_hist08:12:383
68542850,7sleep10-21swapper/111:47:381
278782859,40sleep027885-21modprobe07:47:350
175028526,19sleep00-21swapper/009:48:070
2351428421,18sleep10-21swapper/110:57:391
537428320,44sleep35347-21/usr/sbin/munin11:42:523
2085628319,49sleep320917-21processes10:48:163
2085628319,49sleep320917-21processes10:48:163
1355028326,24sleep30-21swapper/307:02:393
1983728121,31sleep30-21swapper/309:03:073
1983728121,31sleep30-21swapper/309:03:073
3054928022,16sleep00-21swapper/011:18:060
3054928022,16sleep00-21swapper/011:18:060
1226828027,24sleep10-21swapper/106:58:131
2953127924,33sleep329674-21cat07:52:393
2454627914,50sleep224622-21mailstats09:18:042
172502796,62sleep322356-1kworker/3:0H12:18:313
961527818,40sleep18975-21/usr/sbin/munin10:13:081
961527818,40sleep18975-21/usr/sbin/munin10:13:081
2311727830,12sleep00-21swapper/009:13:090
2119427813,51sleep321241-21latency09:07:593
1325227816,42sleep213267-21sort12:07:562
468727716,24sleep20-21swapper/209:58:032
468727716,24sleep20-21swapper/209:58:032
1786127723,29sleep30-21swapper/312:22:513
1786127723,29sleep30-21swapper/312:22:513
1525927728,12sleep00-21swapper/008:48:130
62922766,36sleep00-21swapper/008:18:450
1859627620,26sleep30-21swapper/307:17:373
94227524,29sleep30-21swapper/309:47:383
3171627524,24sleep30-21swapper/311:22:573
2849327519,29sleep30-21swapper/311:12:523
2562527521,35sleep325707-21sed11:03:023
2562527521,35sleep325707-21sed11:03:023
1994627528,23sleep10-21swapper/110:47:371
195727516,36sleep32070-21latency_hist08:07:373
194102758,30sleep00-21swapper/012:27:490
194102758,30sleep00-21swapper/012:27:490
1651727522,14sleep00-21swapper/012:17:550
1282827426,29sleep20-21swapper/206:58:462
739427318,28sleep30-21swapper/310:07:493
706327314,40sleep27133-21/usr/sbin/munin11:47:562
3004427316,18sleep20-21swapper/211:17:512
3004427316,18sleep20-21swapper/211:17:512
2572527318,24sleep10-21swapper/111:03:041
2572527318,24sleep10-21swapper/111:03:041
1507527315,44sleep215116-21ntpq08:48:032
2779327226,29sleep227837-21ntpq09:28:182
888127019,38sleep18891-21ntp_states08:28:031
2067327023,25sleep10-21swapper/109:07:371
2067327023,25sleep10-21swapper/109:07:371
3023326919,32sleep230293-21/usr/sbin/munin07:53:062
2373026918,22sleep10-21swapper/107:32:581
1791426939,15sleep00-21swapper/010:38:110
282226817,33sleep12895-21switchtime11:33:051
282226817,33sleep12895-21switchtime11:33:051
1494026816,37sleep114970-21idleruntime12:12:551
1494026816,37sleep114970-21idleruntime12:12:551
527326713,36sleep15274-21sshd09:58:471
527326713,36sleep15274-21sshd09:58:471
308922676,50sleep00-21swapper/007:56:030
2817326717,25sleep30-21swapper/311:12:353
2730126724,26sleep227342-21tr11:08:022
1827526716,22sleep20-21swapper/208:58:052
1598226719,24sleep20-21swapper/210:33:332
1556126714,15sleep10-21swapper/107:07:461
692426614,38sleep16925-21http11:47:511
2097626619,19sleep20-21swapper/207:23:102
1600926624,21sleep30-21swapper/308:52:373
394926517,33sleep34035-21/usr/sbin/munin11:38:003
2536926519,30sleep325397-21latency_hist09:22:373
2379726519,25sleep30-21swapper/310:58:263
1858826517,31sleep218633-21/usr/sbin/munin10:42:542
134026523,20sleep20-21swapper/208:03:202
1027526518,30sleep29989-21/usr/sbin/munin08:33:012
1027526518,30sleep29989-21/usr/sbin/munin08:33:012
583926422,23sleep20-21swapper/210:02:472
2415826412,27sleep10-21swapper/109:17:551
165272645,13sleep20-21swapper/208:53:002
165272645,13sleep20-21swapper/208:53:002
1499626420,18sleep30-21swapper/310:32:373
1499626420,18sleep30-21swapper/310:32:373
1443926419,27sleep114501-21sendmail-msp12:12:351
1202026412,37sleep112026-21ntp_states08:38:051
1055326418,28sleep110546-21apt-get10:17:491
659626317,29sleep16730-21/usr/sbin/munin10:03:151
321126317,32sleep23217-21ntp_states09:53:022
3176726312,27sleep20-21swapper/207:58:032
1521226317,19sleep30-21swapper/312:13:023
1521226317,19sleep30-21swapper/312:13:023
779226215,33sleep17826-21ntp_kernel_err10:08:041
736026215,26sleep30-21swapper/308:23:043
586826219,27sleep35936-21df10:02:523
2966226215,20sleep30-21swapper/309:33:083
2925226225,22sleep20-21swapper/209:32:592
184472625,12sleep30-21swapper/310:42:383
1326926225,22sleep30-21swapper/312:07:573
120426217,31sleep11294-21grep11:28:081
713426117,24sleep30-21swapper/311:47:573
604526124,15sleep10-21swapper/108:18:101
3050526113,35sleep330545-21ps11:18:053
3050526113,35sleep330545-21ps11:18:053
2614226116,28sleep226211-21/usr/sbin/munin09:23:052
2614226116,28sleep226211-21/usr/sbin/munin09:23:052
2248126114,29sleep122502-21apt-get09:12:461
1438126120,25sleep214420-21modprobe08:47:352
1438126120,25sleep214420-21modprobe08:47:352
1336526121,24sleep20-21swapper/210:27:352
301226018,22sleep30-21swapper/311:33:073
301226018,22sleep30-21swapper/311:33:073
2873726023,20sleep10-21swapper/109:32:371
2867126012,26sleep20-21swapper/207:48:062
2661926018,25sleep126613-21apt-get07:42:521
2203726015,20sleep20-21swapper/210:52:552
2202826018,25sleep122036-21diskstats10:52:541
1994826017,26sleep120017-21sensors_temp12:28:061
1994826017,26sleep120017-21sensors_temp12:28:061
1717526018,24sleep117213-21users12:18:091
1630726015,27sleep216301-21apt-get12:17:492
1064926013,19sleep30-21swapper/310:17:553
3235225922,23sleep10-21swapper/109:42:591
2865925917,18sleep30-21swapper/309:32:353
2584225918,19sleep20-21swapper/211:03:092
2584225918,19sleep20-21swapper/211:03:092
2378725917,22sleep20-21swapper/207:32:592
2333325915,18sleep20-21swapper/207:32:392
2326725916,26sleep323349-21munin-run10:57:373
2173825916,20sleep10-21swapper/110:52:381
2173825916,20sleep10-21swapper/110:52:381
2039425918,20sleep30-21swapper/307:22:543
1033225918,28sleep210376-21latency11:57:582
914625821,21sleep30-21swapper/311:53:103
77025811,31sleep2665-21/usr/sbin/munin11:27:532
385525812,30sleep23861-21df_inode08:12:532
311662587,40sleep10-21swapper/111:20:531
311662587,40sleep10-21swapper/111:20:531
2092825814,23sleep120998-21df09:07:531
1697225817,25sleep20-21swapper/207:12:352
1671625814,23sleep30-21swapper/308:53:053
1671625814,23sleep30-21swapper/308:53:053
128125819,17sleep20-21swapper/209:47:572
1062925820,18sleep30-21swapper/308:33:103
1062925820,18sleep30-21swapper/308:33:103
81072575,11sleep20-21swapper/208:27:072
49232575,10sleep20-21swapper/211:40:372
2876425715,24sleep20-21swapper/211:12:592
283425718,24sleep12840-21diskstats09:52:531
2690825716,24sleep126993-21apt-config09:27:361
2690825716,24sleep126993-21apt-config09:27:361
219625714,26sleep12191-21apt-get08:07:521
219625714,26sleep12191-21apt-get08:07:521
1821025710,31sleep317783-21/usr/sbin/munin08:58:023
167082574,17sleep10-21swapper/110:37:071
1046625716,22sleep10-21swapper/111:58:011
1256725617,23sleep112573-21ntp_states10:23:061
97052554,37sleep10-21swapper/111:56:481
3203125518,21sleep232050-21ntp_states11:23:042
2674025518,21sleep326826-21perl07:42:583
2298125516,19sleep20-21swapper/209:13:072
2089625513,27sleep120975-21systemctl07:23:091
1682325518,16sleep10-21swapper/108:53:071
1682325518,16sleep10-21swapper/108:53:071
1578825518,21sleep30-21swapper/307:07:563
149932555,10sleep30-21swapper/307:07:233
149932555,10sleep30-21swapper/307:07:233
2829425415,15sleep10-21swapper/111:12:371
1786025413,24sleep217854-21apt-get12:22:502
1786025413,24sleep217854-21apt-get12:22:502
1414325412,23sleep113639-21/usr/sbin/munin10:28:111
1414325412,23sleep113639-21/usr/sbin/munin10:28:111
1287925418,17sleep20-21swapper/208:42:382
426925314,23sleep14280-21ntp_states11:38:061
1711925313,22sleep117123-21diskstats10:37:561
1318725316,21sleep113195-21df_inode08:42:531
1318725316,21sleep113195-21df_inode08:42:531
545325215,21sleep25459-21diskstats11:42:562
1776525215,17sleep10-21swapper/107:13:111
1529425211,24sleep215364-21sensors_temp12:13:052
1529425211,24sleep215364-21sensors_temp12:13:052
95112516,34sleep30-21swapper/308:30:193
66425117,20sleep30-21swapper/311:27:463
320552519,24sleep232074-21apt-get09:42:482
3139025120,13sleep10-21swapper/107:57:541
3107025115,17sleep10-21swapper/107:57:381
198882515,10sleep10-21swapper/107:19:531
1287325112,24sleep212919-21systemctl12:07:352
919025017,18sleep30-21swapper/310:12:583
919025017,18sleep30-21swapper/310:12:583
3208225017,16sleep10-21swapper/111:23:071
3029125014,20sleep10-21swapper/109:37:381
2936925012,21sleep129397-21users11:13:101
1167825018,17sleep30-21swapper/312:02:523
405024914,16sleep30-21swapper/308:12:593
2817324913,20sleep10-21swapper/107:47:521
2194124913,20sleep221938-21apt-get07:27:502
1366724813,16sleep20-21swapper/210:27:522
1366724813,16sleep20-21swapper/210:27:522
1022224810,22sleep110274-21cat08:33:001
1022224810,22sleep110274-21cat08:33:001
2507224712,19sleep225068-21apt-get07:37:522
198682478,15sleep20-21swapper/212:28:032
198682478,15sleep20-21swapper/212:28:032
3074424612,18sleep20-21swapper/209:37:592
118424612,17sleep10-21swapper/109:47:521
533224410,17sleep20-21swapper/208:17:502
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional