You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-04 - 00:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Jun 03, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3681247829,4760sleep032763-21kworker/0:110:48:160
283192210118,2067sleep128321-21ntp_states08:38:411
12422210688,1048sleep312464-21ntpq07:48:403
10644265047,481sleep310781-21grep09:23:493
4405252519,488sleep04454-21/usr/sbin/munin12:33:290
21004247634,418sleep321020-21ls11:43:143
1865723778,356sleep318698-21ntpq08:08:433
32442231544,196sleep332584-21head10:33:503
8901230489,195sleep39038-21unixbench_singl09:19:163
29171228818,251sleep129252-21fschecks_time12:08:351
4959227276,81sleep30-21swapper/309:08:343
265652249124,18sleep30-21swapper/308:33:383
23386222286,118sleep323169-21/usr/sbin/munin08:23:413
28334222158,59sleep30-21swapper/308:38:433
20900221511,180sleep30-21swapper/309:58:333
2304221035,111sleep30-21swapper/308:58:453
22804220415,166sleep322901-21gpgv08:23:163
1280322014,187sleep00-21swapper/011:16:050
17924219810,174sleep217933-21unin-run11:33:152
6709219354,82sleep36285-21/usr/sbin/munin09:13:433
6709219354,82sleep36285-21/usr/sbin/munin09:13:433
287421927,174sleep32959-21tr10:43:433
23666217712,146sleep223690-21/usr/sbin/munin08:23:482
31736217313,108sleep331793-21fschecks_count07:08:353
31736217313,108sleep331793-21fschecks_count07:08:353
1526421588,135sleep315365-21fschecks_time07:58:333
29882215527,108sleep329944-21processes08:43:433
22132215412,125sleep222105-21latency_hist10:03:162
30229214918,57sleep30-21swapper/310:28:333
6445214513,116sleep36569-21sed07:28:463
6445214513,116sleep36569-21sed07:28:463
29078214312,21sleep30-21swapper/310:23:443
500214219,43sleep1559-21cat07:13:171
500214219,43sleep1559-21cat07:13:171
32279214220,72sleep332367-21latency_hist08:53:163
12967214125,97sleep212976-21modprobe11:18:132
930121409,114sleep38929-21/usr/sbin/munin07:38:393
14028213921,103sleep30-21swapper/307:53:413
11463213810,113sleep311607-21date11:13:183
23626213742,41sleep30-21swapper/310:08:143
17452213514,106sleep017531-21cat09:48:160
203321329,76sleep32178-21grep12:23:473
7735213171,31sleep00-21swapper/009:18:160
7735213171,31sleep00-21swapper/009:18:160
26370213013,103sleep326404-21sed11:58:403
24425212919,93sleep00-21swapper/010:08:430
20576212927,40sleep30-21swapper/309:58:173
1278521250,10sleep212-21rcu_sched07:48:462
27050212422,85sleep30-21swapper/310:18:273
29613212311,20sleep30-21swapper/312:08:453
3281212224,41sleep30-21swapper/309:03:343
1778821207,11sleep30-21swapper/311:31:593
4218211921,18sleep30-21swapper/312:33:163
323921198,99sleep33319-21tr07:18:443
28195211914,90sleep228304-21sort12:03:442
24525211912,91sleep124599-21find10:08:441
1441211913,57sleep31561-21timerandwakeup10:38:463
21168211642,62sleep321217-21grep08:18:163
15204211637,19sleep30-21swapper/311:23:393
2535921136,10sleep00-21swapper/011:54:290
4480211115,82sleep24490-21sshd12:33:312
10963211021,70sleep210996-21grep07:43:432
19510210913,80sleep219629-21date11:38:162
4848210814,37sleep30-21swapper/307:23:453
24952210728,65sleep325028-21cat11:53:443
17370210611,79sleep317492-21sed08:03:463
860321058,40sleep30-21swapper/311:03:293
8690210411,77sleep08712-21cut11:03:340
605321047,13sleep20-21swapper/207:28:372
605321047,13sleep20-21swapper/207:28:372
30154210412,32sleep20-21swapper/207:03:342
30644210314,72sleep130718-21sshd10:28:431
29951210210,76sleep030021-21sensors_temp08:43:450
16580210220,29sleep30-21swapper/309:43:393
14587210211,37sleep30-21swapper/311:23:153
19933210113,69sleep019964-21idleruntime11:38:340
10536210115,69sleep210642-21ls09:23:482
4187210018,58sleep30-21swapper/310:48:353
3221121007,44sleep332207-21apt-get12:18:263
3221121007,44sleep332207-21apt-get12:18:263
1831721007,42sleep318377-21perl11:33:333
16993210013,74sleep217068-21cat08:03:372
863629911,35sleep30-21swapper/307:38:143
68082997,42sleep36807-21munin-run10:58:163
1765729915,36sleep30-21swapper/309:48:193
2518329810,35sleep30-21swapper/308:28:423
1585429712,72sleep315900-21unin-run09:43:153
128329717,64sleep21301-21apt12:23:262
2907729425,43sleep029071-21apt-get12:08:300
28542948,38sleep30-21swapper/312:28:283
1500929410,66sleep00-21swapper/009:38:390
616829237,30sleep00-21swapper/010:53:450
3109029239,17sleep331152-21ps12:13:423
3109029239,17sleep331152-21ps12:13:423
240672926,74sleep30-21swapper/311:52:473
1749429213,61sleep017624-21uptime08:03:470
74692919,32sleep30-21swapper/310:58:403
2555329121,53sleep225642-21fschecks_time10:13:342
254622917,32sleep30-21swapper/310:13:273
989128934,41sleep029001-1kworker/0:2H09:23:250
100528911,63sleep3512-21munin-node07:13:233
1026128818,54sleep010364-21latency_hist07:43:160
714428613,61sleep17236-21sort10:58:341
2129528619,46sleep221289-21apt-get11:43:292
3189628522,43sleep231937-21cat12:18:132
3189628522,43sleep231937-21cat12:18:132
2415328513,55sleep024265-21cat11:53:160
76662840,7sleep00-21swapper/010:58:440
1283428431,16sleep312979-21modprobe09:33:183
3177928312,54sleep031788-21diskstats10:33:340
2273828314,30sleep00-21swapper/010:03:390
2105528338,26sleep10-21swapper/111:43:161
1630228328,29sleep00-21swapper/011:28:170
1277728314,54sleep212832-21latency_hist09:33:172
1255628315,56sleep30-21swapper/309:31:033
777928213,48sleep17827-21ntp_offset07:33:401
3132228227,36sleep331326-21cron07:08:153
2855028218,37sleep228693-21grep08:38:462
140428211,21sleep30-21swapper/308:58:173
140428211,21sleep30-21swapper/308:58:173
129312826,29sleep30-21swapper/311:17:463
1231728223,31sleep10-21swapper/107:48:381
2263828115,34sleep20-21swapper/210:03:372
222852815,30sleep30-21swapper/311:44:473
645828012,51sleep26520-21/usr/sbin/munin09:13:382
645828012,51sleep26520-21/usr/sbin/munin09:13:382
2483328019,38sleep024898-21fschecks_time08:28:340
1932327925,34sleep00-21swapper/009:53:330
1827627914,27sleep10-21swapper/108:08:291
3039627821,24sleep20-21swapper/212:13:172
177672788,19sleep00-21swapper/011:31:440
327627714,47sleep23280-21diskstats09:03:332
3051227714,48sleep030542-21/usr/sbin/munin10:28:380
241327723,34sleep12437-21apt07:18:141
400527623,22sleep00-21swapper/010:48:320
2950227623,30sleep20-21swapper/208:43:342
1836727619,41sleep218462-21perl08:08:382
1046827613,20sleep10-21swapper/109:23:471
629127516,28sleep00-21swapper/007:28:430
629127516,28sleep00-21swapper/007:28:430
251442756,36sleep30-21swapper/310:12:503
135972757,12sleep10-21swapper/107:53:311
91242748,12sleep20-21swapper/211:03:442
91027413,35sleep20-21swapper/208:53:482
91027413,35sleep20-21swapper/208:53:482
825127420,34sleep08279-21fschecks_count09:18:570
35632748,46sleep00-21swapper/009:03:390
295627313,44sleep23074-21date09:03:162
274952738,49sleep30-21swapper/312:03:243
2020227315,40sleep219726-21/usr/sbin/munin11:38:402
1178427311,47sleep111786-21diskstats11:13:321
56227223,25sleep20-21swapper/207:13:182
56227223,25sleep20-21swapper/207:13:182
672827127,25sleep10-21swapper/110:58:141
3014627121,31sleep130153-21/usr/sbin/munin07:03:331
300302710,8sleep00-21swapper/010:28:160
266122718,48sleep00-21swapper/010:16:430
198127120,29sleep20-21swapper/207:13:472
107892710,7sleep10-21swapper/107:43:371
432327014,23sleep20-21swapper/210:48:382
245427010,45sleep12466-21/usr/sbin/munin10:43:321
1201527020,28sleep10-21swapper/109:28:431
524726913,39sleep05315-21run-parts09:08:440
2859926913,37sleep228666-21df10:23:312
2271426920,27sleep20-21swapper/208:23:152
1609226917,34sleep116130-21http11:28:141
1180426925,24sleep30-21swapper/307:48:163
2316826817,27sleep10-21swapper/108:23:271
143212685,12sleep20-21swapper/211:20:192
91426717,27sleep00-21swapper/010:38:350
322522678,17sleep10-21swapper/110:33:481
168422675,13sleep10-21swapper/111:29:271
1194926716,35sleep012014-21awk09:28:420
713926618,27sleep10-21swapper/107:33:161
713926618,27sleep10-21swapper/107:33:161
317222669,34sleep231836-21unixbench_singl08:48:512
194192667,11sleep00-21swapper/008:11:470
1805826621,21sleep00-21swapper/009:48:370
750026514,20sleep00-21swapper/007:33:350
74826519,26sleep10-21swapper/108:53:471
74826519,26sleep10-21swapper/108:53:471
639826521,22sleep10-21swapper/109:13:371
639826521,22sleep10-21swapper/109:13:371
359226514,22sleep10-21swapper/112:28:461
3197726513,37sleep032067-21sort12:18:150
3197726513,37sleep032067-21sort12:18:150
3107326514,36sleep131081-21ntp_states12:13:401
3107326514,36sleep131081-21ntp_states12:13:401
2878226519,27sleep128833-21unin-run12:08:151
2827026516,33sleep028317-21ntpq08:38:400
2284726523,22sleep10-21swapper/111:48:291
1776926517,26sleep20-21swapper/209:48:312
2049626417,19sleep10-21swapper/109:58:151
151992649,27sleep10-21swapper/109:38:451
1461626418,16sleep00-21swapper/011:23:160
1052026420,23sleep00-21swapper/011:08:380
3268026320,24sleep00-21swapper/012:18:440
3268026320,24sleep00-21swapper/012:18:440
2868326318,23sleep10-21swapper/110:23:341
224392636,12sleep00-21swapper/008:20:190
2029426312,35sleep020358-21ps08:13:440
1963726315,29sleep20-21swapper/209:53:392
57792626,11sleep10-21swapper/109:10:101
2712126213,22sleep20-21swapper/210:18:312
2489926215,26sleep10-21swapper/108:28:351
218392627,43sleep00-21swapper/010:00:070
1516926218,27sleep115164-21apt-get07:58:271
1481126217,29sleep214812-21/usr/sbin/munin11:23:262
1391426218,26sleep114031-21/usr/sbin/munin11:18:451
903626117,30sleep29039-21entropy07:38:342
455426112,31sleep04646-21/usr/sbin/munin07:23:380
37912616,43sleep00-21swapper/007:20:430
3146526113,33sleep231522-21cat10:33:162
2259726115,25sleep00-21swapper/011:48:150
2167826119,23sleep10-21swapper/108:18:381
163826116,28sleep11635-21apt-get08:58:271
1565426118,22sleep00-21swapper/007:58:430
3153126018,24sleep00-21swapper/008:48:490
2763226010,27sleep10-21swapper/108:38:161
2608926015,29sleep226236-21cut08:33:172
2444326012,19sleep20-21swapper/211:53:302
2437626013,33sleep224383-21ntpq10:08:412
1492526013,24sleep20-21swapper/207:58:162
119726018,19sleep10-21swapper/110:38:401
1185326015,31sleep012003-21latency_hist07:48:170
1054426013,32sleep110569-21/usr/sbin/munin11:08:401
476325816,21sleep20-21swapper/209:08:292
439625815,18sleep10-21swapper/110:48:411
3247925818,21sleep20-21swapper/212:18:382
3247925818,21sleep20-21swapper/212:18:382
303302585,12sleep10-21swapper/108:43:501
2643325816,21sleep20-21swapper/211:58:432
2445725819,26sleep0225-21systemd-journal08:28:140
2263025810,23sleep10-21swapper/111:48:161
1505325819,26sleep214568-21/usr/sbin/munin09:38:412
1301525813,27sleep113007-21apt-get09:33:281
798225717,27sleep28062-21timerandwakeup07:33:452
279982575,42sleep10-21swapper/110:19:141
2573625718,19sleep10-21swapper/110:13:371
185725714,26sleep11920-21/usr/sbin/munin12:23:451
1843225715,24sleep10-21swapper/111:33:351
165372575,12sleep20-21swapper/209:43:382
3124125614,26sleep131336-21memory08:48:421
2705225613,25sleep027046-21apt-get10:18:280
2597225615,25sleep025965-21apt-get11:58:290
1979925613,27sleep219818-21apt-get08:13:262
1660225610,31sleep116640-21ps09:43:411
1426625617,18sleep00-21swapper/007:53:460
326725513,25sleep13275-21df_inode09:03:321
324932557,12sleep00-21swapper/008:53:250
324932557,12sleep00-21swapper/008:53:250
3232025513,27sleep232441-21sed10:33:492
285625517,20sleep00-21swapper/012:28:290
283025513,24sleep22939-21perl07:18:352
2675525513,28sleep026769-21sshd08:33:430
944025416,18sleep10-21swapper/111:03:471
334025410,15sleep20-21swapper/212:28:432
3133325412,25sleep031394-21unin-run07:08:160
2448925410,28sleep124575-21perl11:53:331
188042545,11sleep10-21swapper/109:51:241
1301625416,20sleep20-21swapper/209:33:292
55042535,36sleep20-21swapper/207:27:482
3165325313,23sleep031720-21df07:08:320
3165325313,23sleep031720-21df07:08:320
1990725313,24sleep119909-21diskstats08:13:331
1246725314,24sleep012501-21ntp_states07:48:410
812625214,23sleep27598-21apt-get09:18:272
3272225213,22sleep1324-21mailstats12:18:451
3272225213,22sleep1324-21mailstats12:18:451
1762525214,23sleep117639-21vmstat08:03:481
944125115,17sleep10-21swapper/107:38:451
44012517,21sleep10-21swapper/112:33:261
2633225117,19sleep126345-21grep11:58:381
1824225110,18sleep00-21swapper/011:33:300
161625110,16sleep00-21swapper/008:58:260
138225112,22sleep01391-21df_inode12:23:320
105702518,17sleep20-21swapper/211:08:412
546925011,22sleep25465-21apt-get10:53:272
318725011,22sleep13193-21ntp_states07:18:421
1648825012,14sleep20-21swapper/208:03:172
1344325011,15sleep00-21swapper/009:33:390
89824914,20sleep2907-21df_inode10:38:332
437924911,21sleep14464-21perl07:23:351
2081024911,22sleep120807-21apt-get09:58:271
784024810,16sleep10-21swapper/109:18:171
273624713,16sleep20-21swapper/210:43:382
2258824714,16sleep10-21swapper/110:03:361
80824510,20sleep0851-21ntp_states07:13:200
265382447,17sleep20-21swapper/208:33:372
275872439,16sleep00-21swapper/012:03:300
2410224310,14sleep20-21swapper/211:53:142
723824011,11sleep20-21swapper/210:58:352
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional