You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 17:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sat Nov 29, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
619270106,6992sleep131663-21kworker/1:112:04:141
163532246210,2435sleep221461-1kworker/2:1H11:13:552
32508281419,779sleep232539-21memory08:38:552
209902642480,104sleep120992-21sshd08:03:321
32385243047,291sleep132420-21munin-plugin-st10:23:131
26134235111,326sleep026172-21irqstats11:43:500
18389228831,225sleep00-21swapper/009:38:260
4149227980,104sleep14296-21latency_hist12:18:161
23802227314,238sleep0419-21cron11:38:120
23802227314,238sleep0419-21cron11:38:120
26736225675,87sleep10-21swapper/110:03:551
22771225065,170sleep122799-21idleruntime-cro09:53:131
7198223113,24sleep17202-21cron09:03:131
7198223113,24sleep17202-21cron09:03:131
6479222712,192sleep16553-21sort12:23:451
6799222226,180sleep16919-21latency_hist07:18:161
16547221312,182sleep116590-21awk09:33:141
15146220810,187sleep115189-21ntp_kernel_pll_07:43:451
12644220411,176sleep112857-21latency_hist11:03:161
8695220016,112sleep18839-21tail10:48:461
12964219210,108sleep10-21swapper/107:38:151
19124218716,88sleep10-21swapper/109:38:471
2000921828,52sleep120034-21df09:43:321
31832217111,97sleep131856-21modprobe12:03:131
14195217146,29sleep10-21swapper/111:08:131
19673216718,131sleep319697-21basename09:43:133
1796321678,13sleep00-21swapper/007:53:330
1796321678,13sleep00-21swapper/007:53:330
517821649,138sleep10-21swapper/107:13:151
517821649,138sleep10-21swapper/107:13:151
18353216418,129sleep318418-21awk07:53:443
18353216418,129sleep318418-21awk07:53:443
2664121637,29sleep10-21swapper/108:19:271
1898421565,27sleep10-21swapper/107:56:091
1898421565,27sleep10-21swapper/107:56:091
4375215435,29sleep10-21swapper/112:18:321
25677214815,118sleep225670-21apt-get08:18:312
29628214042,80sleep10-21swapper/110:13:321
25406213934,86sleep125431-21munin-run11:43:141
25406213934,86sleep125431-21munin-run11:43:141
13782213811,115sleep113863-21switchtime07:38:441
740321376,73sleep10-21swapper/107:18:411
19528213720,97sleep119617-21fschecks_time07:58:351
31911213645,69sleep10-21swapper/108:38:171
18143213618,100sleep217845-21/usr/sbin/munin11:18:402
2542213322,67sleep11760-21/usr/sbin/munin10:28:511
2542213322,67sleep11760-21/usr/sbin/munin10:28:511
6732212817,91sleep10-21swapper/110:43:371
6732212817,91sleep10-21swapper/110:43:371
3184212831,83sleep13208-21unin-run07:08:151
8764212722,88sleep11-21systemd09:08:141
8764212722,88sleep11-21systemd09:08:141
22224212713,99sleep122270-21modprobe11:33:131
22224212713,99sleep122270-21modprobe11:33:131
5283212665,38sleep10-21swapper/110:38:401
2492621269,44sleep10-21swapper/109:58:381
32437212419,87sleep332525-21latency_hist10:23:163
28555212419,77sleep30-21swapper/308:28:173
28555212419,77sleep30-21swapper/308:28:173
2601121228,13sleep10-21swapper/111:43:391
1684821216,11sleep10-21swapper/107:48:451
10897212112,18sleep10-21swapper/110:58:151
10897212112,18sleep10-21swapper/110:58:151
1988211914,84sleep02025-21fschecks_count07:03:330
6687211721,74sleep2225-21systemd-journal07:18:132
1791521179,96sleep00-21swapper/009:35:310
11951211710,93sleep112088-21date09:18:151
2925321167,72sleep129287-21ps08:28:521
24360211613,86sleep124363-21cron09:58:141
10729211610,82sleep010775-21fschecks_count09:13:340
24782211410,59sleep124905-21timerwakeupswit08:13:451
15756211311,81sleep115827-21http11:13:131
30698211218,77sleep230760-21perl11:58:382
26045211126,24sleep20-21swapper/210:03:152
2808021099,86sleep128172-21tune2fs10:08:361
2563210912,79sleep32650-21latency_hist08:48:173
23513210931,18sleep10-21swapper/108:09:331
14760210915,77sleep314755-21apt-get07:43:303
4754210728,39sleep14793-21ntpq08:53:401
4754210728,39sleep14793-21ntpq08:53:401
24705210714,71sleep024780-21grep08:13:440
29669210512,28sleep029761-21perl10:13:350
28861210530,61sleep328878-21uniq08:28:343
16861210520,18sleep10-21swapper/109:33:261
9863210414,77sleep39909-21sed10:53:413
9863210414,77sleep39909-21sed10:53:413
17641210410,80sleep317675-21cat07:53:163
16284210317,68sleep015802-21apt-get11:13:250
10392210327,64sleep110401-21munin-run09:13:151
10392210327,64sleep110401-21munin-run09:13:151
27004210214,69sleep027074-21/usr/sbin/munin10:04:020
11044210212,18sleep20-21swapper/209:13:402
5188210116,59sleep25327-21latency_hist07:13:162
5188210116,59sleep25327-21latency_hist07:13:162
3820210127,38sleep00-21swapper/007:08:260
3820210127,38sleep00-21swapper/007:08:260
13529210131,22sleep10-21swapper/109:23:161
23806210025,62sleep10-21swapper/111:38:131
23806210025,62sleep10-21swapper/111:38:131
673829917,68sleep00-21swapper/007:18:150
630629912,73sleep06349-21ntpq12:23:400
1972629910,75sleep219763-21sed09:43:162
13732995,12sleep10-21swapper/110:26:451
124629919,18sleep10-21swapper/108:43:301
883329817,63sleep28995-21apt-config12:33:152
1160129812,60sleep30-21swapper/310:58:423
335729722,15sleep10-21swapper/110:33:271
2719629720,61sleep227284-21latency_hist11:48:162
1300929720,51sleep212945-21latency_hist07:38:162
1249729722,52sleep012640-21latency_hist11:03:150
959129611,72sleep25666-21snmpd10:53:322
959129611,72sleep25666-21snmpd10:53:322
85932945,10sleep10-21swapper/112:31:241
492229411,12sleep30-21swapper/308:53:463
492229411,12sleep30-21swapper/308:53:463
490929313,67sleep24979-21/usr/sbin/munin12:18:452
304462939,26sleep130524-21latency_hist11:58:171
633929113,62sleep26373-21/usr/sbin/munin08:58:422
633929113,62sleep26373-21/usr/sbin/munin08:58:422
1330429121,50sleep313396-21fschecks_time07:38:343
1311129118,53sleep0369-21sshd09:18:510
1006229016,13sleep10-21swapper/107:28:301
873428921,50sleep08790-21tr12:33:130
2933328926,51sleep029371-21cat10:13:150
1786928820,15sleep10-21swapper/111:18:311
1172228817,54sleep211808-21switchtime10:58:452
81328729,24sleep20-21swapper/206:58:432
590128713,28sleep35919-21apt-get12:23:273
131322870,11sleep20-21swapper/211:03:482
288022867,64sleep128947-21sed11:53:161
726228517,36sleep30-21swapper/309:03:153
726228517,36sleep30-21swapper/309:03:153
147628518,32sleep10-21swapper/112:08:381
2125628425,29sleep00-21swapper/008:03:390
2640828320,43sleep225924-21apt-get10:03:242
1713728312,52sleep317229-21/usr/sbin/munin09:33:363
1374228317,50sleep313812-21df09:23:303
28252827,40sleep12876-21df08:48:311
852828116,48sleep30-21swapper/307:23:273
852828116,48sleep30-21swapper/307:23:273
78442810,9sleep00-21swapper/010:48:160
78442810,9sleep00-21swapper/010:48:160
439028113,54sleep00-21swapper/010:35:160
298728115,41sleep23042-21/usr/sbin/munin12:13:362
44612800,6sleep30-21swapper/307:08:443
44612800,6sleep30-21swapper/307:08:443
313828017,39sleep23286-21grep10:33:162
313828017,39sleep23286-21grep10:33:162
258228013,50sleep22632-21latency_hist12:13:162
2459328017,17sleep20-21swapper/208:13:422
2431828015,20sleep30-21swapper/308:13:353
1857728023,25sleep20-21swapper/207:53:472
1857728023,25sleep20-21swapper/207:53:472
2036827918,20sleep20-21swapper/209:43:412
829027812,46sleep00-21swapper/007:23:160
3240127812,54sleep032451-21sed12:03:410
304322786,32sleep10-21swapper/108:33:321
162642787,33sleep10-21swapper/111:13:181
555827712,12sleep00-21swapper/008:57:020
555827712,12sleep00-21swapper/008:57:020
3003427720,34sleep230040-21modprobe08:33:132
2218827715,49sleep022317-21sed09:48:480
3111827621,28sleep00-21swapper/011:58:520
655427513,48sleep26672-21wc12:23:462
622027522,30sleep30-21swapper/310:43:153
3187327519,26sleep00-21swapper/010:18:480
2444527515,31sleep30-21swapper/311:38:403
2444527515,31sleep30-21swapper/311:38:403
1192227515,29sleep30-21swapper/307:33:373
3011527438,16sleep00-21swapper/008:33:150
2314127410,51sleep323164-21df09:53:313
820027319,34sleep38340-21grep12:28:483
602927315,39sleep05919-21/usr/sbin/munin08:58:360
602927315,39sleep05919-21/usr/sbin/munin08:58:360
284222738,30sleep10-21swapper/108:27:271
284222738,30sleep10-21swapper/108:27:271
284212736,54sleep00-21swapper/008:27:260
284212736,54sleep00-21swapper/008:27:260
486327213,38sleep20-21swapper/208:53:452
486327213,38sleep20-21swapper/208:53:452
212222728,44sleep30-21swapper/309:48:143
1855227222,24sleep30-21swapper/311:18:493
3275427115,39sleep332759-21diskstats10:23:323
2608627114,38sleep026134-21/usr/sbin/munin08:18:410
206332715,11sleep10-21swapper/111:27:391
206332715,11sleep10-21swapper/111:27:391
3259927016,39sleep232658-21processes12:03:472
1058527011,21sleep20-21swapper/207:28:472
97592695,10sleep10-21swapper/107:27:531
97592695,10sleep10-21swapper/107:27:531
627826911,41sleep06417-21cut10:43:160
568626919,28sleep00-21swapper/012:23:150
279526920,29sleep00-21swapper/012:13:260
1597326931,19sleep30-21swapper/309:28:483
1265326915,36sleep30-21swapper/309:18:423
3161426818,32sleep20-21swapper/208:38:142
2273626813,38sleep022793-21if_eth008:08:350
2002926823,25sleep30-21swapper/307:59:013
1692026815,24sleep20-21swapper/207:48:462
147332685,31sleep10-21swapper/109:24:461
670126714,33sleep36736-21munin-run07:18:143
3254026717,28sleep30-21swapper/308:38:563
2698826710,25sleep30-21swapper/308:23:173
1516726710,28sleep00-21swapper/011:08:480
1124526718,31sleep30-21swapper/309:13:453
73126612,37sleep2821-21grep10:23:432
3103426621,22sleep30-21swapper/308:33:473
280526617,29sleep00-21swapper/008:48:300
234326613,39sleep32416-21cat10:28:493
234326613,39sleep32416-21cat10:28:493
162872668,49sleep00-21swapper/009:29:500
541326511,22sleep20-21swapper/210:38:462
227842658,12sleep20-21swapper/211:33:382
227842658,12sleep20-21swapper/211:33:382
2262326520,25sleep20-21swapper/208:08:332
1568026516,31sleep214735-21/usr/sbin/munin07:43:542
927026412,41sleep01-21systemd10:53:130
959426314,22sleep00-21swapper/010:53:340
959426314,22sleep00-21swapper/010:53:340
858726314,32sleep28607-21/usr/sbin/munin07:23:302
858726314,32sleep28607-21/usr/sbin/munin07:23:302
80052630,8sleep00-21swapper/009:03:440
80052630,8sleep00-21swapper/009:03:440
66162635,14sleep30-21swapper/310:43:353
66162635,14sleep30-21swapper/310:43:353
2997726312,23sleep20-21swapper/211:54:182
28082636,12sleep00-21swapper/010:30:110
28082636,12sleep00-21swapper/010:30:110
1556226314,31sleep215593-21/usr/sbin/munin09:28:382
482826218,19sleep30-21swapper/312:18:423
3172926216,19sleep00-21swapper/008:38:160
295526219,23sleep20-21swapper/208:48:362
2784226218,21sleep20-21swapper/208:23:442
2784226218,21sleep20-21swapper/208:23:442
1675226218,27sleep316786-21/usr/sbin/munin11:14:083
2549526115,23sleep20-21swapper/211:43:152
2549526115,23sleep20-21swapper/211:43:152
2003126115,30sleep20-21swapper/211:23:452
2003126115,30sleep20-21swapper/211:23:452
1994626115,22sleep20-21swapper/207:59:002
950126019,19sleep30-21swapper/309:08:403
950126019,19sleep30-21swapper/309:08:403
2985026017,19sleep30-21swapper/310:13:383
2853826014,28sleep30-21swapper/310:08:463
2345726018,20sleep20-21swapper/209:53:382
2009626018,29sleep020211-21expr07:59:020
1834126018,28sleep018401-21processes11:18:460
1583526018,26sleep215929-21gpgv11:13:142
1386626019,22sleep20-21swapper/207:38:452
13752607,15sleep10-21swapper/107:00:441
936425916,27sleep29290-21latency_hist10:53:152
926425917,21sleep20-21swapper/209:08:352
926425917,21sleep20-21swapper/209:08:352
2448325915,23sleep00-21swapper/009:58:170
240325911,26sleep20-21swapper/207:03:432
327612589,31sleep00-21swapper/010:23:330
3212225815,27sleep332119-21apt-get12:03:283
2644425815,23sleep30-21swapper/310:03:333
263425816,26sleep32773-21cat12:13:173
2106625813,18sleep30-21swapper/311:28:343
2106625813,18sleep30-21swapper/311:28:343
168525813,28sleep01709-21ntp_states08:43:410
1594125815,24sleep316118-21apt-config11:13:153
14102585,10sleep00-21swapper/007:01:110
706725716,21sleep20-21swapper/207:18:342
2869525712,30sleep328767-21systemd-run11:53:143
2164125715,21sleep221736-21timerwakeupswit11:28:452
2164125715,21sleep221736-21timerwakeupswit11:28:452
2145625711,33sleep221494-21ps08:03:452
149242577,17sleep00-21swapper/009:27:150
1382725715,22sleep20-21swapper/209:23:332
1054125716,24sleep010547-21ntp_states07:28:450
322325616,21sleep30-21swapper/307:08:173
3112825612,27sleep231124-21apt-get10:18:302
3057325616,24sleep330566-21apt-get11:58:313
237725610,32sleep22378-21sshd08:46:302
233252566,13sleep30-21swapper/308:08:473
2254625615,25sleep322541-21apt-get11:33:313
2254625615,25sleep322541-21apt-get11:33:313
224802567,11sleep30-21swapper/309:49:273
1941625620,17sleep30-21swapper/311:23:293
1941625620,17sleep30-21swapper/311:23:293
1291025613,17sleep30-21swapper/311:03:263
1034525616,25sleep310365-21cut07:28:403
2096225513,25sleep020958-21apt-get11:28:270
2096225513,25sleep020958-21apt-get11:28:270
90925417,20sleep30-21swapper/306:58:443
708025413,25sleep37178-21grep07:18:353
377625416,21sleep23813-21ntp_states10:33:412
331125415,24sleep33389-21awk08:48:453
213932549,13sleep30-21swapper/308:03:423
177792549,20sleep20-21swapper/209:33:472
1370125411,24sleep00-21swapper/007:38:430
126792547,38sleep00-21swapper/007:35:120
3258725314,24sleep032624-21ntp_states08:38:570
281752539,12sleep20-21swapper/210:08:372
2535225311,21sleep00-21swapper/011:43:130
2535225311,21sleep00-21swapper/011:43:130
1768125313,24sleep017824-21sed07:53:170
1264625314,24sleep212652-21ntp_states09:18:412
951325217,17sleep00-21swapper/009:08:410
951325217,17sleep00-21swapper/009:08:410
749725212,24sleep27495-21apt-get09:03:262
749725212,24sleep27495-21apt-get09:03:262
646225212,23sleep26454-21apt-get10:43:302
646225212,23sleep26454-21apt-get10:43:302
1488725213,19sleep00-21swapper/007:43:400
1368525218,17sleep00-21swapper/011:04:040
2764825114,19sleep00-21swapper/011:48:380
2072625114,18sleep00-21swapper/011:28:150
2072625114,18sleep00-21swapper/011:28:150
3083725012,22sleep230868-21munin-run10:18:142
2938425012,16sleep00-21swapper/011:53:390
2929825010,21sleep329382-21memory11:53:383
289725013,23sleep32903-21df_inode12:13:333
363024917,16sleep30-21swapper/310:33:383
2465224911,22sleep024647-21apt-get09:58:290
244924916,18sleep00-21swapper/012:12:330
2511824812,18sleep30-21swapper/309:58:443
2172524714,14sleep20-21swapper/209:48:372
202672479,15sleep00-21swapper/009:43:390
76252468,13sleep30-21swapper/309:03:353
76252468,13sleep30-21swapper/309:03:353
2595724611,21sleep325964-21df_inode11:43:333
897824510,18sleep00-21swapper/007:23:400
897824510,18sleep00-21swapper/007:23:400
82242456,15sleep30-21swapper/310:48:363
2750724514,16sleep20-21swapper/211:48:322
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional