You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-26 - 21:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sun Oct 26, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25031439917191,1713cyclictest29158042sleep312:10:173
25031439915790,1574cyclictest29378462sleep312:24:083
25031439915620,1557cyclictest26891022sleep309:49:273
25031439915572,1551cyclictest26299082sleep309:10:073
25031439915470,1542cyclictest28618202sleep311:40:073
25031439915470,1542cyclictest27175792sleep310:07:273
25031439915280,1524cyclictest28348902sleep311:22:273
25031439915190,1514cyclictest29019112sleep312:04:013
25031439915121,1506cyclictest27875892sleep310:55:013
25031439915050,1500cyclictest27714092sleep310:45:073
25031439915023,1494cyclictest26130432sleep308:50:193
25031439914461,1440cyclictest28027742sleep311:03:353
25031439914181,1413cyclictest29274942sleep312:17:263
25031439914031,1397cyclictest29634742sleep312:37:353
25031439913720,1367cyclictest2608979-21sleep08:50:003
25031439913651,1359cyclictest25465532sleep307:48:273
25031439913491,1343cyclictest26292342sleep309:05:343
25031439913390,1334cyclictest25879602sleep308:25:343
2502722212154,55sleep30-21swapper/307:06:163
250266728759,10sleep10-21swapper/107:05:331
250269128455,10sleep20-21swapper/207:05:502
250147428252,17sleep00-21swapper/007:05:200
2503143992522,1cyclictest0-21swapper/312:26:553
2503143992512,10cyclictest2674618-21turbostat09:44:543
250314399240,20cyclictest57-1kworker/3:0H-kblockd08:15:173
29622912230,2sleep00-21swapper/012:36:070
2503143992320,1cyclictest0-21swapper/311:15:163
29394362222,6sleep12939662-21smartctl12:25:151
250314399220,18cyclictest2878041-21taskset11:49:573
250314399211,16cyclictest29540702sleep312:31:553
250314399211,16cyclictest2784577-21taskset10:50:573
27324002201,1sleep1171rcu_preempt10:17:291
2503143992016,2cyclictest0-21swapper/312:00:003
2503143992016,2cyclictest0-21swapper/312:00:003
250314399201,16cyclictest2746854-21taskset10:25:563
250314399201,16cyclictest2598665-21sleep08:40:003
250314399201,14cyclictest0-21swapper/307:35:113
250314399200,16cyclictest2755052-21taskset10:33:243
2503143991916,2cyclictest0-21swapper/311:54:543
2503143991915,2cyclictest0-21swapper/309:15:213
2503143991915,2cyclictest0-21swapper/307:25:173
2503143991911,5cyclictest2515458-21turbostat07:20:003
250314399191,14cyclictest0-21swapper/311:25:333
250314399191,14cyclictest0-21swapper/311:07:253
250314399191,14cyclictest0-21swapper/309:53:133
250314399191,13cyclictest0-21swapper/310:00:153
250314399191,13cyclictest0-21swapper/310:00:143
250314399191,13cyclictest0-21swapper/309:30:123
2503143991910,6cyclictest2733724-21chrt10:19:413
29065012181,6sleep02907045-21ssh12:05:180
28025492181,13sleep12629593-21diskmemload11:03:171
2503143991816,1cyclictest0-21swapper/307:50:153
250314399181,4cyclictest0-21swapper/311:14:303
2503143991812,3cyclictest0-21swapper/309:25:353
2503143991811,4cyclictest66150irq/131-enp1s0-TxRx-311:30:203
250314399181,14cyclictest0-21swapper/308:07:023
250314399181,13cyclictest0-21swapper/310:20:323
250314399181,13cyclictest0-21swapper/310:11:493
250314399181,13cyclictest0-21swapper/309:35:593
250314399181,13cyclictest0-21swapper/308:30:133
250314399180,14cyclictest0-21swapper/312:07:583
250314399180,14cyclictest0-21swapper/309:20:273
29563852171,6sleep12956396-21sh12:34:451
27109282178,5sleep02710937-21sh10:04:120
27109282178,5sleep02710937-21sh10:04:120
26199402171,7sleep12620403-21tr09:00:161
250314399171,12cyclictest0-21swapper/310:35:073
250314399170,14cyclictest0-21swapper/310:43:003
28002792161,3sleep20-21swapper/211:00:262
27796662161,4sleep12779822-21ssh10:50:151
27715072161,5sleep22771532-21apt-get10:45:152
27491812162,6sleep22749204-21apt-get10:30:112
26300022161,4sleep12630162-21diskstats09:10:141
250314399162,2cyclictest0-21swapper/311:36:293
250314399161,3cyclictest0-21swapper/308:20:153
2503143991612,2cyclictest0-21swapper/309:00:133
2503143991612,2cyclictest0-21swapper/307:40:023
2503143991612,2cyclictest0-21swapper/307:25:003
250314399161,12cyclictest0-21swapper/308:40:173
29454732150,2sleep02629593-21diskmemload12:27:150
29373722151,8sleep02629593-21diskmemload12:23:340
29251172151,4sleep02925165-21ntpq12:15:200
28090882151,8sleep02629593-21diskmemload11:05:290
27260062151,8sleep02629593-21diskmemload10:14:420
25351282151,9sleep22535176-21ntpq07:35:202
250314399152,9cyclictest2557324-21taskset08:00:133
2503143991512,2cyclictest0-21swapper/307:55:163
250314399151,11cyclictest0-21swapper/308:55:223
250314399151,10cyclictest2503309-21taskset07:10:073
2503143991510,3cyclictest0-21swapper/307:32:443
250314399150,13cyclictest0-21swapper/308:10:173
250314399150,12cyclictest0-21swapper/309:55:143
29266542141,8sleep22629593-21diskmemload12:16:262
28829502141,7sleep02629593-21diskmemload11:50:270
27025612141,9sleep22702573-21ssh09:57:212
26742172140,2sleep10-21swapper/109:39:201
25924792141,8sleep12592519-21ntpq08:30:221
25157422141,8sleep22515888-21df07:20:142
29106502131,5sleep10-21swapper/112:08:521
28854742131,6sleep12629593-21diskmemload11:53:521
28853512131,9sleep2421rcuc/211:53:412
28529692130,1sleep10-21swapper/111:34:371
28258642131,4sleep00-21swapper/011:16:110
26995482131,7sleep12700845-21memory09:55:191
26591462131,8sleep02659159-21ssh09:29:370
26428392131,8sleep12642850-21ssh09:16:281
25732282131,2sleep20-21swapper/208:15:162
29175752121,7sleep02917590-21ssh12:11:310
28763752121,4sleep22629593-21diskmemload11:47:322
28670932121,6sleep02867107-21ssh11:41:090
28520932121,4sleep22629593-21diskmemload11:33:212
28473402121,7sleep02849131-21tr11:30:180
28276322120,6sleep22629593-21diskmemload11:18:462
27774722121,4sleep02629593-21diskmemload10:47:210
27688662120,1sleep10-21swapper/110:41:261
26870272121,4sleep22629593-21diskmemload09:45:342
26862442122,5sleep12686292-21ntpq09:45:211
26809062121,4sleep22629593-21diskmemload09:42:582
26737372121,2sleep20-21swapper/209:38:332
26659742121,6sleep02629593-21diskmemload09:33:020
26508762121,4sleep02629593-21diskmemload09:22:110
26093872121,6sleep22609600-21hddtemp_smartct08:50:152
25672512120,1sleep20-21swapper/208:08:392
25364722121,5sleep12536258-31find07:40:081
25088162121,5sleep02510248-21diskstats07:15:020
29273022110,3sleep10-21swapper/112:17:141
29174722111,3sleep10-21swapper/112:11:241
29030692111,3sleep20-21swapper/212:05:082
28985442111,6sleep22898635-21ntpq12:00:192
28768532116,2sleep12876867-21diskmemload11:48:151
28613772111,3sleep12629593-21diskmemload11:39:531
28536662111,4sleep02853826-21df_inode11:35:150
28430982111,3sleep10-21swapper/111:27:391
28177362111,6sleep22817746-21ssh11:11:162
28174152111,6sleep02817428-21sh11:10:460
28166592111,6sleep12816810-21ssh11:10:231
28012122111,6sleep02629593-21diskmemload11:01:160
27879462111,3sleep20-21swapper/210:55:172
27848152111,4sleep02629593-21diskmemload10:51:180
27624962111,4sleep12629593-21diskmemload10:38:201
27493692111,6sleep02749421-21fschecks_count10:30:130
27486982111,4sleep02629593-21diskmemload10:29:450
27459862111,5sleep22746047-21ntpq10:25:222
27424262111,6sleep12743683-21cut10:25:171
27319032111,3sleep02629593-21diskmemload10:16:390
27182072110,4sleep22629593-21diskmemload10:08:312
27157702111,4sleep12715830-21ntpq10:05:211
26962952111,3sleep10-21swapper/109:54:191
26936542111,5sleep02693695-21ntpq09:50:210
26864202111,5sleep02686568-21grep09:45:240
26738862111,3sleep00-21swapper/009:38:470
26439612111,4sleep02629593-21diskmemload09:18:260
25989352111,5sleep22599090-21diskstats08:40:132
25937472111,5sleep22593903-21diskstats08:35:162
25664092111,4sleep12566463-21ntpq08:05:211
25469602111,4sleep02547115-21diskstats07:50:130
25311422111,5sleep12531161-21apt-get07:35:111
25033162111,4sleep02503335-21apt-get07:10:130
29629542101,2sleep10-21swapper/112:36:581
29625282101,2sleep20-21swapper/212:36:272
29554122101,2sleep20-21swapper/212:33:342
29553372101,3sleep02629593-21diskmemload12:33:290
29450652101,3sleep22629593-21diskmemload12:26:442
29344312101,2sleep10-21swapper/112:20:261
29301972101,5sleep22930377-21cut12:20:152
29006012100,1sleep10-21swapper/112:02:221
28867922101,4sleep22886870-21/usr/sbin/munin11:55:142
28867922101,4sleep22886870-21/usr/sbin/munin11:55:142
28866002100,3sleep02886629-21apt-get11:55:120
28866002100,3sleep02886629-21apt-get11:55:120
28756442100,1sleep00-21swapper/011:46:300
28676922101,3sleep10-21swapper/111:41:581
28659582101,4sleep22866013-21ntpq11:40:222
28587372101,3sleep22629593-21diskmemload11:36:062
28362082100,4sleep02629593-21diskmemload11:24:290
28342712101,2sleep20-21swapper/211:21:302
28248842101,3sleep10-21swapper/111:15:241
27955072101,4sleep02795557-21perf11:00:000
27902822101,4sleep12791520-21memory10:55:201
27865692100,3sleep20-21swapper/210:53:522
27782532101,3sleep12629593-21diskmemload10:48:301
27693012101,3sleep02629593-21diskmemload10:42:040
27678732101,4sleep22767913-21ntpq10:40:222
27630352100,1sleep20-21swapper/210:39:132
27610952100,1sleep00-21swapper/010:35:430
27412062100,1sleep00-21swapper/010:24:200
27347372103,3sleep20-21swapper/210:20:172
27265182101,3sleep22726546-21apt-get10:15:122
27195262100,2sleep2171rcu_preempt10:10:152
27166402100,1sleep00-21swapper/010:05:520
27113272100,1sleep10-21swapper/110:04:571
27113272100,1sleep10-21swapper/110:04:571
27013332100,3sleep02629593-21diskmemload09:55:270
26799912101,3sleep02629593-21diskmemload09:41:070
26660612100,1sleep10-21swapper/109:33:101
26651682101,2sleep20-21swapper/209:31:272
26582782100,1sleep20-21swapper/209:26:582
26504652101,3sleep20-21swapper/209:21:252
26491322101,4sleep12649183-21ntpq09:20:201
26360072100,3sleep20-21swapper/209:12:472
26304662101,6sleep00-21swapper/009:10:170
26195252101,4sleep22619668-21df09:00:132
25838522101,5sleep02585102-21cut08:25:200
25834132101,4sleep12583568-21diskstats08:25:171
25790482101,5sleep12580831-21kthreadcore08:20:201
25766292101,5sleep12576677-21ntpq08:15:191
25441712101,4sleep02545521-21memory07:45:200
25419482100,6sleep12542165-21basename07:45:171
25213012103,3sleep0221rcuc/007:25:160
25106842101,4sleep12510893-21hddtemp_smartct07:15:151
25082542100,1sleep20-21swapper/207:15:002
2917486291,4sleep22917499-21ssh12:11:252
2900747290,1sleep00-21swapper/012:02:330
2892116291,4sleep12892130-21ssh11:56:381
2892116291,4sleep12892130-21ssh11:56:371
2841739290,5sleep02841752-21ssh11:25:370
2840866291,3sleep22840925-21ntpq11:25:192
2836397291,4sleep12836411-21ssh11:24:461
2811244290,2sleep10-21swapper/111:08:441
2809927290,1sleep20-21swapper/211:06:432
2749132290,1sleep10-21swapper/110:30:061
2740234291,2sleep10-21swapper/110:22:371
2725751290,2sleep10-21swapper/110:14:151
2709301291,3sleep22709315-21ssh10:00:572
2709301291,3sleep22709315-21ssh10:00:572
2696623291,2sleep20-21swapper/209:54:522
2680895291,3sleep12680905-21ssh09:42:561
2652731290,2sleep10-21swapper/109:25:051
2644161290,3sleep20-21swapper/209:18:442
2625084291,3sleep22625402-21grep09:05:182
2618247291,4sleep02618298-21awk08:55:220
2609701291,4sleep02611259-21grep08:50:170
2608336291,3sleep12608580-21users08:45:291
2577966291,2sleep00-21swapper/008:20:150
2562472291,2sleep00-21swapper/008:05:120
2557302291,2sleep20-21swapper/208:00:132
2570827280,1sleep10-21swapper/108:10:181
2557474281,3sleep02557562-21tune2fs08:00:150
2536328280,3sleep22536258-31find07:40:012
2529979281,3sleep20-21swapper/207:30:232
2529923281,3sleep12529978-21ntp_offset07:30:221
2525972280,2sleep00-21swapper/007:30:130
2524814281,3sleep22524907-21wc07:25:222
2520297282,3sleep08045-21kworker/0:0+events07:21:230
2515720280,2sleep10-21swapper/107:20:121
2504494280,4sleep12505283-21kthreadcore07:10:181
2629584271,3sleep00-21swapper/009:10:010
2614522271,3sleep20-21swapper/208:55:162
2613141271,2sleep10-21swapper/108:50:231
2608606270,1sleep20-21swapper/208:45:422
2598913270,2sleep10-21swapper/108:40:121
2594187271,3sleep12595616-21kthreadcore08:35:191
2593179270,4sleep20-21swapper/208:31:272
2552508270,3sleep22552605-21ps07:55:162
2546935270,1sleep10-21swapper/107:50:101
2542166271,2sleep20-21swapper/207:45:182
2536471270,2sleep00-21swapper/007:40:070
2619119260,4sleep10-21swapper/108:58:431
2598568260,1sleep00-21swapper/008:39:150
2592433261,1sleep00-21swapper/008:30:210
2588027260,4sleep20-21swapper/208:26:262
2582814260,1sleep20-21swapper/208:24:342
2576450260,2sleep02576623-21strings08:15:180
2572231260,1sleep20-21swapper/208:11:312
2561100260,2sleep12561195-21ntpq08:00:201
2556327261,2sleep02556498-21timerwakeupswit07:55:260
2536081260,3sleep00-21swapper/007:39:560
2522245260,3sleep1321rcuc/107:25:171
2515150260,4sleep20-21swapper/207:16:252
2628719250,1sleep10-21swapper/109:05:251
2623543250,2sleep00-21swapper/009:00:230
2608012250,1sleep00-21swapper/008:45:240
2603327250,1sleep00-21swapper/008:40:270
2569003250,2sleep00-21swapper/008:10:170
2552295250,1sleep10-21swapper/107:55:151
2550800250,1sleep20-21swapper/207:50:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional