You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 12:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Dec 03, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3446728212354,64sleep30-21swapper/319:06:163
3446149211198,6sleep00-21swapper/019:05:240
344691328752,10sleep10-21swapper/119:08:331
344687128352,9sleep20-21swapper/219:08:022
3447147993733,2cyclictest0-21swapper/321:40:283
39019802321,21sleep03575990-21diskmemload00:30:090
3447147992723,2cyclictest0-21swapper/323:30:283
344714799271,15cyclictest228-1kworker/3:1H+kblockd00:06:203
3447147992517,6cyclictest66550irq/131-enp1s0-TxRx-322:40:123
344714799251,20cyclictest3679818-21taskset22:10:523
36333002241,10sleep03575990-21diskmemload21:44:570
344714799242,17cyclictest3592863-21chrt21:20:133
344714799242,15cyclictest3523057-21taskset20:17:493
38533012231,2sleep20-21swapper/223:58:252
36146212230,2sleep30-21swapper/321:31:273
3447147992320,1cyclictest0-21swapper/319:45:223
3447147992319,2cyclictest0-21swapper/322:23:533
38131302221,10sleep03575990-21diskmemload23:34:200
344714799222,16cyclictest3497248-21chrt19:55:123
3447147992213,4cyclictest37065682sleep322:29:143
344714799221,17cyclictest3589460-21chrt21:15:483
344714799221,13cyclictest3842910-21taskset23:50:273
344714799220,14cyclictest3884023-21taskset00:16:453
344714799212,13cyclictest3670731-21taskset22:05:193
3447147992118,1cyclictest0-21swapper/323:46:163
3447147992118,1cyclictest0-21swapper/323:24:413
3447147992117,2cyclictest0-21swapper/322:55:173
3447147992116,2cyclictest0-21swapper/322:54:173
3447147992116,2cyclictest0-21swapper/300:00:193
344714799211,16cyclictest3623330-21chrt21:37:133
38302332201,13sleep03830264-21ssh23:45:120
344714799202,12cyclictest3737291-21taskset22:46:173
344714799202,12cyclictest3638609-21taskset21:45:313
3447147992017,1cyclictest0-21swapper/323:55:153
3447147992017,1cyclictest0-21swapper/320:05:133
3447147992017,1cyclictest0-21swapper/319:20:233
3447147992017,1cyclictest0-21swapper/300:25:143
3447147992016,2cyclictest0-21swapper/320:10:163
3447147992012,4cyclictest37561622sleep323:00:133
344714799201,12cyclictest3821745-21taskset23:40:013
344714799200,15cyclictest0-21swapper/319:15:583
36390902191,7sleep10-21swapper/121:46:181
344714799195,11cyclictest0-21swapper/320:00:143
344714799193,13cyclictest66550irq/131-enp1s0-TxRx-300:10:363
344714799192,13cyclictest3687990-21taskset22:15:463
344714799192,12cyclictest3914311-21taskset00:35:253
344714799192,12cyclictest3787148-21taskset23:17:473
344714799192,12cyclictest3772609-21taskset23:10:103
3447147991916,1cyclictest0-21swapper/323:05:203
3447147991915,2cyclictest0-21swapper/321:50:183
3447147991915,2cyclictest0-21swapper/320:55:223
344714799191,14cyclictest3891825-21chrt00:21:383
3447147991911,4cyclictest38219872sleep323:40:063
344714799191,14cyclictest3801756-21chrt23:25:203
344714799191,14cyclictest0-21swapper/321:25:123
344714799191,14cyclictest0-21swapper/320:50:133
344714799191,13cyclictest3715583-21chrt22:35:143
344714799191,13cyclictest3654986-21chrt21:55:413
39155522181,4sleep03575990-21diskmemload00:36:550
38853252181,2sleep00-21swapper/000:19:030
37810852181,5sleep03781118-21forks23:15:130
37148972181,3sleep10-21swapper/122:34:361
36819442181,2sleep20-21swapper/222:14:082
36664992181,3sleep10-21swapper/122:05:061
344714799182,12cyclictest3582016-21taskset21:11:463
3447147991816,1cyclictest0-21swapper/322:30:163
3447147991816,1cyclictest0-21swapper/322:00:153
3447147991815,1cyclictest0-21swapper/319:10:133
344714799181,13cyclictest0-21swapper/321:00:123
344714799181,13cyclictest0-21swapper/300:30:183
39085612171,3sleep20-21swapper/200:33:212
38832762171,4sleep13575990-21diskmemload00:15:321
38547062171,4sleep13575990-21diskmemload00:00:031
38015782171,10sleep03575990-21diskmemload23:25:190
37948632171,4sleep03575990-21diskmemload23:21:380
37775132171,10sleep03575990-21diskmemload23:10:350
37622772171,3sleep03575990-21diskmemload23:02:290
37380592171,10sleep03575990-21diskmemload22:47:310
36648572171,11sleep03575990-21diskmemload22:02:570
36475012171,10sleep03575990-21diskmemload21:51:420
36213952172,8sleep13621451-21ntpq21:35:201
35956392172,8sleep03596832-21memory21:20:190
344714799172,11cyclictest3532865-21taskset20:25:223
3447147991714,1cyclictest0-21swapper/319:50:163
3447147991713,2cyclictest0-21swapper/320:30:223
344714799171,13cyclictest0-21swapper/319:25:173
344714799170,13cyclictest0-21swapper/319:30:123
38923442161,2sleep10-21swapper/100:22:331
38920342161,10sleep23575990-21diskmemload00:22:022
38461742161,3sleep00-21swapper/023:54:570
38428712161,2sleep20-21swapper/223:50:272
38215822161,3sleep23575990-21diskmemload23:39:532
37898042161,10sleep13790072-21kthreadcore23:20:151
37714232161,4sleep03575990-21diskmemload23:09:080
37611112161,8sleep13575990-21diskmemload23:00:381
37464962160,2sleep10-21swapper/122:53:041
37073492160,4sleep23575990-21diskmemload22:30:042
36556732161,10sleep03575990-21diskmemload21:56:470
36331002161,10sleep23633110-21ssh21:44:392
36225092161,10sleep03622520-21ssh21:35:580
35902922161,4sleep23575990-21diskmemload21:17:092
344714799161,11cyclictest3544397-21turbostat20:40:003
344714799161,11cyclictest3523536-21chrt20:20:133
344714799160,13cyclictest0-21swapper/320:45:233
39079662151,8sleep13575990-21diskmemload00:32:191
38283282151,10sleep03828341-21ssh23:42:380
38220582151,4sleep13822084-21apt-get23:40:111
37780262151,9sleep23778040-21ssh23:11:222
37449862150,9sleep03575990-21diskmemload22:50:390
36894142150,2sleep10-21swapper/122:17:571
36882222151,3sleep03575990-21diskmemload22:16:080
36078412151,9sleep03607851-21ssh21:28:360
3447147991512,1cyclictest0-21swapper/319:40:173
344714799151,11cyclictest3576009-21sleep21:10:003
344714799150,13cyclictest0-21swapper/319:35:173
344714799150,12cyclictest0-21swapper/320:40:143
38452242141,7sleep10-21swapper/123:53:301
38150212141,8sleep13816555-21cut23:35:191
38046182140,2sleep13575990-21diskmemload23:28:531
38039042141,8sleep23803916-21rm23:27:502
37797442141,4sleep13779754-21ssh23:13:581
37390362145,2sleep1171rcu_preempt22:49:041
36418432141,8sleep23641857-21ssh21:50:082
36164192141,8sleep03616431-21rm21:34:140
35834572141,7sleep03575990-21diskmemload21:14:060
35820082140,8sleep20-21swapper/221:11:472
35520052141,3sleep03553721-21latency20:45:190
35452412141,8sleep13546484-21cut20:40:161
34816962143,6sleep03481793-21grep19:40:150
39157052131,2sleep13915713-21ssh00:37:121
38625062132,5sleep13862534-21apt-get00:05:131
37954932131,6sleep23575990-21diskmemload23:22:372
37885852131,7sleep21-21systemd23:20:002
37713192131,8sleep23771329-21ssh23:08:582
37540912131,5sleep00-21swapper/022:57:210
37525542131,7sleep23752559-21ssh22:55:272
37449652131,6sleep20-21swapper/222:50:382
36976112131,5sleep13575990-21diskmemload22:22:541
36965942131,1sleep00-21swapper/022:21:210
36652162131,4sleep23665228-21ssh22:03:302
36398312131,6sleep23575990-21diskmemload21:47:232
35971572131,7sleep13597291-21readlink21:20:241
35896032131,3sleep13589613-21ssh21:16:021
39104372121,7sleep23910769-21kthreadcore00:35:172
38763712121,4sleep20-21swapper/200:12:322
38601162121,6sleep03575990-21diskmemload00:01:100
38516952121,7sleep03575990-21diskmemload23:55:540
38359992121,6sleep13575990-21diskmemload23:46:531
38117682121,4sleep23575990-21diskmemload23:32:192
37129302121,5sleep00-21swapper/022:31:310
36803642120,8sleep13680367-21ssh22:11:441
36731522121,5sleep00-21swapper/022:08:100
36505742121,7sleep13651314-21cut21:55:161
36408612121,7sleep03640871-21ssh21:48:570
36085962120,8sleep13608602-21ssh21:29:451
35650982120,1sleep10-21swapper/120:55:591
35543712121,7sleep13549945-21/usr/sbin/munin20:45:281
34804442121,3sleep10-21swapper/119:35:261
39004792111,4sleep03575990-21diskmemload00:28:010
38996622111,3sleep10-21swapper/100:26:381
38913022111,4sleep03575990-21diskmemload00:20:470
38739982111,6sleep03874332-21needreboot00:10:160
38680862111,3sleep00-21swapper/000:06:530
38597572111,4sleep23575990-21diskmemload00:00:332
38265092111,4sleep23575990-21diskmemload23:40:252
38192832111,4sleep03575990-21diskmemload23:36:180
37859252111,3sleep10-21swapper/123:15:541
37691522111,3sleep10-21swapper/123:05:321
37631842111,4sleep23575990-21diskmemload23:03:572
37544322111,3sleep10-21swapper/122:57:541
37315542111,3sleep20-21swapper/222:44:552
37291182111,4sleep03575990-21diskmemload22:41:130
37279942111,3sleep10-21swapper/122:40:221
37232602111,4sleep03575990-21diskmemload22:39:490
37206792111,4sleep23575990-21diskmemload22:35:542
37154482111,3sleep10-21swapper/122:35:051
37041952111,4sleep03575990-21diskmemload22:25:300
37037512111,6sleep23703781-21ssh22:25:242
37032892111,4sleep13703440-21ntpq22:25:191
36986382111,3sleep20-21swapper/222:24:312
36809252110,1sleep00-21swapper/022:12:350
36600542111,5sleep13661738-21tr22:00:171
36553082111,3sleep20-21swapper/221:56:132
36492672111,3sleep10-21swapper/121:54:281
36304942111,2sleep10-21swapper/121:40:371
36172752111,4sleep23575990-21diskmemload21:35:092
36145102111,4sleep13575990-21diskmemload21:31:191
35928352111,4sleep23575990-21diskmemload21:20:132
35903152111,4sleep03575990-21diskmemload21:17:100
35828662110,7sleep13582877-21diskmemload21:13:111
35654072111,7sleep03560177-21turbostat20:59:540
35293902111,6sleep23530984-21cut20:25:182
35178632110,1sleep10-21swapper/120:13:311
35169422111,4sleep03516993-21ntpq20:10:210
34672882111,6sleep23469105-21kthreadcore19:25:172
34605282111,6sleep23460737-21hddtemp_smartct19:20:172
38999072101,2sleep20-21swapper/200:27:032
38825422101,4sleep23882599-21ntpq00:15:212
38688712101,3sleep23575990-21diskmemload00:08:152
38357902101,3sleep23575990-21diskmemload23:46:342
38116272101,2sleep10-21swapper/123:32:061
37372372101,5sleep23737243-21ssh22:46:132
36879792101,3sleep20-21swapper/222:15:462
36736352101,5sleep23622441-21kworker/u16:1+rpciod22:08:562
36151722101,3sleep20-21swapper/221:32:202
36010562100,1sleep20-21swapper/221:25:142
35591252101,4sleep03559173-21ntpq20:50:210
35570262101,5sleep23558854-21cut20:50:192
34998442101,4sleep03501033-21memory19:55:180
34764152101,4sleep23476625-21hddtemp_smartct19:35:152
34476112101,4sleep23447820-21hddtemp_smartct19:10:152
3876405290,1sleep10-21swapper/100:12:351
3570378290,2sleep20-21swapper/221:00:492
3569827291,4sleep13569938-21processes21:00:241
3554626290,4sleep20-21swapper/220:46:182
3539378291,4sleep03539451-21cpuspeed_turbos20:35:130
3527503291,4sleep23527544-21ntpq20:20:212
3518534291,3sleep23518744-21hddtemp_smartct20:15:162
3502494290,5sleep23502566-21sort20:00:132
3497552291,4sleep13497760-21/usr/sbin/munin19:55:141
3495980291,3sleep23496032-21ntpq19:50:232
3491006291,4sleep13491201-21/usr/sbin/munin19:45:241
3490723291,3sleep03490772-21ntpq19:45:200
3470563291,5sleep00-21swapper/019:30:010
3460228290,2sleep0171rcu_preempt19:20:070
3451407291,3sleep13451463-21ntpq19:10:221
3852431281,2sleep10-21swapper/123:57:031
3528524281,3sleep13528769-21latency_hist20:25:011
3523118280,1sleep10-21swapper/120:18:351
3511766281,3sleep23511823-21ntp_states20:05:222
3495943281,2sleep10-21swapper/119:50:221
3481414280,2sleep20-21swapper/219:40:132
3476717281,3sleep03478224-21kthreadcore19:35:180
3471150282,3sleep23471368-21if_enp1s019:30:152
3452903281,3sleep03452372-21apt-get19:15:020
3447343281,3sleep00-21swapper/019:10:130
3575673270,1sleep00-21swapper/021:06:070
3571135270,3sleep20-21swapper/221:05:142
3565944270,4sleep00-21swapper/021:00:150
3561010271,2sleep23562494-21tr20:55:162
3555754271,3sleep13557023-21kthreadcore20:50:181
3544740270,3sleep20-21swapper/220:40:132
3541241271,3sleep10-21swapper/120:35:191
3538888270,1sleep00-21swapper/020:32:260
3538835270,2sleep20-21swapper/220:31:472
3528786270,2sleep10-21swapper/120:25:121
3528463270,2sleep00-21swapper/020:24:160
3507743271,2sleep10-21swapper/120:05:121
3507721270,2sleep00-21swapper/020:05:110
3497226270,2sleep20-21swapper/219:55:112
3491423270,1sleep20-21swapper/219:46:342
3475544270,1sleep10-21swapper/119:31:021
3459703270,1sleep10-21swapper/119:16:551
3459636270,1sleep20-21swapper/219:16:042
3539042260,1sleep10-21swapper/120:34:231
3518845261,2sleep00-21swapper/020:15:170
3515113260,3sleep2421rcuc/220:10:192
3496703260,4sleep00-21swapper/019:51:420
3486349260,1sleep10-21swapper/119:43:451
3471800261,2sleep00-21swapper/019:30:170
3466101261,2sleep10-21swapper/119:25:161
3465160260,1sleep10-21swapper/119:23:371
3545169250,1sleep00-21swapper/020:40:150
3544147251,2sleep20-21swapper/220:37:012
3533245250,1sleep00-21swapper/020:25:270
3506214250,2sleep03506410-21strings20:00:190
3570711240,1sleep10-21swapper/121:05:011
3502473240,1sleep10-21swapper/120:00:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional