You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-09 - 17:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Tue Dec 09, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
381316822380,2sleep20-21swapper/212:25:172
3386406216152,104sleep30-21swapper/307:05:393
338660128554,9sleep10-21swapper/107:08:101
338657028052,6sleep00-21swapper/007:07:450
338653327862,9sleep20-21swapper/207:07:192
3386859994037,1cyclictest0-21swapper/309:45:173
3386859993128,1cyclictest0-21swapper/310:35:233
37512162268,8sleep03752685-21tr11:45:150
3386859992623,1cyclictest0-21swapper/307:35:003
338685999258,13cyclictest3626952-21chrt10:24:163
3386859992421,1cyclictest0-21swapper/311:35:013
3386859992421,1cyclictest0-21swapper/308:10:233
3386859992421,1cyclictest0-21swapper/307:36:233
3386859992420,2cyclictest0-21swapper/312:20:173
3386859992420,2cyclictest0-21swapper/311:23:303
3386859992420,2cyclictest0-21swapper/311:05:013
3386859992420,2cyclictest0-21swapper/308:05:003
3386859992412,8cyclictest3656480-21taskset10:43:243
338685999241,19cyclictest3580703-21taskset09:50:563
3386859992320,1cyclictest0-21swapper/310:10:003
338685999230,19cyclictest35888692chrt09:58:013
338685999222,12cyclictest3832902-21taskset12:37:373
3386859992219,1cyclictest0-21swapper/309:10:193
3386859992216,3cyclictest0-21swapper/310:45:173
338685999221,17cyclictest3709975-21taskset11:16:513
37557442211,2sleep10-21swapper/111:46:571
3386859992118,1cyclictest0-21swapper/307:25:243
3386859992117,2cyclictest0-21swapper/310:25:183
3386859992113,4cyclictest37022262sleep311:11:483
3386859992112,5cyclictest35662092sleep309:41:353
338685999211,15cyclictest3763428-21chrt11:51:393
338685999211,14cyclictest3749112-21taskset11:42:383
338685999211,13cyclictest0-21swapper/312:02:313
37278672201,13sleep23727984-21fschecks_time11:30:152
36144392205,10sleep23616267-21cut10:15:172
36144392205,10sleep23616267-21cut10:15:172
338685999202,13cyclictest3824296-21taskset12:31:153
338685999202,13cyclictest3773118-21taskset11:59:523
3386859992018,1cyclictest0-21swapper/310:35:023
3386859992017,1cyclictest0-21swapper/310:51:503
3386859992015,1cyclictest0-21swapper/308:35:003
338685999201,14cyclictest3726981-21chrt11:29:343
338685999201,13cyclictest3736156-21taskset11:35:153
338685999201,11cyclictest3458420-21taskset08:15:123
338685999192,11cyclictest3469053-21taskset08:25:103
3386859991915,3cyclictest66550irq/131-enp1s0-TxRx-311:06:343
338685999191,14cyclictest3811758-21taskset12:25:143
338685999191,14cyclictest3544072-21chrt09:26:023
338685999191,14cyclictest0-21swapper/312:15:203
338685999191,14cyclictest0-21swapper/309:16:383
338685999191,13cyclictest3604989-21chrt10:10:083
37189992180,2sleep10-21swapper/111:24:361
36046062181,3sleep20-21swapper/210:09:562
35677162181,2sleep10-21swapper/109:44:301
338685999182,11cyclictest3557794-21taskset09:35:203
3386859991815,1cyclictest0-21swapper/309:10:003
338685999181,14cyclictest3679507-21chrt10:57:593
338685999181,14cyclictest0-21swapper/312:10:513
338685999181,14cyclictest0-21swapper/307:10:213
338685999181,13cyclictest3781133-21chrt12:05:103
338685999181,13cyclictest3781133-21chrt12:05:103
338685999181,13cyclictest3612491-21chrt10:15:023
338685999181,13cyclictest3612491-21chrt10:15:013
338685999180,14cyclictest0-21swapper/309:30:083
338685999180,14cyclictest0-21swapper/309:20:293
338685999180,14cyclictest0-21swapper/309:20:293
38331902171,3sleep00-21swapper/012:38:440
36867352171,4sleep13516452-21diskmemload11:01:511
35961542171,10sleep03516452-21diskmemload10:02:530
35230572171,4sleep13516452-21diskmemload09:12:591
338685999172,12cyclictest3431156-21taskset07:46:433
3386859991715,1cyclictest0-21swapper/310:01:443
3386859991714,1cyclictest0-21swapper/307:50:103
3386859991713,2cyclictest0-21swapper/307:15:123
338685999171,13cyclictest0-21swapper/307:20:443
338685999171,12cyclictest3484845-21turbostat08:40:003
338685999171,10cyclictest3506058-21chrt09:00:013
338685999170,14cyclictest0-21swapper/309:00:253
37953462161,3sleep23516452-21diskmemload12:14:162
37639172161,3sleep10-21swapper/111:52:271
37627872161,5sleep23516452-21diskmemload11:50:382
37185562161,4sleep23516452-21diskmemload11:23:492
37104672161,4sleep03516452-21diskmemload11:17:450
37097562161,2sleep10-21swapper/111:16:261
37006902161,10sleep03700748-21ntpq11:10:190
36791062161,2sleep20-21swapper/210:57:192
36716062161,7sleep00-21swapper/010:52:560
36413532161,2sleep00-21swapper/010:33:070
36341102161,2sleep00-21swapper/010:28:580
36035232161,3sleep03516452-21diskmemload10:07:410
35807202166,4sleep1331ktimers/109:51:001
35749252161,4sleep23516452-21diskmemload09:49:072
35521182161,4sleep23516452-21diskmemload09:32:282
35517972161,10sleep13516452-21diskmemload09:31:481
35448472161,10sleep13544859-21ssh09:27:421
35390022161,2sleep00-21swapper/009:25:120
34581352161,9sleep23458184-21munin-run08:15:002
338685999161,9cyclictest3490387-21taskset08:45:103
3386859991613,1cyclictest0-21swapper/307:40:213
338685999161,11cyclictest0-21swapper/311:45:063
338685999160,12cyclictest0-21swapper/308:40:113
38333882151,2sleep20-21swapper/212:39:132
38087432151,3sleep23516452-21diskmemload12:20:452
37705852150,3sleep10-21swapper/111:55:351
37329792151,2sleep00-21swapper/011:31:440
36961842150,1sleep10-21swapper/111:09:561
36929962151,9sleep23693042-21modprobe11:05:202
36861662151,3sleep03516452-21diskmemload11:00:570
36630992151,10sleep03663105-21ssh10:46:260
36630822151,3sleep23516452-21diskmemload10:46:242
36497422151,2sleep00-21swapper/010:39:490
36362912151,5sleep23637877-21grep10:30:162
36264332151,8sleep03516452-21diskmemload10:23:210
36251212150,1sleep20-21swapper/210:20:502
36113772151,9sleep03516452-21diskmemload10:13:280
35818492151,9sleep03516452-21diskmemload09:53:340
35817332151,3sleep23516452-21diskmemload09:53:202
35599542151,8sleep20-21swapper/209:38:492
338685999151,9cyclictest3500503-21taskset08:52:553
338685999151,9cyclictest3436987-21taskset07:55:133
338685999151,10cyclictest3452587-21turbostat08:10:003
37938362141,8sleep13793846-21ssh12:10:421
37724252141,8sleep03516452-21diskmemload11:58:380
37624322141,8sleep03757903-21/usr/sbin/munin11:50:280
37241542148,3sleep20-21swapper/211:25:252
37183092141,7sleep03516452-21diskmemload11:23:220
36961442141,9sleep03696155-21ssh11:09:510
35739732141,7sleep00-21swapper/009:47:050
35306352141,2sleep13516452-21diskmemload09:18:411
35168182141,9sleep03517033-21expr09:10:130
34269302140,8sleep03427753-21kthreadcore07:45:160
34263922141,8sleep13426650-21df_inode07:45:131
338685999140,11cyclictest0-21swapper/308:20:143
338685999140,11cyclictest0-21swapper/308:20:143
37871142135,2sleep00-21swapper/012:07:290
37871142135,2sleep00-21swapper/012:07:280
37797252131,3sleep23779733-21ssh12:02:532
37706432131,8sleep20-21swapper/211:55:402
37334902130,8sleep13516452-21diskmemload11:32:411
36727872131,6sleep13516452-21diskmemload10:54:531
36168752131,6sleep03616968-21ps10:15:230
36168752131,6sleep03616968-21ps10:15:230
35952192131,6sleep13516452-21diskmemload10:00:571
35455442131,5sleep20-21swapper/209:29:072
35382422131,5sleep13516452-21diskmemload09:23:541
35382422131,5sleep13516452-21diskmemload09:23:531
34889712131,7sleep03489112-21ntpq08:40:190
38114792121,6sleep13811561-21cpuspeed_turbos12:25:131
38025042121,3sleep10-21swapper/112:18:121
37952512121,3sleep00-21swapper/012:14:060
37495852121,4sleep03516452-21diskmemload11:43:390
37208732121,6sleep03722456-21cut11:25:190
37010562121,7sleep23701292-21unixbench_multi11:10:252
36866232121,4sleep23516452-21diskmemload11:01:402
36722352121,4sleep23516452-21diskmemload10:53:582
36091722121,6sleep13609233-21ntpq10:10:231
35865922121,7sleep03586954-21needreboot09:55:190
35829032121,3sleep23582915-21ssh09:55:092
35678672121,6sleep03516452-21diskmemload09:44:480
35373502121,7sleep03537361-21ssh09:21:550
35373502121,7sleep03537361-21ssh09:21:550
35217192121,4sleep23516452-21diskmemload09:10:392
34203272121,6sleep13420373-21wireplumber07:36:231
38261992111,3sleep13516452-21diskmemload12:34:351
38258142111,3sleep20-21swapper/212:33:562
38092682111,4sleep13516452-21diskmemload12:21:521
38091672111,6sleep03516452-21diskmemload12:21:390
38027762111,4sleep03516452-21diskmemload12:18:480
37785762111,3sleep10-21swapper/112:00:521
37568032111,3sleep20-21swapper/211:48:502
37433392110,7sleep13743440-21fschecks_time11:40:151
37109572111,4sleep23516452-21diskmemload11:18:432
37026282111,4sleep13516452-21diskmemload11:12:251
36806362111,4sleep03516452-21diskmemload10:59:570
36561982110,7sleep03516452-21diskmemload10:42:530
36554482111,4sleep13516452-21diskmemload10:41:181
36477352110,3sleep10-21swapper/110:36:091
36326972111,3sleep20-21swapper/210:26:162
36254752111,5sleep13516452-21diskmemload10:21:301
36176672111,3sleep10-21swapper/110:15:481
36176672111,3sleep10-21swapper/110:15:481
36031302111,5sleep13603142-21ssh10:06:581
35891672111,4sleep13516452-21diskmemload09:58:361
35382202111,4sleep23516452-21diskmemload09:23:502
35382202111,4sleep23516452-21diskmemload09:23:502
35302562111,4sleep03516452-21diskmemload09:17:470
35291862111,4sleep23516452-21diskmemload09:15:402
35155482114,3sleep23515636-21processes09:05:222
35049252111,4sleep23504974-21ntpq08:55:242
35012582111,6sleep03501469-21if_enp1s008:55:180
34687742111,5sleep03468791-21munin-run08:25:010
34687742111,5sleep03468791-21munin-run08:25:000
34589442111,6sleep23459451-21tr08:15:162
34571692111,5sleep13457223-21ntpq08:10:221
34516092111,5sleep13451661-21ntpq08:05:211
34330772116,2sleep23433739-21kthreadcore07:50:152
34009852114,3sleep23402740-21kthreadcore07:20:172
38338492101,6sleep10-21swapper/112:40:001
38188772101,3sleep03516452-21diskmemload12:30:050
38019502101,3sleep20-21swapper/212:17:042
37861122101,3sleep10-21swapper/112:05:451
37861122101,3sleep10-21swapper/112:05:451
37811012102,3sleep23781127-21apt-get12:05:102
37811012102,3sleep23781127-21apt-get12:05:102
37429372101,2sleep20-21swapper/211:40:052
37408622101,3sleep20-21swapper/211:36:192
37377202100,6sleep03739408-21latency11:35:170
37254292101,3sleep10-21swapper/111:26:441
36787922101,3sleep13516452-21diskmemload10:56:451
36646602101,3sleep10-21swapper/110:49:241
36565842101,3sleep20-21swapper/210:43:352
36489162101,3sleep23516452-21diskmemload10:38:222
36349852102,4sleep13634997-21apt-get10:30:111
36334322101,3sleep13516452-21diskmemload10:27:391
36049752101,3sleep23516452-21diskmemload10:10:082
35971382101,3sleep23516452-21diskmemload10:04:472
35682912101,3sleep13516452-21diskmemload09:45:101
35609002101,3sleep23516452-21diskmemload09:40:052
35596442101,3sleep00-21swapper/009:38:030
35576732101,4sleep13516452-21diskmemload09:35:191
35516952101,3sleep03516452-21diskmemload09:31:340
35154352101,4sleep03515484-21ntpq09:05:200
34957432101,4sleep13495946-21diskstats08:50:141
34637612101,4sleep13464019-21df_inode08:20:121
34637612101,4sleep13464019-21df_inode08:20:121
34369652101,3sleep20-21swapper/207:55:122
34316752101,4sleep13431694-21apt-get07:50:101
34155992101,4sleep23415618-21apt-get07:35:122
3815484291,4sleep03815582-21ld12:25:210
3778607291,2sleep00-21swapper/012:00:560
3735400290,1sleep10-21swapper/111:35:061
3491661293,2sleep2431ktimers/208:45:182
3484245291,4sleep03484462-21users08:35:260
3474575291,3sleep13474785-21hddtemp_smartct08:30:141
3464020291,4sleep23464123-21fschecks_time08:20:132
3464020291,4sleep23464123-21fschecks_time08:20:132
3443851291,4sleep03445635-21grep08:00:190
3430524291,4sleep23430612-21ps07:45:222
3425485291,4sleep13425729-21users07:40:281
3419688291,4sleep03419735-21awk07:35:220
3415254290,1sleep10-21swapper/107:33:591
3398797291,4sleep03399047-21unixbench_singl07:15:260
3387068291,3sleep03387323-21df07:10:140
3510265280,1sleep20-21swapper/209:00:212
3506709280,4sleep03507070-21cat09:00:150
3501470281,3sleep13501924-21kthreadcore08:55:201
3496230281,3sleep03497513-21kthreadcore08:50:170
3483788281,2sleep10-21swapper/108:35:191
3469031281,2sleep20-21swapper/208:25:102
3458677281,3sleep00-21swapper/008:15:140
3453681281,3sleep00-21swapper/008:10:170
3452213280,4sleep00-21swapper/008:05:280
3442242280,1sleep20-21swapper/208:00:012
3441989280,4sleep13442241-21sed08:00:001
3422949281,3sleep03424760-21kthreadcore07:40:190
3414110280,1sleep00-21swapper/007:30:170
3405066281,3sleep23405050-21/usr/sbin/munin07:25:142
3391432280,1sleep10-21swapper/107:10:261
3511440270,1sleep10-21swapper/109:05:061
3490390270,2sleep10-21swapper/108:45:141
3489404270,4sleep10-21swapper/108:40:241
3486865271,3sleep23487178-21kthreadcore08:40:172
3479271270,1sleep00-21swapper/008:34:420
3474282270,2sleep20-21swapper/208:30:112
3473974270,3sleep10-21swapper/108:29:301
3462297271,2sleep13462438-21ntpq08:15:191
3441080271,3sleep03441137-21cut07:55:220
3436550270,1sleep00-21swapper/007:53:060
3421015270,2sleep20-21swapper/207:40:132
3394946271,2sleep20-21swapper/207:15:162
3392046270,1sleep20-21swapper/207:14:002
3511017260,4sleep10-21swapper/109:02:521
3479853261,2sleep20-21swapper/208:35:122
3452268260,1sleep20-21swapper/208:06:102
3410354261,2sleep20-21swapper/207:30:112
3409912260,1sleep00-21swapper/007:28:010
3409299260,3sleep10-21swapper/107:25:251
3404510260,1sleep10-21swapper/107:21:321
3398355260,2sleep10-21swapper/107:15:191
3499332250,1sleep20-21swapper/208:50:192
3494271250,1sleep00-21swapper/008:45:200
3446646251,2sleep10-21swapper/108:00:281
3469307240,1sleep00-21swapper/008:25:120
3400113240,1sleep00-21swapper/007:20:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional