You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 12:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Thu Nov 20, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
347330523160,2sleep23473304-21ssh21:15:102
3373377212855,68sleep30-21swapper/319:07:203
374175821220,2sleep00-21swapper/000:00:020
337339528755,10sleep10-21swapper/119:07:341
337354427555,14sleep00-21swapper/019:09:320
337350627558,10sleep20-21swapper/219:09:032
3373714993830,3cyclictest3464785-21sleep21:10:013
3373714993320,9cyclictest3529418-21turbostat21:50:013
3373714992622,2cyclictest0-21swapper/320:55:003
337371499262,16cyclictest3678780-21taskset23:20:163
3373714992521,2cyclictest0-21swapper/300:20:163
35705562241,16sleep13570677-21irqstats22:15:161
337371499242,14cyclictest3749619-21taskset00:04:503
37569902231,15sleep03757005-21sshd00:08:320
337371499239,10cyclictest0-21swapper/321:20:153
3373714992317,4cyclictest36091762chrt22:38:343
337371499231,17cyclictest3789000-21sleep00:30:003
337371499231,17cyclictest3789000-21sleep00:29:593
337371499228,10cyclictest3478508-21taskset21:15:593
337371499226,10cyclictest3416699-21chrt20:20:403
3373714992219,2cyclictest0-21swapper/322:30:003
337371499221,15cyclictest3671541-21chrt23:15:183
337371499221,15cyclictest3632863-21taskset22:51:563
35030792210,8sleep244-21ksoftirqd/221:31:272
337371499212,13cyclictest3558064-21taskset22:05:193
3373714992119,1cyclictest0-21swapper/323:00:013
3373714992118,2cyclictest0-21swapper/300:10:173
337371499211,16cyclictest3504777-21chrt21:34:073
337371499211,15cyclictest3698375-21chrt23:32:143
337371499211,15cyclictest3505616-21chrt21:35:033
35930382201,7sleep13464771-21diskmemload22:28:341
33815522200,1sleep20-21swapper/219:51:382
337371499208,8cyclictest3584214-21taskset22:22:263
337371499205,11cyclictest3740883-21taskset23:59:033
337371499202,13cyclictest3730684-21taskset23:51:033
337371499202,13cyclictest3529761-21taskset21:50:113
337371499202,12cyclictest3707158-21taskset23:38:073
3373714992017,2cyclictest0-21swapper/300:35:173
3373714992017,1cyclictest0-21swapper/320:50:003
3373714992016,2cyclictest0-21swapper/323:11:373
3373714992016,2cyclictest0-21swapper/322:30:123
3373714992016,2cyclictest0-21swapper/321:40:163
3373714992016,2cyclictest0-21swapper/321:40:153
3373714992016,2cyclictest0-21swapper/300:19:593
3373714992012,4cyclictest37224322sleep323:45:573
3373714992012,4cyclictest0-21swapper/322:40:243
337371499201,14cyclictest3562495-21chrt22:10:113
37487242191,3sleep00-21swapper/000:03:260
35297922191,5sleep03529904-21sh21:50:130
34160152191,11sleep03416061-21ntpq20:20:230
337371499192,12cyclictest3757604-21taskset00:09:263
3373714991915,2cyclictest0-21swapper/320:00:013
3373714991914,2cyclictest0-21swapper/322:45:123
3373714991914,2cyclictest0-21swapper/321:55:263
337371499191,14cyclictest66550irq/131-enp1s0-TxRx-321:10:143
337371499191,14cyclictest0-21swapper/323:41:053
337371499191,14cyclictest0-21swapper/323:08:133
37963532181,3sleep10-21swapper/100:34:191
36749722181,3sleep10-21swapper/123:19:381
36572922181,3sleep20-21swapper/223:06:562
337371499182,14cyclictest66550irq/131-enp1s0-TxRx-300:30:383
337371499182,12cyclictest3551167-21taskset22:01:133
3373714991815,1cyclictest0-21swapper/321:00:143
3373714991814,2cyclictest0-21swapper/320:55:133
3373714991814,2cyclictest0-21swapper/320:35:143
337371499181,3cyclictest3421720-21taskset20:25:263
337371499181,14cyclictest0-21swapper/323:27:223
337371499181,14cyclictest0-21swapper/322:15:213
337371499181,14cyclictest0-21swapper/321:25:583
337371499181,10cyclictest3390969-21taskset20:00:163
337371499180,15cyclictest0-21swapper/323:00:093
37875642171,3sleep23464771-21diskmemload00:27:392
37875642171,3sleep23464771-21diskmemload00:27:382
37314092171,4sleep03464771-21diskmemload23:52:080
36646492171,3sleep03464771-21diskmemload23:10:560
35847002171,2sleep00-21swapper/022:23:130
35197122171,10sleep23464771-21diskmemload21:42:182
35197122171,10sleep23464771-21diskmemload21:42:172
34008372171,11sleep03400864-21perf20:10:000
3373714991714,1cyclictest0-21swapper/320:40:183
337371499171,11cyclictest3377516-21sleep19:24:593
337371499171,10cyclictest3411601-21taskset20:18:133
37797182161,9sleep03464771-21diskmemload00:22:220
37379102162,7sleep13737973-21ntp_states23:55:221
36820882161,10sleep13682098-21ssh23:22:351
36335702161,10sleep03633581-21sh22:53:040
36106612160,1sleep00-21swapper/022:40:170
36073692161,3sleep13464771-21diskmemload22:35:401
35996892161,3sleep13464771-21diskmemload22:31:161
35652122161,10sleep23565975-21cat22:10:152
35615792160,9sleep272-21kcompactd022:09:562
35354812161,10sleep13464771-21diskmemload21:51:541
35052902160,2sleep00-21swapper/021:34:580
35015762161,10sleep13501726-21ntp_192.168.11421:30:181
337371499161,9cyclictest3395825-21taskset20:05:103
337371499161,9cyclictest3380904-21taskset19:50:063
337371499161,12cyclictest0-21swapper/320:15:003
337371499161,11cyclictest3380031-21turbostat19:45:003
337371499161,11cyclictest3379379-21turbostat19:40:003
337371499161,10cyclictest3378640-21taskset19:33:393
337371499160,13cyclictest0-21swapper/319:20:013
37158132151,1sleep10-21swapper/123:43:481
36833222151,9sleep23683332-21ssh23:24:332
36493082151,3sleep13464771-21diskmemload23:02:111
36409392150,2sleep00-21swapper/022:56:470
36253022151,8sleep23464771-21diskmemload22:48:002
36165412150,3sleep13464771-21diskmemload22:42:261
34542292151,11sleep20-21swapper/221:00:002
337371499159,4cyclictest3380505-21chrt19:48:113
337371499151,9cyclictest3422567-21taskset20:30:063
337371499151,9cyclictest3373880-21taskset19:10:143
337371499151,11cyclictest3378146-21sleep19:30:003
38025572141,2sleep00-21swapper/000:36:300
37893902143,7sleep03789578-21expr00:30:170
37634782141,8sleep03464771-21diskmemload00:10:520
37414322141,5sleep20-21swapper/223:59:552
36700882141,3sleep23464771-21diskmemload23:15:182
36601662141,8sleep13661654-21kthreadcore23:10:161
36492722141,7sleep23464771-21diskmemload23:02:082
35823382141,8sleep13578112-21/usr/sbin/munin22:20:201
35277692141,9sleep03527774-21ssh21:47:250
35106092141,8sleep13510618-21ssh21:35:391
34874552140,6sleep13487461-21sh21:22:131
34781242141,2sleep13464771-21diskmemload21:15:261
34706802141,8sleep13464771-21diskmemload21:11:341
34482472141,8sleep03448467-21sed20:50:290
38025702131,5sleep10-21swapper/100:36:311
37640642131,6sleep23464771-21diskmemload00:11:472
37565812131,8sleep23756592-21ssh00:07:532
37251062131,8sleep13725347-21latency_hist23:50:001
36984322131,7sleep03464771-21diskmemload23:32:200
36923632131,5sleep13464771-21diskmemload23:30:131
36575482131,6sleep13464771-21diskmemload23:07:231
36414512131,2sleep2171rcu_preempt22:57:382
35528252131,5sleep00-21swapper/022:03:520
35518662131,6sleep23464771-21diskmemload22:02:212
35437702135,2sleep23464771-21diskmemload21:57:212
35370172131,5sleep20-21swapper/221:54:242
34584902131,7sleep23458540-21ntpq21:00:212
34267172131,7sleep13426800-21/usr/sbin/munin20:30:261
34110512130,1sleep10-21swapper/120:15:261
34072562132,7sleep03408017-21cut20:15:170
37873922121,7sleep03464771-21diskmemload00:27:210
37873922121,7sleep03464771-21diskmemload00:27:200
37734162121,7sleep13773419-21ssh00:19:471
37732162120,2sleep23773225-21ssh00:19:242
37705332122,5sleep03770593-21/usr/sbin/munin00:15:220
37633612121,7sleep13464771-21diskmemload00:10:421
37489602121,4sleep13464771-21diskmemload00:03:481
37205522121,7sleep03721327-21missed_timers23:45:220
37097232121,7sleep03711302-21cut23:40:130
37057092121,4sleep23464771-21diskmemload23:35:542
36426592121,7sleep13642662-21ssh22:59:331
36258742121,4sleep03464771-21diskmemload22:48:540
35766352121,3sleep00-21swapper/022:18:150
35700892122,4sleep23570148-21cpu22:15:132
35608112122,2sleep0171rcu_preempt22:08:410
35204252122,4sleep0151ktimers/021:43:320
35204252122,4sleep0151ktimers/021:43:310
35191962121,4sleep13464771-21diskmemload21:41:291
35191962121,4sleep13464771-21diskmemload21:41:281
34955202121,7sleep13495530-21ssh21:27:131
34178802121,6sleep13419327-21kthreadcore20:25:161
38030602110,5sleep20-21swapper/200:37:252
37958952111,3sleep20-21swapper/200:33:312
37799432111,4sleep13464771-21diskmemload00:22:481
37788522111,6sleep23778862-21ssh00:20:492
37500962111,4sleep13750123-21apt-get00:05:121
37317852111,4sleep23731796-21sshd23:52:432
37214442111,5sleep23721498-21ntpq23:45:242
37040512111,6sleep13704592-21ssh23:35:171
36965412110,1sleep20-21swapper/223:30:222
36898432111,3sleep20-21swapper/223:26:362
36810032111,3sleep00-21swapper/023:20:540
36733992111,5sleep00-21swapper/023:17:040
36594602111,3sleep20-21swapper/223:10:052
36506852110,5sleep00-21swapper/023:04:180
36322282111,4sleep23464771-21diskmemload22:51:002
36241512111,3sleep13464771-21diskmemload22:46:141
36166402111,4sleep23464771-21diskmemload22:42:342
36086202111,3sleep20-21swapper/222:37:422
36023532111,3sleep00-21swapper/022:35:070
35995302111,3sleep23464771-21diskmemload22:31:002
35935972111,3sleep20-21swapper/222:29:282
35934872111,4sleep03464771-21diskmemload22:29:170
35780532111,2sleep20-21swapper/222:20:052
35635392111,5sleep13565211-21cut22:10:141
35616892111,4sleep03464771-21diskmemload22:10:010
35581942111,3sleep10-21swapper/122:05:221
35526202111,6sleep13464771-21diskmemload22:03:331
35277002111,6sleep23527711-21ssh21:47:192
35118672111,3sleep00-21swapper/021:37:380
35108762111,4sleep23464771-21diskmemload21:36:042
34965542111,4sleep03464771-21diskmemload21:28:520
34856722111,5sleep03485726-21ntpq21:20:190
34775642111,5sleep03464771-21diskmemload21:15:210
34708472111,3sleep00-21swapper/021:11:490
34532492111,5sleep13453296-21ntpq20:55:181
34479212111,4sleep23447969-21ntpq20:50:232
34424062111,5sleep13442460-21ntpq20:45:201
34332242111,5sleep03433427-21diskstats20:40:150
33997932111,5sleep13399850-21ntpq20:05:201
33822062111,6sleep03383903-21tr19:52:250
33809232111,5sleep13380926-21apt-get19:50:091
37814422101,2sleep10-21swapper/100:25:051
37814422101,2sleep10-21swapper/100:25:041
37472372101,2sleep20-21swapper/200:01:092
37127432101,5sleep23712895-21awk23:40:162
37059472101,2sleep00-21swapper/023:36:160
36919522101,3sleep10-21swapper/123:29:561
36841432101,3sleep00-21swapper/023:25:020
36571852101,3sleep00-21swapper/023:06:460
36320112101,2sleep10-21swapper/122:50:401
35943192100,3sleep00-21swapper/022:30:110
35428412101,3sleep00-21swapper/021:55:500
35382952101,4sleep13538374-21irqstats21:55:161
35287502100,1sleep10-21swapper/121:48:581
34896022101,4sleep23489631-21apt-get21:25:132
34878492100,2sleep20-21swapper/221:22:502
34707332101,3sleep23464771-21diskmemload21:11:392
34435222101,3sleep13443385-21apt-get20:50:021
34411692101,4sleep23442214-21memory20:45:172
34370812101,3sleep23437129-21ntpq20:40:212
34225762101,3sleep23422595-21apt-get20:30:152
34017032101,4sleep23403211-21cut20:10:182
34013932101,4sleep03401603-21hddtemp_smartct20:10:170
33955662101,4sleep13395595-21/usr/sbin/munin20:05:001
33895282100,1sleep00-21swapper/019:56:530
33742842101,4sleep03374680-21hddtemp_smartct19:15:010
3725805290,2sleep10-21swapper/123:50:171
3464068291,4sleep23464166-21sed21:05:252
3459779291,4sleep13459832-21cpuspeed_turbos21:05:121
3458363291,4sleep13458484-21ntpq21:00:201
3432855291,4sleep23433067-21sed20:40:002
3429168291,4sleep03430871-21grep20:35:170
3427588290,1sleep00-21swapper/020:34:460
3417299291,3sleep03417318-21apt-get20:25:110
3391922291,4sleep23393637-21tr20:00:202
3449698281,4sleep03449778-21kernelversion20:55:130
3443373280,3sleep03443521-21apt-config20:50:000
3422276280,1sleep20-21swapper/220:29:382
3412857281,3sleep13414525-21cut20:20:201
3390604280,2sleep00-21swapper/020:00:150
3385279280,2sleep10-21swapper/119:55:061
3380019281,3sleep13380159-21cat19:45:001
3459228270,1sleep00-21swapper/021:01:350
3432642270,1sleep10-21swapper/120:37:181
3401100270,2sleep10-21swapper/120:10:141
3395606270,4sleep20-21swapper/220:05:022
3379859270,1sleep00-21swapper/019:42:560
3379279270,1sleep10-21swapper/119:38:511
3379160270,1sleep20-21swapper/219:37:162
3378107270,1sleep00-21swapper/019:29:560
3377885270,1sleep10-21swapper/119:27:001
3377836270,1sleep20-21swapper/219:26:222
3376485270,4sleep10-21swapper/119:15:271
3464641260,4sleep00-21swapper/021:08:320
3436737260,2sleep13437075-21strings20:40:201
3411393260,1sleep20-21swapper/220:15:352
3390227260,1sleep20-21swapper/219:58:532
3380442260,1sleep00-21swapper/019:47:190
3380358260,1sleep10-21swapper/119:46:131
3380271260,1sleep20-21swapper/219:45:062
3378698260,1sleep10-21swapper/119:34:261
3378490260,1sleep00-21swapper/019:31:410
3373927260,1sleep20-21swapper/219:10:502
3378553250,3sleep20-21swapper/219:32:312
3412317240,1sleep20-21swapper/220:20:162
3379877240,1sleep20-21swapper/219:43:092
3377317240,1sleep10-21swapper/119:22:481
3377137240,1sleep00-21swapper/019:20:260
3375402240,1sleep20-21swapper/219:15:042
3374132240,1sleep00-21swapper/019:13:330
3373637240,1sleep10-21swapper/119:10:011
3378990230,1sleep00-21swapper/019:35:010
3377109230,1sleep20-21swapper/219:20:042
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional