You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-19 - 00:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot3s.osadl.org (updated Sat Oct 18, 2025 12:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28390172183161,15sleep00-21swapper/007:07:470
28390612156107,16sleep30-21swapper/307:08:223
2838837214799,15sleep10-21swapper/107:05:311
28389402146102,14sleep20-21swapper/207:06:492
31135452690,2sleep20-21swapper/210:53:212
2839323993630,4cyclictest3034252-21kthreadcore10:00:173
2839323993630,3cyclictest0-21swapper/311:45:003
2839323993328,3cyclictest2881728-21dump-pmu-power07:49:593
2839323993121,7cyclictest2983795-21cat09:25:003
2839323992724,2cyclictest2839316-21cyclictest08:25:143
2839323992722,3cyclictest3189396-21gltestperf11:45:133
2839323992520,3cyclictest2050013-21systemd-journal07:37:293
283932399250,25cyclictest0-21swapper/310:32:563
2839323992420,2cyclictest0-21swapper/310:55:303
2839323992419,3cyclictest0-21swapper/309:34:553
283932399241,7cyclictest3271159-21ssh12:36:263
2839323992415,7cyclictest0-21swapper/310:55:003
2839323992413,7cyclictest2976849-21sed09:20:013
283932399241,19cyclictest3146521-21tr11:15:213
283932399240,11cyclictest3163512-21taskset11:25:323
2839323992320,2cyclictest0-21swapper/309:55:143
2839323992315,6cyclictest0-21swapper/312:10:213
2839323992312,7cyclictest3176512-21tr11:35:203
283932399231,19cyclictest3173352-21awk11:35:003
2839323992311,8cyclictest3074845-21sshd10:25:423
2839323992310,9cyclictest3091533-21seq10:38:413
283932399230,9cyclictest3044936-21ntpq10:05:223
2839323992218,2cyclictest0-21swapper/311:50:193
2839323992217,4cyclictest2839316-21cyclictest12:25:203
2839323992217,3cyclictest720-21tpm2-abrmd09:45:003
283932399221,6cyclictest2946896-21/usr/sbin/munin08:50:253
283932399221,18cyclictest3137382-21/usr/sbin/munin11:10:173
283932399219,8cyclictest3155964-21seq11:20:463
283932399216,2cyclictest3060318-21sh10:15:563
2839323992116,3cyclictest720-21tpm2-abrmd11:00:253
2839323992116,3cyclictest2839869-21grep07:10:153
2839323992116,3cyclictest0-21swapper/311:55:173
2839323992115,4cyclictest511ktimers/309:25:013
2839323992112,6cyclictest3066902-21ntp_states10:20:243
283932399210,18cyclictest3024520-21ssh09:51:523
283932399209,7cyclictest3259418-21tr12:30:163
2839323992017,2cyclictest3055242-21perf10:15:013
2839323992017,2cyclictest2962761-21apt-key09:10:093
283932399201,6cyclictest2882369-21hddtemp_smartct07:50:163
2839323992016,2cyclictest0-21swapper/311:05:203
2839323992015,3cyclictest2927395-21cpuspeed_turbos08:35:143
2839323992015,3cyclictest2865761-21unixbench_multi07:30:263
2839323992014,4cyclictest2949102-21cut08:55:173
283932399201,17cyclictest3245971-21grep12:20:173
283932399201,17cyclictest2876351-21runrttasks07:40:293
2839323992011,7cyclictest0-21swapper/310:49:063
283932399200,17cyclictest3235423-21ssh12:15:163
283932399200,17cyclictest2912449-21df08:20:153
283932399198,7cyclictest733-21gdbus08:44:593
2839323991915,3cyclictest3218996-21cpuspeed_turbos12:05:163
2839323991915,2cyclictest0-21swapper/308:19:593
283932399191,12cyclictest3092969-21/usr/sbin/munin10:40:133
283932399190,6cyclictest3003131-21ssh09:36:303
283932399189,7cyclictest2961361-21ntpq09:05:233
283932399187,8cyclictest703-21dbus-daemon07:30:003
2839323991814,2cyclictest0-21swapper/308:05:013
2839323991812,4cyclictest943-21runrttasks07:55:133
2839323991810,6cyclictest2851570-21cpuspeed_turbos07:20:133
2839323991810,6cyclictest0-21swapper/309:45:093
283932399180,6cyclictest3211285-21ssh12:00:153
31789522171,4sleep22962305-21diskmemload11:36:512
2839323991714,2cyclictest2927044-21cut08:35:003
2839323991713,2cyclictest0-21swapper/309:05:003
283932399171,14cyclictest2897694-21irqstats08:05:173
2839323991710,5cyclictest2902204-21/usr/sbin/munin08:10:133
32398162161,3sleep23239826-21ssh12:15:452
32090782160,2sleep10-21swapper/111:57:001
30318282160,13sleep00-21swapper/009:57:090
30269132161,10sleep23028448-21cut09:55:182
28887942161,2sleep20-21swapper/207:55:172
2839323991614,1cyclictest492-21hwrng07:15:253
2839323991612,2cyclictest0-21swapper/308:45:273
32648922151,4sleep10-21swapper/112:33:471
32394192151,9sleep13239619-21users12:15:281
32249662150,2sleep2421ktimers/212:07:542
31970042151,2sleep20-21swapper/211:50:192
31733072151,10sleep23173339-21perf11:35:002
31662662151,5sleep03166291-21apt-get11:30:130
30984872150,3sleep00-21swapper/010:42:430
30758142150,2sleep20-21swapper/210:28:102
30976842141,8sleep12962305-21diskmemload10:41:141
30243672141,2sleep10-21swapper/109:51:371
31408552131,8sleep13140980-21sendmail_mailqu11:10:261
30339082138,1sleep20-21swapper/210:00:152
30170612130,2sleep10-21swapper/109:46:151
30113452131,2sleep04186928-21kworker/0:1+events09:43:490
29863592130,7sleep12871073-21kworker/1:2+events09:25:171
29319992130,1sleep00-21swapper/008:39:450
32631232120,1sleep00-21swapper/012:30:580
32497122121,6sleep12962305-21diskmemload12:24:011
32489952121,7sleep03249009-21ssh12:22:520
32286742121,7sleep23228985-21kthreadcore12:10:192
31932212121,7sleep13193306-21/usr/sbin/munin11:45:251
31556172121,7sleep23155810-21users11:20:292
31502502120,1sleep20-21swapper/211:18:522
31274342120,1sleep20-21swapper/211:01:162
31266772121,7sleep13126697-21awk11:00:271
31202632121,6sleep02962305-21diskmemload10:56:420
31127052121,7sleep02962305-21diskmemload10:51:490
31066062123,5sleep23097961-21kworker/u16:2+xprtiod10:49:102
30478402121,7sleep23047851-21ssh10:09:582
30326092121,3sleep13032618-21ssh09:58:381
29902522120,2sleep0191rcu_preempt09:29:100
29900312121,6sleep20-21swapper/209:28:442
28924112121,7sleep22893229-21cat08:00:152
32560112111,3sleep10-21swapper/112:26:341
32452562111,3sleep20-21swapper/212:20:172
32243392111,5sleep02962305-21diskmemload12:06:570
31885132111,6sleep03188523-21ssh11:44:480
31788542111,3sleep10-21swapper/111:36:421
31643222111,3sleep10-21swapper/111:27:041
31575382110,1sleep10-21swapper/111:23:361
31364172111,6sleep23136428-21ssh11:09:462
31074542112,4sleep13107478-21apt-get10:50:131
30684062110,7sleep13068412-21ssh10:22:141
30539042112,4sleep22962305-21diskmemload10:12:302
30180862111,5sleep02962305-21diskmemload09:48:190
30159412111,5sleep23016000-21ntpq09:45:192
29981202111,5sleep02998144-21apt-get09:35:120
29915132111,6sleep12992424-21kthreadcore09:30:161
29770292111,7sleep12977124-21tr09:20:141
29738772111,5sleep02973923-21ntpq09:15:190
29682492111,3sleep10-21swapper/109:12:041
29181762110,7sleep02919889-21grep08:25:180
29159322111,5sleep02916005-21ntpq08:20:200
28502362111,5sleep12850286-21ntpq07:15:211
32718002101,3sleep00-21swapper/012:37:350
32711842101,2sleep20-21swapper/212:36:292
32645192101,4sleep22962305-21diskmemload12:33:122
32396552101,5sleep02962305-21diskmemload12:15:310
32245332101,3sleep12962305-21diskmemload12:07:141
32183812101,3sleep20-21swapper/212:04:502
32101262100,1sleep00-21swapper/011:59:030
32025362101,4sleep02962305-21diskmemload11:53:290
31951592101,4sleep265-21kcompactd011:48:382
31884862101,3sleep20-21swapper/211:44:462
31701502101,5sleep13170208-21ntpq11:30:221
31502712101,4sleep02962305-21diskmemload11:18:530
31419922100,3sleep20-21swapper/211:11:512
31292412101,4sleep02962305-21diskmemload11:04:350
31199462101,4sleep12962305-21diskmemload10:56:071
31197012101,3sleep20-21swapper/210:55:352
31055312101,4sleep12962305-21diskmemload10:47:131
31049302101,4sleep02962305-21diskmemload10:46:070
30898412101,4sleep02962305-21diskmemload10:35:450
30889722101,5sleep23089019-21ntpq10:35:232
30834622101,3sleep20-21swapper/210:33:222
30823062101,4sleep02962305-21diskmemload10:31:120
30762272101,3sleep02962305-21diskmemload10:28:520
30749782101,3sleep10-21swapper/110:26:001
30608312101,4sleep12962305-21diskmemload10:16:581
30607322101,4sleep02962305-21diskmemload10:16:450
30600962101,5sleep22962305-21diskmemload10:15:352
30546872101,3sleep10-21swapper/110:14:041
30465742101,6sleep02962305-21diskmemload10:07:190
29973772100,2sleep20-21swapper/209:34:102
29759112101,2sleep10-21swapper/109:18:211
29682412101,3sleep00-21swapper/009:12:030
29260062101,4sleep12926052-21ntpq08:30:231
29058702101,4sleep12905944-21ntpq08:10:201
28843092101,4sleep02885564-21memory07:50:190
28786892101,6sleep12880504-21grep07:45:181
28719292101,4sleep22871948-21apt-get07:40:152
28711572100,1sleep10-21swapper/107:37:331
28514712101,4sleep02851490-21apt-get07:20:130
3270677291,2sleep10-21swapper/112:35:401
3257126291,2sleep00-21swapper/012:28:210
3256315291,2sleep20-21swapper/212:27:032
3231799291,2sleep00-21swapper/012:10:410
3227017291,2sleep10-21swapper/112:10:141
3214772291,4sleep13214892-21ntpq12:00:191
3211271291,3sleep00-21swapper/012:00:140
3208792291,2sleep20-21swapper/211:56:252
3196182290,2sleep10-21swapper/111:50:021
3195530291,2sleep00-21swapper/011:49:170
3186201291,3sleep10-21swapper/111:40:321
3177639291,3sleep02962305-21diskmemload11:35:260
3164825290,3sleep00-21swapper/011:27:550
3159676291,4sleep23161267-21tr11:25:172
3155965290,2sleep00-21swapper/011:20:470
3150275291,2sleep10-21swapper/111:18:541
3141666291,3sleep02962305-21diskmemload11:11:090
3136535291,4sleep03136554-21munin-run11:10:000
3133996291,4sleep13134092-21sed11:05:291
3097742291,2sleep20-21swapper/210:41:192
3091278291,3sleep12962305-21diskmemload10:38:151
3082210291,2sleep10-21swapper/110:31:001
3046380290,2sleep10-21swapper/110:07:001
3037906291,4sleep12962305-21diskmemload10:00:261
3025681291,3sleep02962305-21diskmemload09:54:230
3023964291,3sleep22962305-21diskmemload09:50:492
3009752290,2sleep20-21swapper/209:40:552
3005041290,1sleep10-21swapper/109:40:061
3002824291,3sleep12962305-21diskmemload09:35:441
2996455291,2sleep00-21swapper/009:32:110
2982283290,1sleep00-21swapper/009:21:450
2981857291,3sleep22962305-21diskmemload09:20:592
2975342290,3sleep20-21swapper/209:17:182
2968338291,3sleep20-21swapper/209:12:132
2942247290,1sleep00-21swapper/008:49:030
2927287291,3sleep12927306-21apt-get08:35:131
2887429291,4sleep12888792-21grep07:55:171
2877004291,4sleep02877130-21grep07:45:130
2839676291,3sleep12839886-21hddtemp_smartct07:10:151
3069686280,2sleep00-21swapper/010:24:420
3067679281,3sleep22962305-21diskmemload10:20:552
3053132281,3sleep03053141-21ssh10:11:070
3039659281,3sleep02962305-21diskmemload10:03:090
2998077280,2sleep20-21swapper/209:35:052
2958301281,3sleep12959995-21kthreadcore09:05:181
2951275281,3sleep10-21swapper/108:55:211
2948131281,4sleep22949789-21tr08:55:182
2942823281,4sleep02942584-21/usr/sbin/munin08:50:150
2942583280,2sleep10-21swapper/108:50:121
2936988280,5sleep10-21swapper/108:40:411
2916060281,3sleep22916117-21ntpq08:20:232
2916006281,3sleep12916053-21ntpq08:20:211
2907212280,3sleep12907231-21apt-get08:15:131
2867535281,3sleep22869268-21kthreadcore07:35:182
2866215281,3sleep02866240-21perf07:35:000
2860350281,4sleep12860433-21proc_pri07:25:231
2860294281,3sleep02860348-21ntpq07:25:210
2956431270,3sleep12956519-21ps09:00:261
2956243270,2sleep20-21swapper/209:00:232
2952312271,3sleep02952413-21latency_hist09:00:000
2942066270,1sleep20-21swapper/208:46:442
2934835270,3sleep22936188-21memory08:40:182
2927996271,2sleep20-21swapper/208:35:182
2921988270,1sleep10-21swapper/108:29:321
2920838270,3sleep22920964-21ntpq08:25:202
2902196270,2sleep20-21swapper/208:10:132
2901930271,3sleep22902027-21grep08:10:002
2896336270,1sleep10-21swapper/108:00:301
2882382271,2sleep20-21swapper/207:50:172
2876580270,1sleep00-21swapper/007:43:180
2861904271,3sleep12862073-21kthreadcore07:30:161
2860161271,3sleep22860237-21gcc07:25:202
2855926270,1sleep20-21swapper/207:21:152
2962288261,2sleep20-21swapper/209:10:012
2957579260,2sleep00-21swapper/009:05:130
2946361261,2sleep20-21swapper/208:50:202
2932582260,2sleep00-21swapper/008:40:130
2922771261,2sleep20-21swapper/208:30:202
2907897260,2sleep20-21swapper/208:15:182
2906695260,1sleep00-21swapper/008:11:460
2901210261,2sleep00-21swapper/008:05:260
2891417260,1sleep00-21swapper/007:56:450
2871167260,2sleep00-21swapper/007:37:400
2861523262,1sleep20-21swapper/207:30:132
2853105261,2sleep10-21swapper/107:20:181
2846426260,2sleep00-21swapper/007:15:120
2956983250,1sleep00-21swapper/009:00:450
2937675250,2sleep10-21swapper/108:45:151
2880829250,2sleep20-21swapper/207:45:232
2872318250,1sleep10-21swapper/107:40:181
2850839250,1sleep20-21swapper/207:15:432
2844233250,1sleep00-21swapper/007:14:480
2922480240,1sleep00-21swapper/008:30:180
2907401240,1sleep00-21swapper/008:15:160
2897214240,1sleep10-21swapper/108:05:031
2891675240,1sleep00-21swapper/008:00:000
2881961240,1sleep10-21swapper/107:50:041
2839416240,1sleep20-21swapper/207:10:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional