You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-19 - 06:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Sat Jul 19, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
33215612151105,16sleep20-21swapper/219:07:192
33217212148102,15sleep10-21swapper/119:09:241
3321450214399,15sleep30-21swapper/319:05:563
3317088213999,13sleep00-21swapper/019:05:060
354511621350,2sleep00-21swapper/022:07:010
37979352470,1sleep30-21swapper/300:35:273
3321919994524,15cyclictest717-21systemd-logind23:24:593
3321919993621,13cyclictest3612751-21grep22:45:273
3321919993523,10cyclictest0-21swapper/319:35:143
3321919993429,3cyclictest0-21swapper/321:30:183
3321919993422,10cyclictest3509635-21unixbench_multi21:45:283
3321919993422,10cyclictest262-21systemd-journal21:25:003
3321919993419,13cyclictest3749904-21grep00:05:303
3321919993416,15cyclictest3535302-21grep22:00:283
332191999327,14cyclictest3629190-21head22:55:203
332191999321,25cyclictest3621846-21taskset22:51:033
3321919993211,16cyclictest3769641-21tail00:20:163
332191999320,1cyclictest0-21swapper/321:29:103
3321919993122,6cyclictest3490098-21cut21:35:193
3321919993120,9cyclictest0-21swapper/300:00:003
332191999311,9cyclictest3354385-21tune2fs19:40:163
3321919993118,11cyclictest0-21swapper/300:00:193
332191999308,16cyclictest3444587-21diskmemload21:57:183
3321919993025,3cyclictest3521802-21cron21:55:013
3321919993010,14cyclictest943-21runrttasks00:11:143
3321919993010,10cyclictest0-21swapper/322:05:183
332191999297,17cyclictest0-21swapper/321:16:513
332191999296,17cyclictest3648533-21kworker/u16:0+xprtiod23:12:573
332191999291,7cyclictest33946571latency_hist20:20:013
332191999290,8cyclictest36078541latency_hist22:45:013
332191999290,8cyclictest36078541latency_hist22:45:003
332191999290,8cyclictest3449712-21taskset21:10:383
332191999290,7cyclictest36855431cpu23:30:123
332191999290,27cyclictest3635682-21grep23:00:163
332191999289,17cyclictest3722878-21cut23:50:193
332191999281,25cyclictest3718879-21runrttasks23:49:113
332191999281,25cyclictest3444587-21diskmemload23:37:003
332191999281,22cyclictest3414894-21chrt20:40:133
3321919992810,15cyclictest3552636-21cat22:10:313
332191999280,7cyclictest3650170-21ssh23:09:143
332191999280,7cyclictest3435427-21grep21:00:153
332191999280,6cyclictest33660061kthreadcore19:50:173
332191999280,20cyclictest3683532-21ssh23:28:013
332191999280,18cyclictest0-21swapper/322:38:453
332191999280,17cyclictest3502060-21taskset21:41:393
332191999280,17cyclictest3502060-21taskset21:41:383
332191999277,9cyclictest0-21swapper/322:26:083
332191999271,6cyclictest35572661kthreadcore22:15:153
332191999271,6cyclictest33309161kthreadcore19:15:173
3321919992714,11cyclictest3419951-21cpuspeed_turbos20:45:133
3321919992714,10cyclictest3374846-21gltestperf20:00:153
3321919992713,12cyclictest3590705-21latency_hist22:35:003
332191999271,2cyclictest0-21swapper/323:15:183
332191999271,23cyclictest3791453-21sh00:32:433
332191999270,2cyclictest0-21swapper/300:26:343
332191999270,19cyclictest3383530-21ntpq20:05:213
3321919992613,11cyclictest0-21swapper/321:00:003
332191999261,22cyclictest0-21swapper/323:40:083
3321919992612,12cyclictest0-21swapper/322:23:193
332191999260,7cyclictest3343603-21sshd19:27:103
332191999260,5cyclictest37633511kthreadcore00:15:173
3321919992514,9cyclictest3322569-21kthreadcore19:10:173
332191999251,21cyclictest3344458-21sed19:30:143
332191999251,21cyclictest3344458-21sed19:30:143
3321919992512,11cyclictest3440016-21cpuspeed_turbos21:05:123
36088162240,2sleep00-21swapper/022:45:170
332191999240,10cyclictest3336603-21cut19:20:213
332191999239,12cyclictest717-21systemd-logind20:35:003
3321919992319,2cyclictest717-21systemd-logind20:24:593
332191999231,6cyclictest33872531kthreadcore20:10:153
332191999231,18cyclictest3369805-21head19:55:133
332191999225,14cyclictest3400064-21uniq20:25:163
3321919992210,10cyclictest0-21swapper/320:35:253
3321919992210,10cyclictest0-21swapper/319:45:123
332191999211,17cyclictest3428592-21grep20:50:213
37118082201,7sleep13713081-21cat23:45:151
35991912200,1sleep20-21swapper/222:39:542
35110452201,14sleep03444587-21diskmemload21:47:130
36577392191,3sleep20-21swapper/223:13:012
359139121910,4sleep13591592-21if_err_enp1s022:35:171
37993082181,2sleep00-21swapper/000:37:180
37028172170,12sleep10-21swapper/123:40:151
36936032171,3sleep00-21swapper/023:34:520
35651362171,3sleep10-21swapper/122:20:011
37833082160,2sleep10-21swapper/100:27:291
37676752161,8sleep10-21swapper/100:18:061
37174192161,3sleep20-21swapper/223:47:212
36663862161,3sleep20-21swapper/223:18:072
35905832161,4sleep13444587-21diskmemload22:34:521
34536882161,11sleep00-21swapper/021:15:150
37284092151,9sleep23728422-21ssh23:55:062
36807672151,9sleep23680824-21ssh23:25:212
36323922151,4sleep23632403-21ssh22:58:342
35814252151,2sleep20-21swapper/222:29:092
35346522151,7sleep23534711-21ntpq22:00:222
34682072150,6sleep10-21swapper/121:22:241
36379822141,7sleep13638070-21/usr/sbin/munin23:00:211
35463382141,2sleep10-21swapper/122:08:341
34781252141,2sleep10-21swapper/121:28:551
37868422130,2sleep2411rcuc/200:30:182
37586332131,2sleep2191rcu_preempt00:11:432
37576532131,6sleep13757849-21users00:10:301
37521292131,7sleep10-21swapper/100:08:521
37507762131,3sleep23750786-21ssh00:06:452
36855362131,6sleep23685693-21df23:30:132
36591042131,8sleep03444587-21diskmemload23:14:430
35572312131,8sleep13558518-21tr22:15:171
35548502131,6sleep10-21swapper/122:13:071
35367372131,6sleep00-21swapper/022:02:120
35203732130,1sleep1321rcuc/121:53:111
34964292131,8sleep13496443-21ssh21:40:051
34964292131,8sleep13496443-21ssh21:40:041
34789182131,8sleep03478929-21ssh21:29:540
34399092130,2sleep10-21swapper/121:05:131
37823302120,8sleep03782338-21rm00:26:000
37753012122,2sleep1191rcu_preempt00:22:311
37612842120,6sleep281-21kswapd000:15:022
37525712120,2sleep00-21swapper/000:09:310
36844492121,7sleep03684456-21ssh23:29:120
36664782121,3sleep10-21swapper/123:18:141
36582162121,7sleep10-21swapper/123:13:361
36217892121,7sleep03444587-21diskmemload22:50:580
35702332121,7sleep03444587-21diskmemload22:20:520
35618572121,2sleep00-21swapper/022:16:080
34784612127,2sleep2914-21ntpd21:29:192
34669222120,8sleep03466935-21sh21:20:430
33744842122,5sleep13374503-21apt-get20:00:121
33342572120,1sleep20-21swapper/219:20:182
37743052110,3sleep20-21swapper/200:21:052
37387982111,6sleep03740494-21grep00:00:180
37102482111,6sleep03444587-21diskmemload23:44:110
37003762111,3sleep10-21swapper/123:37:331
36816032111,6sleep13681611-21ssh23:25:361
36694812111,6sleep03671178-21kthreadcore23:20:170
36318532110,1sleep10-21swapper/122:57:541
36246952111,3sleep10-21swapper/122:54:401
36214952111,5sleep20-21swapper/222:50:372
36124852111,6sleep13612569-21ssh22:45:271
36057662111,3sleep20-21swapper/222:42:232
36057662111,3sleep20-21swapper/222:42:222
36002232111,6sleep13600278-21kthreadcore22:40:171
36002232111,6sleep13600278-21kthreadcore22:40:171
35793962111,4sleep13444587-21diskmemload22:26:351
35717602111,3sleep20-21swapper/222:22:462
35557812111,3sleep00-21swapper/022:14:180
35279042111,4sleep23444587-21diskmemload21:56:532
35195592110,1sleep20-21swapper/221:52:092
35104532110,1sleep20-21swapper/221:46:262
35015582111,3sleep20-21swapper/221:41:022
35015582111,3sleep20-21swapper/221:41:012
34976762111,6sleep03499339-21tr21:40:160
34976762111,6sleep03499339-21tr21:40:150
34670432111,3sleep20-21swapper/221:20:582
34586312111,5sleep23444587-21diskmemload21:16:082
34454362111,7sleep03445733-21cut21:10:160
34103032115,2sleep10-21swapper/120:35:191
33961942111,6sleep13397965-21kthreadcore20:20:161
33504312111,6sleep03350973-21kthreadcore19:35:190
33428532111,5sleep03342902-21ntpq19:25:220
38006752100,6sleep20-21swapper/200:39:252
37924272100,3sleep03792441-21ssh00:34:090
37853862102,3sleep13785410-21apt-get00:30:131
37843112101,4sleep23444587-21diskmemload00:28:572
37678542101,3sleep03444587-21diskmemload00:18:220
37586962101,4sleep03444587-21diskmemload00:11:480
37425292101,2sleep10-21swapper/100:01:201
37423322101,2sleep20-21swapper/200:01:042
37340832100,3sleep10-21swapper/123:56:421
37269232100,3sleep10-21swapper/123:53:341
37265342101,5sleep03444587-21diskmemload23:53:050
37161082100,1sleep00-21swapper/023:45:420
37009972101,3sleep03444587-21diskmemload23:38:210
36752142101,3sleep20-21swapper/223:23:232
36730332101,3sleep13444587-21diskmemload23:20:381
36654592101,3sleep00-21swapper/023:16:570
36501652101,3sleep00-21swapper/023:09:150
36475842101,3sleep23444587-21diskmemload23:06:012
36402542101,2sleep20-21swapper/223:02:352
36306262101,3sleep03444587-21diskmemload22:56:200
36122642100,6sleep23612392-21sendmail_mailqu22:45:232
35898972101,4sleep03444587-21diskmemload22:33:590
35807182100,1sleep00-21swapper/022:28:140
35614572101,3sleep23444587-21diskmemload22:15:372
35481272100,3sleep20-21swapper/222:10:132
35054632101,4sleep13505591-21irqstats21:45:171
34879472101,3sleep10-21swapper/121:35:121
34833742101,5sleep03483434-21/usr/sbin/munin21:30:210
34602582101,4sleep13444587-21diskmemload21:18:111
34334682101,2sleep2191rcu_preempt20:55:202
33902382100,6sleep03390714-21cat20:15:140
33578792101,4sleep23357926-21ntpq19:40:232
3793294290,1sleep10-21swapper/100:35:031
3735256290,2sleep00-21swapper/023:58:090
3726779291,4sleep23726790-21ssh23:53:232
3706738291,4sleep23706834-21ssh23:40:232
3700181291,2sleep20-21swapper/223:37:192
3645434291,2sleep10-21swapper/123:05:191
3638646291,2sleep00-21swapper/023:00:320
3605321291,3sleep00-21swapper/022:41:480
3605321291,3sleep00-21swapper/022:41:480
3591301290,6sleep03591388-21fschecks_time22:35:160
3590287291,2sleep20-21swapper/222:34:292
3546342290,2sleep20-21swapper/222:08:352
3536433291,2sleep10-21swapper/122:01:501
3525989291,4sleep03526047-21ntpq21:55:180
3522132290,3sleep10-21swapper/121:55:071
3521564290,2sleep00-21swapper/021:54:430
3493390291,3sleep03444587-21diskmemload21:36:310
3492749290,1sleep20-21swapper/221:35:402
3486662290,1sleep10-21swapper/121:33:511
3483572290,2sleep20-21swapper/221:30:232
3450131291,2sleep20-21swapper/221:11:122
3445734291,4sleep10-21swapper/121:10:181
3432350291,4sleep13433467-21memory20:55:191
3419867291,3sleep03419886-21apt-get20:45:120
3415261291,4sleep13415332-21irqstats20:40:171
3413510291,4sleep03413583-21ntpq20:35:220
3394841291,3sleep03394863-21apt-get20:20:110
3366806291,4sleep03368061-21memory19:50:170
3354305291,3sleep03354518-21hddtemp_smartct19:40:160
3334080291,2sleep00-21swapper/019:20:160
3775289280,1sleep00-21swapper/000:22:300
3685099281,3sleep10-21swapper/123:30:001
3434894280,3sleep13434913-21apt-get21:00:131
3430667281,3sleep03432349-21cut20:55:180
3424909281,3sleep23424928-21apt-get20:50:162
3400026281,3sleep03400236-21hddtemp_smartct20:25:160
3379756280,2sleep10-21swapper/120:05:141
3348720282,4sleep03321825-21kworker/0:3+events19:33:470
3348720282,4sleep03321825-21kworker/0:3+events19:33:460
3332813281,4sleep23332865-21gcc19:15:212
3326685280,1sleep00-21swapper/019:12:340
3429548270,1sleep10-21swapper/120:53:551
3400336271,2sleep20-21swapper/220:25:192
3385178271,3sleep13385255-21kernelversion20:10:141
3364411270,2sleep20-21swapper/219:50:112
3359113270,2sleep20-21swapper/219:45:102
3329029270,2sleep13329048-21apt-get19:15:131
3322069270,2sleep20-21swapper/219:10:122
3443569261,2sleep00-21swapper/021:05:200
3440057261,2sleep20-21swapper/221:05:142
3420708261,2sleep10-21swapper/120:45:171
3414872260,2sleep20-21swapper/220:40:142
3409556260,1sleep10-21swapper/120:34:351
3408723261,2sleep20-21swapper/220:30:232
3405627261,2sleep03407323-21kthreadcore20:30:180
3399559260,1sleep20-21swapper/220:24:502
3390715261,2sleep10-21swapper/120:15:161
3380610261,2sleep20-21swapper/220:05:192
3375618261,2sleep20-21swapper/220:00:172
3374048260,1sleep00-21swapper/019:58:080
3364219260,2sleep10-21swapper/119:50:021
3363187261,2sleep00-21swapper/019:45:250
3348791260,1sleep10-21swapper/119:34:401
3348791260,1sleep10-21swapper/119:34:391
3343821261,2sleep23343846-21perf19:30:002
3333785261,2sleep03333858-21munin-run19:20:010
3438656250,2sleep20-21swapper/221:00:222
3427136250,2sleep00-21swapper/020:50:220
3424634251,1sleep20-21swapper/220:49:552
3418395250,1sleep00-21swapper/020:40:200
3392428250,1sleep20-21swapper/220:15:172
3385268250,2sleep20-21swapper/220:10:152
3384964250,1sleep00-21swapper/020:10:130
3379213250,2sleep0221rcuc/020:05:000
3373220250,2sleep10-21swapper/119:55:201
3362758250,1sleep10-21swapper/119:45:171
3358578251,2sleep10-21swapper/119:41:301
3344087250,1sleep20-21swapper/219:30:132
3344087250,1sleep20-21swapper/219:30:122
3342904250,1sleep10-21swapper/119:25:231
3326013250,1sleep10-21swapper/119:10:231
3434892240,1sleep00-21swapper/021:00:120
3409880240,1sleep20-21swapper/220:35:162
3399842240,1sleep10-21swapper/120:25:131
3379949240,1sleep00-21swapper/020:05:170
3369459240,1sleep20-21swapper/219:55:082
3349260240,1sleep10-21swapper/119:35:161
3348820240,1sleep20-21swapper/219:35:012
3333859240,1sleep10-21swapper/119:20:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional