You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-05 - 05:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Nov 05, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2806335213176,51sleep30-21swapper/319:06:023
280663328358,19sleep00-21swapper/019:09:480
280659127862,9sleep10-21swapper/119:09:171
280648427863,9sleep20-21swapper/219:07:542
29985532520,3sleep30-21swapper/321:50:143
2806774992724,1cyclictest0-21swapper/321:35:003
32453012261,10sleep22932147-21diskmemload00:15:372
280677499232,12cyclictest3165135-21taskset23:30:163
2806774992320,1cyclictest0-21swapper/323:40:003
2806774992319,2cyclictest0-21swapper/322:10:013
2806774992218,2cyclictest0-21swapper/323:07:553
280677499221,17cyclictest3254095-21taskset00:21:233
280677499221,17cyclictest3212304-21taskset23:55:373
30225482211,9sleep03022810-21df22:05:150
280677499219,7cyclictest2856814-21turbostat19:55:003
280677499217,7cyclictest2851167-21taskset19:45:333
280677499216,7cyclictest2866381-21taskset20:00:203
2806774992118,2cyclictest0-21swapper/319:15:003
28663312201,4sleep22866379-21ntpq20:00:202
280677499207,1cyclictest0-21swapper/323:25:173
2806774992017,1cyclictest0-21swapper/319:30:153
2806774992017,1cyclictest0-21swapper/319:30:153
2806774992016,2cyclictest0-21swapper/300:12:493
280677499201,15cyclictest0-21swapper/321:35:533
280677499201,15cyclictest0-21swapper/321:25:083
280677499201,14cyclictest0-21swapper/322:46:563
280677499201,13cyclictest0-21swapper/323:44:233
280677499200,17cyclictest66550irq/131-enp1s0-TxRx-323:22:043
280677499200,12cyclictest2857068-21taskset19:55:153
32294332190,3sleep23229450-21ssh00:06:232
2806774991916,1cyclictest0-21swapper/321:40:283
2806774991915,2cyclictest0-21swapper/300:25:163
2806774991912,4cyclictest31358012sleep323:10:273
280677499191,15cyclictest0-21swapper/322:12:353
280677499191,15cyclictest0-21swapper/322:00:193
280677499191,14cyclictest2845980-21turbostat19:45:003
280677499191,14cyclictest0-21swapper/322:50:563
280677499191,14cyclictest0-21swapper/322:40:413
280677499191,14cyclictest0-21swapper/321:16:333
280677499191,14cyclictest0-21swapper/321:13:103
280677499191,14cyclictest0-21swapper/300:34:063
280677499191,10cyclictest3006650-21taskset21:55:153
280677499187,9cyclictest3119680-21chrt23:01:133
2806774991815,1cyclictest0-21swapper/322:30:173
2806774991815,1cyclictest0-21swapper/321:45:253
2806774991815,1cyclictest0-21swapper/321:05:083
280677499181,14cyclictest0-21swapper/323:45:213
280677499181,14cyclictest0-21swapper/322:55:373
280677499181,14cyclictest0-21swapper/321:21:443
280677499181,13cyclictest0-21swapper/323:50:103
280677499181,13cyclictest0-21swapper/322:20:063
280677499181,13cyclictest0-21swapper/322:15:123
280677499181,13cyclictest0-21swapper/320:30:153
280677499181,13cyclictest0-21swapper/320:15:163
280677499181,13cyclictest0-21swapper/300:05:303
280677499181,10cyclictest2835772-21taskset19:35:153
2806774991810,1cyclictest0-21swapper/320:30:003
28251072177,6sleep12825250-21/usr/sbin/munin19:25:121
280677499176,8cyclictest2883534-21chrt20:20:083
2806774991715,1cyclictest0-21swapper/322:25:233
280677499171,4cyclictest0-21swapper/300:00:163
2806774991714,1cyclictest0-21swapper/320:55:153
2806774991713,2cyclictest0-21swapper/319:15:023
280677499171,15cyclictest0-21swapper/300:16:043
280677499171,14cyclictest0-21swapper/323:16:163
280677499171,13cyclictest0-21swapper/322:35:123
280677499171,13cyclictest0-21swapper/300:35:023
280677499170,13cyclictest0-21swapper/320:40:153
31019312161,10sleep23102109-21/usr/sbin/munin22:50:272
31017192166,6sleep03101754-21sh22:50:250
30132832161,10sleep13013298-21ssh21:58:551
280677499161,10cyclictest2872196-21chrt20:05:293
280677499160,13cyclictest0-21swapper/319:25:133
32633962151,9sleep13263406-21ssh00:29:121
32383642151,9sleep03238374-21ssh00:12:270
31701312151,7sleep12932147-21diskmemload23:30:511
30691402151,9sleep23069154-21sh22:31:592
30375242151,9sleep03037535-21ssh22:13:410
30068462153,7sleep03007505-21kthreadcore21:55:170
29578502151,10sleep00-21swapper/021:25:150
29001772151,10sleep02901358-21kthreadcore20:35:170
28911832150,4sleep0221rcuc/020:25:180
28888252150,3sleep20-21swapper/220:25:132
28465022150,2sleep20-21swapper/219:45:132
28362512151,9sleep22837609-21cut19:35:192
280677499151,9cyclictest2920532-21taskset20:50:303
2806774991511,2cyclictest0-21swapper/320:10:253
280677499151,10cyclictest2910585-21chrt20:45:133
2806774991510,3cyclictest0-21swapper/320:35:123
280677499150,13cyclictest0-21swapper/319:20:183
32486622141,8sleep13248827-21ssh00:20:131
32378962140,8sleep22932147-21diskmemload00:11:452
32054342140,9sleep03205444-21ssh23:53:070
31820582141,8sleep13182325-21df23:40:131
31650622140,7sleep22932147-21diskmemload23:30:162
31374972141,9sleep13137512-21ssh23:12:471
31267732141,8sleep03126824-21ntpq23:05:200
31114902141,9sleep23111504-21ssh22:56:362
29952882141,9sleep22995300-21sh21:46:052
29647472141,4sleep22964757-21ssh21:29:102
29368002141,8sleep12936897-21proc_pri21:10:251
29330142141,8sleep22933138-21/usr/sbin/munin21:10:182
29286962141,8sleep22930488-21grep21:05:132
28998952140,6sleep20-21swapper/220:35:152
28946512141,4sleep02896160-21tr20:30:200
280677499141,10cyclictest2926844-21turbostat21:05:003
32786292131,8sleep23278644-21ssh00:37:582
32277152131,4sleep1802-21snmpd00:05:181
32228242131,8sleep03222837-21ssh00:03:530
31981932131,7sleep02932147-21diskmemload23:49:380
31978722131,7sleep23197887-21ssh23:49:122
31486812131,2sleep10-21swapper/123:20:161
30437392131,3sleep03043753-21ssh22:15:380
30305402131,8sleep23030545-21ssh22:10:022
30280472130,3sleep23028058-21ssh22:06:382
30149592131,7sleep13015701-21kthreadcore22:00:151
29977502131,7sleep02997785-21awk21:50:000
29808062131,7sleep22980820-21ssh21:39:002
29779162130,2sleep10-21swapper/121:35:221
29158622131,6sleep00-21swapper/020:50:120
32325352121,7sleep13232713-21grep00:10:151
31631602121,4sleep20-21swapper/223:27:592
31620412120,1sleep00-21swapper/023:26:250
31446392121,3sleep13144649-21ssh23:15:561
31313392121,7sleep23131507-21grep23:10:152
31201992121,7sleep03120207-21sh23:02:000
31103282121,7sleep13110568-21awk22:55:271
31014882121,7sleep13101547-21open_inodes22:50:221
30961952121,2sleep02932147-21diskmemload22:49:330
30622312121,6sleep13062233-21ssh22:28:571
30216482121,6sleep22932147-21diskmemload22:04:172
29981172122,3sleep00-21swapper/021:50:120
29508802121,7sleep12952627-21kthreadcore21:20:191
29474802121,6sleep22947486-21ssh21:18:032
29461392120,7sleep12932147-21diskmemload21:16:061
28082432120,7sleep22809995-21kthreadcore19:10:182
32711102111,2sleep23271120-21ssh00:33:412
32542872111,4sleep02932147-21diskmemload00:21:450
32220692110,1sleep10-21swapper/100:02:421
32130952111,6sleep03213109-21ssh23:56:510
31878792111,6sleep22932147-21diskmemload23:41:592
31875782110,1sleep00-21swapper/023:41:320
31720362110,7sleep03172050-21ssh23:33:300
31316212111,5sleep02932147-21diskmemload23:10:170
30878002111,6sleep13087814-21ssh22:44:491
30799822112,4sleep23080003-21apt-get22:40:172
30065502110,7sleep23006648-21fschecks_time21:55:152
30054672111,3sleep20-21swapper/221:54:232
29980772111,6sleep12998088-21sh21:50:091
29726002111,3sleep00-21swapper/021:33:530
29423522110,7sleep02944125-21kthreadcore21:15:170
28109052111,5sleep02811018-21ntpq19:10:200
32631992100,1sleep00-21swapper/000:28:500
32445712101,5sleep03244630-21/usr/sbin/munin00:15:240
32158342101,4sleep23215990-21diskstats00:00:142
31613742100,2sleep10-21swapper/123:25:301
31286572101,3sleep12932147-21diskmemload23:07:141
30888972101,5sleep13089139-21grep22:45:171
30702502101,2sleep00-21swapper/022:33:450
30673582101,4sleep13067456-21ntpq22:30:221
30593612101,5sleep23059418-21ntpq22:25:222
30543402101,2sleep00-21swapper/022:24:150
30536502101,6sleep12806767-21cyclictest22:23:081
29270862101,4sleep12927104-21apt-get21:05:081
29211502101,4sleep02921169-21apt-get20:55:130
29158842101,4sleep12916136-21df20:50:131
29052622101,4sleep02905365-21perl20:40:180
28943612101,5sleep22894571-21hddtemp_smartct20:30:182
28808982101,4sleep12882101-21memory20:15:221
28773752101,4sleep02877619-21users20:10:400
28717202101,4sleep12871770-21ntpq20:05:241
28610332101,4sleep22861086-21ntpq19:55:232
28069482101,4sleep12806967-21apt-get19:10:131
3272798291,4sleep03272906-21tail00:35:160
3267691291,4sleep03268264-21/usr/sbin/munin00:30:190
3254129290,1sleep20-21swapper/200:21:272
3244304294,2sleep10-21swapper/100:15:201
3231274291,2sleep00-21swapper/000:09:100
3213578291,4sleep23213589-21ssh23:57:332
3205305291,3sleep22932147-21diskmemload23:52:572
3197817291,4sleep13197832-21ssh23:49:071
3179753291,4sleep03179771-21ssh23:37:230
3179109291,3sleep22932147-21diskmemload23:36:292
3153999291,4sleep23154010-21ssh23:22:022
3146011290,3sleep02932147-21diskmemload23:17:470
3127979291,2sleep20-21swapper/223:06:142
3120258290,1sleep20-21swapper/223:02:062
3119350291,2sleep10-21swapper/123:00:441
3094314291,4sleep23094328-21ssh22:46:582
3084493291,4sleep03084590-21cat22:40:310
3076593291,4sleep03076604-21ssh22:35:500
3076526291,4sleep23076540-21ssh22:35:442
3045425290,1sleep10-21swapper/122:18:111
3036036290,1sleep10-21swapper/122:11:221
3019979290,1sleep00-21swapper/022:01:430
2996740290,1sleep10-21swapper/121:48:251
2987419290,1sleep00-21swapper/021:41:300
2957046291,3sleep20-21swapper/221:24:532
2949206291,4sleep02949266-21grep21:20:140
2940479291,4sleep02940508-21munin-run21:15:010
2914588291,4sleep22914634-21ntpq20:45:212
2888240290,1sleep20-21swapper/220:20:562
2878261291,4sleep02878513-21df20:15:170
2872966291,3sleep12872985-21apt-get20:10:251
2860847293,3sleep12861029-21needreboot19:55:221
2822565291,4sleep12823348-21missed_timers19:20:191
3277959281,2sleep13277966-21ssh00:36:491
3212658281,2sleep13212668-21ssh23:56:121
3147970281,2sleep00-21swapper/023:20:110
3143502281,2sleep20-21swapper/223:15:202
3110742281,3sleep03110750-21ssh22:55:340
3076562281,3sleep13076564-21ssh22:35:471
3061235281,3sleep03061249-21ssh22:27:240
3054041280,1sleep20-21swapper/222:23:452
3042747281,3sleep23042866-21netstat22:15:202
2982221281,3sleep12982349-21perl21:40:191
2981898281,3sleep22981906-21ssh21:40:122
2978776281,2sleep00-21swapper/021:35:490
2970092281,4sleep22965695-21/usr/sbin/munin21:30:262
2965631281,3sleep12965643-21ssh21:30:091
2962890281,3sleep12962897-21ssh21:26:161
2926664281,3sleep22926665-21pool-udisksd21:02:562
2915473280,1sleep00-21swapper/020:48:230
2904981281,2sleep20-21swapper/220:40:162
2901359281,3sleep12901701-21kthreadcore20:35:181
2898382281,3sleep12898482-21timerandwakeup20:30:291
2888558280,1sleep10-21swapper/120:24:561
2867643280,2sleep00-21swapper/020:05:150
2855836280,4sleep10-21swapper/119:50:211
2830368280,2sleep10-21swapper/119:30:131
2830368280,2sleep10-21swapper/119:30:131
2825074280,1sleep00-21swapper/019:25:030
2824545280,1sleep20-21swapper/219:24:572
2814840281,3sleep02816150-21kthreadcore19:15:170
3263056270,3sleep20-21swapper/200:28:362
3198721271,2sleep10-21swapper/123:50:031
3173498271,2sleep13173529-21apt-get23:35:121
3029841271,2sleep10-21swapper/122:09:211
2926537270,1sleep10-21swapper/121:01:231
2920639270,1sleep20-21swapper/220:51:542
2904970270,1sleep10-21swapper/120:40:071
2883541270,1sleep00-21swapper/020:20:140
2878232270,1sleep20-21swapper/220:15:112
2851384270,1sleep00-21swapper/019:48:200
2840961270,2sleep20-21swapper/219:40:122
2835679270,2sleep10-21swapper/119:35:141
2829926270,1sleep20-21swapper/219:27:452
2819006270,1sleep20-21swapper/219:17:072
3269244260,3sleep10-21swapper/100:30:401
2914474260,2sleep10-21swapper/120:45:201
2893733260,1sleep10-21swapper/120:29:091
2873538261,2sleep20-21swapper/220:10:302
2867350260,4sleep10-21swapper/120:04:401
2862455260,2sleep00-21swapper/020:00:140
2861936261,2sleep02861937-21sshd19:58:210
2852348261,2sleep00-21swapper/019:50:160
2851286260,1sleep10-21swapper/119:47:041
2839451260,2sleep02839633-21strings19:35:210
2835215260,4sleep20-21swapper/219:32:262
2835215260,4sleep20-21swapper/219:32:262
2812631260,2sleep10-21swapper/119:15:021
2925853250,3sleep20-21swapper/220:56:072
2856449250,3sleep20-21swapper/219:50:502
2841251250,1sleep10-21swapper/119:40:151
2820813250,2sleep00-21swapper/019:20:180
2927495240,1sleep00-21swapper/021:05:110
2926236240,1sleep00-21swapper/021:00:020
2920983240,1sleep10-21swapper/120:55:021
2867925240,1sleep20-21swapper/220:05:172
2841147240,1sleep00-21swapper/019:40:140
2830753240,1sleep00-21swapper/019:30:160
2830753240,1sleep00-21swapper/019:30:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional