You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-24 - 12:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Thu Apr 24, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
245334399293282,8cyclictest2502484-21kworker/u16:3+i915-dp00:37:502
251595722460,2sleep10-21swapper/121:22:201
245334399175166,7cyclictest2530727-21kworker/u16:2+i915-dp00:23:232
245334399173167,4cyclictest2457461-21kworker/u16:0+i915-dp19:27:322
245334399173165,5cyclictest2530727-21kworker/u16:2+i915-dp23:33:312
245334399172166,4cyclictest2626147-21kworker/u16:1+i915-dp00:28:262
245334399172166,4cyclictest2502484-21kworker/u16:3+i915-dp00:08:172
245334399172166,4cyclictest2463666-21kworker/u16:2+i915-dp20:57:512
245334399172164,6cyclictest2502484-21kworker/u16:3+i915-dp00:16:002
245334399172163,7cyclictest2530727-21kworker/u16:2+i915-dp21:41:222
245334399172163,6cyclictest2605550-21kworker/u16:1+i915-dp23:07:502
245334399171165,4cyclictest2638806-21kworker/u16:4+i915-dp23:58:322
245334399171165,4cyclictest2437033-21kworker/u16:3+i915-dp19:55:152
245334399170158,8cyclictest2530727-21kworker/u16:2+i915-dp22:19:082
2453343991700,164cyclictest2485750-21kworker/2:0+i915-unordered23:52:122
245334399169163,4cyclictest2530727-21kworker/u16:2+i915-dp23:10:112
245334399169163,4cyclictest2502484-21kworker/u16:3+i915-dp22:23:002
245334399169161,6cyclictest2530727-21kworker/u16:2+i915-dp23:27:292
245334399169160,6cyclictest2476884-21kworker/u16:0+i915-dp21:38:192
245334399169156,10cyclictest2502484-21kworker/u16:3+i915-dp22:49:422
2453343991690,166cyclictest2452894-21kworker/2:0+i915-unordered19:43:022
2453343991690,163cyclictest2485750-21kworker/2:0+i915-unordered21:11:412
245334899168165,1cyclictest0-21swapper/321:14:403
245334899167164,1cyclictest0-21swapper/319:18:183
245334399167161,4cyclictest2502484-21kworker/u16:3+i915-dp22:09:342
245334399167160,4cyclictest2502484-21kworker/u16:3+i915-dp00:31:262
245334399167157,8cyclictest2474656-21kworker/u16:1+i915-dp21:27:042
2453343991671,162cyclictest2477296-21kworker/2:1+i915-unordered20:19:482
245334899166163,1cyclictest0-21swapper/323:21:253
245334399166161,3cyclictest2463666-21kworker/u16:2+i915-dp21:03:252
245334399166160,4cyclictest2474656-21kworker/u16:1+i915-dp21:32:182
245334399166160,4cyclictest2457461-21kworker/u16:0+i915-dp19:31:442
2453343991660,163cyclictest2485750-21kworker/2:0+i915-unordered21:56:212
2453343991660,162cyclictest2485750-21kworker/2:0+i915-unordered21:09:592
2453343991660,162cyclictest2452894-21kworker/2:0+i915-unordered19:10:482
245334399165159,4cyclictest2502484-21kworker/u16:3+i915-dp23:36:122
245334399165159,4cyclictest2502484-21kworker/u16:3+i915-dp23:36:122
245334399165159,4cyclictest2474656-21kworker/u16:1+i915-dp21:21:332
245334399165157,6cyclictest2476884-21kworker/u16:0+i915-dp20:30:202
2453343991650,162cyclictest2485750-21kworker/2:0+i915-unordered23:18:472
2453343991650,162cyclictest2485750-21kworker/2:0+i915-unordered22:10:572
245334399164158,4cyclictest2502484-21kworker/u16:3+i915-dp22:54:432
2453343991640,161cyclictest2485750-21kworker/2:0+i915-unordered23:00:482
2453343991640,161cyclictest2452894-21kworker/2:0+i915-unordered19:51:042
2453343991640,160cyclictest2485750-21kworker/2:0+i915-unordered23:46:402
2453343991630,160cyclictest2485750-21kworker/2:0+i915-unordered22:29:252
2453343991630,160cyclictest2477296-21kworker/2:1+i915-unordered20:26:012
2453343991630,158cyclictest2485750-21kworker/2:0+i915-unordered21:53:502
245334899162159,1cyclictest0-21swapper/319:31:143
2453343991620,159cyclictest2485750-21kworker/2:0+i915-unordered21:16:322
2453343991620,159cyclictest2452894-21kworker/2:0+i915-unordered19:47:442
2453343991620,159cyclictest2452894-21kworker/2:0+i915-unordered19:47:432
2453343991620,159cyclictest2452894-21kworker/2:0+i915-unordered19:35:492
2453343991620,158cyclictest2477296-21kworker/2:1+i915-unordered20:10:232
245334899161158,1cyclictest0-21swapper/322:35:363
245334899161158,1cyclictest0-21swapper/300:27:553
245334399161155,4cyclictest2451418-21kworker/u16:1+i915-dp19:15:572
2453343991611,157cyclictest2485750-21kworker/2:0+i915-unordered23:43:282
2453343991610,160cyclictest2485750-21kworker/2:0+i915-unordered00:12:412
2453343991610,157cyclictest2485750-21kworker/2:0+i915-unordered22:55:572
2453343991610,157cyclictest2452894-21kworker/2:0+i915-unordered20:04:202
245334899160157,1cyclictest0-21swapper/323:00:163
245334399160154,4cyclictest2502484-21kworker/u16:3+i915-dp22:33:042
2453343991600,157cyclictest2485750-21kworker/2:0+i915-unordered22:41:302
2453343991600,157cyclictest2485750-21kworker/2:0+i915-unordered22:35:082
2453343991600,157cyclictest2485750-21kworker/2:0+i915-unordered20:39:472
245334399159153,4cyclictest2476884-21kworker/u16:0+i915-dp20:05:192
2453343991590,156cyclictest2485750-21kworker/2:0+i915-unordered23:21:582
2453343991590,156cyclictest2485750-21kworker/2:0+i915-unordered20:54:132
2453343991590,156cyclictest2485750-21kworker/2:0+i915-unordered00:01:362
245334899158155,1cyclictest0-21swapper/321:33:383
245334899158155,1cyclictest0-21swapper/320:03:073
2453343991570,154cyclictest2485750-21kworker/2:0+i915-unordered20:44:492
245334899156153,1cyclictest0-21swapper/319:46:013
245334899156153,1cyclictest0-21swapper/319:46:013
245334899156153,1cyclictest0-21swapper/300:35:393
2453343991561,152cyclictest2485750-21kworker/2:0+i915-unordered22:04:552
2453343991550,151cyclictest2485750-21kworker/2:0+i915-unordered21:46:052
245334899154151,1cyclictest0-21swapper/321:58:583
2453343991540,151cyclictest2477296-21kworker/2:1+i915-unordered20:22:392
245334899153149,2cyclictest0-21swapper/300:00:533
245334899151148,1cyclictest0-21swapper/323:43:363
245334899150147,1cyclictest0-21swapper/300:31:073
256961921490,3sleep12569604-21perf22:20:011
2453343991490,146cyclictest2485750-21kworker/2:0+i915-unordered20:48:012
245334899146143,1cyclictest0-21swapper/319:13:353
2453343991460,143cyclictest2452894-21kworker/2:0+i915-unordered19:23:232
245334899144141,1cyclictest0-21swapper/319:41:393
245334899144139,2cyclictest0-21swapper/321:41:523
245334899142139,1cyclictest0-21swapper/319:57:063
245334899137134,1cyclictest0-21swapper/320:59:433
245334899135132,1cyclictest0-21swapper/323:47:473
245334899135132,1cyclictest0-21swapper/321:17:203
245334899134131,1cyclictest0-21swapper/322:52:533
245334899133130,1cyclictest0-21swapper/322:18:183
245334899132129,1cyclictest0-21swapper/323:17:343
245334899132129,1cyclictest0-21swapper/320:46:373
245334899128123,2cyclictest0-21swapper/321:39:303
245334899127124,1cyclictest0-21swapper/322:47:313
245334899125122,1cyclictest0-21swapper/323:36:433
245334899125122,1cyclictest0-21swapper/323:36:423
245334899124121,1cyclictest0-21swapper/322:26:113
245334899123120,1cyclictest0-21swapper/321:50:263
245334899122117,2cyclictest0-21swapper/323:33:503
245334899119116,1cyclictest0-21swapper/323:56:113
245334899118115,1cyclictest0-21swapper/322:07:123
245334899115112,1cyclictest0-21swapper/320:19:463
245334899114111,1cyclictest0-21swapper/323:50:393
245334899114111,1cyclictest0-21swapper/322:02:413
245334899114111,1cyclictest0-21swapper/321:21:433
2452999211458,52sleep30-21swapper/319:07:513
245334899113110,1cyclictest0-21swapper/320:35:433
245334899113110,1cyclictest0-21swapper/300:10:483
245334899111108,1cyclictest0-21swapper/320:52:503
245334899105102,1cyclictest0-21swapper/319:35:363
245334899104101,1cyclictest0-21swapper/320:40:553
24533489910299,1cyclictest0-21swapper/321:27:143
24533489910198,1cyclictest0-21swapper/320:23:163
24533489910198,1cyclictest0-21swapper/300:17:113
24533489910097,1cyclictest0-21swapper/320:12:413
2453348998986,1cyclictest0-21swapper/300:06:263
245312728453,9sleep20-21swapper/219:09:102
245309928456,9sleep10-21swapper/119:08:531
245141028455,6sleep00-21swapper/019:05:060
2453348998178,1cyclictest0-21swapper/322:20:493
2453348997673,1cyclictest0-21swapper/300:24:043
2453348997168,1cyclictest0-21swapper/321:01:543
2453348997168,1cyclictest0-21swapper/319:26:023
2453348996966,1cyclictest0-21swapper/321:45:433
2453348996964,2cyclictest0-21swapper/323:12:213
2453348996461,1cyclictest0-21swapper/322:40:383
2453348996057,1cyclictest0-21swapper/323:09:213
2453348994845,1cyclictest0-21swapper/323:26:183
2453348994138,1cyclictest0-21swapper/322:10:143
2453348994037,1cyclictest0-21swapper/319:53:243
2453348993835,1cyclictest0-21swapper/321:08:573
24705212370,2sleep00-21swapper/019:47:030
24705212370,2sleep00-21swapper/019:47:020
24821302231,16sleep12482149-21apt-get20:20:121
25812792203,8sleep136-21ksoftirqd/122:31:561
2453348992015,2cyclictest0-21swapper/320:30:163
2453348991916,1cyclictest0-21swapper/322:31:163
2453348991916,1cyclictest0-21swapper/320:06:023
2453348991916,1cyclictest0-21swapper/319:21:313
2453348991915,2cyclictest0-21swapper/320:25:213
2453348991914,2cyclictest0-21swapper/322:56:373
26300502181,3sleep02502500-21diskmemload23:23:230
25537012180,6sleep00-21swapper/022:02:120
26400112170,2sleep00-21swapper/023:34:260
25906972171,4sleep02502500-21diskmemload22:40:580
26183772160,1sleep10-21swapper/123:11:131
26062482160,2sleep00-21swapper/022:58:380
25580382161,2sleep00-21swapper/022:08:060
25116822161,2sleep00-21swapper/021:18:510
26650842150,3sleep00-21swapper/000:00:090
26151222150,2sleep00-21swapper/023:08:480
25726632151,4sleep02572674-21ssh22:22:080
25669082150,1sleep00-21swapper/022:16:160
25495472150,2sleep0241rcuc/021:58:350
25382982150,2sleep00-21swapper/021:46:530
25335652150,2sleep00-21swapper/021:40:280
25332012151,10sleep12533343-21/usr/sbin/munin21:40:251
25145182150,6sleep00-21swapper/021:20:270
25044632150,2sleep00-21swapper/021:10:370
26919682141,4sleep02477977-21kworker/0:0+i915-unordered00:30:580
25429522140,2sleep10-21swapper/121:51:181
26782442130,1sleep00-21swapper/000:15:190
26623712131,2sleep1191rcu_preempt23:56:401
26532142131,2sleep10-21swapper/123:48:141
26382872131,8sleep12638297-21ssh23:31:581
25986432131,4sleep02598761-21grep22:50:200
25930502135,4sleep12577604-21kworker/u16:1+i915-dp22:44:181
26889582121,2sleep10-21swapper/100:28:391
26862922121,6sleep02686302-21ssh00:24:400
26699022121,7sleep02669908-21ssh00:05:290
26409952121,5sleep00-21swapper/023:35:180
26409952121,5sleep00-21swapper/023:35:170
26406282120,7sleep10-21swapper/123:35:031
26406282120,7sleep10-21swapper/123:35:031
26312312121,7sleep12631280-21https23:25:001
26182212121,3sleep00-21swapper/023:11:000
25937872120,3sleep00-21swapper/022:45:020
25849592121,7sleep12585214-21/usr/sbin/munin22:35:251
25577592121,7sleep12557773-21ssh22:07:431
25306462120,2sleep00-21swapper/021:38:350
26935212111,6sleep12502500-21diskmemload00:33:591
26884792110,1sleep00-21swapper/000:27:500
26746312114,2sleep0241rcuc/000:10:410
26674712111,4sleep12502500-21diskmemload00:01:481
26515442113,4sleep02626147-21kworker/u16:1+xprtiod23:45:480
26487372110,2sleep00-21swapper/023:42:260
26482922111,5sleep12502500-21diskmemload23:41:481
26094612110,2sleep0191rcu_preempt23:00:580
26094202111,4sleep12502500-21diskmemload23:00:551
25965502111,4sleep12502500-21diskmemload22:49:041
25762372111,2sleep02576248-21ssh22:26:540
25636352110,5sleep12502500-21diskmemload22:13:481
25500402111,4sleep12502500-21diskmemload21:59:171
25442122110,4sleep02474656-21kworker/u16:1+i915-dp21:53:060
25397022114,4sleep12519963-21kworker/u16:4+i915-dp21:48:551
25296622111,4sleep12502500-21diskmemload21:37:121
25252732110,7sleep02525284-21ssh21:33:100
25006932111,5sleep02500712-21apt-get21:05:120
24971772111,5sleep12497263-21ntpq20:55:181
24675252113,4sleep02467651-21/usr/sbin/munin19:40:180
24541432111,4sleep02454217-21ntpq19:10:200
26850522100,4sleep12502500-21diskmemload00:22:361
26806692101,2sleep10-21swapper/100:18:081
26762662101,3sleep12502500-21diskmemload00:13:261
26701852100,2sleep10-21swapper/100:05:581
26639682100,2sleep02663977-21ssh23:58:590
26320432101,4sleep12502500-21diskmemload23:25:251
26235882101,3sleep12502500-21diskmemload23:16:221
26221022101,4sleep02622184-21ntpq23:15:170
26062582101,3sleep12502500-21diskmemload22:58:391
26013932101,2sleep10-21swapper/122:53:521
25808422101,2sleep00-21swapper/022:31:180
25509782101,3sleep12502500-21diskmemload22:00:131
25190562101,2sleep00-21swapper/021:26:290
25101282101,3sleep12502500-21diskmemload21:16:341
25045712101,3sleep12502500-21diskmemload21:10:461
24717042101,4sleep12471778-21ntpq19:50:201
24568482104,2sleep12456904-21ntp_states19:15:221
2698054291,4sleep02698060-21ssh00:39:240
2697931291,3sleep12502500-21diskmemload00:39:091
2657349291,3sleep12502500-21diskmemload23:51:571
2632268291,2sleep00-21swapper/023:25:430
2613725291,4sleep12613731-21ssh23:06:441
2587034291,2sleep00-21swapper/022:38:010
2577214290,2sleep10-21swapper/122:28:201
2574406290,2sleep10-21swapper/122:24:381
2562425290,2sleep02562439-21ssh22:12:060
2523933291,3sleep12502500-21diskmemload21:31:151
2518425291,3sleep12502500-21diskmemload21:25:321
2484582291,4sleep12484651-21irqstats20:25:171
2476999291,4sleep12477010-21sh20:05:001
2471083290,1sleep00-21swapper/019:50:160
2466590295,1sleep0102324-21kworker/0:1+i915-unordered19:39:090
2464417290,1sleep00-21swapper/019:32:570
2462281291,4sleep02462353-21/usr/sbin/munin19:29:590
2496563280,2sleep00-21swapper/020:55:130
2495199281,3sleep02495241-21ntpq20:50:210
2493864280,1sleep00-21swapper/020:46:070
2492451281,2sleep10-21swapper/120:45:121
2490565280,1sleep00-21swapper/020:40:150
2478010281,3sleep12478160-21df_inode20:10:101
2475819281,3sleep02475910-21ntpq20:00:180
2473299283,3sleep12473378-21fschecks_time19:55:141
2459108282,2sleep02459156-21ntpq19:20:220
2657673270,3sleep02657680-21ssh23:52:250
2500343270,1sleep00-21swapper/021:04:080
2499634270,1sleep10-21swapper/121:00:281
2488306270,1sleep00-21swapper/020:35:050
2485947270,1sleep00-21swapper/020:29:320
2480229271,3sleep12480049-21/usr/sbin/munin20:15:151
2478447271,3sleep00-21swapper/020:10:130
2477361270,1sleep00-21swapper/020:06:120
2474507270,1sleep00-21swapper/019:55:460
2466424270,1sleep10-21swapper/119:37:261
2464014271,3sleep12464173-21vmstat19:30:291
2458629272,2sleep1341rcuc/119:20:191
2455214270,1sleep10-21swapper/119:14:061
2494509260,1sleep10-21swapper/120:50:141
2492079260,3sleep10-21swapper/120:43:571
2488033260,4sleep00-21swapper/020:34:500
2483708260,3sleep00-21swapper/020:22:470
2469846261,2sleep1351ktimers/119:45:241
2469846261,2sleep1351ktimers/119:45:231
2468337260,4sleep10-21swapper/119:40:431
2501304250,1sleep10-21swapper/121:05:161
2486440250,1sleep10-21swapper/120:30:181
2460489250,1sleep10-21swapper/119:25:111
2456269251,2sleep00-21swapper/019:15:160
2488299240,1sleep10-21swapper/120:35:021
2479786240,1sleep00-21swapper/020:15:010
2477472240,1sleep10-21swapper/120:07:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional