You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 08:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot4s.osadl.org (updated Wed Nov 26, 2025 00:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
781983996559,4cyclictest808370-21kworker/u8:2+events_unbound21:21:281
781983996460,3cyclictest1072196-21kworker/u8:4+flush-8:000:31:271
781983996256,5cyclictest4150989-21kworker/u8:4+events_unbound20:26:121
781980996150,8cyclictest1036293-21ntpq23:21:260
781983995953,5cyclictest808370-21kworker/u8:2+flush-8:000:01:251
781983995948,8cyclictest877057-21ntpq20:41:211
781983995853,4cyclictest781854-21kworker/u8:1+events_unbound20:11:321
781983995853,3cyclictest1072196-21kworker/u8:4+events_unbound00:06:271
781983995847,8cyclictest857272-21ntpq20:21:211
781983995752,4cyclictest1041738-21kworker/u8:1+events_unbound23:41:121
781983995745,9cyclictest1026321-21ntpq23:11:211
781983995744,10cyclictest882814-21apt-get20:50:001
781980995749,6cyclictest4150989-21kworker/u8:4+flush-8:019:26:240
781983995650,4cyclictest808370-21kworker/u8:2+events_unbound21:36:291
781983995649,5cyclictest913414-21kworker/u8:3+flush-8:021:26:221
781983995551,3cyclictest808370-21kworker/u8:2+events_unbound23:31:231
781983995550,4cyclictest903505-21kworker/u8:0+events_unbound22:46:111
781983995550,4cyclictest1081632-21kworker/u8:0+flush-8:000:21:221
781983995550,3cyclictest808370-21kworker/u8:2+events_unbound23:56:101
781983995550,3cyclictest781854-21kworker/u8:1+flush-8:019:56:231
781983995545,8cyclictest282-21systemd-journal20:10:001
781983995449,4cyclictest992548-21kworker/u8:3+events_unbound23:16:261
781983995448,5cyclictest992548-21kworker/u8:3+events_unbound23:21:141
8698182530,6sleep3451irq_work/320:36:193
781983995349,3cyclictest781854-21kworker/u8:1+flush-8:020:51:291
781983995348,4cyclictest1081632-21kworker/u8:0+events_unbound00:11:271
781983995347,4cyclictest781854-21kworker/u8:1+flush-8:019:41:231
781983995347,4cyclictest1072196-21kworker/u8:4+flush-8:000:16:241
781983995342,8cyclictest1006594-21ntpq22:51:211
781983995248,3cyclictest4150989-21kworker/u8:4+events_unbound20:56:111
781983995247,4cyclictest781854-21kworker/u8:1+events_unbound20:31:131
781983995246,4cyclictest903505-21kworker/u8:0+flush-8:022:26:131
781983995246,4cyclictest432086-21kworker/u8:0+flush-8:019:31:251
781983995245,5cyclictest808370-21kworker/u8:2+events_unbound21:11:231
781983995241,8cyclictest897196-21ntpq21:01:221
781983995145,4cyclictest903505-21kworker/u8:0+flush-8:022:36:241
781983995140,8cyclictest1042009-21latency_hist23:30:011
781983995046,3cyclictest808370-21kworker/u8:2+events_unbound22:56:251
781983995045,4cyclictest903505-21kworker/u8:0+events_unbound22:21:281
781983995044,4cyclictest808370-21kworker/u8:2+events_unbound23:51:211
781983995044,4cyclictest757585-21kworker/u8:2+events_unbound19:11:131
781983995043,6cyclictest808370-21kworker/u8:2+events_unbound22:31:241
781983995043,5cyclictest1081632-21kworker/u8:0+flush-8:000:26:241
781983994944,4cyclictest808370-21kworker/u8:2+events_unbound19:36:131
781983994944,3cyclictest781854-21kworker/u8:1+events_unbound20:05:001
781983994942,5cyclictest903505-21kworker/u8:0+flush-8:022:41:251
781983994942,5cyclictest808370-21kworker/u8:2+flush-8:023:46:231
781983994942,5cyclictest808370-21kworker/u8:2+flush-8:023:46:231
781983994942,5cyclictest4150989-21kworker/u8:4+flush-8:019:51:211
781983994942,4cyclictest903270-21latency_hist21:10:011
781983994842,5cyclictest4150989-21kworker/u8:4+events_unbound19:26:141
781983994740,5cyclictest933278-21kworker/u8:1+flush-8:021:51:251
781983994740,5cyclictest933278-21kworker/u8:1+flush-8:021:51:251
781983994740,5cyclictest781854-21kworker/u8:1+flush-8:019:46:291
781983994738,7cyclictest4150989-21kworker/u8:4+flush-8:020:36:231
781983994736,8cyclictest282-21systemd-journal21:17:381
781980994736,8cyclictest1062625-21diskstats23:51:120
781983994641,4cyclictest757585-21kworker/u8:2+events_unbound19:15:041
781983994640,4cyclictest808370-21kworker/u8:2+events_unbound23:06:151
781983994640,4cyclictest432086-21kworker/u8:0+events_unbound19:21:231
781980994635,8cyclictest789356-21diskstats19:16:100
781983994539,4cyclictest808370-21kworker/u8:2+events_unbound22:11:231
781983994538,5cyclictest781854-21kworker/u8:1+flush-8:020:16:231
781980994438,4cyclictest992548-21kworker/u8:3+flush-8:023:11:240
781980994437,5cyclictest937362-21ntpq21:41:220
781980994435,6cyclictest813471-21latency_hist19:40:010
781988994339,3cyclictest1066700-21kworker/u8:0+events_unbound00:00:012
781983994333,7cyclictest1016431-21ntpq23:01:221
781983994333,7cyclictest1016431-21ntpq23:01:221
781980994335,6cyclictest903505-21kworker/u8:0+flush-8:021:46:250
781980994333,8cyclictest282-21systemd-journal00:40:010
781980994234,6cyclictest898080-21latency_hist21:05:000
781983994134,5cyclictest808370-21kworker/u8:2+events_unbound22:01:221
781980994134,5cyclictest781854-21kworker/u8:1+flush-8:020:31:140
781983994035,3cyclictest903505-21kworker/u8:0+events_unbound21:46:251
781983994032,6cyclictest903505-21kworker/u8:0+flush-8:021:41:241
781980994034,5cyclictest781854-21kworker/u8:1+events_unbound19:41:250
781980994033,5cyclictest903505-21kworker/u8:0+flush-8:022:46:100
781980994031,7cyclictest1092129-21/usr/sbin/munin00:21:210
781980993933,4cyclictest808370-21kworker/u8:2+flush-8:023:06:310
781980993932,5cyclictest903505-21kworker/u8:0+events_unbound22:06:270
781980993930,7cyclictest968195-21apt-get22:16:080
781980993928,9cyclictest822734-21ntpq19:46:210
781983993833,3cyclictest808370-21kworker/u8:2+events_unbound22:16:131
781983993733,3cyclictest933278-21kworker/u8:1+flush-8:021:56:241
781983993733,3cyclictest933278-21kworker/u8:1+flush-8:021:56:241
781980993730,5cyclictest917349-21ntpq21:21:250
781980993730,5cyclictest4150989-21kworker/u8:4+events_unbound19:51:250
781980993727,7cyclictest808515-21latency_hist19:35:010
781980993631,3cyclictest992548-21kworker/u8:3+flush-8:022:51:100
781980993629,5cyclictest4150989-21kworker/u8:4+flush-8:020:26:140
781983993530,3cyclictest1041738-21kworker/u8:1+flush-8:023:36:171
7816752354,9sleep10-21swapper/119:08:251
781980993429,4cyclictest808370-21kworker/u8:2+flush-8:023:41:290
781980993427,5cyclictest808370-21kworker/u8:2+flush-8:021:31:310
781988993329,3cyclictest757585-21kworker/u8:2+flush-8:019:15:062
781983993327,4cyclictest903505-21kworker/u8:0+events_unbound22:06:191
781980993328,4cyclictest1081632-21kworker/u8:0+flush-8:000:11:390
781980993327,4cyclictest808370-21kworker/u8:2+flush-8:022:31:210
781980993326,5cyclictest4150989-21kworker/u8:4+flush-8:020:41:220
781980993326,5cyclictest4150989-21kworker/u8:4+events_unbound20:51:130
781980993227,3cyclictest982646-21idleruntime-cro22:30:000
781980993227,3cyclictest903275-21munin-plugin-st21:10:000
781980993227,3cyclictest1081632-21kworker/u8:0+events_unbound00:26:140
781980993225,5cyclictest1042012-21apt-get23:30:010
781988993124,5cyclictest1076647-21grep00:01:332
781983993127,3cyclictest808370-21kworker/u8:2+events_unbound21:31:211
781983993127,3cyclictest1081632-21kworker/u8:0+events_unbound00:36:181
781980993126,3cyclictest948096-21cron21:55:000
781980993126,3cyclictest948096-21cron21:55:000
781980993126,3cyclictest1087305-21aten2_r5power_v00:16:110
781980993126,3cyclictest1057662-21cpuspeed_turbos23:46:110
781980993126,3cyclictest1057662-21cpuspeed_turbos23:46:110
781980993125,5cyclictest808370-21kworker/u8:2+events_unbound21:20:000
781980993022,6cyclictest4150989-21kworker/u8:4+flush-8:020:46:230
781980993021,6cyclictest903505-21kworker/u8:0+flush-8:022:21:090
781988992925,2cyclictest1086536-21unixbench_singl00:11:302
781980992823,3cyclictest808370-21kworker/u8:2+events_unbound22:11:140
781980992823,3cyclictest1047244-21systemd-run23:35:000
781980992822,4cyclictest808370-21kworker/u8:2+flush-8:022:56:390
781980992822,4cyclictest808370-21kworker/u8:2+events_unbound23:56:220
781980992819,7cyclictest929489-21apt-get21:36:210
781988992723,3cyclictest992515-21sendmail22:40:002
781988992723,3cyclictest938661-21aten2_r5power_p21:46:122
781988992722,3cyclictest897738-21unixbench_singl21:01:302
781980992721,4cyclictest1105786-21ntp_kernel_pll_00:31:210
781980992720,5cyclictest432086-21kworker/u8:0+events_unbound19:21:250
781980992718,2cyclictest421rcuog/219:15:010
781988992623,2cyclictest918846-21aten2_r5power_v21:26:102
781988992623,2cyclictest782326-21aten2_r5power_v19:11:122
781988992622,2cyclictest1092420-21fschecks_count00:21:132
781988992619,5cyclictest948096-21cron21:55:002
781988992619,5cyclictest948096-21cron21:54:592
781988992616,8cyclictest928254-21cat21:35:002
78198099264,19cyclictest3619722-21kworker/0:0+events20:05:050
781980992621,4cyclictest781854-21kworker/u8:1+flush-8:020:21:120
781980992621,4cyclictest781854-21kworker/u8:1+flush-8:020:01:270
781980992621,3cyclictest781854-21kworker/u8:1+flush-8:019:56:250
781980992621,3cyclictest1072196-21kworker/u8:4+events_unbound00:06:130
781980992620,4cyclictest4150989-21kworker/u8:4+flush-8:020:56:240
781988992522,2cyclictest1097376-21fschecks_count00:26:142
781988992520,4cyclictest867995-21turbostat.cron20:35:002
781988992520,3cyclictest933682-21aten2_r5power_p21:41:122
781988992515,8cyclictest282-21systemd-journal20:40:002
78198099252,22cyclictest3619722-21kworker/0:0+events23:36:220
78198099252,18cyclictest3619722-21kworker/0:0+events22:40:530
781980992520,4cyclictest4150989-21kworker/u8:4+events_unbound20:36:330
781980992516,6cyclictest1017310-21latency_hist23:05:010
781980992516,6cyclictest1017310-21latency_hist23:05:010
781980992515,8cyclictest953348-21/usr/sbin/munin22:01:210
781988992421,2cyclictest948553-21aten2_r5power_v21:56:112
781988992421,2cyclictest948553-21aten2_r5power_v21:56:102
781988992421,2cyclictest1051806-21unixbench_singl23:36:292
78198099242,21cyclictest0-21swapper/023:16:280
781980992418,5cyclictest992150-21unixbench_multi22:36:290
78198099241,22cyclictest3619722-21kworker/0:0+events21:11:090
781988992320,2cyclictest977396-21unixbench_singl22:21:342
781988992320,2cyclictest819050-21aten2_r5power_e19:46:112
781988992319,3cyclictest902934-21unixbench_singl21:06:282
781988992319,3cyclictest0-21swapper/219:45:012
781988992317,5cyclictest1027201-21cron23:15:012
781980992319,3cyclictest847925-21unixbench_multi20:11:330
78198099230,21cyclictest953061-21munin-run22:00:000
78198099230,21cyclictest953061-21munin-run21:59:590
781988992218,3cyclictest808370-21kworker/u8:2+flush-8:021:16:222
781988992218,2cyclictest0-21swapper/219:24:592
78198099224,17cyclictest3619722-21kworker/0:0+events00:01:220
781980992217,4cyclictest781854-21kworker/u8:1+flush-8:020:16:280
78198899212,13cyclictest978061-21apt-get22:26:102
781988992118,2cyclictest1052600-21aten2_r5power_v23:41:112
781988992117,2cyclictest0-21swapper/221:25:002
781988992117,2cyclictest0-21swapper/220:50:002
781988992116,4cyclictest828184-21unixbench_singl19:51:292
781988992115,4cyclictest996820-21sendmail_mailqu22:41:292
781988992114,5cyclictest843039-21head20:06:282
781988992114,5cyclictest282-21systemd-journal22:48:292
781988992112,2cyclictest0-21swapper/221:11:182
781988992110,8cyclictest883381-21apt-get20:51:102
78198899209,9cyclictest928937-21fschecks_time21:36:172
781988992017,2cyclictest1102156-21aten2_r5power_v00:31:102
781988992017,2cyclictest0-21swapper/223:20:012
781988992016,2cyclictest0-21swapper/220:05:002
781988992015,3cyclictest0-21swapper/223:00:002
781988992014,4cyclictest853205-21idleruntime-cro20:20:002
781988992012,6cyclictest813267-21sshd19:37:352
781988992010,8cyclictest858745-21fschecks_count20:26:132
78198899200,9cyclictest10174112latency_hist23:05:002
78198899200,9cyclictest10174112latency_hist23:05:002
78198099201,14cyclictest587-21runrttasks21:26:080
7815692202,12sleep00-21swapper/019:07:040
7774942204,9sleep20-21swapper/219:06:132
78198899192,12cyclictest957207-21ntpq22:01:242
781988991916,2cyclictest1017762-21aten2_r5power_v23:06:142
781988991910,7cyclictest1007020-21unixbench_multi22:51:272
9243642187,7sleep30-21swapper/321:31:223
78198899182,12cyclictest1077283-21apt-get00:06:092
78198899182,11cyclictest968195-21apt-get22:16:102
781988991815,1cyclictest0-21swapper/200:40:002
781988991814,2cyclictest962588-21unixbench_singl22:06:312
7815472183,8sleep30-21swapper/319:06:473
781988991714,2cyclictest0-21swapper/223:21:202
781988991713,2cyclictest0-21swapper/223:46:102
781988991713,2cyclictest0-21swapper/223:46:102
78198899167,8cyclictest799291-21fschecks_count19:26:152
78198899167,7cyclictest984676-21cut22:31:172
78198899167,7cyclictest829147-21fschecks_count19:56:132
78198899166,2cyclictest0-21swapper/220:55:012
78198899162,13cyclictest0-21swapper/222:10:272
781988991613,2cyclictest0-21swapper/223:26:292
781988991613,2cyclictest0-21swapper/200:17:202
781988991612,2cyclictest0-21swapper/223:51:132
781988991612,2cyclictest0-21swapper/223:30:052
781988991612,2cyclictest0-21swapper/220:21:212
781988991612,2cyclictest0-21swapper/219:31:212
781988991611,3cyclictest0-21swapper/220:11:212
781988991610,4cyclictest0-21swapper/220:41:192
9081912152,8sleep30-21swapper/321:15:013
9171232142,8sleep30-21swapper/321:21:233
8827992142,7sleep30-21swapper/320:50:003
8408122143,6sleep30-21swapper/320:06:183
8049092144,6sleep30-21swapper/319:31:193
9826412132,6sleep30-21swapper/322:30:013
8931312132,7sleep30-21swapper/321:00:003
8838802133,7sleep30-21swapper/320:51:173
8729932132,6sleep30-21swapper/320:40:013
8227872132,7sleep30-21swapper/319:46:243
11066592132,6sleep30-21swapper/300:35:023
10628592133,6sleep30-21swapper/323:51:153
10065182132,7sleep30-21swapper/322:51:203
9879372122,6sleep30-21swapper/322:36:113
9785942122,6sleep30-21swapper/322:26:163
9717652122,6sleep30-21swapper/322:16:203
9670632122,6sleep30-21swapper/322:11:273
9580172122,6sleep30-21swapper/322:05:013
9571682122,7sleep30-21swapper/322:01:243
9522582122,6sleep30-21swapper/321:56:263
9522582122,6sleep30-21swapper/321:56:263
9455362122,6sleep30-21swapper/321:51:203
9455362122,6sleep30-21swapper/321:51:203
9290272125,4sleep30-21swapper/321:36:203
9121572122,6sleep30-21swapper/321:16:213
8584532122,6sleep30-21swapper/320:26:093
8561702122,6sleep30-21swapper/320:21:193
8292972122,6sleep30-21swapper/319:56:153
8141972122,6sleep30-21swapper/319:41:133
8092312122,6sleep30-21swapper/319:36:173
7994542122,6sleep30-21swapper/319:26:173
7843852122,6sleep30-21swapper/319:11:213
11026842122,6sleep30-21swapper/300:31:163
10973972122,6sleep30-21swapper/300:26:153
10957692122,6sleep30-21swapper/300:21:213
10872012122,6sleep30-21swapper/300:16:113
10766582122,6sleep30-21swapper/300:01:343
10571542122,6sleep30-21swapper/323:45:013
10571542122,6sleep30-21swapper/323:45:013
10525852122,6sleep30-21swapper/323:41:123
10374312122,6sleep30-21swapper/323:26:113
10321022122,6sleep30-21swapper/323:20:003
10225832122,6sleep30-21swapper/323:11:103
10187742122,6sleep30-21swapper/323:06:213
10164532122,6sleep30-21swapper/323:01:233
10164532122,6sleep30-21swapper/323:01:233
10023132122,6sleep30-21swapper/322:48:313
9928232112,6sleep30-21swapper/322:40:573
9773072112,5sleep30-21swapper/322:21:343
9384942113,5sleep30-21swapper/321:45:273
9332262112,5sleep30-21swapper/321:40:013
9189822112,5sleep30-21swapper/321:26:123
8982832111,5sleep30-21swapper/321:05:013
8935052112,5sleep30-21swapper/321:01:123
8643832112,5sleep30-21swapper/320:31:183
8482962112,5sleep30-21swapper/320:15:013
8436562112,5sleep30-21swapper/320:11:123
8337752112,6sleep30-21swapper/320:01:013
8241952112,5sleep30-21swapper/319:51:133
7939662112,5sleep30-21swapper/319:20:113
7892202112,5sleep30-21swapper/319:16:103
10824812112,5sleep30-21swapper/300:11:143
10777822112,5sleep30-21swapper/300:06:163
10675552112,5sleep30-21swapper/323:56:133
10518672112,6sleep30-21swapper/323:36:473
10425492111,6sleep30-21swapper/323:31:093
10370442113,5sleep30-21swapper/323:24:353
10074252112,5sleep30-21swapper/322:55:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional