You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-04 - 23:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Core i7-X990 @3470 MHz, Linux 5.10.47-rt46 (Profile)

Latency plot of shadow in rack #0, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q --smi
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Sun Dec 04, 2022 12:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
868729482,10sleep110-21swapper/1107:08:303
299891890,63ptp4l0-21swapper/107:06:461
81722730,1sleep90-21swapper/907:05:2611
870525941,10sleep100-21swapper/1007:08:412
551625847,8sleep70-21swapper/707:05:139
870025644,10sleep50-21swapper/507:08:377
857625443,9sleep60-21swapper/607:06:498
676825442,9sleep00-21swapper/007:05:180
113382540,0sleep100-21swapper/1008:35:142
853325342,9sleep40-21swapper/407:06:116
60272470,0sleep80-21swapper/810:35:0010
917199440,7cyclictest0-21swapper/1011:30:202
142142260,0sleep90-21swapper/911:25:2011
913799255,7cyclictest0-21swapper/309:20:015
914399221,16cyclictest0-21swapper/412:05:006
915699202,10cyclictest0-21swapper/707:40:009
916199194,9cyclictest0-21swapper/809:00:0010
913799183,9cyclictest0-21swapper/308:40:015
913299184,8cyclictest0-21swapper/211:20:004
913299183,9cyclictest0-21swapper/210:05:004
9117991813,4cyclictest15173-21turbostat10:09:560
224972180,1sleep6501rcuc/607:25:158
155792180,0sleep100-21swapper/1008:40:152
916199172,9cyclictest0-21swapper/811:30:0110
915699173,9cyclictest0-21swapper/712:30:009
913299174,9cyclictest0-21swapper/208:00:014
913299173,9cyclictest0-21swapper/209:05:004
911799174,8cyclictest0-21swapper/009:20:000
299891170,0ptp4l0-21swapper/112:30:001
299891170,0ptp4l0-21swapper/111:35:231
299891170,0ptp4l0-21swapper/108:25:231
299891170,0ptp4l0-21swapper/108:15:231
299891170,0ptp4l0-21swapper/108:00:271
146182170,0sleep100-21swapper/1012:10:012
9165991615,1cyclictest0-21swapper/912:34:5611
916599160,15cyclictest0-21swapper/908:40:1911
916199163,8cyclictest0-21swapper/812:20:0010
916199163,8cyclictest0-21swapper/810:40:0010
916199163,8cyclictest0-21swapper/810:15:0010
916199163,8cyclictest0-21swapper/810:15:0010
9161991614,1cyclictest0-21swapper/810:05:0110
915699163,8cyclictest0-21swapper/709:30:019
915699162,9cyclictest23194-21kworker/u24:107:50:009
915699162,8cyclictest0-21swapper/711:50:009
915699162,8cyclictest0-21swapper/711:40:009
9156991614,1cyclictest0-21swapper/712:10:009
914799163,8cyclictest0-21swapper/512:35:007
914799162,9cyclictest0-21swapper/512:40:017
914799161,11cyclictest0-21swapper/511:40:017
913799162,8cyclictest0-21swapper/311:20:015
911799162,9cyclictest0-21swapper/009:10:010
911799161,2cyclictest0-21swapper/012:05:180
299891160,1ptp4l21-21ksoftirqd/109:50:211
299891160,1ptp4l21-21ksoftirqd/108:59:361
299891160,0ptp4l0-21swapper/111:20:251
299891160,0ptp4l0-21swapper/110:30:231
916199153,8cyclictest0-21swapper/812:20:0010
916199151,8cyclictest0-21swapper/809:25:0010
916199151,5cyclictest0-21swapper/811:40:0010
915699153,7cyclictest0-21swapper/710:05:009
915299152,9cyclictest0-21swapper/610:45:008
914799153,7cyclictest0-21swapper/508:55:007
914799152,8cyclictest0-21swapper/509:45:007
914399152,8cyclictest0-21swapper/411:20:316
913799152,7cyclictest0-21swapper/307:50:285
913299153,8cyclictest0-21swapper/211:35:004
913299153,8cyclictest0-21swapper/208:20:294
913299153,7cyclictest0-21swapper/210:25:014
913299153,7cyclictest0-21swapper/210:25:004
913299152,9cyclictest0-21swapper/212:25:274
913299152,8cyclictest0-21swapper/211:15:004
911799157,3cyclictest13786-21load10:00:260
55032157,6sleep80-21swapper/807:05:0210
307112150,0sleep30-21swapper/311:45:275
299891150,2ptp4l21-21ksoftirqd/109:19:111
299891150,1ptp4l21-21ksoftirqd/111:50:251
299891150,1ptp4l21-21ksoftirqd/108:05:151
299891150,0ptp4l0-21swapper/112:15:271
299891150,0ptp4l0-21swapper/112:10:221
299891150,0ptp4l0-21swapper/109:40:251
917599142,8cyclictest0-21swapper/1112:25:003
916599140,1cyclictest0-21swapper/907:25:1811
916199142,7cyclictest0-21swapper/811:05:0010
916199141,8cyclictest0-21swapper/807:30:0010
915699143,7cyclictest0-21swapper/709:50:299
915699142,7cyclictest0-21swapper/711:05:009
915699142,7cyclictest0-21swapper/708:05:019
915299142,7cyclictest0-21swapper/610:35:018
914799142,8cyclictest0-21swapper/511:15:017
914799142,7cyclictest0-21swapper/509:50:007
9147991412,2cyclictest0-21swapper/511:25:187
913799143,7cyclictest0-21swapper/311:45:015
913799142,8cyclictest0-21swapper/307:35:245
913799142,7cyclictest0-21swapper/308:50:265
913299143,8cyclictest0-21swapper/207:30:304
912699142,7cyclictest0-21swapper/110:25:331
912699142,7cyclictest0-21swapper/110:25:331
911799148,2cyclictest0-21swapper/009:28:070
911799142,7cyclictest0-21swapper/010:35:250
911799142,7cyclictest0-21swapper/008:30:000
299891140,5ptp4l21-21ksoftirqd/112:05:161
299891140,1ptp4l21-21ksoftirqd/110:15:271
299891140,1ptp4l21-21ksoftirqd/110:15:271
299891140,1ptp4l21-21ksoftirqd/108:45:021
299891140,10ptp4l21-21ksoftirqd/107:23:431
299891140,0ptp4l0-21swapper/110:45:261
299891140,0ptp4l0-21swapper/109:45:251
917599133,7cyclictest0-21swapper/1108:15:003
917599132,6cyclictest0-21swapper/1107:25:003
917599130,8cyclictest0-21swapper/1109:00:013
917199132,7cyclictest0-21swapper/1009:25:322
916599132,8cyclictest0-21swapper/909:10:0111
916599132,7cyclictest0-21swapper/911:35:2711
916599132,6cyclictest0-21swapper/910:15:0011
916599132,6cyclictest0-21swapper/910:15:0011
916199133,6cyclictest0-21swapper/807:25:0110
916199132,7cyclictest0-21swapper/808:05:2610
915699132,6cyclictest0-21swapper/709:30:329
915299132,6cyclictest0-21swapper/609:15:018
914799132,7cyclictest0-21swapper/508:15:017
914399132,6cyclictest0-21swapper/411:15:346
913799132,7cyclictest0-21swapper/310:20:005
913799132,7cyclictest0-21swapper/310:20:005
913799131,7cyclictest0-21swapper/309:50:005
913299132,7cyclictest0-21swapper/209:05:234
913299131,8cyclictest0-21swapper/208:40:014
912699132,7cyclictest0-21swapper/107:55:001
299891130,5ptp4l21-21ksoftirqd/110:20:181
299891130,5ptp4l21-21ksoftirqd/110:20:181
299891130,5ptp4l21-21ksoftirqd/108:40:181
299891130,1ptp4l21-21ksoftirqd/111:30:291
299891130,1ptp4l21-21ksoftirqd/111:25:251
299891130,1ptp4l21-21ksoftirqd/111:10:181
299891130,1ptp4l21-21ksoftirqd/111:05:161
299891130,1ptp4l21-21ksoftirqd/109:56:511
299891130,1ptp4l21-21ksoftirqd/108:20:181
299891130,1ptp4l21-21ksoftirqd/107:25:221
299891130,1ptp4l21-21ksoftirqd/107:10:331
299891130,0ptp4l0-21swapper/112:35:011
299891130,0ptp4l0-21swapper/111:15:221
299891130,0ptp4l0-21swapper/110:10:231
299891130,0ptp4l0-21swapper/110:10:231
299891130,0ptp4l0-21swapper/108:30:271
299891130,0ptp4l0-21swapper/107:15:341
916199122,6cyclictest0-21swapper/812:10:0110
915699121,6cyclictest28173-21kworker/u24:009:50:019
9156991211,1cyclictest0-21swapper/711:14:429
915699120,11cyclictest21697-21hddtemp_smartct08:05:189
914399122,6cyclictest0-21swapper/411:34:566
914399121,6cyclictest0-21swapper/409:55:336
912699120,2cyclictest23878-21ps12:20:201
911799122,5cyclictest0-21swapper/009:40:000
45312120,1sleep40-21swapper/409:50:166
299891120,2ptp4l26818-21hddtemp_smartct11:00:231
299891120,1ptp4l21-21ksoftirqd/111:59:181
299891120,1ptp4l21-21ksoftirqd/109:22:481
299891120,1ptp4l21-21ksoftirqd/109:10:181
299891120,0ptp4l0-21swapper/108:50:131
299891120,0ptp4l0-21swapper/108:10:251
299891120,0ptp4l0-21swapper/107:55:221
299891120,0ptp4l0-21swapper/107:41:341
917599117,3cyclictest16956-21cat09:25:013
917599110,1cyclictest0-21swapper/1109:40:393
917199110,1cyclictest0-21swapper/1012:11:212
916599112,6cyclictest0-21swapper/911:00:3411
916599112,5cyclictest0-21swapper/911:10:0111
916599111,6cyclictest0-21swapper/910:30:0111
916599111,6cyclictest0-21swapper/910:30:0111
916599110,1cyclictest0-21swapper/907:33:1311
9156991110,1cyclictest0-21swapper/712:18:569
915699110,6cyclictest0-21swapper/710:35:019
915699110,11cyclictest0-21swapper/707:55:139
915299112,5cyclictest0-21swapper/609:25:008
915299112,1cyclictest0-21swapper/611:05:188
915299111,6cyclictest0-21swapper/610:55:018
915299111,6cyclictest0-21swapper/609:10:018
9152991110,1cyclictest0-21swapper/609:25:238
915299110,11cyclictest0-21swapper/612:32:228
915299110,11cyclictest0-21swapper/612:03:378
915299110,10cyclictest20773-21proc_pri12:15:298
915299110,10cyclictest11025-21taskset11:20:278
914799113,7cyclictest18494-21cut09:25:197
914799113,5cyclictest0-21swapper/507:45:007
914799112,6cyclictest0-21swapper/510:20:317
914799112,6cyclictest0-21swapper/510:20:317
914799111,6cyclictest0-21swapper/510:05:337
913799112,5cyclictest0-21swapper/309:25:005
913799110,10cyclictest0-21swapper/311:15:265
913299118,2cyclictest25817-21latency11:40:204
913299112,8cyclictest5237-21grep09:50:184
913299110,1cyclictest0-21swapper/211:57:164
913299110,11cyclictest0-21swapper/210:59:504
912699118,2cyclictest24236-21cpuspeed_turbos11:40:131
911799113,4cyclictest0-21swapper/011:55:000
9117991110,1cyclictest0-21swapper/011:20:020
911799110,4cyclictest1497-21/usr/sbin/munin12:35:280
299891110,7ptp4l21-21ksoftirqd/110:44:461
299891110,5ptp4l21-21ksoftirqd/110:52:331
299891110,4ptp4l21-21ksoftirqd/107:30:241
299891110,1ptp4l2297-21df_inode12:35:161
299891110,1ptp4l21-21ksoftirqd/110:00:021
299891110,1ptp4l21-21ksoftirqd/109:03:591
299891110,1ptp4l21-21ksoftirqd/107:47:371
299891110,0ptp4l0-21swapper/109:34:271
277091110,0getstats0-21swapper/209:35:144
917599107,2cyclictest24425-21grep10:15:183
917599107,2cyclictest24425-21grep10:15:173
917599103,7cyclictest0-21swapper/1112:35:413
917599102,5cyclictest0-21swapper/1110:00:013
917599101,5cyclictest0-21swapper/1108:55:003
917599100,9cyclictest20933-21tr08:05:133
917599100,9cyclictest15999-21gltestperf12:10:163
917599100,9cyclictest0-21swapper/1108:15:143
917199109,1cyclictest0-21swapper/1011:45:542
917199107,2cyclictest1026-21runrttasks08:10:442
917199103,7cyclictest0-21swapper/1011:25:042
917199102,6cyclictest0-21swapper/1011:20:002
917199100,9cyclictest9111-21cpuspeed_turbos11:20:162
917199100,7cyclictest22789-21fschecks_time10:55:232
917199100,4cyclictest0-21swapper/1007:24:362
917199100,1cyclictest0-21swapper/1007:56:242
917199100,10cyclictest0-21swapper/1012:20:222
917199100,10cyclictest0-21swapper/1010:31:012
916599109,1cyclictest0-21swapper/911:23:2911
916599109,1cyclictest0-21swapper/907:47:0111
916599104,2cyclictest29485-21sensors_fan10:20:2711
916599104,2cyclictest29485-21sensors_fan10:20:2711
916599103,7cyclictest0-21swapper/910:47:4911
916599101,6cyclictest0-21swapper/908:25:2611
916599101,6cyclictest0-21swapper/908:10:0011
916599100,10cyclictest0-21swapper/910:19:1811
916599100,10cyclictest0-21swapper/910:19:1811
916199109,1cyclictest0-21swapper/811:55:2310
916199109,1cyclictest0-21swapper/809:40:2910
916199109,1cyclictest0-21swapper/808:33:1010
916199107,2cyclictest21675-21latency_hist10:55:0010
916199100,9cyclictest27376-21switchtime09:35:2810
916199100,9cyclictest1993-21fw_forwarded_lo11:10:1810
916199100,1cyclictest0-21swapper/811:05:2310
916199100,10cyclictest0-21swapper/808:49:0310
916199100,10cyclictest0-21swapper/808:22:2910
915699108,1cyclictest0-21swapper/711:55:369
915699107,2cyclictest0-21swapper/711:25:139
915699106,3cyclictest15697-21/usr/sbin/munin10:05:279
915699106,3cyclictest11088-21df_inode10:40:159
915699106,3cyclictest10436-21taskset08:31:249
915699103,1cyclictest0-21swapper/709:44:059
915699100,9cyclictest6609-21runrttasks08:26:259
915699100,9cyclictest31791-21kernelversion12:30:279
915699100,9cyclictest29925-21grep08:15:199
915699100,9cyclictest29544-21sed07:35:009
915699100,9cyclictest27547-21fschecks_time08:55:159
915699100,1cyclictest0-21swapper/709:15:159
915699100,10cyclictest0-21swapper/709:10:249
915299108,1cyclictest21828-21if_err_enp1s008:05:198
915299107,2cyclictest0-21swapper/609:47:348
915299106,3cyclictest3119-21irqrtprio12:35:228
915299105,2cyclictest32043-21idleruntime-cro09:45:008
915299104,6cyclictest0-21swapper/610:49:558
915299103,7cyclictest0-21swapper/610:57:068
915299102,7cyclictest19380-21cstates08:45:148
915299101,9cyclictest113650irq/18-p34p112:24:228
915299101,8cyclictest51-21ksoftirqd/611:45:138
915299100,9cyclictest24612-21grep08:10:148
915299100,9cyclictest2120-21sed11:50:348
915299100,10cyclictest0-21swapper/610:35:208
914799109,1cyclictest0-21swapper/511:22:217
914799103,1cyclictest0-21swapper/511:59:427
914799102,5cyclictest0-21swapper/509:00:007
914799101,5cyclictest0-21swapper/510:45:007
914799100,1cyclictest0-21swapper/510:02:287
914799100,1cyclictest0-21swapper/508:48:077
914399109,1cyclictest0-21swapper/407:55:496
914399100,9cyclictest381rcuc/407:50:166
914399100,9cyclictest0-21swapper/407:35:016
914399100,10cyclictest381rcuc/408:50:206
914399100,10cyclictest0-21swapper/411:58:506
913799109,1cyclictest0-21swapper/310:31:145
913799109,1cyclictest0-21swapper/307:57:105
913799105,5cyclictest0-21swapper/308:17:155
913799105,1cyclictest0-21swapper/311:33:585
913799104,2cyclictest8448-21latency_hist07:50:005
913799103,2cyclictest25744-21irqrtprio08:10:205
913799102,6cyclictest0-21swapper/311:50:335
913799102,3cyclictest28595-21timerwakeupswit07:30:305
913799101,5cyclictest0-21swapper/310:10:005
913799100,9cyclictest0-21swapper/307:20:175
913799100,5cyclictest0-21swapper/307:20:005
913799100,10cyclictest0-21swapper/312:00:295
913799100,10cyclictest0-21swapper/310:30:005
913799100,10cyclictest0-21swapper/310:30:005
913299109,1cyclictest0-21swapper/208:53:484
913299109,1cyclictest0-21swapper/208:41:384
913299106,3cyclictest20872-21cpuspeed_turbos08:05:134
913299106,3cyclictest14260-21/usr/sbin/munin09:20:154
913299106,3cyclictest1335-21fw_conntrack09:45:174
913299101,2cyclictest0-21swapper/208:25:234
913299100,9cyclictest7634-21awk08:30:154
913299100,9cyclictest21058-21proc_pri08:45:224
913299100,9cyclictest19186-21timerandwakeup08:00:264
913299100,9cyclictest15437-21sed11:25:264
913299100,9cyclictest12936-21munin-run09:20:014
913299100,10cyclictest0-21swapper/209:43:354
913299100,10cyclictest0-21swapper/209:33:524
911799106,3cyclictest1026-21runrttasks08:18:350
911799102,2cyclictest20466-21cpuspeed_turbos11:35:160
911799100,9cyclictest32706-21irqrtprio10:25:250
911799100,9cyclictest32706-21irqrtprio10:25:240
911799100,9cyclictest24900-21expr08:10:140
911799100,9cyclictest15172-21cpuspeed_turbos12:10:140
911799100,9cyclictest1026-21runrttasks12:24:210
911799100,4cyclictest31107-21munin-run11:50:010
911799100,3cyclictest8459-21latency_hist11:20:000
911799100,2cyclictest0-21swapper/011:35:010
911799100,10cyclictest0-21swapper/007:22:550
299891100,1ptp4l3271-21irqrtprio09:05:181
299891100,0ptp4l0-21swapper/112:00:141
299891100,0ptp4l0-21swapper/111:45:161
299891100,0ptp4l0-21swapper/110:55:291
299891100,0ptp4l0-21swapper/110:35:131
299891100,0ptp4l0-21swapper/110:10:011
299891100,0ptp4l0-21swapper/109:35:131
299891100,0ptp4l0-21swapper/109:25:241
299891100,0ptp4l0-21swapper/108:35:141
299891100,0ptp4l0-21swapper/107:35:141
91759998,1cyclictest0-21swapper/1107:40:163
91759995,3cyclictest29928-21cpuspeed_turbos07:35:133
91759995,3cyclictest17536-21cpuspeed_turbos09:25:153
91759992,6cyclictest6385-21cut09:50:243
91759992,5cyclictest0-21swapper/1108:30:003
91759992,1cyclictest26137-21phc2sys11:40:223
91759990,9cyclictest0-21swapper/1112:05:153
91759990,9cyclictest0-21swapper/1108:00:153
91759990,8cyclictest7611-21fw_forwarded_lo10:35:153
91759990,8cyclictest596-21cat09:00:293
91759990,8cyclictest5693-21cstates07:45:143
91759990,8cyclictest28642-21switchtime12:25:273
91759990,8cyclictest23734-21missed_timers10:55:283
91759990,8cyclictest23668-21sort09:30:313
91759990,8cyclictest21363-21sed10:55:003
91759990,8cyclictest1683-21cut09:45:203
91759990,8cyclictest16183-21grep10:05:153
91759990,8cyclictest12727-21irqrtprio08:35:203
91759990,8cyclictest12356-21grep10:00:203
91759990,7cyclictest1081-21cpuspeed_turbos11:10:143
91759990,3cyclictest5288-21diskmemload11:15:553
91759990,3cyclictest32414-21expr11:50:203
91759990,3cyclictest22033-21taskset07:25:093
91759990,3cyclictest19579-21grep12:15:213
91759990,3cyclictest14221-21uname07:15:023
91759990,3cyclictest12489-21ps10:40:243
91759990,3cyclictest1013-21snmpd07:58:533
91759990,1cyclictest0-21swapper/1110:45:133
91719998,1cyclictest0-21swapper/1009:20:002
91719998,1cyclictest0-21swapper/1008:19:122
91719997,1cyclictest23674-21cpuspeed_turbos10:15:142
91719997,1cyclictest23674-21cpuspeed_turbos10:15:132
91719994,1cyclictest0-21swapper/1008:20:292
91719993,5cyclictest22281-21latency_hist12:20:002
91719990,8cyclictest30051-21/usr/sbin/munin12:30:262
91719990,8cyclictest14768-21cpuspeed_turbos07:15:142
91719990,6cyclictest4011-21grep09:05:202
91719990,3cyclictest8838-21/usr/sbin/munin09:55:182
91659997,1cyclictest21992-21fschecks_count09:30:1911
91659996,2cyclictest30478-21df_inode07:35:1411
91659996,2cyclictest25007-21/usr/sbin/munin08:50:2311
91659995,3cyclictest20324-21irqstats12:15:2511
91659992,5cyclictest1026-21runrttasks09:36:5811
91659992,4cyclictest0-21swapper/911:00:0011
91659991,5cyclictest0-21swapper/912:00:0011
91659990,9cyclictest0-21swapper/910:40:1711
91659990,9cyclictest0-21swapper/909:45:1611
91659990,8cyclictest30451-21irqrtprio09:40:2211
91659990,3cyclictest7076-21unixbench-2d09:50:3011
91659990,3cyclictest31137-21/usr/sbin/munin09:00:1911
91659990,3cyclictest21997-21latency_hist07:25:0011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional