You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-10-08 - 07:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Tue Oct 08, 2024 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
275991830,62ptp4l0-21swapper/119:07:391
676726655,9sleep110-21swapper/1119:07:103
694926557,6sleep50-21swapper/519:09:537
674826452,9sleep00-21swapper/019:06:590
675526351,10sleep70-21swapper/719:07:069
680726251,9sleep100-21swapper/1019:07:442
666126250,10sleep60-21swapper/619:05:418
678626052,6sleep40-21swapper/419:07:276
175542520,0sleep70-21swapper/722:10:009
236462500,0sleep00-21swapper/021:35:140
236462500,0sleep00-21swapper/021:35:140
740099320,1cyclictest0-21swapper/1000:10:232
736899243,17cyclictest0-21swapper/422:05:016
7400992012,8cyclictest0-21swapper/1021:35:242
7400992012,8cyclictest0-21swapper/1021:35:242
735899203,11cyclictest0-21swapper/219:30:004
735899203,11cyclictest0-21swapper/200:20:014
735899203,10cyclictest0-21swapper/223:10:244
735899193,10cyclictest0-21swapper/223:40:014
735899193,10cyclictest0-21swapper/222:10:244
275991190,1ptp4l21-21ksoftirqd/120:47:191
735899183,9cyclictest0-21swapper/220:50:004
735899182,10cyclictest0-21swapper/200:15:014
735899180,12cyclictest0-21swapper/219:15:014
735899172,9cyclictest0-21swapper/220:55:244
7358991714,2cyclictest14256-21turbostat20:00:004
734799172,9cyclictest0-21swapper/020:04:590
275991170,1ptp4l21-21ksoftirqd/123:45:121
275991170,0ptp4l0-21swapper/122:20:211
740499162,8cyclictest0-21swapper/1122:10:013
739499163,7cyclictest0-21swapper/921:30:0111
736899160,2cyclictest0-21swapper/421:05:326
736399164,7cyclictest0-21swapper/321:00:005
736399163,8cyclictest0-21swapper/320:10:005
736399163,7cyclictest0-21swapper/319:10:005
734799163,8cyclictest0-21swapper/021:30:000
275991160,0ptp4l0-21swapper/123:55:141
275991160,0ptp4l0-21swapper/123:35:181
275991160,0ptp4l0-21swapper/123:05:151
275991160,0ptp4l0-21swapper/120:00:231
275991160,0ptp4l0-21swapper/100:10:251
17192160,0sleep90-21swapper/923:55:1411
740499151,10cyclictest6650-21kworker/u24:221:40:013
740499151,10cyclictest6650-21kworker/u24:221:40:013
739499153,7cyclictest0-21swapper/922:40:0111
739499153,7cyclictest0-21swapper/922:05:0011
739499152,8cyclictest0-21swapper/920:50:0011
737799152,6cyclictest0-21swapper/600:00:018
737299153,8cyclictest0-21swapper/519:15:007
735899152,9cyclictest0-21swapper/220:40:004
735899151,2cyclictest0-21swapper/219:30:174
734799153,7cyclictest0-21swapper/021:25:000
734799152,8cyclictest0-21swapper/023:35:240
734799152,8cyclictest0-21swapper/022:20:010
734799152,8cyclictest0-21swapper/022:15:010
734799152,8cyclictest0-21swapper/000:00:010
276402150,0sleep111020-21runrttasks21:40:143
276402150,0sleep111020-21runrttasks21:40:143
275991150,1ptp4l21-21ksoftirqd/121:35:161
275991150,1ptp4l21-21ksoftirqd/121:35:151
275991150,1ptp4l21-21ksoftirqd/100:35:191
275991150,0ptp4l0-21swapper/123:10:121
275991150,0ptp4l0-21swapper/120:27:241
275991150,0ptp4l0-21swapper/119:55:211
275991150,0ptp4l0-21swapper/119:50:291
275991150,0ptp4l0-21swapper/119:30:241
740499142,8cyclictest0-21swapper/1123:25:013
740499142,7cyclictest0-21swapper/1119:40:013
739499142,7cyclictest0-21swapper/922:55:2311
739499142,7cyclictest0-21swapper/922:55:0111
739499142,7cyclictest0-21swapper/921:50:0011
739499142,7cyclictest0-21swapper/921:49:5911
739499142,7cyclictest0-21swapper/921:15:0011
739499142,7cyclictest0-21swapper/920:40:0111
739499142,7cyclictest0-21swapper/920:15:0011
737799142,7cyclictest0-21swapper/621:00:018
736899142,7cyclictest0-21swapper/421:52:306
736899142,7cyclictest0-21swapper/421:52:306
736399143,6cyclictest0-21swapper/322:40:015
736399142,7cyclictest0-21swapper/300:25:015
735899143,8cyclictest0-21swapper/221:09:594
734799143,6cyclictest0-21swapper/020:35:010
734799142,8cyclictest0-21swapper/019:55:010
734799142,8cyclictest0-21swapper/019:40:000
275991140,6ptp4l21-21ksoftirqd/122:45:191
275991140,4ptp4l0-21swapper/123:50:141
275991140,0ptp4l0-21swapper/123:00:231
275991140,0ptp4l0-21swapper/122:25:231
275991140,0ptp4l0-21swapper/122:15:161
275991140,0ptp4l0-21swapper/121:00:131
275991140,0ptp4l0-21swapper/120:05:151
15552140,0sleep80-21swapper/819:40:2210
740099132,8cyclictest0-21swapper/1021:20:252
739499132,7cyclictest0-21swapper/922:00:0111
739499132,7cyclictest0-21swapper/922:00:0111
739499132,7cyclictest0-21swapper/919:10:2411
739499132,6cyclictest0-21swapper/923:15:0111
739499132,6cyclictest0-21swapper/919:45:0111
739499130,13cyclictest0-21swapper/920:25:1511
739499130,12cyclictest17120-21awk22:10:0011
737799133,6cyclictest0-21swapper/622:25:018
737799132,7cyclictest0-21swapper/619:35:008
737299133,6cyclictest0-21swapper/520:05:007
736899132,7cyclictest0-21swapper/423:30:226
736399132,7cyclictest0-21swapper/320:30:015
736399132,7cyclictest0-21swapper/319:40:255
736399132,7cyclictest0-21swapper/300:05:015
736399132,6cyclictest0-21swapper/323:05:015
735899130,13cyclictest0-21swapper/223:31:474
734799130,7cyclictest0-21swapper/020:45:010
275991130,0ptp4l0-21swapper/122:14:441
275991130,0ptp4l0-21swapper/122:05:121
275991130,0ptp4l0-21swapper/121:55:131
275991130,0ptp4l0-21swapper/121:55:131
275991130,0ptp4l0-21swapper/121:15:131
275991130,0ptp4l0-21swapper/119:47:141
275991130,0ptp4l0-21swapper/119:15:171
275991130,0ptp4l0-21swapper/100:30:151
275991130,0ptp4l0-21swapper/100:25:441
275991130,0ptp4l0-21swapper/100:00:161
740499122,6cyclictest0-21swapper/1123:45:243
740499122,6cyclictest0-21swapper/1121:10:003
740499122,6cyclictest0-21swapper/1120:35:013
740499121,7cyclictest0-21swapper/1122:05:013
740499121,7cyclictest0-21swapper/1121:35:003
740099122,7cyclictest0-21swapper/1020:05:262
740099121,6cyclictest0-21swapper/1021:30:262
740099120,11cyclictest0-21swapper/1021:45:002
740099120,11cyclictest0-21swapper/1021:45:002
739499122,6cyclictest0-21swapper/922:30:0111
739499122,6cyclictest0-21swapper/919:40:0011
7387991211,1cyclictest0-21swapper/821:32:3710
738799120,1cyclictest0-21swapper/820:05:5210
7381991210,1cyclictest0-21swapper/719:40:009
737799120,8cyclictest0-21swapper/619:45:248
737799120,12cyclictest0-21swapper/623:26:448
737799120,11cyclictest0-21swapper/622:45:008
737299121,7cyclictest0-21swapper/523:15:247
737299121,6cyclictest0-21swapper/520:10:007
736899122,6cyclictest0-21swapper/420:00:236
736399122,6cyclictest0-21swapper/321:55:015
736399122,6cyclictest0-21swapper/321:55:015
735899129,2cyclictest0-21swapper/200:30:014
735899122,7cyclictest0-21swapper/200:25:014
7352991211,1cyclictest0-21swapper/120:55:241
7352991211,1cyclictest0-21swapper/119:14:571
734799122,7cyclictest0-21swapper/022:35:000
734799122,6cyclictest0-21swapper/021:45:250
734799122,6cyclictest0-21swapper/021:45:250
275991120,3ptp4l6328-21/usr/sbin/munin20:30:141
275991120,1ptp4l0-21swapper/119:25:001
275991120,1ptp4l0-21swapper/100:20:191
275991120,0ptp4l0-21swapper/123:20:131
275991120,0ptp4l0-21swapper/122:38:471
275991120,0ptp4l0-21swapper/121:25:261
275991120,0ptp4l0-21swapper/121:06:431
275991120,0ptp4l0-21swapper/120:10:151
275991120,0ptp4l0-21swapper/119:28:301
275991120,0ptp4l0-21swapper/100:15:181
740499112,6cyclictest0-21swapper/1120:45:253
740099110,9cyclictest15508-21latency_hist22:50:012
740099110,6cyclictest0-21swapper/1000:39:552
740099110,2cyclictest17850-21ps19:20:222
7387991110,1cyclictest0-21swapper/820:47:1510
738799110,3cyclictest934-21grep22:30:1110
738799110,1cyclictest0-21swapper/821:06:5610
738799110,11cyclictest0-21swapper/822:10:0310
738799110,10cyclictest31686-21tr23:10:1210
7381991110,1cyclictest0-21swapper/723:09:179
738199110,1cyclictest0-21swapper/700:39:579
737799118,2cyclictest18323-21if_enp1s020:45:178
737799110,1cyclictest0-21swapper/621:13:298
737299117,3cyclictest7625-21switchtime00:00:237
737299113,5cyclictest0-21swapper/520:15:007
737299111,7cyclictest6650-21kworker/u24:221:20:007
736899117,3cyclictest17203-21cat21:25:196
735299119,2cyclictest31114-21turbostat19:40:001
735299118,2cyclictest28037-21gltestperf21:40:161
735299118,2cyclictest28037-21gltestperf21:40:161
734799110,10cyclictest0-21swapper/023:45:140
734799110,10cyclictest0-21swapper/020:50:150
67182116,4sleep90-21swapper/919:06:3211
275991110,6ptp4l3233-21diskmemload23:15:371
275991110,3ptp4l16835-21dnf22:50:131
275991110,2ptp4l22229-21grep23:40:121
275991110,1ptp4l0-21swapper/123:25:251
275991110,1ptp4l0-21swapper/121:20:151
275991110,1ptp4l0-21swapper/120:50:251
275991110,0ptp4l0-21swapper/121:50:181
275991110,0ptp4l0-21swapper/121:50:181
275991110,0ptp4l0-21swapper/121:32:341
275991110,0ptp4l0-21swapper/121:10:171
275991110,0ptp4l0-21swapper/120:43:301
275991110,0ptp4l0-21swapper/120:35:121
740499102,5cyclictest0-21swapper/1121:10:273
740499100,9cyclictest0-21swapper/1123:10:213
740499100,9cyclictest0-21swapper/1121:45:133
740499100,9cyclictest0-21swapper/1121:45:123
740499100,1cyclictest0-21swapper/1119:30:173
740499100,10cyclictest0-21swapper/1123:37:573
740099109,1cyclictest0-21swapper/1022:56:282
740099109,1cyclictest0-21swapper/1022:09:192
740099107,2cyclictest13927-21turbostat23:30:002
740099106,4cyclictest0-21swapper/1021:25:472
740099100,3cyclictest20933-21munin-run22:15:002
740099100,10cyclictest0-21swapper/1023:49:222
740099100,10cyclictest0-21swapper/1020:52:302
739499100,9cyclictest0-21swapper/921:00:1411
739499100,9cyclictest0-21swapper/900:05:1511
738799109,1cyclictest0-21swapper/823:37:5510
738799109,1cyclictest0-21swapper/823:16:3910
738799109,1cyclictest0-21swapper/822:45:3310
738799107,2cyclictest23315-21hddtemp_smartct23:40:1610
738799106,3cyclictest8734-21/usr/sbin/munin21:15:1510
738799100,9cyclictest5186-21hddtemp_smartct21:10:1810
738799100,9cyclictest0-21swapper/800:30:1310
738799100,6cyclictest0-21swapper/822:35:2710
738799100,6cyclictest0-21swapper/820:11:3210
738799100,5cyclictest0-21swapper/822:55:0110
738799100,2cyclictest5063-21awk20:30:0010
738199109,1cyclictest0-21swapper/721:10:179
738199108,1cyclictest0-21swapper/721:09:559
738199106,3cyclictest25744-21kernelversion19:30:189
738199106,3cyclictest25101-21/usr/sbin/munin20:10:259
738199100,9cyclictest5999-21df_inode00:00:129
738199100,9cyclictest3154-21fschecks_time21:50:169
738199100,9cyclictest3154-21fschecks_time21:50:169
738199100,9cyclictest27708-21irqrtprio20:15:199
738199100,9cyclictest1942-21timerandwakeup119:40:259
738199100,9cyclictest16831-21latency_hist20:45:019
738199100,9cyclictest16787-21cut19:20:179
738199100,9cyclictest16563-21sed00:15:019
738199100,9cyclictest16531-21perl21:25:159
738199100,3cyclictest11779-21/usr/sbin/munin23:25:159
738199100,10cyclictest0-21swapper/722:30:129
737799109,1cyclictest0-21swapper/622:15:508
737799106,3cyclictest8977-21sed20:35:008
737799106,3cyclictest17187-21entropy19:20:188
737799103,7cyclictest0-21swapper/621:28:038
737799101,6cyclictest0-21swapper/622:35:018
737799101,6cyclictest0-21swapper/600:35:018
737299109,1cyclictest0-21swapper/520:19:167
737299100,9cyclictest3233-21diskmemload22:58:587
737299100,9cyclictest21227-21grep19:25:147
737299100,9cyclictest1623-21missed_timers00:35:217
737299100,3cyclictest32379-21fschecks_time23:10:147
737299100,1cyclictest26388-21strings00:25:207
737299100,1cyclictest0-21swapper/520:45:207
737299100,10cyclictest0-21swapper/521:53:157
737299100,10cyclictest0-21swapper/521:53:157
736899107,2cyclictest2313-21df20:25:136
736899102,5cyclictest0-21swapper/419:20:276
736899100,10cyclictest0-21swapper/419:28:186
736399107,2cyclictest0-21swapper/323:30:145
736399103,5cyclictest0-21swapper/323:40:015
736399100,9cyclictest9973-21processes21:15:215
736399100,9cyclictest658-21cpuspeed22:30:115
736399100,9cyclictest17405-21grep19:20:195
736399100,10cyclictest0-21swapper/320:27:245
735899105,1cyclictest0-21swapper/223:57:514
735899101,5cyclictest0-21swapper/223:00:244
735899100,9cyclictest19330-21aten2_r3power_v21:30:144
735899100,9cyclictest0-21swapper/223:15:144
735899100,9cyclictest0-21swapper/222:06:254
735899100,6cyclictest0-21swapper/222:25:004
735299100,9cyclictest16215-21latency23:30:171
735299100,9cyclictest10449-21tr22:00:121
734799107,2cyclictest0-21swapper/022:35:190
734799106,3cyclictest24627-21/usr/sbin/munin20:55:200
734799106,3cyclictest2138-21/usr/sbin/munin21:50:120
734799106,3cyclictest2138-21/usr/sbin/munin21:50:110
734799106,3cyclictest10312-21idleruntime00:05:170
734799100,9cyclictest7031-21taskset21:12:120
734799100,9cyclictest533-21irqrtprio23:10:180
734799100,9cyclictest32327-21timerandwakeup23:50:230
734799100,9cyclictest30762-21sendmail_mailtr21:00:220
734799100,9cyclictest2570-21users00:35:270
734799100,9cyclictest0-21swapper/023:20:160
734799100,2cyclictest0-21swapper/019:15:180
734799100,1cyclictest0-21swapper/023:35:010
734799100,10cyclictest0-21swapper/022:42:480
66762101,7sleep80-21swapper/819:05:5510
275991100,2ptp4l9136-21/usr/sbin/munin00:05:121
275991100,0ptp4l0-21swapper/122:55:121
275991100,0ptp4l0-21swapper/122:40:141
275991100,0ptp4l0-21swapper/121:46:171
275991100,0ptp4l0-21swapper/121:46:161
275991100,0ptp4l0-21swapper/120:20:151
275991100,0ptp4l0-21swapper/120:15:221
275991100,0ptp4l0-21swapper/119:40:141
74049996,2cyclictest13410-21cstates00:10:153
74049991,2cyclictest0-21swapper/1119:15:163
74049990,1cyclictest0-21swapper/1119:10:013
74009998,1cyclictest0-21swapper/1023:55:162
74009998,1cyclictest0-21swapper/1020:37:222
74009996,2cyclictest27331-21perl20:15:162
74009991,7cyclictest0-21swapper/1020:25:182
74009990,9cyclictest0-21swapper/1019:20:012
74009990,8cyclictest8069-21timerandwakeup20:30:252
74009990,8cyclictest5127-21grep21:10:182
74009990,8cyclictest0-21swapper/1022:37:502
74009990,8cyclictest0-21swapper/1022:25:202
74009990,8cyclictest0-21swapper/1022:20:182
74009990,8cyclictest0-21swapper/1019:39:282
74009990,6cyclictest28134-21/usr/sbin/munin00:30:202
74009990,3cyclictest18360-21smart_sda22:50:212
74009990,1cyclictest0-21swapper/1019:50:152
74009990,1cyclictest0-21swapper/1000:00:252
73949998,1cyclictest0-21swapper/921:07:0811
73949998,1cyclictest0-21swapper/920:30:1911
73949997,1cyclictest0-21swapper/922:25:0011
73949992,6cyclictest0-21swapper/919:50:0011
73949990,8cyclictest0-21swapper/921:15:2111
73949990,7cyclictest25342-21irqrtprio23:00:1811
73949990,2cyclictest0-21swapper/921:50:1911
73949990,2cyclictest0-21swapper/921:50:1911
73949990,1cyclictest0-21swapper/919:55:1511
73879998,1cyclictest0-21swapper/820:22:2210
73879996,2cyclictest26379-21switchtime19:30:2310
73879996,2cyclictest17504-21fschecks_time19:20:1910
73879995,3cyclictest555-21grep00:35:1610
73879995,3cyclictest11890-21/usr/sbin/munin00:05:2510
73879990,9cyclictest0-21swapper/823:55:1610
73879990,9cyclictest0-21swapper/823:45:1810
73879990,9cyclictest0-21swapper/822:40:1810
73879990,8cyclictest7304-21fw_conntrack21:55:1410
73879990,8cyclictest7304-21fw_conntrack21:55:1410
73879990,8cyclictest3233-21diskmemload22:55:1010
73879990,8cyclictest29265-21/usr/sbin/munin21:00:1510
73879990,8cyclictest25478-21cat23:00:1810
73879990,8cyclictest22037-21expr22:15:1310
73879990,8cyclictest20133-21latency_hist00:20:0010
73879990,8cyclictest15623-21cpuspeed21:25:1310
73879990,8cyclictest14125-21timerandwakeup21:20:2510
73879990,8cyclictest13155-21sed20:40:0110
73879990,8cyclictest0-21swapper/819:45:1810
73879990,8cyclictest0-21swapper/819:25:5310
73879990,3cyclictest15459-21unixbench_singl19:15:2410
73879990,3cyclictest14861-21aten2_r3power_v20:00:1010
73879990,2cyclictest0-21swapper/821:50:2510
73879990,2cyclictest0-21swapper/821:50:2410
73879990,1cyclictest0-21swapper/823:53:2710
73879990,1cyclictest0-21swapper/820:19:5110
73819998,1cyclictest0-21swapper/720:30:209
73819995,3cyclictest18604-21/usr/sbin/munin20:05:279
73819991,7cyclictest32619-21if_enp1s023:10:169
73819990,9cyclictest0-21swapper/720:35:249
73819990,8cyclictest8325-21cstates21:15:129
73819990,8cyclictest29530-21sed21:40:259
73819990,8cyclictest29530-21sed21:40:259
73819990,8cyclictest25113-21df_inode00:25:149
73819990,8cyclictest24452-21sed20:55:009
73819990,8cyclictest22236-21irqstats00:20:199
73819990,8cyclictest19633-21tr21:30:159
73819990,8cyclictest16689-21/usr/sbin/munin00:15:259
73819990,8cyclictest14444-21/usr/sbin/munin23:30:139
73819990,8cyclictest13021-21diskstats22:45:139
73819990,8cyclictest12922-21sed19:55:199
73819990,8cyclictest0-21swapper/722:00:139
73819990,3cyclictest32694-21sed21:45:209
73819990,3cyclictest32694-21sed21:45:209
73819990,3cyclictest30763-21timerandwakeup100:30:249
73819990,3cyclictest2901-21/usr/sbin/munin20:25:169
73819990,2cyclictest25702-21latency_hist23:45:019
73779998,1cyclictest0-21swapper/622:12:298
73779998,1cyclictest0-21swapper/620:35:098
73779996,2cyclictest14216-21timerandwakeup21:20:258
73779996,2cyclictest1009-21snmpd23:43:048
73779995,3cyclictest7799-21expr19:50:148
73779995,3cyclictest6346-21latency22:35:188
73779995,3cyclictest27996-21perl21:40:158
73779995,3cyclictest27996-21perl21:40:158
73779995,3cyclictest20099-21latency_hist00:20:018
73779995,3cyclictest16030-21/usr/sbin/munin22:50:248
73779995,3cyclictest13993-21latency22:45:188
73779991,7cyclictest21066-21gltestperf22:55:168
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional