Istio代理级指标
代理级指标
-
Envoy会生成其资源级别（例如Listener、Cluster等）的指标
-
获取Envoy统计信息的常用方式有两程
-
Admin Interface的/stats或/stats/prometheus
-
用于接收统计信息的StatsD集群
-
-
Envoy常用的统计指标的说明
-
HTTP连接器相关
-
https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats#config-http-conn-man-stats
-
-
Cluster管理器相关
-
https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats#config-cluster-manager-cluster-stats
-
-
-
-
网格上的代理级指标
- 默认情况下，Istio仅启用了Envoy生成的统计信息的一小部分，以免导致指标后端过载，也能够减少与指标收集相关的CPU开销
-
同时，Istio也支持由用户在指定的Envoy实例上自定义要收集统计指标
Istio启用的代理级指标
-
解Envoy启用的统计指标
-
相关的端点：/stats和/stats/prometheus
-
获取命令：
-
kubectl exec "$POD" -c istio-proxy -- pilot-agent request GET /stats
-
kubectl exec "$POD" -- curl localhost:15000/stats
- curl $POD_IP:15020/stats/prometheus
-
-
查看启用的统计指标
-
命令：~$ istioctl proxy-config bootstrap $POD.$NAMESPACE | jq .bootstrap.statsConfig
-
statsMatcher.inclusionList段：要暴露的各指标
- statsTags段：支持在指标上使用的标签
-
-
-
-
默认情况下，Istio仅在Envoy上启用了统计信息中的一个较小的子集，相关的指标键包括
-
键前缀匹配
-
cluster_manager、listener_manager、server、cluster.xds-grpc、wasm、component
-
-
键后缀匹配
-
rbac.allowed、rbac.denied、shadow_allowed和shadow_denied
-
-
-
Istio Proxy暴露指标的位置
-
网格中的各Envoy实例使用15020端口上的/stats/prometheus端口对外暴露指标
-
相关工作负载的Pod上会借助于Annotation向Prometheus的Scraper指定该位置
-
在网格上配置代理级指标
- Istio上与网格代理相关的默认配置定义在MeshConfig.defaultConfig配置段中，而每个Pod之上Sidecar Envoy的指标定义则要通过Annotations完成
-
要自定义启用的统计指标可以使用MeshConfig.defautConfig.proxyStatsMatcher进行定义
-
inclusionRegexps：基于正则表达式模式指定要启用的统计指标键
-
inclusionPrefixes：基于前缀匹配指定要启用的统计指标键
-
inclusionSuffixes：基于后缀匹配指定要启用的统计指标键
-
查看默认指标
~# curl -sS `kubectl get pod -l app=proxy -o jsonpath='{.items[0].status.podIP}'`:15020/stats/prometheus查看默认信息
~# istioctl profile dump demo
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
components:
base:
enabled: true
cni:
enabled: false
egressGateways:
- enabled: true
k8s:
resources:
requests:
cpu: 10m
memory: 40Mi
name: istio-egressgateway
ingressGateways:
- enabled: true
k8s:
resources:
requests:
cpu: 10m
memory: 40Mi
service:
ports:
- name: status-port
port: 15021
targetPort: 15021
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tcp
port: 31400
targetPort: 31400
- name: tls
port: 15443
targetPort: 15443
name: istio-ingressgateway
istiodRemote:
enabled: false
pilot:
enabled: true
k8s:
env:
- name: PILOT_TRACE_SAMPLING
value: "100"
resources:
requests:
cpu: 10m
memory: 100Mi
hub: docker.io/istio
meshConfig:
accessLogFile: /dev/stdout
defaultConfig:
proxyMetadata: {}
enablePrometheusMerge: true
extensionProviders:
- envoyOtelAls:
port: 4317
service: opentelemetry-collector.istio-system.svc.cluster.local
name: otel
profile: demo
tag: 1.15.2
values:
base:
enableCRDTemplates: false
validationURL: ""
defaultRevision: ""
gateways:
istio-egressgateway:
autoscaleEnabled: false
env: {}
name: istio-egressgateway
secretVolumes:
- mountPath: /etc/istio/egressgateway-certs
name: egressgateway-certs
secretName: istio-egressgateway-certs
- mountPath: /etc/istio/egressgateway-ca-certs
name: egressgateway-ca-certs
secretName: istio-egressgateway-ca-certs
type: ClusterIP
istio-ingressgateway:
autoscaleEnabled: false
env: {}
name: istio-ingressgateway
secretVolumes:
- mountPath: /etc/istio/ingressgateway-certs
name: ingressgateway-certs
secretName: istio-ingressgateway-certs
- mountPath: /etc/istio/ingressgateway-ca-certs
name: ingressgateway-ca-certs
secretName: istio-ingressgateway-ca-certs
type: LoadBalancer
global:
configValidation: true
defaultNodeSelector: {}
defaultPodDisruptionBudget:
enabled: true
defaultResources:
requests:
cpu: 10m
imagePullPolicy: ""
imagePullSecrets: []
istioNamespace: istio-system
istiod:
enableAnalysis: false
jwtPolicy: third-party-jwt
logAsJson: false
logging:
level: default:info
meshNetworks: {}
mountMtlsCerts: false
multiCluster:
clusterName: ""
enabled: false
network: ""
omitSidecarInjectorConfigMap: false
oneNamespace: false
operatorManageWebhooks: false
pilotCertProvider: istiod
priorityClassName: ""
proxy:
autoInject: enabled
clusterDomain: cluster.local
componentLogLevel: misc:error
enableCoreDump: false
excludeIPRanges: ""
excludeInboundPorts: ""
excludeOutboundPorts: ""
image: proxyv2
includeIPRanges: '*'
logLevel: warning
privileged: false
readinessFailureThreshold: 30
readinessInitialDelaySeconds: 1
readinessPeriodSeconds: 2
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 40Mi
statusPort: 15020
tracer: zipkin
proxy_init:
image: proxyv2
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 10Mi
sds:
token:
aud: istio-ca
sts:
servicePort: 0
tracer:
datadog: {}
lightstep: {}
stackdriver: {}
zipkin: {}
useMCP: false
istiodRemote:
injectionURL: ""
pilot:
autoscaleEnabled: false
autoscaleMax: 5
autoscaleMin: 1
configMap: true
cpu:
targetAverageUtilization: 80
enableProtocolSniffingForInbound: true
enableProtocolSniffingForOutbound: true
env: {}
image: pilot
keepaliveMaxServerConnectionAge: 30m
nodeSelector: {}
podLabels: {}
replicaCount: 1
traceSampling: 1
telemetry:
enabled: true
v2:
enabled: true
metadataExchange:
wasmEnabled: false
prometheus:
enabled: true
wasmEnabled: false
stackdriver:
configOverride: {}
enabled: false
logging: false
monitoring: false
topology: false应用指标
# istioctl install -f filename还原指标
# istioctl install --set profile=demo -y代理级指标示例
Annotations
期望配置单个工作负载上的统计指标时，可通过在相应的Pod上添加或修改特定的Annotaion实现.
apiVersion: v1
kind: Pod
metadata:
name: client
annotations:
proxy.istio.io/config: |-
proxyStatsMatcher:
inclusionRegexps:
- ".*circuit_breakers.*"
inclusionPrefixes:
- "upstream_rq_retry"
- "upstream_cx"
spec:
containers:
- image: ikubernetes/admin-box:v1.2
name: client
command: ['/bin/bash','-c','sleep 99999']
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: NeverMeshConfig
该配置意味着要在网格级别启用断路器、重试和上游连接的统计信息.
全局级别的统计信息的修改，需要重新重新启用实例才能生效 .
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
meshConfig:
defaultConfig:
proxyStatsMatcher:
inclusionRegexps:
- ".*circuit_breakers.*"
inclusionPrefixes:
- "upstream_rq_retry"
- "upstream_cx"参考文档
https://istio.io/latest/zh/docs/tasks/observability/metrics/
