使用k8s-prometheus-adapter实现HPA

环境：

kubernetes 1.11+/openshift3.11

自定义metric HPA原理：

首选需要注册一个apiservice(custom metrics API)。

当HPA请求metrics时，kube-aggregator(apiservice的controller)会将请求转发到adapter，adapter作为kubernentes集群的pod，实现了Kubernetes resource metrics API 和custom metrics API，它会根据配置的rules从Prometheus抓取并处理metrics，在处理(如重命名metrics等)完后将metric通过custom metrics API返回给HPA。最后HPA通过获取的metrics的value对Deployment/ReplicaSet进行扩缩容。

adapter作为extension-apiserver(即自己实现的pod)，充当了代理kube-apiserver请求Prometheus的功能。

如下是k8s-prometheus-adapter apiservice的定义，kube-aggregator通过下面的service将请求转发给adapter。v1beta1.custom.metrics.k8s.io是写在k8s-prometheus-adapter代码中的，因此不能任意改变。

apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  service:
    name: custom-metrics-apiserver
    namespace: custom-metrics
  group: custom.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

部署：

github下载k8s-prometheus-adapter

参照官方文档部署adapter：

pull镜像：directxman12/k8s-prometheus-adapter:latest，修改镜像tag并push到本地镜像仓库

生成证书：运行如下shell脚本(来自官方)生成cm-adapter-serving-certs.yaml，并将其拷贝到manifests/目录下，该证书用于kube-aggregator与adapter通信时认证adapter。注意下面证书有效时间为5年(43800h)以及授权的域名。

#!/usr/bin/env bash
# exit immediately when a command fails
set -e
# only exit with zero if all commands of the pipeline exit successfully
set -o pipefail
# error on unset variables
set -u

# Detect if we are on mac or should use GNU base64 options
case $(uname) in
        Darwin)
            b64_opts='-b=0'
            ;; 
        *)
            b64_opts='--wrap=0'
esac

go get -v -u github.com/cloudflare/cfssl/cmd/...

export PURPOSE=metrics
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","'${PURPOSE}'"]}}}' > "ca-config.json"

export SERVICE_NAME=custom-metrics-apiserver
export ALT_NAMES='"custom-metrics-apiserver.custom-metrics","custom-metrics-apiserver.custom-metrics.svc"'
echo "{\"CN\":\"${SERVICE_NAME}\", \"hosts\": [${ALT_NAMES}], \"key\": {\"algo\": \"rsa\",\"size\": 2048}}" | \
       	cfssl gencert -ca=ca.crt -ca-key=ca.key -config=ca-config.json - | cfssljson -bare apiserver

cat <<-EOF > cm-adapter-serving-certs.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cm-adapter-serving-certs
data:
  serving.crt: $(base64 ${b64_opts} < apiserver.pem)
  serving.key: $(base64 ${b64_opts} < apiserver-key.pem)
EOF

可以在custom-metrics-apiservice.yaml中设置insecureSkipTLSVerify: true时，kube-aggregator不会校验adapter的如上证书。如果需要启用校验，则需要在caBundle中添加openshift集群的ca证书(非openshift集群的自签证书会被认为是不可信任的证书)，将openshift集群master节点的/etc/origin/master/ca.crt进行base64转码黏贴到caBundle字段即可。

base64 ca.crt

也可以黏贴openshift集群master节点的/root/.kube/config文件中的clusters.cluster.certificate-authority-data字段

创建命名空间：kubectl create namespace custom-metrics

openshift的kube-system下面可能没有role extension-apiserver-authentication-reader,如果不存在，则需要创建

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: extension-apiserver-authentication-reader
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resourceNames:
  - extension-apiserver-authentication
  resources:
  - configmaps
  verbs:
  - get

修改custom-metrics-apiserver-deployment.yaml的--prometheus-url字段，指向正确的prometheus
创建其他组件：kubectl create -f manifests/

在部署时会创建一个名为custom-metrics-resource-reader的clusterRole，用于授权adapter读取kubernetes cluster的资源，可以看到其允许读取的资源为namespaces/pods/services
```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-metrics-resource-reader
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  - services
  verbs:
  - get
  - list
```

部署demo：

部署官方demo

# cat sample-app.deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - image: docker-local.art.aliocp.csvw.com/openshift3/autoscale-demo:v0.1.2
        name: metrics-provider
        ports:
        - name: http
          containerPort: 8080

创建service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: sample-app
  name: sample-app
  namespace: custom-metrics
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: sample-app
  type: ClusterIP

在custom-metrics命名空间下验证可以获取到metrics

curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics

部署serviceMonitor

由于HPA需要用到namespace和pod等kubernetes的资源信息，因此需要使用servicemonitor注册方式来为metrics添加这些信息

openshift Prometheus operator对servicemonitor的限制如下

  serviceMonitorNamespaceSelector:
    matchExpressions:
    - key: openshift.io/cluster-monitoring
      operator: Exists
  serviceMonitorSelector:
    matchExpressions:
    - key: k8s-app
      operator: Exists

因此需要给custom-metrics命名空间添加标签

oc label namespace custom-metrics openshift.io/cluster-monitoring=true

在openshift-monitoring命名空间中创建service-monitor

# cat service-monitor.yaml
kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  name: sample-app
  labels:
    k8s-app: testsample
    app: sample-app
spec:
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      app: sample-app
  endpoints:
  - port: http

添加权限

oc adm policy add-cluster-role-to-user view system:serviceaccount:openshift-monitoring:prometheus-k8s

oc adm policy add-role-to-user view system:serviceaccount:openshift-monitoring:prometheus-k8s -n custom-metrics

测试HPA

创建HPA，表示1秒请求大于0.5个时开始扩容

# cat sample-app-hpa.yaml
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: sample-app
spec:
  scaleTargetRef:
    # point the HPA at the sample application
    # you created above
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  # autoscale between 1 and 10 replicas
  minReplicas: 1
  maxReplicas: 10
  metrics:
  # use a "Pods" metric, which takes the average of the
  # given metric across all pods controlled by the autoscaling target
  - type: Pods
    pods:
      # use the metric that you used above: pods/http_requests
      metricName: http_requests_per_second
      # target 500 milli-requests per second,
      # which is 1 request every two seconds
      targetAverageValue: 500m

通过oc describe hpa sample-app查看hpa是否运行正常

持续执行命令curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics发出请求

通过命令kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/custom-metrics/pods/*/http_requests_per_second"查看其对应的value值，当其值大于500m时开始扩容

# oc get pod
NAME                          READY     STATUS    RESTARTS   AGE
sample-app-6d55487cdd-dc6qz   1/1       Running   0          18h
sample-app-6d55487cdd-w6bbb   1/1       Running   0          5m
sample-app-6d55487cdd-zbdbr   1/1       Running   0          5m

过段时间，当kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/custom-metrics/pods/*/http_requests_per_second"的值持续低于500m时进行缩容，缩容时间由--horizontal-pod-autoscaler-downscale-stabilization指定，默认5分钟。

提供oc get hpa的TARGETS字段可以查看扩缩容比例
```
# oc get hpa
NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
sample-app   Deployment/sample-app   66m/500m   1         10        1          3h
```

Adapter config

部署adapter前需要配置adapter的rule，用于预处理metrics，默认配置为manifests/custom-metrics-config-map.yaml。adapter的配置主要分为4个:

Discovery：指定需要处理的Prometheus的metrics。通过seriesQuery挑选需要处理的metrics集合，可以通过seriesFilters精确过滤metrics。

seriesQuery可以根据标签进行查找(如下)，也可以直接指定metric name查找

seriesQuery: '{__name__=~"^container_.*_total",container_name!="POD",namespace!="",pod_name!=""}'
seriesFilters:
  - isNot: "^container_.*_seconds_total"

seriesFilters：

is: <regex>, 匹配包含该正则表达式的metrics.
isNot: <regex>, 匹配不包含该正则表达式的metrics.

Association：设置metric与kubernetes resources的映射关系，kubernetes resorces可以通过kubectl api-resources命令查看。overrides会将Prometheus metric label与一个kubernetes resource(下例为deployment)关联。需要注意的是该label必须是一个真实的kubernetes resource，如metric的pod_name可以映射为kubernetes的pod resource，但不能将container_image映射为kubernetes的pod resource，映射错误会导致无法通过custom metrics API获取正确的值。这也表示metric中必须存在一个真实的resource 名称，将其映射为kubernetes resource。
```
resources:
  overrides:
    microservice: {group: "apps", resource: "deployment"}
```
Naming：用于将prometheus metrics名称转化为custom metrics API所使用的metrics名称，但不会改变其本身的metric名称，即通过curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics获得的仍然是老的metric名称。如果不需要可以不执行这一步。
```
# match turn any name <name>_total to <name>_per_second
# e.g. http_requests_total becomes http_requests_per_second
name:
  matches: "^(.*)_total$"
  as: "${1}_per_second"
```
如本例中HPA后续可以通过/apis/{APIService-name}/v1beta1/namespaces/{namespaces-name}/pods/*/http_requests_per_second获取metrics

Querying：处理调用custom metrics API获取到的metrics的value，该值最终提供给HPA进行扩缩容

# convert cumulative cAdvisor metrics into rates calculated over 2 minutes
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[2m])) by (<<.GroupBy>>)"

metricsQuery 字段使用Go template将URL请求转变为Prometheus的请求，它会提取custom metrics API请求中的字段，并将其划分为metric name,group-resource,以及group-resource中的一个或多个objects，对应如下字段：

Series: metric名称
LabelMatchers: 以逗号分割的objects，当前表示特定group-resource加上命名空间的label(如果该group-resource 是namespaced的)
GroupBy：以逗号分割的label的集合，当前表示LabelMatchers中的group-resource label

假设metrics http_requests_per_second如下

http_requests_per_second{pod="pod1",service="nginx1",namespace="somens"}
http_requests_per_second{pod="pod2",service="nginx2",namespace="somens"}

当调用kubectl get --raw "/apis/{APIService-name}/v1beta1/namespaces/somens/pods/*/http_request_per_second"时，metricsQuery字段的模板的实际内容如下：

Series: "http_requests_total"
LabelMatchers: "pod=~\"pod1|pod2",namespace="somens"
GroupBy:pod

adapter使用字段rules和externalRules分别表示custom metrics和external metrics，如本例中

apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    externalRules:
    - seriesQuery: '{namespace!="",pod!=""}'
      seriesFilters: []
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[22m])) by (<<.GroupBy>>)
    rules:
    - seriesQuery: '{namespace!="",pod!=""}'
      seriesFilters: []
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)

HPA的配置

HPA通常会根据type从aggregated APIs (metrics.k8s.io, custom.metrics.k8s.io, external.metrics.k8s.io)的资源路径上拉取metrics

HPA支持的metrics类型有4种(下述为v2beta2的格式)：

resource：目前仅支持cpu和memory。target可以指定数值(targetAverageValue)和比例(targetAverageUtilization)进行扩缩容

HPA从metrics.k8s.io获取resource metrics
pods：custom metrics，这类metrics描述了pod类型，target仅支持按指定数值(targetAverageValue)进行扩缩容。targetAverageValue 用于计算所有相关pods上的metrics的平均值
```
type: Pods
pods:
  metric:
    name: packets-per-second
  target:
    type: AverageValue
    averageValue: 1k
```
HPA从custom.metrics.k8s.io获取custom metrics
object：custom metrics，这类metrics描述了相同命名空间下的(非pod)类型。target支持通过value和AverageValue进行扩缩容，前者直接将metric与target比较进行扩缩容，后者通过metric/相关的pod数目与target比较进行扩缩容
```
type: Object
object:
  metric:
    name: requests-per-second
  describedObject:
    apiVersion: extensions/v1beta1
    kind: Ingress
    name: main-route
  target:
    type: Value
    value: 2k
```
external：kubernetes 1.10+。这类metrics与kubernetes集群无关(pods和object需要与kubernetes中的某一类型关联)。与object类似，target支持通过value和AverageValue进行扩缩容。由于external会尝试匹配所有kubernetes资源的metrics，因此实际中不建议使用该类型。

HPA从external.metrics.k8s.io获取external metrics
```
- type: External
  external:
    metric:
      name: queue_messages_ready
      selector: "queue=worker_tasks"
    target:
      type: AverageValue
      averageValue: 30
```
1.6版本支持多metrics的扩缩容，当其中一个metrics达到扩容标准时就会创建pod副本(当前副本<maxReplicas)

注：target的value的一个单位可以划分为1000份，每一份以m为单位，如500m表示1/2个单位。参见Quantity

kubernetes HPA的算法如下：

desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

当使用targetAverageValue 或targetAverageUtilization时，currentMetricValue会取HPA指定的所有pods的metric的平均值

Kubernetes metrics的获取

假设注册的APIService为custom.metrics.k8s.io/v1beta1，在注册好APIService后HorizontalPodAutoscaler controller会从以/apis/custom.metrics.k8s.io/v1beta1为根API的路径上抓取metrics。metrics的API path可以分为namespaced和non-namespaced类型的。通过如下方式校验HPA是否可以获取到metrics：

namespaced

获取指定namespace下指定object类型和名称的metrics

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/{object-type}/{object-name}/{metric-name...}"

如获取monitor命名空间下名为grafana的pod的start_time_seconds metric

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitor/pods/grafana/start_time_seconds"

获取指定namespace下所有特定object类型的metrics

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/pods/*/{metric-name...}"

如获取monitor命名空间下名为所有pod的start_time_seconds metric

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitor/pods/*/start_time_seconds"

使用labelSelector可以选择带有特定label的object

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/{object-type}/{object-name}/{metric-name...}?labelSelector={label-name}"

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/pods/*/{metric-name...}?labelSelector={label-name}"

non-namespaced

non-namespaced和namespaced的类似，主要有node，namespace，PersistentVolume等。non-namespaced访问有些与custom metrics API描述不一致。

访问object为namespace的方式如下如下

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/metrics/{metric-name...}"

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/*/metrics/{metric-name...}"

访问node的方式如下

 kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/nodes/{node-name}/{metric-name...}"

DEBUG:

使用如下方式查看注册的APIService发现的所有rules
```
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
```
如果获取失败，可以看下使用oc get apiservice v1beta1.custom.metrics.k8s.io -oyaml查看status和message的相关信息

如果获取到的resource为空，则需要校验deploy中的Prometheus url是否正确，是否有权限等

通过如下方式查看完整的请求过程(--v=8)

kubectl get --raw “/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/pods/*/{metric-name...}" --v=8

如果上述过程正确，但获取到的items为空
- 首先保证k8s-prometheus-adapter的参数--metrics-relist-interval设置值大于Prometheus的参数scrape_interval
- 确保k8s-prometheus-adapter rules的seriesQuery规则可以抓取到Prometheus的数据
- 确保k8s-prometheus-adapter rules的metricsQuery规则可以抓取到计算出数据，此处需要注意的是，如果使用到了计算某段时间的数据，如果时间设置过短，可能导致没有数据生成

TIPS:

官方提供了End-to-end walkthrough，但需要采集的metrics中包含pod和namespace label，否则在官方默认配置下无法采集到metrics。
Configuration Walkthroughs一步步讲解了如何配置adapter config
在goland里面使用如下参数可以远程调试adapter：

--secure-port=6443 --tls-cert-file=D:\adapter\serving.crt --tls-private-key-file=D:\adapter\serving.key --logtostderr=true --prometheus-url=${prometheus-url} --metrics-relist-interval=70s --v=10 --config=D:\adapter\config.yaml --lister-kubeconfig=D:\adapter\k8s-config.yaml --authorization-kubeconfig=D:\adapter\k8s-config.yaml --authentication-kubeconfig=D:\adapter\k8s-config.yaml