Kubernetes-基于ingress实现灰度发布
1、基于ingress实现灰度发布介绍
- Ingress-Nginx是一个K8S ingress工具,支持配置Ingress Annotations来实现不同场景下的灰度发布和测试。
1.1、Nginx Annotations支持以下4种Canary规则
- nginx.ingress.kubernetes.io/canary:开启灰度发布功能,如果没有开启此属性,则如下属性不生效。
- nginx.ingress.kubernetes.io/canary-by-header:基于Request Header的流量切分,适用于灰度发布以及A/B测试。当Request Header设置为 always时,请求将会被一直发送到Canary版本;当 equest Header设置为never时,请求不会被发送到Canary入口;对于任何其他Header值,将忽略Header,并通过优先级将请求与其他金丝雀规则进行优先级的比较。
- nginx.ingress.kubernetes.io/canary-by-header-value:要匹配的Request Header的值,用于通知Ingress将请求路由到Canary Ingress中指定的服务。当Request Header设置为此值时,它将被路由到Canary入口。该规则允许用户自定义Request Header的值,必须与上一个annotation (即canary-by-header)一起使用。
- nginx.ingress.kubernetes.io/canary-by-header-pattern:正则表达式匹配的Request Header的值,用于通知Ingress将请求路由到Canary Ingress中指定的服务。当Request Header设置为此值时,它将被路由到Canary入口。该规则允许用户自定义Request Header的值,必须与上一个annotation(即canary-by-header)一起使用,如果已经使用canary-by-header-value,那么此属性将被忽略。
- nginx.ingress.kubernetes.io/canary-by-cookie:基于Cookie的流量切分,适用于灰度发布与A/B测试。用于通知Ingress将请求路由到Canary Ingress中指定的服务的cookie。当cookie值设置为always时,它将被路由到Canary入口;当cookie值设置为never时,请求不会被发送到Canary入口;对于任何其他值,将忽略cookie并将请求与其他金丝雀规则进行优先级的比较。
- nginx.ingress.kubernetes.io/canary-weight:基于服务权重的流量切分,适用于蓝绿部署,权重范围0-100按百分比将请求路由到Canary Ingress中指定的服务。权重为0意味着该金丝雀规则不会向Canary入口的服务发送任何请求。权重为100意味着所有请求都将被发送到Canary入口。
- 金丝雀规则按优先顺序进行如下排序:
- canary-by-header - > canary-by-cookie - > canary-weight
1.2、基于权重或用户实现灰度发布
- 可以把上面四种annotation规则划分为两类。
2、ingress基于用户实现灰度发布
- 基于不同的场景,灰度发布有四种:
//第一种,所有的请求都会被转发到灰度(Canary)版本 nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-by-header: "always" //第二种,所有的请求都不会被转发到灰度(Canary)版本 nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-by-header: "never" //第三种,如果请求的header头包含"user-id: user_1",该请求会被转发到灰度(Canary)版本 nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-by-header: "user_id" nginx.ingress.kubernetes.io/canary-by-header-value: "user_1" //第四种,如果请求的header头包含"user-id: user_2"或"user-id: user-3"或"user-id: user4",该请求会被转发到灰度(Canary)版 nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-by-header: "user_id" #在networking.k8s.io/v1中,canary-by-header的值尽量不用使用下划线“_”。如果使用了,请求头中要使用中横线“-”代替。 nginx.ingress.kubernetes.io/canary-by-header-pattern: "user_2|user-3|user4"
2.1、发布现网服务
1、创建Deployment和Service的yaml文件(nginx-deployment.yaml)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
apiVersion: v1 kind: Service metadata: name: nginx-service spec: selector: app: nginx-pod ports: - port: 11180 targetPort: 80 protocol: TCP --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx-pod replicas: 2 strategy: #! type: RollingUpdate rollingUpdate: #先减一个pod,再加一个pod maxSurge: 0 maxUnavailable: 1 template: metadata: labels: app: nginx-pod spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx-pod topologyKey: "kubernetes.io/hostname" volumes: - name: html hostPath: path: /apps/html/ containers: - name: nginx-container image: nginx:04.26 imagePullPolicy: Never command: ["/usr/sbin/nginx", "-g", "daemon off;"] ports: - containerPort: 80 volumeMounts: - mountPath: /usr/share/nginx/html name: html
- 准备镜像和index.html文件
]# docker image tag nginx:latest nginx:04.26 ]# echo "04.26" > /apps/html/index.html
2、创建Ingress的yaml文件(nginx-ingress.yaml)
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx-ingress annotations: kubernetes.io/ingress.class: "nginx" #必须有,否则创建出的ingress可能不正常 spec: rules: - http: paths: - path: / pathType: Prefix backend: service: name: nginx-service port: number: 11180
3、创建Deployment、Service和Ingress
]# kubectl apply -f nginx-deployment.yaml ]# kubectl apply -f nginx-ingress.yaml
4、查看Deployment、Pod、Service和Ingress
]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-deployment-7448cc74f6-t5gbd 1/1 Running 0 49s 10.10.36.68 k8s-node1 <none> <none> nginx-deployment-7448cc74f6-znfzv 1/1 Running 0 49s 10.10.169.133 k8s-node2 <none> <none> ]# kubectl get deployment -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR nginx-deployment 2/2 2 2 34s nginx-container nginx:04.26 app=nginx-pod ]# kubectl get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR nginx-service ClusterIP 10.20.50.80 <none> 11180/TCP 57s app=nginx-pod ]# kubectl get ingress -o wide NAME CLASS HOSTS ADDRESS PORTS AGE nginx-ingress <none> * 10.1.1.13 80 44s
5、使用nodeIP+ingressPort访问服务
]# curl 10.1.1.11:32080 04.26
2.2、发布灰度服务
1、创建Deployment和Service的yaml文件(canary-nginx-deployment.yaml)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
apiVersion: v1 kind: Service metadata: name: nginx-service-canary spec: selector: app: nginx-pod-canary ports: - port: 11180 targetPort: 80 protocol: TCP --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment-canary spec: selector: matchLabels: app: nginx-pod-canary replicas: 2 strategy: #! type: RollingUpdate rollingUpdate: #先减一个pod,再加一个pod maxSurge: 0 maxUnavailable: 1 template: metadata: labels: app: nginx-pod-canary spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx-pod-canary topologyKey: "kubernetes.io/hostname" volumes: - name: html hostPath: path: /apps/html/canary/ containers: - name: nginx-container-canary image: nginx:04.27 imagePullPolicy: Never command: ["/usr/sbin/nginx", "-g", "daemon off;"] ports: - containerPort: 80 volumeMounts: - mountPath: /usr/share/nginx/html name: html
- 准备镜像和index.html文件
]# docker image tag nginx:latest nginx:04.27 ]# echo "04.27" > /apps/html/canary/index.html
2、创建Ingress的yaml文件(canary-nginx-ingress.yaml)
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx-ingress-canary annotations: kubernetes.io/ingress.class: "nginx" #必须有,否则创建出的ingress可能不正常 nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-by-header: "user_id" #如果请求的header头包含"user-id: user_2"或"user-id: user-3"或"user-id: user4",该请求会被转发到灰度(Canary)版 nginx.ingress.kubernetes.io/canary-by-header-pattern: "user_2|user-3|user4" spec: rules: - http: paths: - path: / pathType: Prefix backend: service: name: nginx-service-canary port: number: 11180
3、创建Deployment、Service和Ingress
]# kubectl apply -f canary-nginx-deployment.yaml ]# kubectl apply -f canary-nginx-ingress.yaml
4、查看Deployment、Pod、Service和Ingress
]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-deployment-7448cc74f6-t5gbd 1/1 Running 0 13m 10.10.36.68 k8s-node1 <none> <none> nginx-deployment-7448cc74f6-znfzv 1/1 Running 0 13m 10.10.169.133 k8s-node2 <none> <none> nginx-deployment-canary-c4884b965-bkzt9 1/1 Running 0 38s 10.10.36.69 k8s-node1 <none> <none> nginx-deployment-canary-c4884b965-gkjzt 1/1 Running 0 38s 10.10.169.134 k8s-node2 <none> <none> ]# kubectl get deployment -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR nginx-deployment 2/2 2 2 13m nginx-container nginx:04.26 app=nginx-pod nginx-deployment-canary 2/2 2 2 29s nginx-container-canary nginx:04.27 app=nginx-pod-canary ]# kubectl get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR nginx-service ClusterIP 10.20.50.80 <none> 11180/TCP 13m app=nginx-pod nginx-service-canary ClusterIP 10.20.227.2 <none> 11180/TCP 43s app=nginx-pod-canary ]# kubectl get ingress -o wide NAME CLASS HOSTS ADDRESS PORTS AGE nginx-ingress <none> * 10.1.1.13 80 14m nginx-ingress-canary <none> * 10.1.1.13 80 107s
5、使用nodeIP+ingressPort访问服务
]# curl 10.1.1.11:32080 04.26 //虽然nginx.ingress.kubernetes.io/canary-by-header: "user_id",但在访问时要使用user-id。尽量不要使用下划线“_” //nginx.ingress.kubernetes.io/canary-by-header-pattern: "user_2|user-3|user4",可以使用下划线“_”和中划线“-” ]# curl -H "user_id: user_2" 10.1.1.11:32080 04.26 ]# curl -H "user-id: user_2" 10.1.1.11:32080 04.27 ]# curl -H "user-id: user-3" 10.1.1.11:32080 04.27 ]# curl -H "user-id: user4" 10.1.1.11:32080 04.27
2.3、滚动升级现网服务
1、滚动升级
]# kubectl set image deployment/nginx-deployment nginx-container=nginx:04.27 --record
2、查看滚动升级后端的pod和image
]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-deployment-6cff55d5cc-299sr 1/1 Running 0 29s 10.10.36.70 k8s-node1 <none> <none> nginx-deployment-6cff55d5cc-nsdmp 1/1 Running 0 24s 10.10.169.135 k8s-node2 <none> <none> nginx-deployment-canary-c4884b965-bkzt9 1/1 Running 0 12m 10.10.36.69 k8s-node1 <none> <none> nginx-deployment-canary-c4884b965-gkjzt 1/1 Running 0 12m 10.10.169.134 k8s-node2 <none> <none> ]# kubectl describe pods nginx-deployment-6cff55d5cc-299sr | grep 'Image:' Image: nginx:04.27
3、使用nodeIP+ingressPort访问服务
]# curl 10.1.1.11:32080 04.26 ]# curl -H "user_id: user_2" 10.1.1.11:32080 04.26 ]# curl -H "user-id: user_2" 10.1.1.11:32080 04.27 ]# curl -H "user-id: user-3" 10.1.1.11:32080 04.27 ]# curl -H "user-id: user4" 10.1.1.11:32080 04.27
2.4、删除灰度服务的ingress
1、删除灰度服务的ingress
]# kubectl delete -f canary-nginx-ingress.yaml ]# kubectl get ingress -o wide NAME CLASS HOSTS ADDRESS PORTS AGE nginx-ingress <none> * 10.1.1.13 80 35m
2、使用nodeIP+ingressPort访问服务
]# curl 10.1.1.11:32080 04.26 ]# curl -H "user_id: user_2" 10.1.1.11:32080 04.26 ]# curl -H "user-id: user_2" 10.1.1.11:32080 04.26 ]# curl -H "user-id: user-3" 10.1.1.11:32080 04.26 ]# curl -H "user-id: user4" 10.1.1.11:32080 04.26
3、ingress基于权重实现灰度发布
1、创建Ingress的yaml文件(canary-weight-nginx-ingress.yaml)
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx-ingress-canary-weight annotations: kubernetes.io/ingress.class: "nginx" #必须有,否则创建出的ingress可能不正常 nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "20" #20%的请求会被转发到灰度版本,80%的请求会被转发到非灰度版本 spec: rules: - http: paths: - path: / pathType: Prefix backend: service: name: nginx-service-canary port: number: 11180
2、创建Ingress
]# kubectl apply -f canary-weight-nginx-ingress.yaml
3、查看Ingress
]# kubectl get ingress -o wide NAME CLASS HOSTS ADDRESS PORTS AGE nginx-ingress <none> * 10.1.1.13 80 72m nginx-ingress-canary-weight <none> * 10.1.1.13 80 45s
4、使用nodeIP+ingressPort访问服务
]# for i in {1..100}; do curl 10.1.1.11:32080 2>/dev/null; done