Kubernetes-基于ingress实现灰度发布
1、基于ingress实现灰度发布介绍
- Ingress-Nginx是一个K8S ingress工具，支持配置Ingress Annotations来实现不同场景下的灰度发布和测试。
1.1、Nginx Annotations支持以下4种Canary规则
- nginx.ingress.kubernetes.io/canary：开启灰度发布功能，如果没有开启此属性，则如下属性不生效。
- nginx.ingress.kubernetes.io/canary-by-header：基于Request Header的流量切分，适用于灰度发布以及A/B测试。当Request Header设置为 always时，请求将会被一直发送到Canary版本；当 equest Header设置为never时，请求不会被发送到Canary入口；对于任何其他Header值，将忽略Header，并通过优先级将请求与其他金丝雀规则进行优先级的比较。
- nginx.ingress.kubernetes.io/canary-by-header-value：要匹配的Request Header的值，用于通知Ingress将请求路由到Canary Ingress中指定的服务。当Request Header设置为此值时，它将被路由到Canary入口。该规则允许用户自定义Request Header的值，必须与上一个annotation (即canary-by-header）一起使用。
- nginx.ingress.kubernetes.io/canary-by-header-pattern：正则表达式匹配的Request Header的值，用于通知Ingress将请求路由到Canary Ingress中指定的服务。当Request Header设置为此值时，它将被路由到Canary入口。该规则允许用户自定义Request Header的值，必须与上一个annotation(即canary-by-header）一起使用，如果已经使用canary-by-header-value，那么此属性将被忽略。
- nginx.ingress.kubernetes.io/canary-by-cookie：基于Cookie的流量切分，适用于灰度发布与A/B测试。用于通知Ingress将请求路由到Canary Ingress中指定的服务的cookie。当cookie值设置为always时，它将被路由到Canary入口；当cookie值设置为never时，请求不会被发送到Canary入口；对于任何其他值，将忽略cookie并将请求与其他金丝雀规则进行优先级的比较。
- nginx.ingress.kubernetes.io/canary-weight：基于服务权重的流量切分，适用于蓝绿部署，权重范围0-100按百分比将请求路由到Canary Ingress中指定的服务。权重为0意味着该金丝雀规则不会向Canary入口的服务发送任何请求。权重为100意味着所有请求都将被发送到Canary入口。
- 金丝雀规则按优先顺序进行如下排序：
- canary-by-header - > canary-by-cookie - > canary-weight
1.2、基于权重或用户实现灰度发布
- 可以把上面四种annotation规则划分为两类。
2、ingress基于用户实现灰度发布
- 基于不同的场景，灰度发布有四种：
//第一种，所有的请求都会被转发到灰度（Canary）版本
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "always"
//第二种，所有的请求都不会被转发到灰度（Canary）版本
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "never"
//第三种，如果请求的header头包含"user-id: user_1"，该请求会被转发到灰度（Canary）版本
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "user_id"
nginx.ingress.kubernetes.io/canary-by-header-value: "user_1"
//第四种，如果请求的header头包含"user-id: user_2"或"user-id: user-3"或"user-id: user4"，该请求会被转发到灰度（Canary）版
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "user_id" #在networking.k8s.io/v1中，canary-by-header的值尽量不用使用下划线“_”。如果使用了，请求头中要使用中横线“-”代替。
nginx.ingress.kubernetes.io/canary-by-header-pattern: "user_2|user-3|user4"
2.1、发布现网服务
1、创建Deployment和Service的yaml文件（nginx-deployment.yaml）
apiVersion: v1 kind: Service metadata: name: nginx-service spec: selector: app: nginx-pod ports: - port: 11180 targetPort: 80 protocol: TCP --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx-pod replicas: 2 strategy: #! type: RollingUpdate rollingUpdate: #先减一个pod，再加一个pod maxSurge: 0 maxUnavailable: 1 template: metadata: labels: app: nginx-pod spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx-pod topologyKey: "kubernetes.io/hostname" volumes: - name: html hostPath: path: /apps/html/ containers: - name: nginx-container image: nginx:04.26 imagePullPolicy: Never command: ["/usr/sbin/nginx", "-g", "daemon off;"] ports: - containerPort: 80 volumeMounts: - mountPath: /usr/share/nginx/html name: html
- 准备镜像和index.html文件
]# docker image tag nginx:latest nginx:04.26 ]# echo "04.26" > /apps/html/index.html
2、创建Ingress的yaml文件（nginx-ingress.yaml）
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
kubernetes.io/ingress.class: "nginx" #必须有，否则创建出的ingress可能不正常
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 11180
3、创建Deployment、Service和Ingress
]# kubectl apply -f nginx-deployment.yaml ]# kubectl apply -f nginx-ingress.yaml
4、查看Deployment、Pod、Service和Ingress
]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-deployment-7448cc74f6-t5gbd 1/1 Running 0 49s 10.10.36.68 k8s-node1 <none> <none> nginx-deployment-7448cc74f6-znfzv 1/1 Running 0 49s 10.10.169.133 k8s-node2 <none> <none> ]# kubectl get deployment -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR nginx-deployment 2/2 2 2 34s nginx-container nginx:04.26 app=nginx-pod ]# kubectl get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR nginx-service ClusterIP 10.20.50.80 <none> 11180/TCP 57s app=nginx-pod ]# kubectl get ingress -o wide NAME CLASS HOSTS ADDRESS PORTS AGE nginx-ingress <none> * 10.1.1.13 80 44s
5、使用nodeIP+ingressPort访问服务
]# curl 10.1.1.11:32080 04.26
2.2、发布灰度服务
1、创建Deployment和Service的yaml文件（canary-nginx-deployment.yaml）
apiVersion: v1 kind: Service metadata: name: nginx-service-canary spec: selector: app: nginx-pod-canary ports: - port: 11180 targetPort: 80 protocol: TCP --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment-canary spec: selector: matchLabels: app: nginx-pod-canary replicas: 2 strategy: #! type: RollingUpdate rollingUpdate: #先减一个pod，再加一个pod maxSurge: 0 maxUnavailable: 1 template: metadata: labels: app: nginx-pod-canary spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx-pod-canary topologyKey: "kubernetes.io/hostname" volumes: - name: html hostPath: path: /apps/html/canary/ containers: - name: nginx-container-canary image: nginx:04.27 imagePullPolicy: Never command: ["/usr/sbin/nginx", "-g", "daemon off;"] ports: - containerPort: 80 volumeMounts: - mountPath: /usr/share/nginx/html name: html
- 准备镜像和index.html文件
]# docker image tag nginx:latest nginx:04.27 ]# echo "04.27" > /apps/html/canary/index.html
2、创建Ingress的yaml文件（canary-nginx-ingress.yaml）
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress-canary
annotations:
kubernetes.io/ingress.class: "nginx" #必须有，否则创建出的ingress可能不正常
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "user_id" #如果请求的header头包含"user-id: user_2"或"user-id: user-3"或"user-id: user4"，该请求会被转发到灰度（Canary）版
nginx.ingress.kubernetes.io/canary-by-header-pattern: "user_2|user-3|user4"
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-service-canary
port:
number: 11180
3、创建Deployment、Service和Ingress
]# kubectl apply -f canary-nginx-deployment.yaml ]# kubectl apply -f canary-nginx-ingress.yaml
4、查看Deployment、Pod、Service和Ingress
]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-deployment-7448cc74f6-t5gbd 1/1 Running 0 13m 10.10.36.68 k8s-node1 <none> <none> nginx-deployment-7448cc74f6-znfzv 1/1 Running 0 13m 10.10.169.133 k8s-node2 <none> <none> nginx-deployment-canary-c4884b965-bkzt9 1/1 Running 0 38s 10.10.36.69 k8s-node1 <none> <none> nginx-deployment-canary-c4884b965-gkjzt 1/1 Running 0 38s 10.10.169.134 k8s-node2 <none> <none> ]# kubectl get deployment -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR nginx-deployment 2/2 2 2 13m nginx-container nginx:04.26 app=nginx-pod nginx-deployment-canary 2/2 2 2 29s nginx-container-canary nginx:04.27 app=nginx-pod-canary ]# kubectl get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR nginx-service ClusterIP 10.20.50.80 <none> 11180/TCP 13m app=nginx-pod nginx-service-canary ClusterIP 10.20.227.2 <none> 11180/TCP 43s app=nginx-pod-canary ]# kubectl get ingress -o wide NAME CLASS HOSTS ADDRESS PORTS AGE nginx-ingress <none> * 10.1.1.13 80 14m nginx-ingress-canary <none> * 10.1.1.13 80 107s
5、使用nodeIP+ingressPort访问服务
]# curl 10.1.1.11:32080 04.26 //虽然nginx.ingress.kubernetes.io/canary-by-header: "user_id"，但在访问时要使用user-id。尽量不要使用下划线“_” //nginx.ingress.kubernetes.io/canary-by-header-pattern: "user_2|user-3|user4",可以使用下划线“_”和中划线“-” ]# curl -H "user_id: user_2" 10.1.1.11:32080 04.26 ]# curl -H "user-id: user_2" 10.1.1.11:32080 04.27 ]# curl -H "user-id: user-3" 10.1.1.11:32080 04.27 ]# curl -H "user-id: user4" 10.1.1.11:32080 04.27
2.3、滚动升级现网服务
1、滚动升级
]# kubectl set image deployment/nginx-deployment nginx-container=nginx:04.27 --record
2、查看滚动升级后端的pod和image
]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-6cff55d5cc-299sr 1/1 Running 0 29s 10.10.36.70 k8s-node1 <none> <none>
nginx-deployment-6cff55d5cc-nsdmp 1/1 Running 0 24s 10.10.169.135 k8s-node2 <none> <none>
nginx-deployment-canary-c4884b965-bkzt9 1/1 Running 0 12m 10.10.36.69 k8s-node1 <none> <none>
nginx-deployment-canary-c4884b965-gkjzt 1/1 Running 0 12m 10.10.169.134 k8s-node2 <none> <none>
]# kubectl describe pods nginx-deployment-6cff55d5cc-299sr | grep 'Image:'
Image: nginx:04.27
3、使用nodeIP+ingressPort访问服务
]# curl 10.1.1.11:32080 04.26 ]# curl -H "user_id: user_2" 10.1.1.11:32080 04.26 ]# curl -H "user-id: user_2" 10.1.1.11:32080 04.27 ]# curl -H "user-id: user-3" 10.1.1.11:32080 04.27 ]# curl -H "user-id: user4" 10.1.1.11:32080 04.27
2.4、删除灰度服务的ingress
1、删除灰度服务的ingress
]# kubectl delete -f canary-nginx-ingress.yaml ]# kubectl get ingress -o wide NAME CLASS HOSTS ADDRESS PORTS AGE nginx-ingress <none> * 10.1.1.13 80 35m
2、使用nodeIP+ingressPort访问服务
]# curl 10.1.1.11:32080 04.26 ]# curl -H "user_id: user_2" 10.1.1.11:32080 04.26 ]# curl -H "user-id: user_2" 10.1.1.11:32080 04.26 ]# curl -H "user-id: user-3" 10.1.1.11:32080 04.26 ]# curl -H "user-id: user4" 10.1.1.11:32080 04.26
3、ingress基于权重实现灰度发布
1、创建Ingress的yaml文件（canary-weight-nginx-ingress.yaml）
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress-canary-weight
annotations:
kubernetes.io/ingress.class: "nginx" #必须有，否则创建出的ingress可能不正常
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "20" #20%的请求会被转发到灰度版本，80%的请求会被转发到非灰度版本
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-service-canary
port:
number: 11180
2、创建Ingress
]# kubectl apply -f canary-weight-nginx-ingress.yaml
3、查看Ingress
]# kubectl get ingress -o wide NAME CLASS HOSTS ADDRESS PORTS AGE nginx-ingress <none> * 10.1.1.13 80 72m nginx-ingress-canary-weight <none> * 10.1.1.13 80 45s
4、使用nodeIP+ingressPort访问服务
]# for i in {1..100}; do curl 10.1.1.11:32080 2>/dev/null; done
