Kubernetes 使用Nginx-Ingress实现蓝绿发布/金丝雀发布/AB测试
https://juejin.cn/post/6844903927318577159
背景介绍
某些情况下,我们在使用Kubernetes作为业务应用的云平台,想要实现应用的蓝绿部署用来迭代应用版本,用lstio太重太复杂,而且它本身定位于流控和网格治理;Ingress-Nginx在0.21版本引入了Canary功能,可以为网关入口配置多个版本的应用程序,使用annotation来控制多个后端服务的流量分配
Ingress-Nginx-Annotation Canary 功能介绍
如果想启用Canary功能,要先设置
nginx.ingress.kubernetes.io/canary: "true"
,然后可以启用以下注释来配置Canary
nginx.ingress.kubernetes.io/canary-weight
请求到Canary ingress中指定的服务的请求百分比,值为0-100的整数,根据设置的值来决定大概有百分之多少的流量会分配Canary Ingress中指定的后端s服务nginx.ingress.kubernetes.io/canary-by-header
基于request header 的流量切分,适用于灰度发布或者A/B测试,当设定的hearder值为always是,请求流量会被一直分配到Canary入口,当hearder值被设置为never时,请求流量不会分配到Canary入口,对于其他hearder值,将忽略,并通过优先级将请求流量分配到其他规则nginx.ingress.kubernetes.io/canary-by-header-value
这个配置要和nginx.ingress.kubernetes.io/canary-by-header
一起使用,当请求中的hearder key和value 和nginx.ingress.kubernetes.io/canary-by-header
nginx.ingress.kubernetes.io/canary-by-header-value
匹配时,请求流量会被分配到Canary Ingress入口,对于其他任何hearder值,将忽略,并通过优先级将请求流量分配到其他规则nginx.ingress.kubernetes.io/canary-by-cookie
这个配置是基于cookie的流量切分,也适用于灰度发布或者A/B测试,当cookie值设置为always时,请求流量将被路由到Canary Ingress入口,当cookie值设置为never时,请求流量将不会路由到Canary入口,对于其他值,将忽略,并通过优先级将请求流量分配到其他规则
金丝雀规则按优先顺序进行如下排序:canary-by-header - > canary-by-cookie - > canary-weight
1.基于权重的小规模版本测试
- v1版本编排文件
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
labels:
app: echoserverv1
name: echoserverv1
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv1
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv1
namespace: echoserver
spec:
selector:
name: echoserverv1
type: ClusterIP
ports:
- name: echoserverv1
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv1
namespace: echoserver
labels:
name: echoserverv1
spec:
template:
metadata:
labels:
name: echoserverv1
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv1
ports:
- containerPort: 8080
name: echoserverv1
复制代码
- 查看v1版本创建的资源
$ [K8sSj] kubectl get pod,service,ingress -n echoserver
NAME READY STATUS RESTARTS AGE
pod/echoserverv1-657b966cb5-7grqs 1/1 Running 0 24h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/echoserverv1 ClusterIP 10.99.68.72 <none> 8080/TCP 24h
NAME HOSTS ADDRESS PORTS AGE
ingress.extensions/echoserverv1 echo.chulinx.com 80 24h
复制代码
- 访问v1的服务,可以看到10个请求都是访问到一个pod上也就是v1版本的服务
$ [K8sSj] for i in `seq 10`;do curl -s echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
复制代码
- 创建v2版本的服务
我们开启canary功能,将v2版本的权重设置为50%,这个百分比并不能精确的将请求平均分配到两个版本的服务,而是在50%上下浮动
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
复制代码
- 再次查看创建的资源
$ [K8sSj] kubectl get pod,service,ingress -n echoserver
NAME READY STATUS RESTARTS AGE
pod/echoserverv1-657b966cb5-7grqs 1/1 Running 0 24h
pod/echoserverv2-856bb5758-f9tqn 1/1 Running 0 4s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/echoserverv1 ClusterIP 10.99.68.72 <none> 8080/TCP 24h
service/echoserverv2 ClusterIP 10.111.103.170 <none> 8080/TCP 4s
NAME HOSTS ADDRESS PORTS AGE
ingress.extensions/echoserverv1 echo.chulinx.com 80 24h
ingress.extensions/echoserverv2 echo.chulinx.com 80 4s
复制代码
- 访问测试
可以看到请求有4个落到v2版本,6个落到v1版本,理论上来说,请求说越多,落到v2版本的请求数越接近设置的权重50%
$ [K8sSj] for i in `seq 10`;do curl -s echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
复制代码
2.基于header的A/B测试
- 更改v2版本的编排文件
增加header
nginx.ingress.kubernetes.io/canary-by-header: "v2"
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
nginx.ingress.kubernetes.io/canary-by-header: "v2"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
复制代码
- 更新访问测试
测试了header 为
v2:always
v2:never
v2:true
这三个hearder值,可以看到当hearder为v2:always
时,流量会全部流入v2,当v2:never
时,流量会全部流入v1,当v2:true
时,也就是非always/never
,流量会按照配置的权重流入对应版本的服务
$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:never" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:true" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
复制代码
- 自定义header-value
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
nginx.ingress.kubernetes.io/canary-by-header: "v2"
nginx.ingress.kubernetes.io/canary-by-header-value: "true"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
复制代码
- 更新测试
可以看到只有header为
v2:never
时,请求流量才会流入v2版本,其他值流量都会按照权重设置流入不通版本的服务
$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:true" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:never" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
复制代码
3.基于cookie的流控
cookie其实和header原理大致相同,也是ingress自动cookie值,客户访问如果cookie匹配,流量就会流入与之匹配的后端服务
- 更新v2版本的编排文件
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
nginx.ingress.kubernetes.io/canary-by-header: "v2"
nginx.ingress.kubernetes.io/canary-by-header-value: "true"
nginx.ingress.kubernetes.io/canary-by-cookie: "user_from_shanghai"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
复制代码
- 访问测试
可以看和header的访问效果是一样的,只不过cookie不能自定义value
$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged
$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
# zlx @ zlxdeMacBook-Pro in ~/Desktop/unicom/k8syml/nginx-ingress-canary-deployment [16:01:52]
$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
# zlx @ zlxdeMacBook-Pro in ~/Desktop/unicom/k8syml/nginx-ingress-canary-deployment [16:02:25]
$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai=always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn