Ingress-nginx进阶
一、Ingress-nginx和Nginx-ingress的区别
Ingress-nginx：kubernetes官方维护的ingress
Nginx-ingress：nginx官方维护的ingress
# Ingress-nginx的官方文档：
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#rewrite
# Nginx-ingress的官方文档：
https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
# Ingress-nginx源码地址
https://github.com/kubernetes/ingress-nginx
# Nginx-ingress源码地址
Github：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/nginx-ingress-controllers.md
部署建议：
DaemonSet，找几台专门的服务器进行配置ingress (如果没有的足够的资源，就设置QoS，保证ingress最后删除的那个策略)
hostNetwork: true # 这个设置为true
二、常用ingress配置
2.1、创建一个简单的ingress实例
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-test
namespace: ratel-test1
spec:
rules:
- host: ingress.test.com
http:
paths:
- backend:
serviceName: ingress-test # 代理名字为ingress-test
servicePort: 80 # port为80的svc
path: /
# 创建ingress
kubectl create -f ingress-demo.yaml
2.2、Redirect
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com # 重定向到想去的url
name: ingress-test
namespace: ratel-test1
spec:
rules:
- host: ingress.test.com
http:
paths:
- backend:
serviceName: ingress-test
servicePort: 80
path: /
kind: List
metadata:
resourceVersion: ""
selfLink: ""
2.3、Rewrite
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
generation: 4
name: ingress-test
namespace: ratel-test1
spec:
rules:
- host: rewrite.test.com
http:
paths:
- backend:
serviceName: ingress-test
servicePort: 80
path: /something(/|$)(.*) #($1)($2)
2.4、https
# 官网：https://kubernetes.github.io/ingress-nginx/user-guide/tls/
# 禁用https强制跳转
nginx.ingress.kubernetes.io/ssl-redirect: "false"
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false" # 禁用https强制跳转
generation: 1
name: test-tls
namespace: ratel-test1
spec:
rules:
- host: test-tls.test.com
http:
paths:
- backend:
serviceName: ingress-test
servicePort: 80
path: /
tls:
- hosts:
- test-tls.test.com
secretName: ca-cert
设置默认证书：--default-ssl-certificate=default/foo-tls
更改的ingress-controller的启动参数
2.5、Dashboard自定义证书
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: normal
operator: In
values:
- "true"
containers:
- args:
- --auto-generate-certificates=false
- --tls-key-file=server.key
- --tls-cert-file=server.pem
- --token-ttl=21600
- --authentication-mode=basic,token
- --namespace=kubernetes-dashboard
image: kubernetesui/dashboard:v2.0.0-rc5
imagePullPolicy: Always
lifecycle: {}
livenessProbe:
failureThreshold: 3
httpGet:
path: /
port: 8443
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 30
name: kubernetes-dashboard
ports:
- containerPort: 8443
protocol: TCP
resources: {}
securityContext:
privileged: false
procMount: Default
runAsNonRoot: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /certs
name: kubernetes-dashboard-new
- mountPath: /tmp
name: tmp-volume
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: kubernetes-dashboard
serviceAccountName: kubernetes-dashboard
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
volumes:
- name: kubernetes-dashboard-new
secret:
defaultMode: 420
secretName: kubernetes-dashboard-new
- emptyDir: {}
name: tmp-volume
2.7、黑白名单
黑名单：拒绝某段IP访问
白名单：只允许某段IP访问
Annotations：只对指定的ingress生效
ConfigMap：全局生效
黑名单可以使用ConfigMap去配置，白名单建议使用Annotations去配置
2.7.1、白名单配置（建议使用Annotations）
# 官网：https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#whitelist-source-range
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range 10.0.0.0/24,172.10.0.1 # 后面可以跟一个或者多个IP
2.7.2、黑名单设置（建议使用ConfigMap）（这是全局生效的）
# 因为黑名单可能会不定时加上去，防止恶意攻击的。所以用ConfigMap（热更新）
# 1、更改ingress-nginx的cm
[root@k8s-master01 ~]# kubectl edit cm -n ingress-nginx ingress-nginx-controller -oyaml
apiVersion: v1
data: # 加上data
block-cidrs: 192.168.1.201 # 加上block-cidrs，后面也可以跟多个IP，隔开
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/version: 0.43.0
helm.sh/chart: ingress-nginx-3.20.1
name: ingress-nginx-controller
namespace: ingress-nginx
# 2、删除ingress-nginx的pod，重新加载配置
[root@k8s-master01 ~]# kubectl get po -n ingress-nginx
[root@k8s-master01 ~]# kubectl delete po -n ingress-nginx --all # 生产一个一个删，防止配置错误都挂了
# 3、在192.168.1.201节点上，访问ingress代理的域名，然后403表示配置成功
2.7.2、针对某个域名设置黑名单--snippet
官网参考：https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#canary
# 比如ingress代理了www.test.com这个域名，那么想针对这个域名（www.test.com）设置访问黑名单，就编辑这个ingress即可
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
# 在annotations下面加上这几行配置，有多个IP可以deny多个
annotations:
nginx.ingress.kubernetes.io/server-snippet: |-
deny 192.168.1.101;
deny 192.168.1.102;
allow all;
# 然后在deny的主机上访问 www.test.com 就403
[root@k8s-master02 ~]# curl ngdemo.qikqiak.com
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
2.8、匹配请求头设置
annotations:
nginx.ingress.kubernetes.io/server-snippet: |
set $agentflag 0;
if ($http_user_agent ~* "(iPhone)" ){ # 匹配规则设置（设置匹配为iPhone的手机，则重定向到下面的url）
set $agentflag 1;
}
if ( $agentflag = 1 ) {
return 301 https://www.baidu.com; # 重定向到指定的url
}
2.8、速率限制，其他的也一样参考官网吧
官网参考：https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#canary
2.9、ingress-nginx基本认证（Basic Auth）
同样我们还可以在 Ingress Controller 上面配置一些基本的 Auth 认证，比如 Basic Auth，可以用 htpasswd 生成一个密码文件来验证身份验证。
[root@k8s-master01 ~]# htpasswd -c auth foo # 账号foo 密码 123456
New password: # 123456
Re-type new password: # 123456
Adding password for user foo
# 生成一个auth文件
[root@k8s-master01 ~]# ls auth
auth
然后根据上面的 auth 文件创建一个 secret 对象：
[root@k8s-master01 ~]# kubectl create secret generic basic-auth --from-file=auth
secret/basic-auth created
[root@k8s-master01 ~]# kubectl get secret basic-auth -o yaml
apiVersion: v1
data:
auth: Zm9vOiRhcHIxJGZzREw0b0xmJHNuUUNDTFkxbTE2N1BkNUdEMHIwcC8K
kind: Secret
metadata:
name: basic-auth
namespace: default
type: Opaque
然后对上面的 my-nginx 应用创建一个具有 Basic Auth 的 Ingress 对象：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-with-auth
annotations:
# 认证类型
nginx.ingress.kubernetes.io/auth-type: basic
# 包含 user/password 定义的 secret 对象名
nginx.ingress.kubernetes.io/auth-secret: basic-auth
# 要显示的带有适当上下文的消息，说明需要身份验证的原因
nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo'
spec:
rules:
- host: foo.bar.com
http:
paths:
- path: /
backend:
serviceName: my-nginx
servicePort: 80
直接创建上面的资源对象，然后通过下面的命令或者在浏览器中直接打开配置的域名：
➜ curl -v http://k8s.qikqiak.com -H 'Host: foo.bar.com'
* Rebuilt URL to: http://k8s.qikqiak.com/
* Trying 123.59.188.12...
* TCP_NODELAY set
* Connected to k8s.qikqiak.com (123.59.188.12) port 80 (#0)
> GET / HTTP/1.1
> Host: foo.bar.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Server: openresty/1.15.8.2
< Date: Sun, 08 Dec 2019 06:44:35 GMT
< Content-Type: text/html
< Content-Length: 185
< Connection: keep-alive
< WWW-Authenticate: Basic realm="Authentication Required - foo"
<
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>openresty/1.15.8.2</center>
</body>
</html>
我们可以看到出现了 401 认证失败错误，然后带上我们配置的用户名和密码进行认证：
➜ curl -v http://k8s.qikqiak.com -H 'Host: foo.bar.com' -u 'foo:foo'
* Rebuilt URL to: http://k8s.qikqiak.com/
* Trying 123.59.188.12...
* TCP_NODELAY set
* Connected to k8s.qikqiak.com (123.59.188.12) port 80 (#0)
* Server auth using Basic with user 'foo'
> GET / HTTP/1.1
> Host: foo.bar.com
> Authorization: Basic Zm9vOmZvbw==
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.2
< Date: Sun, 08 Dec 2019 06:46:27 GMT
< Content-Type: text/html
< Content-Length: 612
< Connection: keep-alive
< Vary: Accept-Encoding
< Last-Modified: Tue, 19 Nov 2019 12:50:08 GMT
< ETag: "5dd3e500-264"
< Accept-Ranges: bytes
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
可以看到已经认证成功了。当然出来 Basic Auth 这一种简单的认证方式之外，NGINX Ingress Controller 还支持一些其他高级的认证，比如 OAUTH 认证之类的。
三、灰度发布
3.1、准备2个svc，用于演示
[root@k8s-master01 app]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-nginx ClusterIP 10.104.87.14 <none> 80/TCP 91m
my-nginx1 ClusterIP 10.103.175.67 <none> 80/TCP 2m12s
[root@k8s-master01 app]# curl 10.104.87.14
v1
[root@k8s-master01 app]# curl 10.103.175.67
v2
3.2、开启基于ingress的灰度发布
官网参考：https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#canary
# 开启了灰度发布，才能在同一个ns下创建2个同样域名
# 先创建一个普通的ingress，通过canary.test.com就可以访问到svc my-nginx，版本是v1
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-nginx
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: canary.test.com # 将域名映射到 my-nginx 服务
http:
paths:
- path: /
backend:
serviceName: my-nginx # 将所有请求发送到 my-nginx 服务的 80 端口
servicePort: 80
3.2.1、基于权重的流量调度
基于权重：基于权重的流量切分的典型应用场景就是蓝绿部署，可通过将权重设置为 0 或 100 来实现。例如，可将 Green 版本设置为主要部分，并将 Blue 版本的入口配置为 Canary。最初，将权重设置为 0，因此不会将流量代理到 Blue 版本。一旦新版本测试和验证都成功后，即可将 Blue 版本的权重设置为 100，即所有流量从 Green 版本转向 Blue。
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-nginx1
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: canary.test.com
http:
paths:
- path: /
backend:
serviceName: my-nginx1
servicePort: 80
# 这个是代理V2的ingress，如果使用相同域名，创建会报错
[root@k8s-master01 app]# kubectl apply -f bbb.yaml
Error from server (BadRequest): error when creating "bbb.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "canary.test.com" and path "/" is already defined in ingress default/my-nginx
# 基于权重
annotations:
nginx.ingress.kubernetes.io/canary: "true" # 要开启灰度发布机制，首先需要启用 Canary
nginx.ingress.kubernetes.io/canary-weight: "30" # 切30%的流量到v2去，设置100就全部切过去了
# 创建后查看ingress，可以看到创建了2个代理相同域名的ingress
[root@k8s-master01 app]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
my-nginx <none> canary.test.com 10.101.29.125 80 20m
my-nginx1 <none> canary.test.com 10.101.29.125 80 3m6s
验证是否配置成功：
[root@k8s-master02 ~]# for i in $(seq 1 3); do curl -s -H canary.test.com; done
v1
v2
v1
3.2.2、基于Request Header（还有个never、always不进行演示）
基于 Request Header: 基于 Request Header 进行流量切分的典型应用场景即灰度发布或 A/B 测试场景
注意：当 Request Header 设置为 never 或 always 时，请求将不会或一直被发送到 Canary 版本，对于任何其他 Header 值，将忽略 Header，并通过优先级将请求与其他 Canary 规则进行优先级的比较。
# 基于 Request Header
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true" # 要开启灰度发布机制，首先需要启用 Canary
nginx.ingress.kubernetes.io/canary-by-header-value: canary # 基于header的流量切分 value
nginx.ingress.kubernetes.io/canary-by-header: user # 基于header的流量切分 key
nginx.ingress.kubernetes.io/canary-weight: "30" # 会被忽略，因为配置了 canary-by-headerCanary版本
验证：
➜ for i in $(seq 1 10); do curl -s -H "canary: never" echo.qikqiak.com | grep "Hostname"; done
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
# 流量全部切到V2了，从而实现灰度发布
[root@k8s-master02 ~]# for i in $(seq 1 5); do curl -s -H "user: canary" canary.test.com; done
v2
v2
v2
v2
v2
3.2.3、基于 Cookie
基于 Cookie：与基于 Request Header 的 annotation 用法规则类似。例如在 A/B 测试场景下，需要让地域为北京的用户访问 Canary 版本。那么当 cookie 的 annotation 设置为 nginx.ingress.kubernetes.io/canary-by-cookie: "users_from_Beijing"，此时后台可对登录的用户请求进行检查，如果该用户访问源来自北京则设置 cookie users_from_Beijing 的值为 always，这样就可以确保北京的用户仅访问 Canary 版本。
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true" # 要开启灰度发布机制，首先需要启用 Canary
nginx.ingress.kubernetes.io/canary-by-cookie: "users_from_Beijing" # 基于 cookie
nginx.ingress.kubernetes.io/canary-weight: "30" # 会被忽略，因为配置了 canary-by-cookie
➜ for i in $(seq 1 10); do curl -s -b "users_from_Beijing=always" echo.qikqiak.com | grep "Hostname"; done
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
四、ingress-nginx监控
https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/
https://github.com/kubernetes/ingress-nginx/tree/master/deploy
https://github.com/kubernetes/ingress-nginx/tree/master/deploy/grafana/dashboards
