Ingress
Service一般情况下只作用于内部Pod的代理调度,就算有NodePort类型,其访问节点相对复杂,流程大概如下:
但是我们知道,如果只指定一个NodeIP,随着业务量增大,这个Node的压力就会很大,所以我们可能会在前端再加一个代理,代理几个Node,比如我们在前端加一个NG,流程如下:
在应用小的情况下,这种架构虽然调度复杂,但是也可以使用,但是如果有大量应用,这种管理就非常麻烦,因为 我们要管理大量的NodePort,这个时候使用Ingress就非常方便。
再则,Service的转发不论是iptables还是ipvs,都是4层的,但是在实际中基本都https请求,https是7层,4层是没办法对起进行SSL校验的,如果我们是第二幅流程图,我们可以在前置NG上配置SSL,但是如果我们是第一幅图的流程,我们只能在Pod上配置SSL,因为Service上是无法进行校验,那么就会出现一个问题,SSL的校验是很耗资源的,我们的客户端访问Pod,如果Pod非常多并且访问模式是轮询,那么每访问一次就要做一次SSL校验,这就非常不科学,我们就只有用类似于第二副图的架构,Ingress可以很完美的解决这种问题。
一、Ingress
流程图如上,其中Ingress代理的并不是Pod的Service,而是Pod,之所以在配置的时候是配置的Service,是为了获取Pod的信息。
Ingress提供外部访问集群的入口,将外部的HTTP或者HTTPS请求转发到集群内Service上,流量规则是在Ingress资源上定义。
配置Ingress资源的必要条件是你的kubernetes集群种由Ingress controller。其中Ingress Controller常用的有如下:
- HAProxy Ingress Controller
- Nginx Ingress Controller
- Traefik Ingress Controller
- Kong Ingress Controller
其中最常用的是Nginx Controller和Traefik Ingress Controller。
定义一个简单的Ingresss:
[root@master ingress]# cat ingress-simple-daemo.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-simple-daemo annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - http: paths: - path: /joker backend: serviceName: nginx servicePort: 80
简要说明:
apiVersion,kind,metadata,spec都是Kubernetes YAML文件的标准字段,Ingress经常通过annotations来配置一些选项,比如rewrite-target,不同的Ingress Controller支持不同的annotations。对于规则而言,每个HTTP都有如下规则:
- 主机:主机是可选参数,如果不配置表示适用于所有主机HTTP通信,如果配置了表示只适用于该主机;
- 路径:类似于NG的location,每个路径后面都有后端ServiceName和ServicePort;
- 后端:后端是ServiceName和ServicePort组合,符合该规则的流量会转发到这个后端Service上。通常会在Ingress中配置默认后端,以匹配任何不符合规则的请求流量转发;
- 具体的语法规则可以通过kubectl explain ingress来查看。
1.1、Ingress 类型
1.1.1、单服务Ingress
Kubernetes中已经存在一些概念可以暴露单个service(查看替代方案),但是你仍然可以通过Ingress来实现,通过指定一个没有rule的默认backend的方式。比如:
[root@master ingress]# cat single-service-ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: single-service-ingress spec: backend: serviceName: nginx servicePort: 80
1.1.2、简单展开
如前面描述的那样,kubernete pod中的IP只在集群网络内部可见,我们需要在边界设置一个东西,让它能够接收ingress的流量并将它们转发到正确的端点上。这个东西一般是高可用的loadbalancer。使用Ingress能够允许你将loadbalancer的个数降低到最少,例如,假如你想要创建这样的一个设置:
foo.bar.com -> 178.91.123.132 -> / foo service1:4200
/ bar service2:8080
我们就可以这样配置Ingress:
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: simple-fanout-example annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: foo.bar.com http: paths: - path: /foo backend: serviceName: service1 servicePort: 4200 - path: /bar backend: serviceName: service2 servicePort: 8080
1.1.3、基于名称的虚拟主机
如果想实现下面这种需求:
foo.bar.com --| |-> foo.bar.com s1:80
| 178.91.123.132 |
bar.foo.com --| |-> bar.foo.com s2:80
我们就可以这样配置Ingress:
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: name-virtual-host-ingress spec: rules: - host: foo.bar.com http: paths: - backend: serviceName: service1 servicePort: 80 - host: bar.foo.com http: paths: - backend: serviceName: service2 servicePort: 80
如果要增加配置默认的backend,可以配置成如下Ingress:
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: name-virtual-host-ingress spec: rules: - host: first.bar.com http: paths: - backend: serviceName: service1 servicePort: 80 - host: second.foo.com http: paths: - backend: serviceName: service2 servicePort: 80 - http: paths: - backend: serviceName: service3 servicePort: 80
默认backend:一个没有rule的ingress,所有流量都将发送到一个默认backend。你可以用该技巧通知loadbalancer如何找到你网站的404页面,通过制定一些列rule和一个默认backend的方式。如果请求header中的host不能跟ingress中的host匹配,并且/或请求的URL不能与任何一个path匹配,则流量将路由到你的默认backend。
1.1.4、TLS
你可以通过指定包含TLS私钥和证书的secret来加密Ingress。 目前,Ingress仅支持单个TLS端口443,并假定TLS termination。 如果Ingress中的TLS配置部分指定了不同的主机,则它们将根据通过SNI TLS扩展指定的主机名(假如Ingress controller支持SNI)在多个相同端口上进行复用。 TLS secret中必须包含名为tls.crt
和tls.key
的密钥,这里面包含了用于TLS的证书和私钥,例如:
apiVersion: v1 kind: Secret metadata: name: testsecret-tls namespace: default data: tls.crt: base64 encoded cert tls.key: base64 encoded key type: kubernetes.io/tls
在Ingress中引用这个secret将通知Ingress controller使用TLS加密从将客户端到loadbalancer的channel:
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: tls-example-ingress spec: tls: - hosts: - sslexample.foo.com secretName: testsecret-tls rules: - host: sslexample.foo.com http: paths: - path: / backend: serviceName: service1 servicePort: 80
注意:各种Ingress controller支持的TLS功能之间存在差距。 请参阅有关nginx,GCE或任何其他平台特定Ingress controller的文档,以了解TLS在你的环境中的工作原理。
1.2、Ingress更新
如果你想更改你现在正工作的Ingress,比如新增一个HOST,可以使用kubectl edit ingress my-ingress进行更新,在保存退出后其会触发Ingress Controller重新配置LB。比如我们现有一个ingress:
[root@master ingress]# kubectl get ingresses. NAME HOSTS ADDRESS PORTS AGE * 80 58m
然后我们使用kubectl edit ingress ingress-simple-daemo来新增一个HOST:
...... spec: rules: - http: paths: - backend: serviceName: nginx-service servicePort: 8000 path: /joker - host: bar.baz.com http: paths: - backend: serviceName: nginx servicePort: 80 path: /foo status: loadBalancer: {}
保存退出后就会生效:
[root@master ingress]# kubectl describe ingresses ingress-simple-daemo Name: ingress-simple-daemo Namespace: default Address: Default backend: default-http-backend:80 (<none>) Rules: Host Path Backends ---- ---- -------- * /joker nginx-service:8000 (172.20.2.84:80) bar.baz.com /foo nginx:80 (172.20.2.79:80,172.20.2.82:80) Annotations: kubernetes.io/ingress.class: traefik kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"traefik"},"name":"ingress-simple-daemo","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"nginx-service","servicePort":8000},"path":"/joker"}]}}]}} Events: <none>
二、Nginx Ingress
2.1、安装
2.1.1 在线安装
在线安装直接执行以下命令:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
然后安装NodePort:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml
2.1.2、离线安装
到https://github.com/kubernetes/ingress-nginx/tree/master/deploy/static这个下面下载对应的YAML文件,有configmap.yaml,namespace.yaml,rbac.yaml,with-rbac.yaml,可以写一个如下循环下载:
for yaml in configmap.yaml namespace.yaml rbac.yaml with-rbac.yaml; do wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/$yaml;done
然后执行如下命令安装,前提先执行namespace.yaml这个YAML文件:
kubectl apply -f namespace.yaml kubectl apply -f .
然后我们可以执行以下命令查看:
[root@master nginx]# kubectl get pod -n ingress-nginx NAME READY STATUS RESTARTS AGE nginx-ingress-controller-799dbf6fbd-zxvmp 0/1 ContainerCreating 0 12s
然后安装nodePort:
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml kubectl apply -f service-nodeport.yaml
然后我修改了nodePort的端口,如下:
apiVersion: v1 kind: Service metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: type: NodePort ports: - name: http port: 80 targetPort: 80 nodePort: 30080 protocol: TCP - name: https port: 443 targetPort: 443 nodePort: 30443 protocol: TCP selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx ---
然后查看结果:
[root@master nginx]# kubectl get pod -n ingress-nginx NAME READY STATUS RESTARTS AGE nginx-ingress-controller-799dbf6fbd-zxvmp 1/1 Running 0 52m [root@master nginx]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx NodePort 10.68.194.177 <none> 80:30080/TCP,443:30443/TCP 46m
2.2、使用
先创建Pod,service,定义如下YAML:
[root@master test]# cat nginx-ingress-demo.yaml --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-deploy namespace: default spec: replicas: 3 selector: matchLabels: app: nginx-demo release: canary template: metadata: name: my-nginx labels: app: nginx-demo release: canary spec: containers: - name: my-nginx image: nginx imagePullPolicy: IfNotPresent ports: - name: http containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-svc namespace: default spec: selector: app: nginx-demo release: canary ports: - name: http port: 80 targetPort: 80
然后执行kubectl apply -f nginx-ingress-demo.yaml,查看结果:
[root@master test]# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-deploy-549cb5c7ff-65fbb 1/1 Running 0 21s nginx-deploy-549cb5c7ff-75fmt 1/1 Running 0 21s nginx-deploy-549cb5c7ff-wg4w7 1/1 Running 0 17s [root@master test]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.68.0.1 <none> 443/TCP 5d2h nginx-svc ClusterIP 10.68.214.28 <none> 80/TCP 6m21s
然后定义Ingress Nginx,YAML文件如下:
[root@master test]# cat ingress-nginx.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-nginx annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: http: paths: - path: backend: serviceName: nginx-svc servicePort: 80
然后查看结果:
[root@master test]# kubectl describe ingresses ingress-nginx Name: ingress-nginx Namespace: default Address: 10.68.194.177 Default backend: default-http-backend:80 (<none>) Rules: Host Path Backends ---- ---- -------- * nginx-svc:80 (172.20.1.61:80,172.20.1.62:80,172.20.2.87:80) Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-nginx","namespace":"default"},"spec":{"rules":[{"host":null,"http":{"paths":[{"backend":{"serviceName":"nginx-svc","servicePort":80},"path":null}]}}]}} kubernetes.io/ingress.class: nginx Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 11m nginx-ingress-controller Ingress default/ingress-nginx Normal UPDATE 77s (x3 over 11m) nginx-ingress-controller Ingress default/ingress-nginx
2.3、TLS
Nginx Ingress不止支持HTTP,还支持HTTPS。我们使用如下命令手动创建证书:
openssl genrsa -out tls.key 2048 openssl req -new -x509 -key tls.key -out tls.crt -subj "/C=CN/ST=Chongqing/L=Chongqing/OU=DevOps/CN=nginx.joker.com"
然后配置Secrect:
kubectl create secret tls tls-secret --key tls.key --cert tls.crt
然后查看:
[root@master test]# kubectl get secret NAME TYPE DATA AGE default-token-gmdbb kubernetes.io/service-account-token 3 5d3h tls-secret kubernetes.io/tls 2 20s [root@master test]# kubectl describe secrets tls-secret Name: tls-secret Namespace: default Labels: <none> Annotations: <none> Type: kubernetes.io/tls Data ==== tls.crt: 1302 bytes tls.key: 1679 bytes
配置Ingress nginx:
[root@master test]# cat ingress-nginx-tls.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-nginx-tls annotations: kubernetes.io/ingress.class: "nginx" spec: tls: - host: secretName: tls-secret rules: - host: http: paths: - path: backend: serviceName: nginx-svc servicePort: 80
然后查看结果:
[root@master test]# kubectl get ingresses. NAME HOSTS ADDRESS PORTS AGE ingress-nginx-tls * 80, 443 4s
三、Traefik Ingress
3.1、安装
创建namespace(namespace.yaml):
apiVersion: v1 kind: Namespace metadata: name: ingress-traefik labels: app.kubernetes.io/name: ingress-traefik app.kubernetes.io/part-of: ingress-traefik ---
创建RBAC安全认证(rdac.yaml):
--- apiVersion: v1 kind: ServiceAccount metadata: name: traefik-ingress-controller namespace: ingress-traefik --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: traefik-ingress-controller rules: - apiGroups: - "" resources: - services - endpoints - secrets verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: traefik-ingress-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: traefik-ingress-controller subjects: - kind: ServiceAccount name: traefik-ingress-controller namespace: ingress-traefik
配置traefik(traefik.yaml):
--- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: traefik-ingress-controller namespace: ingress-traefik labels: k8s-app: traefik-ingress-lb spec: replicas: 1 selector: matchLabels: k8s-app: traefik-ingress-lb template: metadata: labels: k8s-app: traefik-ingress-lb name: traefik-ingress-lb spec: serviceAccountName: traefik-ingress-controller terminationGracePeriodSeconds: 60 # tolerations: # - operator: "Exists" # nodeSelector: # kubernetes.io/hostname: master containers: - image: traefik:v1.7.17 name: traefik-ingress-lb ports: - name: http containerPort: 80 - name: admin containerPort: 8080 args: - --api - --kubernetes - --logLevel=INFO --- kind: Service apiVersion: v1 metadata: name: traefik-ingress-service namespace: ingress-traefik spec: selector: k8s-app: traefik-ingress-lb ports: - protocol: TCP port: 80 name: web nodePort: 38000 - protocol: TCP port: 8080 nodePort: 38080 name: admin type: NodePort
然后执行如下命令创建:
kubectl apply -f .
查看其结果:
[root@master traefik]# kubectl get svc -n ingress-traefik NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE traefik-ingress-service NodePort 10.68.173.196 <none> 80:38000/TCP,8080:38080/TCP 9m19s [root@master traefik]# kubectl get pod -n ingress-traefik NAME READY STATUS RESTARTS AGE traefik-ingress-controller-86769d5d99-fvf82 1/1 Running 0 7m10s
Traefik提供一个WEB UI工具,就是上面8080端口,可以通过其映射的NodePort端口在浏览器访问.
3.2、使用
使用和上面nginx ingress一样,只是在annotations里配置的kubernetes.io/ingress.class: "traefik"。
3.3、配置
生成SSL证书:
openssl req -newkey rsa:2048 -nodes -keyout tls.key -x509 -days 365 -out tls.crt
创建Secret来存储证书:
kubectl create secret generic traefik-cert --from-file=tls.crt --from-file=tls.key -n ingress-traefik
配置traefik,让其支持https,配置文件如下:
defaultEntryPoints = ["http", "https"] [entryPoints] [entryPoints.http] address = ":80" [entryPoints.http.redirect] entryPoint = "https" [entryPoints.https] address = ":443" [entryPoints.https.tls] [[entryPoints.https.tls.certificates]] CertFile = "/ssl/tls.crt" KeyFile = "/ssl/tls.key"
将上面的 traefik.toml 配置文件通过一个 ConfigMap 对象挂载到 traefik pod 中去。
创建configMap:
kubectl create configmap traefik-conf --from-file=traefik.toml -n ingress-traefik
更改traefik的YAML文件:
--- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: traefik-ingress-controller namespace: ingress-traefik labels: k8s-app: traefik-ingress-lb spec: replicas: 1 selector: matchLabels: k8s-app: traefik-ingress-lb name: traefik-ingress-lb template: metadata: labels: k8s-app: traefik-ingress-lb name: traefik-ingress-lb spec: serviceAccountName: traefik-ingress-controller terminationGracePeriodSeconds: 60 # tolerations: # - operator: "Exists" # nodeSelector: # kubernetes.io/hostname: master containers: - image: traefik:v1.7.17 name: traefik-ingress-lb volumeMounts: - name: ssl mountPath: "/ssl" - name: config mountPath: "/config" ports: - name: http containerPort: 80 - name: https containerPort: 443 - name: admin containerPort: 8080 args: - --configfile=/config/traefik.toml - --api - --kubernetes - --logLevel=INFO volumes: - name: ssl secret: secretName: traefik-cert - name: config configMap: name: traefik-conf --- kind: Service apiVersion: v1 metadata: name: traefik-ingress-service namespace: ingress-traefik spec: selector: k8s-app: traefik-ingress-lb ports: - protocol: TCP port: 80 name: web nodePort: 38000 - protocol: TCP port: 8080 nodePort: 38080 name: admin type: NodePort
3.4、Rewrite
测试demo:
apiVersion: v1 kind: Service metadata: name: nginx-demo spec: selector: app: nginx ports: - name: nginx port: 80 --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent
正常访问的ingress如下:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx-ingress-demo annotations: kubernetes.io/ingress.class: "traefik" spec: rules: - host: http: paths: - backend: serviceName: nginx-demo servicePort: 80 path: /
但是如果我们把path改为如下:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx-ingress-demo annotations: kubernetes.io/ingress.class: "traefik" spec: rules: - host: http: paths: - backend: serviceName: nginx-demo servicePort: 80 path: /app
那么这时候访问就会报404.
这时候我们可以在配置里加上rewrite规则,如下:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx-ingress-demo annotations: kubernetes.io/ingress.class: "traefik" traefik.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: http: paths: - backend: serviceName: nginx-demo servicePort: 80 path: /app
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx-ingress-demo annotations: kubernetes.io/ingress.class: "traefik" traefik.ingress.kubernetes.io/rewrite-target: /$2 spec: rules: - host: http: paths: - backend: serviceName: nginx-demo servicePort: 80 path: /app(/|$)(.*)
但是这种还是会有问题,比如有的静态资源无法正常显示,比如静态资源的路径在/static下面,现在我们做了 url rewrite 过后,要正常访问也需要带上前缀才可以:http://xxx.xxx/static/screen.css,对于图片或者其他静态资源也是如此,当然我们去更改页面引入静态资源的方式为相对路径也是可以的,但是毕竟要修改代码,这个时候我们可以借助 ingress-traefik 中的 configuration-snippet 来对静态资源做一次跳转,如下所示:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx-ingress-demo annotations: kubernetes.io/ingress.class: "traefik" traefik.ingress.kubernetes.io/rewrite-target: /$2 traefik.ingress.kubernetes.io/configuration-snippet: | rewrite ^/static/(.*)$ /app/static/$1 redirect; rewrite ^/image/(.*)$ /app/image/$1 redirect; spec: rules: - host: http: paths: - backend: serviceName: nginx-demo servicePort: 80 path: /app(/|$)(.*)
这时候就可以正常加载静态资源了。
但是我们还想直接访问主域名,但是会报404,要解决这种问题,可以app-root的注解,这样我们访问主域名的时候就可以跳转到app-root目录(真实存在的路径)下面,如下:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx-ingress-demo annotations: kubernetes.io/ingress.class: "traefik" traefik.ingress.kubernetes.io/app-root: /app/ traefik.ingress.kubernetes.io/rewrite-target: /$2 traefik.ingress.kubernetes.io/configuration-snippet: | rewrite ^/static/(.*)$ /app/static/$1 redirect; rewrite ^/image/(.*)$ /app/image/$1 redirect; spec: rules: - host: http: paths: - backend: serviceName: nginx-demo servicePort: 80 path: /app(/|$)(.*)
但是还有一个问题是我们的 path 路径其实也匹配了 /app 这样的路径,可能我们更加希望我们的应用在最后添加一个 / 这样的 slash,同样我们可以通过 configuration-snippet 配置来完成,如下 Ingress 对象:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx-ingress-demo annotations: kubernetes.io/ingress.class: "traefik" traefik.ingress.kubernetes.io/app-root: /app/ traefik.ingress.kubernetes.io/rewrite-target: /$2 traefik.ingress.kubernetes.io/configuration-snippet: | rewrite ^(/app)$ $1/ redirect; rewrite ^/static/(.*)$ /app/static/$1 redirect; rewrite ^/image/(.*)$ /app/image/$1 redirect; spec: rules: - host: http: paths: - backend: serviceName: nginx-demo servicePort: 80 path: /app(/|$)(.*)