Daemonset方式部署ingress-nginx结合keepalived实现ingress高可用

概述：

官方原始文件使用的是deployment，replicate 为 1，这样将会在某一台节点上启动对应的nginx-ingress-controller pod。外部流量访问至该节点，由该节点负载分担至内部的service。考虑到单点故障的问题，改为DaemonSet然后删掉replicate ，配合亲和性部署在指定节点上启动nginx-ingress-controller pod，确保有多个节点启动nginx-ingress-controller pod，生产环境中建议ingress节点打上污点不允许业务pod进行调度，以避免业务应用与Ingress服务发生资源争抢。后续将这些节点加入到外部硬件负载均衡组实现高可用性。
 

 

云服务器方式介绍：

1.使用daemonset方式将ingress-controller部署在相应节点，一般在k8s-node节点上，master节点为了集群稳定不建议部署。

2.申请使用SLB负载均衡高可用IP对应解析访问域名，指向三台ingress节点主机(把ingress节点主机绑定至后端服务器组)，达到高可用目的。

大概结构图如下：

 

 

　　

私有服务器方式　

下面针对的是企业自建服务器部署高可用架构(云服务器部署架构也基本差不多)

 

大致结构：

在Kubernetes中添加了ingress后，公网域名解析设置为：域名解析到机房公网IP服务器(nginx服务器)，配置nginx转发到keepalived的VIP

这样集群外部就可以通过域名来访问你的服务，也解决了单点故障。

注意：本实验针对生产环境，ingress服务器没有公网IP，备案的域名解析到Nginx服务器上的，所以没使用到ingress的域名，

没使用到ingress的spec.rules.host，也可以不用配置域名，通过nginx反向代理到ingress宿主机的IP访问。

或者VIP使用公网IP，把VIP解析到域名，使用域名访问(可以不用nginx)

选择Kubernetes部署了ingress的三个node作为节点，都安装keepalived。

 

修改配置文件/etc/keepalived/keepalived.conf

除了priority优先级不一样，其他三个node节点都一样

注意：修改如下所示

 

启动keeplived

systemctl start keepalived
systemctl enable keepalived

 

创建deployment和service

(用于测试ingress请求后端业务pod)

$ vim deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment 
metadata: 
  name: deployment 
spec: 
  replicas: 2 
  template: 
    metadata: 
      labels: 
        name: nginx 
    spec: 
      containers: 
      - name: nginx 
        image: wangyanglinux/myapp:v3 
        imagePullPolicy: IfNotPresent 
        ports: 
        - containerPort: 80

---
apiVersion: v1 
kind: Service 
metadata: 
  name: svc-1 
spec: 
  ports: 
  - port: 80 
    targetPort: 80 
    protocol: TCP 
  selector: 
    name: nginx #当name=nginx时匹配

$ kubectl apply -f deployment.yaml

 

 

安装ingress-nginx-controller

官网安装文件地址:

https://github.com/kubernetes/ingress-nginx/tree/main/deploy/static/provider/baremetal/deploy.yaml

 

1.给要部署的ingress节点打标签

nginx-ingress-controller会随意选择一个node节点运行pod，为此需要我们把nginx-ingress-controller运行到指定的node节点上。
首先需要给需要运行nginx-ingress-controller的node节点打标签

kubectl label nodes k8s-node01 edgenode=true
kubectl label nodes k8s-node02 edgenode=true
kubectl label nodes k8s-node03 edgenode=true

查看node标签

kubectl get node --show-labels

 

2.daemonset形式安装ingress-nginx-controller(修改原来ingress部署的yaml文件，注意修改标红处)

• deployment改为daemonset　　
• 注释replicate　　#注销此行，DaemonSet不需要此参数
• 添加hostNetwork　　#添加该字段让pod使用物理机网络，在物理机暴露服务端口80，注意：物理机80端口不能被占用
• dnsPolicy:ClusterFirstWithHostNet　　#使用hostNetwork后容器会使用物理机网络包括DNS，会无法解析内部service，使用此参数可以让容器同时使用 hostNetwork 与 kube-dns 作为 Pod 预设 DNS 配置。
• 添加节点亲和性属性　　

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend 
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---
 
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend
---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
　　　 dnsPolicy: ClusterFirstWithHostNet　
      nodeSelector:
        edgenode: 'true'
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.20.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          # livenessProbe:
          #   failureThreshold: 3
          #   httpGet:
          #     path: /healthz
          #     port: 10254
          #     scheme: HTTP
          #   initialDelaySeconds: 10
          #   periodSeconds: 10
          #   successThreshold: 1
          #   timeoutSeconds: 1
          # readinessProbe:
          #   failureThreshold: 3
          #   httpGet:
          #     path: /healthz
          #     port: 10254
          #     scheme: HTTP
          #   periodSeconds: 10
          #   successThreshold: 1
          #   timeoutSeconds: 1
---

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx

 

应用资源清单

kubectl apply -f ingress-nginx.yaml

 

查看安装是否成功

kubectl get ds -n ingress-nginx
kubectl get pods -n ingress-nginx -o wide
[root@master ingress]# kubectl get pod -n ingress-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ingress-controller-3sfom 1/1 Running 0 13m 192.168.3.1 node1 <none> <none>
nginx-ingress-controller-5jdeq 1/1 Running 0 13m 192.168.3.2 node2 <none> <none>
nginx-ingress-controller-1hdkr 1/1 Running 0 13m 192.168.3.3 node3 <none> <none>

可以看到三个ingress-controller已经根据我们选择，部署在3个node节点上，使用宿主机的网络

 

Ingress HTTPS 代理访问

创建https证书的secret

kubectl create secret tls tls-secret --key tls.key --cert tls.crt

 

创建ingress策略

$ vim https_ingress.yaml

apiVersion: extensions/v1beta1 
kind: Ingress 
metadata: 
  name: https 
spec: 
  tls:
  - hosts: 
    - www.test.com 
    secretName: tls-secret #上面创建时保存的secret名称
  rules: 
    - host: www.test.com 
      http: 
        paths: 
        - path: / 
          backend: 
            serviceName: svc-1
            servicePort: 80

$ kubectl apply -f https_ingress.yaml

 

然后就是分别在三台ingress服务器上部署keepalived，使用VIP (略)

 

最后部署nginx转发，把业务请求路径转发到VIP(略)

 

测试