kubernetes系列(十) - 通过Ingress实现七层代理

1. Ingress入门
2. Ingress部署的几种方式
3. Deployment+NodePort模式
4. DaemonSet+HostNetwork+nodeSelector模式(推荐)

1. Ingress入门

1.1 Ingress简介

kubernetes自带的service概念只有四层代理，即表现形式为IP:Port.

如果需要实现七层代理，即绑定到域名的话，则需要另一个了，即ingress api

官方在v1.11推出了ingress api接口，既而达到七层代理的效果
对于ingress来说，必须要绑定一个域名

1.2 原理和组成部分

Ingress可以理解为Service的Service。它由两部分组成

Ingress Controller
- 这是一个标准，可以有很多实现，其中ingress-nginx是最常用的
- 以pod形式运行的
Ingress策略设置
- 以yaml形式为载体的一组声明式的策略
- ingress-controller会动态地按照策略生成配置文件(如：nginx.conf)

1.3 资料信息

Ingress-Nginx github repo

https://github.com/kubernetes/ingress-nginx

Ingress-Nginx官方网站

https://kubernetes.github.io/ingress-nginx

2. Ingress部署的几种方式

2.1 前言

ingress的部署，需要考虑两个方面：

ingress-controller是作为pod来运行的，那么以什么方式部署比较好？
ingress解决了把如何请求路由到集群内部，那它自己怎么暴露给外部比较好？

下面列举一些目前常见的部署和暴露方式，具体使用哪种方式还是得根据实际需求来考虑决定。

2.1 Deployment+LoadBalancer模式的Service

如果要把ingress部署在公有云，那可以选择这种方式。用Deployment部署ingress-controller，创建一个type为LoadBalancer的service关联这组pod。大部分公有云，都会为LoadBalancer的service自动创建一个负载均衡器，通常还绑定了公网地址。只要把域名解析指向该地址，就实现了集群服务的对外暴露。

需要额外购买公有云的服务！

2.2 Deployment+NodePort模式的Service

同样用deployment模式部署ingress-controller，并创建对应的服务，但是type为NodePort。这样，ingress就会暴露在集群节点ip的特定端口上。由于nodeport暴露的端口是随机端口，一般会在前面再搭建一套负载均衡器来转发请求。该方式一般用于宿主机是相对固定的环境ip地址不变的场景。

缺点：

NodePort方式暴露ingress虽然简单方便，但是NodePort多了一层NAT，在请求量级很大时可能对性能会有一定影响。
请求节点会是类似https://www.xx.com:30076,其中30076是kubectl get svc -n ingress-nginx的svc暴露出来的nodeport端口

2.3 DaemonSet+HostNetwork+nodeSelector(推荐)

用DaemonSet结合nodeselector来部署ingress-controller到特定的node上，然后使用HostNetwork直接把该pod与宿主机node的网络打通，直接使用宿主机的80/433端口就能访问服务。这时，ingress-controller所在的node机器就很类似传统架构的边缘节点，比如机房入口的nginx服务器

优点

该方式整个请求链路最简单，性能相对NodePort模式更好

缺点

由于直接利用宿主机节点的网络和端口，一个node只能部署一个ingress-controller pod

3. Deployment+NodePort模式

3.1. 官网下载yaml，安装ingress-nginx

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml

3.2. 创建deployment和service

这里service定义为clusterip

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tocgenerator-deploy
  namespace: default
  labels:
    app: tocgenerator-deploy
spec:
  replicas: 2
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: tocgenerator-server
  template:
    metadata:
      labels:
        app: tocgenerator-server
    spec:
      containers:        
        - name: tocgenerator
          image: lzw5399/tocgenerator:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: tocgenerator-svc
spec:
  selector:
    app: tocgenerator-server
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

3.3 创建https证书的secret

方式1：直接指定文件创建

kubectl create secret tls mywebsite-secret --key tls.key --cert tls.crt

方式2: 以yaml资源清单方式创建

apiVersion: v1
kind: Secret
metadata:
  name: mywebsite-secret
data:
  tls.crt: **************************
  tls.key: **************************

3.4. 创建ingress策略

ingress策略必须和service同一个namespace！

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tocgenerator-ingress
spec:
  tls:
    - hosts:
      - toc.codepie.fun
      secretName: toc-secret
  rules:
    - host: toc.codepie.fun
      http:
        paths:
          - path: /
            backend:
              serviceName: tocgenerator-svc
              servicePort: 80

3.5 查看ingress-controller的nodeport端口，并访问

查看ingress-controller的nodeport端口

$ kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.104.80.142   <none>        80:30122/TCP,443:30577/TCP   21h

访问

https://toc.codepie.fun:30577

4. DaemonSet+HostNetwork+nodeSelector模式(推荐)

4.1 前言

为了配置kubernetes中的ingress的高可用，对于kubernetes集群以外只暴露一个访问入口，需要使用keepalived排除单点问题。需要使用daemonset方式将ingress-controller部署在边缘节点上。

4.2 边缘节点

首先解释下什么叫边缘节点Edge Node，所谓的边缘节点即集群内部用来向集群外暴露服务能力的节点，集群外部的服务通过该节点来调用集群内部的服务，边缘节点是集群内外交流的一个Endpoint

边缘节点要考虑两个问题

边缘节点的高可用，不能有单点故障，否则整个kubernetes集群将不可用
对外的一致暴露端口，即只能有一个外网访问IP和端口

4.3 架构

为了满足边缘节点的以上需求，我们使用keepalived来实现。

在Kubernetes中添加了ingress后，在DNS中添加A记录，域名为你的ingress中host的内容，IP为你的keepalived的VIP，这样集群外部就可以通过域名来访问你的服务，也解决了单点故障。

选择Kubernetes的node作为边缘节点，并安装keepalived。

4.4 安装keepalived服务

注意：keepalived服务，每个想要当作边缘节点的机器都要安装，一般是node节点

安装keeplived

yum install -y keepalived

将下图配置修改为edgenode

启动keeplived并设置为开机启动

systemctl start keepalived
systemctl enable keepalived

4.5 安装ingress-nginx-controller

给边缘节点打标签

kubectl label nodes k8s-node01 edgenode=true
kubectl label nodes k8s-node02 edgenode=true

daemonset形式安装ingress-nginx-controller

注意：以下的资源清单很长，可以直接复制过来用，会创建一系列ingess-nginx0controller相关的
创建资源清单，并命名为ingress-nginx.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend 
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---
 
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend
---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
      nodeSelector:
        edgenode: 'true'
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.20.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          # livenessProbe:
          #   failureThreshold: 3
          #   httpGet:
          #     path: /healthz
          #     port: 10254
          #     scheme: HTTP
          #   initialDelaySeconds: 10
          #   periodSeconds: 10
          #   successThreshold: 1
          #   timeoutSeconds: 1
          # readinessProbe:
          #   failureThreshold: 3
          #   httpGet:
          #     path: /healthz
          #     port: 10254
          #     scheme: HTTP
          #   periodSeconds: 10
          #   successThreshold: 1
          #   timeoutSeconds: 1
---

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx

应用资源清单

kubectl apply -f ingress-nginx.yaml

4.6 查看安装是否成功

[root@k8s-master01 ingress]# kubectl get ds -n ingress-nginx
NAME                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
nginx-ingress-controller   2         2         2       2            2           edgenode=true   57m

[root@k8s-master01 ingress]# kubectl get pods -n ingress-nginx -o wide
NAME                                    READY   STATUS    RESTARTS   AGE   IP            NODE          NOMINATED NODE
default-http-backend-86569b9d95-x4bsn   1/1     Running   12         24d   172.17.65.6   10.40.0.105   <none>
nginx-ingress-controller-5b7xg          1/1     Running   0          58m   10.40.0.105   10.40.0.105   <none>
nginx-ingress-controller-b5mxc          1/1     Running   0          58m   10.40.0.106   10.40.0.106   <none>

posted @ 2020-07-06 17:28 宝树呐阅读(1685) 评论(0) 编辑收藏举报

刷新页面返回顶部

宝树呐