【K8s教程】使用Helm部署Nginx Ingress控制器（DaemonSet hostNetwork 模式）

参考：
https://cloud.tencent.com/developer/article/1761376
https://kubernetes.github.io/ingress-nginx/
https://github.com/kubernetes/ingress-nginx
https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx

NodePort 和 LoadBlancer 类型的 Service 可以把应用暴露给外部用户使用，除此之外，Kubernetes 还为我们提供了一个非常重要的资源对象可以用来暴露服务给外部用户，那就是 Ingress。对于小规模的应用我们使用 NodePort 或许能够满足我们的需求，但是当你的应用越来越多的时候，你就会发现对于 NodePort 的管理就非常麻烦了，这个时候使用 Ingress 就非常方便了，可以避免管理大量的端口。

安装 NGINX Ingress Controller

生产环境可以使用 HA + LB + DaemonSet hostNetwork 模式。

1.获取资源清单文件：

# helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
# helm repo update
# helm pull ingress-nginx/ingress-nginx
# tar -xvf ingress-nginx-3.35.0.tgz

2.新建一个名为 values-prod.yaml 的 Values 文件，用来覆盖 ingress-nginx 默认的 Values 值，对应的数据如下所示：

# cat ingress-nginx/values-prod.yaml 
controller:
  name: controller
  image:
    repository: registry.cn-hangzhou.aliyuncs.com/varden/ingress-nginx-controller
    tag: "v0.48.1"
    digest: 

  dnsPolicy: ClusterFirstWithHostNet
   
  hostNetwork: true

  publishService:  # hostNetwork 模式下设置为false，通过节点IP地址上报ingress status数据
    enabled: false

  kind: DaemonSet

  nodeSelector: 
    role: ingressproxy

  service:  # HostNetwork 模式不需要创建service
    enabled: false

  tolerations:
    - operator: "Exists"

  resources:
    limits:
      cpu: 100m
      memory: 90Mi
    requests:
      cpu: 100m
      memory: 90Mi

  terminationGracePeriodSeconds: 300

  admissionWebhooks:
    patch:
      enabled: true
      image:
        registry: docker.io
        image: jettech/kube-webhook-certgen
        tag: v1.5.1

defaultBackend:
  enabled: true
  name: defaultbackend
  image:
    repository: registry.cn-hangzhou.aliyuncs.com/varden/defaultbackend-amd64
    tag: "1.5"

3.然后使用如下命令安装 ingress-nginx 应用到 ingress-nginx 的命名空间中：

# kubectl create ns ingress-nginx
# kubectl label nodes k8s-master role=ingressproxy
# helm install ingress-nginx ./ingress-nginx -f ./ingress-nginx/values-prod.yaml --namespace ingress-nginx
NAME: ingress-nginx
LAST DEPLOYED: Wed Aug 11 09:53:41 2021
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'

An example Ingress that makes use of the controller:

  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

更新部署命令：helm upgrade ingress-nginx ./ingress-nginx -f ./ingress-nginx/values-prod.yaml --namespace ingress-nginx

4.部署完成后查看 Pod 的运行状态：

# kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
ingress-nginx-controller-admission   ClusterIP   10.110.55.35    <none>        443/TCP   5h22m
ingress-nginx-defaultbackend         ClusterIP   10.98.122.133   <none>        80/TCP    5h22m

# kubectl get pods -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-zv9fd                  1/1     Running   0          57m
ingress-nginx-defaultbackend-7b85879754-4khmd   1/1     Running   0          54m

# POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -n ingress-nginx -o jsonpath='{.items[0].metadata.name}')
# kubectl exec -it $POD_NAME -n ingress-nginx -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v0.48.1
  Build:         30809c066cd027079cbb32dccc8a101d6fbffdcb
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.20.1

-------------------------------------------------------------------------------

当看到上面的信息证明 ingress-nginx 部署成功了。

5.创建一个 Ingress 资源并简单测试

# vim example-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: example
  namespace: default
spec:
  rules:
  - host: www.example.com
    http:
      paths:
        - backend:
            service:
              name: nginx
              port:
                number: 80
          path: /
          pathType: Prefix

# kubectl apply -f example-ingress.yaml
ingress.networking.k8s.io/example configured

# kubectl get ingress example -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"example","namespace":"default"},"spec":{"rules":[{"host":"www.example.com","http":{"paths":[{"backend":{"service":{"name":"nginx","port":{"number":80}}},"path":"/","pathType":"Prefix"}]}}]}}
    kubernetes.io/ingress.class: nginx
  creationTimestamp: "2021-08-11T02:13:54Z"
  generation: 3
  name: example
  namespace: default
  resourceVersion: "1944004"
  uid: 26279640-a55e-46a8-8b07-9569c513306a
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - backend:
          service:
            name: nginx
            port:
              number: 80
        path: /
        pathType: Prefix
status:
  loadBalancer:
    ingress:
    - ip: 192.168.100.10

# kubectl describe ingress example 
Name:             example
Namespace:        default
Address:          192.168.100.10
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host             Path  Backends
  ----             ----  --------
  www.example.com  
                   /   nginx:80 (10.244.0.42:80,10.244.0.49:80)
Annotations:       kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age                  From                      Message
  ----    ------  ----                 ----                      -------
  Normal  Sync    3m51s (x3 over 73m)  nginx-ingress-controller  Scheduled for sync

# curl -H "Host: www.example.com" http://192.168.100.10
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

下图显示了客户端是如果通过 Ingress Controller 连接到其中一个 Pod 的流程，客户端首先对 ngdemo.qikqiak.com 执行 DNS 解析，得到 Ingress Controller 所在节点的 IP，然后客户端向 Ingress Controller 发送 HTTP 请求，然后根据 Ingress 对象里面的描述匹配域名，找到对应的 Service 对象，并获取关联的 Endpoints 列表，将客户端的请求转发给其中一个 Pod。