k8s 1.28.2 集群部署 ingress 1.11.1 包含 admission-webhook

文章目录

• • @[toc]
  • 证书创建
  • 部署 ingress-controller
  • ingress 验证
  • • 创建测试 nginx pod
    • 创建错误的 ingress 配置
    • 创建正确的 ingress 配置

• ingress 官方 yaml 文件：deploy.yaml
• 基于官方 yaml 文件做了一些修改
  • 官方的 svc 是 ClusterIP 和 LoadBalancer，我这边把 LoadBalancer 改成 NodePort
  • 官方的 yaml 没有给 deployment 配置副本数，我这边改成双副本了
    • 增加了亲和性和反亲和，把 controller 绑定到固定的两个节点，方便前面加反代
  • 官方的镜像是 registry.k8s.io 仓库，国内咱们也不好拉，这边改用 k8s.m.daocloud.io 国内的 daocloud 的仓库
  • 开启 admission-webhook 需要自己生成 tls 证书，官方的模板有名字定义了，sercret 的名字是 ingress-nginx-admission
• 开启了 admission-webhook 功能，这个功能类似于 nginx -s reload 之前去执行 nginx -t 检查配置文件是否正确，如果 admission-webhook 发现配置不正确就不会更新 ingress 的配置，就不会因为错误的配置导致 ingress reload 失败，全部崩了，影响到业务的访问

证书创建

证书这块，我用的是 cert-manager 创建的，可以看我之前的博客：cert-manager - kubernetes 集群中 TLS 证书管理工具

部署 ingress-controller

节点打标签

k label node 192.168.22.112 ingress=true
k label node 192.168.22.113 ingress=true

通过官方 helm 做了一些修改

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
automountServiceAccountToken: true
---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: "true"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
      - namespaces
    verbs:
      - list
      - watch
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices
    verbs:
      - list
      - watch
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  # Omit Ingress status permissions if `--update-status` is disabled.
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    resourceNames:
      - ingress-nginx-leader
    verbs:
      - get
      - update
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices
    verbs:
      - list
      - watch
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-metrics
  namespace: ingress-nginx
spec:
  type: ClusterIP
  ports:
    - name: metrics
      port: 10254
      protocol: TCP
      targetPort: metrics
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/component: controller
---
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission
  namespace: ingress-nginx
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/component: controller
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
      nodePort: 31080
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
      nodePort: 31443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/component: controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/component: controller
  replicas: 2
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        helm.sh/chart: ingress-nginx-4.11.1
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/version: "1.11.1"
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: controller
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: ingress
                operator: In
                values:
                - "true"
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - ingress-nginx
            topologyKey: kubernetes.io/hostname
      dnsPolicy: ClusterFirst
      containers:
        - name: controller
          image: k8s.m.daocloud.io/ingress-nginx/controller:v1.11.1
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
            - --election-id=ingress-nginx-leader
            - --controller-class=k8s.io/ingress-nginx
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
          securityContext:
            runAsNonRoot: true
            runAsUser: 101
            allowPrivilegeEscalation: false
            seccompProfile:
              type: RuntimeDefault
            capabilities:
              drop:
              - ALL
              add:
              - NET_BIND_SERVICE
            readOnlyRootFilesystem: false
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LD_PRELOAD
              value: /usr/local/lib/libmimalloc.so
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
            - name: metrics
              containerPort: 10254
              protocol: TCP
            - name: webhook
              containerPort: 8443
              protocol: TCP
          volumeMounts:
            - name: webhook-cert
              mountPath: /usr/local/certificates/
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 90Mi
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: ingress
                operator: In
                values:
                - "true"
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - ingress-nginx
            topologyKey: kubernetes.io/hostname
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
            items:
              - key: tls.crt
                path: cert
              - key: tls.key
                path: key
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: nginx
spec:
  controller: k8s.io/ingress-nginx
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ingress-nginx-root-cert
  namespace: ingress-nginx
spec:
  secretName: ingress-nginx-root-cert
  duration: "43800h0m0s"
  issuerRef:
    name: ingress-nginx-self-signed-issuer
  commonName: "ca.webhook.ingress-nginx"
  isCA: true
  subject:
    organizations:
      - ingress-nginx
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
spec:
  secretName: ingress-nginx-admission
  duration: "8760h0m0s"
  issuerRef:
    name: ingress-nginx-root-issuer
  dnsNames:
    - ingress-nginx-controller-admission
    - ingress-nginx-controller-admission.ingress-nginx
    - ingress-nginx-controller-admission.ingress-nginx.svc
  subject:
    organizations:
      - ingress-nginx-admission
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ingress-nginx-self-signed-issuer
  namespace: ingress-nginx
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ingress-nginx-root-issuer
  namespace: ingress-nginx
spec:
  ca:
    secretName: ingress-nginx-root-cert
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    certmanager.k8s.io/inject-ca-from: "ingress-nginx/ingress-nginx-admission"
    cert-manager.io/inject-ca-from: "ingress-nginx/ingress-nginx-admission"
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    matchPolicy: Equivalent
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
    clientConfig:
      service:
        name: ingress-nginx-controller-admission
        namespace: ingress-nginx
        path: /networking/v1/ingresses

ingress 验证

创建测试 nginx pod

cat << EOF | kubectl apply -f -
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: nginx
  name: nginx-test
  namespace: default
spec:
  internalTrafficPolicy: Cluster
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    app: nginx
  name: nginx-test
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: docker.m.daocloud.io/nginx:1.26.0
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
EOF

创建错误的 ingress 配置

cat << EOF | kubectl apply -f -
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_pass          http://$targetbackend;
      proxy_set_header    X-Real-IP       $remote_addr;
      proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
  name: ingress-test
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - backend:
          service:
            name: nginx-test
            port:
              number: 80
        path: /
        pathType: Prefix
EOF

• 此时会出现报错，这个报错是 admission webhook 触发的，并且 ingress-controller 也不会触发重启，因为不会更新和创建 ingress 规则，所以不会影响到业务

Error from server (BadRequest): error when creating "STDIN": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:
-------------------------------------------------------------------------------
Error: exit status 1
2024/09/26 07:30:35 [emerg] 166#166: no host in upstream "" in /tmp/nginx/nginx-cfg2607502217:404
nginx: [emerg] no host in upstream "" in /tmp/nginx/nginx-cfg2607502217:404
nginx: configuration file /tmp/nginx/nginx-cfg2607502217 test failed

创建正确的 ingress 配置

cat << EOF | kubectl apply -f -
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
  name: ingress-test
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - backend:
          service:
            name: nginx-test
            port:
              number: 80
        path: /
        pathType: Prefix
EOF

此时会返回 created，说明 ingress 的规则配置成功了

ingress.networking.k8s.io/ingress-test created

验证 ingress

curl 192.168.22.112:31080

正常会返回下面的内容，是 nginx 的默认页面

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>