容器云平台No.8~kubernetes负载均衡之ingress-nginx

Ingress 是什么？

Ingress 公开了从集群外部到集群内服务的 HTTP 和 HTTPS 路由。 流量路由由 Ingress 资源上定义的规则控制。
可以将 Ingress 配置为服务提供外部可访问的 URL、负载均衡流量、终止 SSL/TLS，以及提供基于名称的虚拟主机等能力。 Ingress 控制器 通常负责通过负载均衡器来实现 Ingress，尽管它也可以配置边缘路由器或其他前端来帮助处理流量。

本文使用host network模式，示意图如下

下载部署文件

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.35.0/deploy/static/provider/baremetal/deploy.yaml

修改images为国内仓库

k8s.gcr.io/ingress-nginx/controller:v0.35.0@sha256:fc4979d8b8443a831c9789b5155cded454cb7de737a8b727bc2ba0106d2eae8b

修改为，也可以自行使用魔法到k8s.gcr.io下载

scofield/ingress-nginx-controller:v0.35.0

修改网络模式为host network

template:
  spec:
    hostNetwork: true
    dnsPolicy: ClusterFirstWithHostNet

执行部署

kubectl apply -f deploy.yaml

[root@k8s-master001 ingress-nginx]# kubectl  get po -n ingress-nginx
[root@k8s-master001 ingress-nginx]# kubectl  get po,svc  -n ingress-nginx -o wide 
NAME                                            READY   STATUS      RESTARTS   AGE     IP             NODE            NOMINATED NODE   READINESS GATES
pod/ingress-nginx-admission-create-dfg8g        0/1     Completed   0          47m     10.244.2.155   k8s-master003   <none>           <none>
pod/ingress-nginx-admission-patch-cfl4r         0/1     Completed   1          47m     10.244.1.134   k8s-master002   <none>           <none>
pod/ingress-nginx-controller-6fdd8c7f88-5gzdv   1/1     Running     0          2m42s   10.26.25.21    k8s-master002   <none>           <none>

注意：ingress-nginx-controller的IP应该是宿主机IP，这里是10.26.25.21,至此，ingress-nginx就部署好了

使用ingress-nginx暴露http服务

部署一个最常用的http服务nginx,使用ingress-nginx暴露http服务
1、编写demo.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  serviceName: nginx
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 180
      initContainers:
        - name: init
          image: busybox
          command: ["chmod","777","-R","/var/www"]
          imagePullPolicy: Always
          volumeMounts:
          - name: volume
            mountPath: /var/www/html
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          name: port
        volumeMounts:
        - name: volume
          mountPath: /var/www/html
  volumeClaimTemplates:
  - metadata:
      name: volume
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: rook-ceph
      resources:
        requests:
          storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx

2、执行部署

[root@k8s-master001 ingress-nginx]# kubectl apply -f  demo.yaml
statefulset.apps/nginx configured
service/nginx created

[root@k8s-master001 ~]# kubectl get po,svc
NAME          READY   STATUS    RESTARTS   AGE
pod/nginx-0   1/1     Running   0          21m

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        5d21h
service/nginx        NodePort    10.106.146.193   <none>        80:31389/TCP   21m

[root@k8s-master001 ~]# curl  -I  10.106.146.193
HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Wed, 16 Sep 2020 07:03:26 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 11 Aug 2020 14:50:35 GMT
Connection: keep-alive
ETag: "5f32b03b-264"
Accept-Ranges: bytes

3、nginx已经部署好，而且访问已经OK，接下来创建Ingress
demo-ingress.yaml

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: nginx.text.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 80

[root@k8s-master001 ~]# kubectl  apply -f nginx-ingress.yaml
error: error validating "nginx-ingress.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]; if you choose to ignore these errors, turn validation off with --validate=false

修改apiVersion为networking.k8s.io/v1

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: nginx.text.cn
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: nginx
            port:
              number: 80

[root@k8s-master001 ingress-nginx]# kubectl  apply -f demo-ingress.yaml
Error from server (InternalError): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{\"kubernetes.io/ingress.class\":\"nginx\"},\"name\":\"nginx\",\"namespace\":\"default\"},\"spec\":{\"rules\":[{\"host\":\"nginx.ieasou.cn\",\"http\":{\"paths\":[{\"backend\":{\"service\":{\"name\":\"nginx\",\"port\":{\"number\":80}}},\"path\":\"/\",\"pathType\":\"Prefix\"}]}}]}}\n","kubernetes.io/ingress.class":"nginx"}},"spec":{"rules":[{"host":"nginx.ieasou.cn","http":{"paths":[{"backend":{"service":{"name":"nginx","port":{"number":80}}},"path":"/","pathType":"Prefix"}]}}]}}
to:
Resource: "networking.k8s.io/v1, Resource=ingresses", GroupVersionKind: "networking.k8s.io/v1, Kind=Ingress"
Name: "nginx", Namespace: "default"
for: "demo-ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s": x509: certificate is valid for k8s-master002, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not ingress-nginx-controller-admission.ingress-nginx.svc

还是不行。。。

解决办法1、把Webhook删了

admission webhook 传送门

[root@k8s-master001 ingress-nginx]# kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted

再来

[root@k8s-master001 ingress-nginx]# kubectl  apply -f demo-ingress.yaml
ingress.networking.k8s.io/nginx configured

解决办法2、降级为0.32.0（未测）

传送门了解更多

4、现在来查看创建好的ingress,已经创建好了

[root@k8s-master001 ingress-nginx]# kubectl  get ing
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME    CLASS    HOSTS             ADDRESS       PORTS   AGE
nginx   <none>   nginx.ieasou.cn   10.26.25.21   80      3d19h

5、访问验证,现在在外部就可以通过域名nginx.text.cn访问到nginx了

[root@k8s-master001 ingress-nginx]# vim /etc/hosts
10.26.25.21 nginx.text.cn

[root@k8s-master001 ingress-nginx]# curl -I   nginx.text.cn
HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Wed, 16 Sep 2020 08:05:06 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 11 Aug 2020 14:50:35 GMT
ETag: "5f32b03b-264"
Accept-Ranges: bytes

使用ingress-nginx暴露TCP服务

部署一个常用的redis服务,使用ingress-nginx暴露tcp服务
1、编写redis.yaml文件

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  labels:
    app: redis
spec:
  serviceName: redis
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      terminationGracePeriodSeconds: 180
      initContainers:
        - name: init
          image: busybox
          command: ["chmod","777","-R","/var/www"]
          imagePullPolicy: Always
          volumeMounts:
          - name: volume
            mountPath: /data
      containers:
      - name: redis
        image: redis
        imagePullPolicy: Always
        ports:
        - containerPort: 6379
          name: port
        volumeMounts:
        - name: volume
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: volume
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: rook-ceph
      resources:
        requests:
          storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  type: NodePort
  ports:
  - port: 6379
    targetPort: 6379
  selector:
    app: redis

查看并查看结果

[root@k8s-master001 ingress-nginx]# kubectl apply -f redis.yaml

[root@k8s-master001 ingress-nginx]# kubectl  get po,svc
NAME          READY   STATUS    RESTARTS   AGE
pod/redis-0   1/1     Running   0          104s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/redis        NodePort    10.98.28.146     <none>        80:32193/TCP   104s

测试连接redis，能够连接
[root@k8s-master001 ingress-nginx]# telnet 10.26.25.20 32193
Trying 10.26.25.20...
Connected to 10.26.25.20.
Escape character is '^]'.
info
$3615
# Server
redis_version:6.0.8

2、默认ingress-nginx的deploy.yaml部署文件并没有开启tcp服务支持，这里需要修改部署文件并重新部署。一般情形，如果需要支持tcp,udp等转发，提前规划并修改deploy.yaml文件
修改如下：--tcp-services-configmap,这里顺便把udp也开启了--udp-services-configmap

      containers:
        - name: controller
          image: scofield/ingress-nginx-controller:v0.35.0
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-service
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-service
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key

重新部署ingress-nginx

kubectl apply -f deploy.yaml

3、创建tcp服务需要的configmap,注意namespace一定要和ingress-nginx部署的namespace一致
tcp-service.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-service
  namespace: ingress-nginx
data:
  6379: "default/redis:6379"

[root@k8s-master001 ingress-nginx]# kubectl  apply -f tcp-service.yaml
configmap/tcp-service created

4、创建redis-ingress.yaml文件

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: redis
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: redis.test.cn
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: redis
            port:
              number: 6379

5、创建ingress

kubectl apply -f redis-ingress.yaml

[root@k8s-master001 ingress-nginx]# kubectl  get ing 
NAME    CLASS    HOSTS             ADDRESS       PORTS   AGE
redis   <none>   redis.ieasou.cn   10.26.25.20   80      13m

6、验证

[root@k8s-master001 ingress-nginx]# telnet redis.test.cn 6379
Trying 10.26.25.20...
Connected to redis.test.cn.
Escape character is '^]'.
info
$3622
# Server
redis_version:6.0.8
redis_git_sha1:00000000

更多用法，请移步