容器云平台No.8~kubernetes负载均衡之ingress-nginx

Ingress 是什么？

Ingress 公开了从集群外部到集群内服务的 HTTP 和 HTTPS 路由。流量路由由 Ingress 资源上定义的规则控制。
可以将 Ingress 配置为服务提供外部可访问的 URL、负载均衡流量、终止 SSL/TLS，以及提供基于名称的虚拟主机等能力。 Ingress 控制器通常负责通过负载均衡器来实现 Ingress，尽管它也可以配置边缘路由器或其他前端来帮助处理流量。

本文使用host network模式，示意图如下

下载部署文件

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.35.0/deploy/static/provider/baremetal/deploy.yaml

修改images为国内仓库

k8s.gcr.io/ingress-nginx/controller:v0.35.0@sha256:fc4979d8b8443a831c9789b5155cded454cb7de737a8b727bc2ba0106d2eae8b

修改为，也可以自行使用魔法到http://k8s.gcr.io下载

scofield/ingress-nginx-controller:v0.35.0

修改网络模式为host network

template:
  spec:
    hostNetwork: true
    dnsPolicy: ClusterFirstWithHostNet

执行部署

kubectl apply -f deploy.yaml

[root@k8s-master001 ingress-nginx]# kubectl  get po -n ingress-nginx
[root@k8s-master001 ingress-nginx]# kubectl  get po,svc  -n ingress-nginx -o wide 
NAME                                            READY   STATUS      RESTARTS   AGE     IP             NODE            NOMINATED NODE   READINESS GATES
pod/ingress-nginx-admission-create-dfg8g        0/1     Completed   0          47m     10.244.2.155   k8s-master003   <none>           <none>
pod/ingress-nginx-admission-patch-cfl4r         0/1     Completed   1          47m     10.244.1.134   k8s-master002   <none>           <none>
pod/ingress-nginx-controller-6fdd8c7f88-5gzdv   1/1     Running     0          2m42s   10.26.25.21    k8s-master002   <none>           <none>

注意：ingress-nginx-controller的IP应该是宿主机IP，这里是10.26.25.21,至此，ingress-nginx就部署好了

使用ingress-nginx暴露http服务

部署一个最常用的http服务nginx,使用ingress-nginx暴露http服务
1、编写demo.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  serviceName: nginx
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 180
      initContainers:
        - name: init
          image: busybox
          command: ["chmod","777","-R","/var/www"]
          imagePullPolicy: Always
          volumeMounts:
          - name: volume
            mountPath: /var/www/html
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          name: port
        volumeMounts:
        - name: volume
          mountPath: /var/www/html
  volumeClaimTemplates:
  - metadata:
      name: volume
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: rook-ceph
      resources:
        requests:
          storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx

2、执行部署

[root@k8s-master001 ingress-nginx]# kubectl apply -f  demo.yaml
statefulset.apps/nginx configured
service/nginx created


[root@k8s-master001 ~]# kubectl get po,svc

NAME          READY   STATUS    RESTARTS   AGE

pod/nginx-0   1/1     Running   0          21m




NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE

service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        5d21h

service/nginx        NodePort    10.106.146.193   <none>        80:31389/TCP   21m


[root@k8s-master001 ~]# curl  -I  10.106.146.193

HTTP/1.1 200 OK

Server: nginx/1.19.2

Date: Wed, 16 Sep 2020 07:03:26 GMT

Content-Type: text/html

Content-Length: 612

Last-Modified: Tue, 11 Aug 2020 14:50:35 GMT

Connection: keep-alive

ETag: "5f32b03b-264"

Accept-Ranges: bytes

3、nginx已经部署好，而且访问已经OK，接下来创建Ingress
demo-ingress.yaml

---

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: nginx

annotations:

kubernetes.io/ingress.class: nginx

spec:

rules:





host: nginx.text.cn

http:

paths:



path: /

backend:

serviceName: nginx

servicePort: 80







[root@k8s-master001 ~]# kubectl  apply -f nginx-ingress.yaml

error: error validating "nginx-ingress.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]; if you choose to ignore these errors, turn validation off with --validate=false

修改apiVersion为http://networking.k8s.io/v1

---

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

name: nginx

annotations:

kubernetes.io/ingress.class: nginx

spec:

rules:





host: nginx.text.cn

http:

paths:



pathType: Prefix

path: /

backend:

service:

name: nginx

port:

number: 80







[root@k8s-master001 ingress-nginx]# kubectl  apply -f demo-ingress.yaml

Error from server (InternalError): error when applying patch:

{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"nginx","namespace":"default"},"spec":{"rules":[{"host":"nginx.ieasou.cn","http":{"paths":[{"backend":{"service":{"name":"nginx","port":{"number":80}}},"path":"/","pathType":"Prefix"}]}}]}}\n","kubernetes.io/ingress.class":"nginx"}},"spec":{"rules":[{"host":"nginx.ieasou.cn","http":{"paths":[{"backend":{"service":{"name":"nginx","port":{"number":80}}},"path":"/","pathType":"Prefix"}]}}]}}

to:

Resource: "networking.k8s.io/v1, Resource=ingresses", GroupVersionKind: "networking.k8s.io/v1, Kind=Ingress"

Name: "nginx", Namespace: "default"

for: "demo-ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s": x509: certificate is valid for k8s-master002, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not ingress-nginx-controller-admission.ingress-nginx.svc

还是不行。。。

解决办法1、把Webhook删了

admission webhook 传送门

[root@k8s-master001 ingress-nginx]# kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted

再来

[root@k8s-master001 ingress-nginx]# kubectl  apply -f demo-ingress.yaml

ingress.networking.k8s.io/nginx configured

解决办法2、降级为0.32.0（未测）

传送门了解更多

4、现在来查看创建好的ingress,已经创建好了

[root@k8s-master001 ingress-nginx]# kubectl  get ing

Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

NAME    CLASS    HOSTS             ADDRESS       PORTS   AGE

nginx   <none>   nginx.ieasou.cn   10.26.25.21   80      3d19h

5、访问验证,现在在外部就可以通过域名http://nginx.text.cn访问到nginx了

[root@k8s-master001 ingress-nginx]# vim /etc/hosts

10.26.25.21 nginx.text.cn


[root@k8s-master001 ingress-nginx]# curl -I   nginx.text.cn

HTTP/1.1 200 OK

Server: nginx/1.19.2

Date: Wed, 16 Sep 2020 08:05:06 GMT

Content-Type: text/html

Content-Length: 612

Connection: keep-alive

Vary: Accept-Encoding

Last-Modified: Tue, 11 Aug 2020 14:50:35 GMT

ETag: "5f32b03b-264"

Accept-Ranges: bytes

使用ingress-nginx暴露TCP服务

部署一个常用的redis服务,使用ingress-nginx暴露tcp服务
1、编写redis.yaml文件

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

name: redis

labels:

app: redis

spec:

serviceName: redis

replicas: 1

selector:

matchLabels:

app: redis

template:

metadata:

labels:

app: redis

spec:

terminationGracePeriodSeconds: 180

initContainers:

- name: init

image: busybox

command: ["chmod","777","-R","/var/www"]

imagePullPolicy: Always

volumeMounts:

- name: volume

mountPath: /data

containers:

- name: redis

image: redis

imagePullPolicy: Always

ports:

- containerPort: 6379

name: port

volumeMounts:

- name: volume

mountPath: /data

volumeClaimTemplates:





metadata:

name: volume

spec:

accessModes: ["ReadWriteOnce"]

storageClassName: rook-ceph

resources:

requests:

storage: 1Gi







apiVersion: v1

kind: Service

metadata:

name: redis

labels:

app: redis

spec:

type: NodePort

ports:



port: 6379

targetPort: 6379

selector:

app: redis

查看并查看结果

[root@k8s-master001 ingress-nginx]# kubectl apply -f redis.yaml



[root@k8s-master001 ingress-nginx]# kubectl  get po,svc

NAME          READY   STATUS    RESTARTS   AGE

pod/redis-0   1/1     Running   0          104s




NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE

service/redis        NodePort    10.98.28.146     <none>        80:32193/TCP   104s




测试连接redis，能够连接

[root@k8s-master001 ingress-nginx]# telnet 10.26.25.20 32193

Trying 10.26.25.20...

Connected to 10.26.25.20.

Escape character is '^]'.

info

$3615



Server

redis_version:6.0.8

2、默认ingress-nginx的deploy.yaml部署文件并没有开启tcp服务支持，这里需要修改部署文件并重新部署。一般情形，如果需要支持tcp,udp等转发，提前规划并修改deploy.yaml文件
修改如下：--tcp-services-configmap,这里顺便把udp也开启了--udp-services-configmap

      containers:

- name: controller

image: scofield/ingress-nginx-controller:v0.35.0

imagePullPolicy: IfNotPresent

lifecycle:

preStop:

exec:

command:

- /wait-shutdown

args:

- /nginx-ingress-controller

- --election-id=ingress-controller-leader

- --ingress-class=nginx

- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller

- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-service

- --udp-services-configmap=$(POD_NAMESPACE)/udp-service

- --validating-webhook=:8443

- --validating-webhook-certificate=/usr/local/certificates/cert

- --validating-webhook-key=/usr/local/certificates/key

重新部署ingress-nginx

kubectl apply -f deploy.yaml

3、创建tcp服务需要的configmap,注意namespace一定要和ingress-nginx部署的namespace一致
tcp-service.yaml

apiVersion: v1

kind: ConfigMap

metadata:

name: tcp-service

namespace: ingress-nginx

data:

6379: "default/redis:6379"


[root@k8s-master001 ingress-nginx]# kubectl  apply -f tcp-service.yaml

configmap/tcp-service created

4、创建redis-ingress.yaml文件

---

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

name: redis

annotations:

kubernetes.io/ingress.class: nginx

spec:

rules:



host: redis.test.cn

http:

paths:

pathType: Prefix

path: /

backend:

service:

name: redis

port:

number: 6379

5、创建ingress

kubectl apply -f redis-ingress.yaml

[root@k8s-master001 ingress-nginx]# kubectl  get ing

NAME    CLASS    HOSTS             ADDRESS       PORTS   AGE

redis   <none>   redis.ieasou.cn   10.26.25.20   80      13m

6、验证

[root@k8s-master001 ingress-nginx]# telnet redis.test.cn 6379

Trying 10.26.25.20...

Connected to redis.test.cn.

Escape character is '^]'.

info

$3622




Server

redis_version:6.0.8

redis_git_sha1:00000000