Ingress+hostNetwork方式实现nodeIP:80访问

Ingress+ hostNetwork方式实现nodeIP:80访问

node ip + 非80端口，访问k8s集群内部的服务。可是，我们实际生产中更希望的是node ip + 80端口的方式，访问k8s集群内的服务

随机方式很麻烦，我是真不知道该怎么去解析

80端口，可以通过云平台/F5直接解析到边缘节点80去实现访问。

相比较起来，nodePort部署模式中需要部署的ingress-controller容器较少。一个集群可以部署几个就可以了。而hostNetwork模式需要在每个节点部署一个ingress-controller容器，因此总起来消耗资源较多。另外一个比较直观的区别，nodePort模式主要占用的是svc的nodePort端口。而hostNetwork则需要占用物理机的80和443端口。

从网络流转来说，通过nodePort访问时，该node节点不一定部署了ingress-controller容器。因此还需要iptables将其转发到部署有ingress-controller的节点上去，多了一层流转。

另外，通过nodePort访问时，nginx接收到的http请求中的source ip将会被转换为接受该请求的node节点的ip，而非真正的client端ip。

而使用hostNetwork的方式，ingress-controller将会使用的是物理机的DNS域名解析(即物理机的/etc/resolv.conf)。而无法使用内部的比如coredns的域名解析。

我们重新部署一下，如果已经部署完成，修改配置文件，重新加载即可。

1 下载yaml文件

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

#这个文件，实际上是ingress-nginx/deploy/static 下  config.yaml rbac.yaml… 那一堆文件集合

2 修改文件

Deployment 部署的副本 Pod 会分布在各个 Node 上，每个 Node 都可能运行好几个副本。DaemonSet 的不同之处在于：每个 Node 上最多只能运行一个副本。

kind: DaemonSet：官方原始文件使用的是deployment，replicate 为 1，这样将会在某一台节点上启动对应的nginx-ingress-controller pod。外部流量访问至该节点，由该节点负载分担至内部的service。测试环境考虑防止单点故障，改为DaemonSet然后删掉replicate ，配合亲和性部署在制定节点上启动nginx-ingress-controller pod，确保有多个节点启动nginx-ingress-controller pod，后续将这些节点加入到外部硬件负载均衡组实现高可用性。
hostNetwork: true：添加该字段，暴露nginx-ingress-controller pod的服务端口（80）
nodeSelector: 增加亲和性部署，有custom/ingress-controller-ready 标签的节点才会部署该DaemonSet

为需要部署nginx-ingress-controller的节点设置lable

kubectl label nodes node2 custom/ingress-controller-ready=true
kubectl label nodes node3 custom/ingress-controller-ready=true
kubectl label nodes node4 custom/ingress-controller-ready=true

如果你想除了master节点其他的都部署，那直接对master设置污点就行了

kubectl taint nodes k8s-master node-role.kubernetes.io/master=true:NoSchedule

我的是禁止master执行

3 执行yaml文件

[root@k8s-master tmp]# kubectl apply -f mandatory.yaml
namespace/ingress-nginx unchanged
configmap/nginx-configuration unchanged
configmap/tcp-services unchanged
configmap/udp-services unchanged
serviceaccount/nginx-ingress-serviceaccount unchanged
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole unchanged
role.rbac.authorization.k8s.io/nginx-ingress-role unchanged
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding unchanged
daemonset.apps/nginx-ingress-controller created
[root@k8s-master tmp]#

4 查看ingress服务

[root@k8s-master tmp]# kubectl get pods   -n ingress-nginx  -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
nginx-ingress-controller-8n6fz   1/1     Running   0          51s   10.6.76.23   k8s-node-1   <none>           <none>
nginx-ingress-controller-jt82z   1/1     Running   0          51s   10.6.76.24   k8s-node-2   <none>           <none>
[root@k8s-master tmp]#

5 创建一个Nginx测试服务svc和deployment

apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  selector: #标签选择
    name: nginx
  ports:
  - port: 80 #服务器端口
    name: http #名称
    targetPort: 80 #容器端口
    protocol: TCP #协议,默认TCP
 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx-deploy
spec:
  replicas: 3
  selector: 
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
        name: nginx 
    spec:
      containers: 
      - name: nginx
        image: nginx:alpine #镜像
        ports:
        - name: http
          containerPort: 80 #容器端口

[root@k8s-master tmp]# vim nginx-test.yaml
[root@k8s-master tmp]#
[root@k8s-master tmp]# kubectl apply -f nginx-test.yaml
service/nginx-svc created
deployment.apps/my-nginx-deploy created
[root@k8s-master tmp]#
[root@k8s-master tmp]# kubectl get pod,svc -o wide | grep nginx
pod/my-nginx-deploy-b97f5f447-65xh7           1/1     Running   0          7m43s   10.254.1.46    k8s-node-1   <none>           <none>

pod/my-nginx-deploy-b97f5f447-rmsqc           1/1     Running   0          7m43s   10.254.2.77    k8s-node-2   <none>           <none>
pod/my-nginx-deploy-b97f5f447-z57f8           1/1     Running   0          7m43s   10.254.2.78    k8s-node-2   <none>           <none>
service/nginx-svc    ClusterIP   10.105.38.50    <none>        80/TCP           7m43s   name=nginx
[root@k8s-master tmp]#

6 创建一个基于Nginx-test的 ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-nginx
spec:
  rules:
  - host: nginx.wangxu.com
    http:
      paths:
      - backend:
          serviceName: nginx-svc
          servicePort: 80

[root@k8s-master tmp]# kubectl apply -f nginx-test-ingress.yaml
ingress.extensions/ingress-nginx created
[root@k8s-master tmp]#
[root@k8s-master tmp]# kubectl get ingresses
NAME            HOSTS              ADDRESS   PORTS   AGE
ingress-nginx   nginx.wangxu.com             80      22s
[root@k8s-master tmp]#
[root@k8s-master tmp]# kubectl describe ingresses ingress-nginx
Name:             ingress-nginx
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host              Path  Backends
  ----              ----  --------
  nginx.wangxu.com
                       nginx-svc:80 (10.254.1.46:80,10.254.2.77:80,10.254.2.78:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"ingress-nginx","namespace":"default"},"spec":{"rules":[{"host":"nginx.wangxu.com","http":{"paths":[{"backend":{"serviceName":"nginx-svc","servicePort":80}}]}}]}}

Events:
  Type    Reason  Age    From                      Message
  ----    ------  ----   ----                      -------
  Normal  CREATE  5m11s  nginx-ingress-controller  Ingress default/ingress-nginx
  Normal  CREATE  5m11s  nginx-ingress-controller  Ingress default/ingress-nginx
[root@k8s-master tmp]#

7 测试Nginx-test服务

对nodeIP（边缘节点）进行host解析

10.6.76.23 nginx.wangxu.com

10.6.76.24 nginx.wangxu.com

[root@k8s-master tmp]# curl -I  nginx.wangxu.com
HTTP/1.1 200 OK
Server: openresty/1.15.8.2
Date: Mon, 14 Oct 2019 03:31:02 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 24 Sep 2019 16:01:13 GMT
ETag: "5d8a3dc9-264"
Accept-Ranges: bytes

[root@k8s-master tmp]#

8 实现HTTPS

8.1 创建nginx-ssl的service和deployment

apiVersion: v1
kind: Service
metadata:
  name: myapp-ssl
  namespace: default
spec:
  selector:
    app: myhttps
  ports:
  - name: http
    targetPort: 80
    port: 80
  - name: https
    targetPort: 443
    port: 443
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deploy
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myhttps
  template:
    metadata:
      labels:
        app: myhttps
    spec:
      containers:
      - name: myhttps
        image: nginx:alpine
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443

[root@k8s-master tmp]# kubectl apply -f www.yaml
service/myapp-ssl created
deployment.apps/myapp-deploy created
[root@k8s-master tmp]# kubectl get pod,svc -o wide | grep myapp
pod/myapp-deploy-f4b7cc99-8mnzc               1/1     Running   0          7s    10.254.2.81    k8s-node-2   <none>           <none>

pod/myapp-deploy-f4b7cc99-9g2dp               1/1     Running   0          7s    10.254.1.48    k8s-node-1   <none>           <none>
pod/myapp-deploy-f4b7cc99-p9m75               1/1     Running   0          7s    10.254.2.82    k8s-node-2   <none>           <none>
service/myapp-ssl    ClusterIP   10.102.105.57    <none>        80/TCP,443/TCP   7s      app=myhttps
[root@k8s-master tmp]#

8.2 创建ssl的证书和secret

# 创建一个基于自身域名的证书
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout nginxssl.wangxu.com.key -out nginxssl.wangxu.com.pem -subj "/CN=nginxssl.wangxu.com"

# 导入 域名的证书到secret 中
kubectl create secret tls nginxssl-secret  --cert nginxssl.wangxu.com.pem --key nginxssl.wangxu.com.key
#查看
kubectl  get secret | grep nginxssl

[root@k8s-master tmp]# kubectl  get secret | grep nginxssl
nginxssl-secret                      kubernetes.io/tls                     2      39s
[root@k8s-master tmp]#

8.3 配置https的ingress

[root@k8s-master tmp]# cat www-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: www-https
spec:
  tls:
  - hosts:
    - nginxssl.wangxu.com
    secretName: nginxssl-secret
  rules:
  - host: nginxssl.wangxu.com
    http:
      paths:
      - path: /
        backend:
          serviceName:
          servicePort: 443
        backend:
          serviceName: myapp-ssl
          servicePort: 80

[root@k8s-master tmp]#
[root@k8s-master tmp]# kubectl  get ingresses
NAME            HOSTS                 ADDRESS   PORTS     AGE
ingress-nginx   nginx.wangxu.com                80        43m
www-https       nginxssl.wangxu.com             80, 443   9s

9 测试NginxSSL服务

对nodeIP（边缘节点）进行host解析

10.6.76.23 nginxssl.wangxu.com

10.6.76.24 nginxssl.wangxu.com

[root@k8s-master tmp]# curl -k -I https://nginxssl.wangxu.com
HTTP/1.1 200 OK
Server: openresty/1.15.8.2
Date: Mon, 14 Oct 2019 04:00:52 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Strict-Transport-Security: max-age=15724800; includeSubDomains
Last-Modified: Tue, 24 Sep 2019 16:01:13 GMT
ETag: "5d8a3dc9-264"
Accept-Ranges: bytes

[root@k8s-master tmp]#

10 公网发布配置

将云平台的负载均衡/F5 映射到node节点ip上，再把slb/f5公网地址解析出去，外网就能通过域名访问k8s服务了，生产中，证书一般是花钱的拉，并非我们测试的自签名证书。

域名—slb/f5外网ip—nodeIP

ngi

posted on 2019-10-14 12:15 光阴8023 阅读(7083) 评论(0) 编辑收藏举报