kubernetes之部署traefik-ingress分为http和https

1 理解Ingress

简单的说,ingress就是从kubernetes集群外访问集群的入口,将用户的URL请求转发到不同的service上。Ingress相当于nginx、apache等负载均衡方向代理服务器,其中还包括规则定义,即URL的路由信息,路由信息得的刷新由Ingress controller来提供。

2 理解Ingress Controller

Ingress Controller 实质上可以理解为是个监视器,Ingress Controller 通过不断地跟 kubernetes API 打交道,实时的感知后端 service、pod 等变化,比如新增和减少 pod,service 增加与减少等;当得到这些变化信息后,Ingress Controller 再结合下文的 Ingress 生成配置,然后更新反向代理负载均衡器,并刷新其配置,达到服务发现的作用。

3 RBAC

在开始之前,需要先了解一下什么是RBAC。RBAC(基于角色的访问控制)使用 rbac.authorization.k8s.io API 组来实现权限控制,RBAC 允许管理员通过 Kubernetes API 动态的配置权限策略。在 1.6 版本中 RBAC 还处于 Beat 阶段,如果想要开启 RBAC 授权模式需要在 apiserver 组件中指定 --authorization-mode=RBAC 选项。

在 RBAC API 的四个重要概念: 

Role:是一系列的权限的集合,例如一个角色可以包含读取 Pod 的权限和列出 Pod 的权限 
ClusterRole: 跟 Role 类似,但是可以在集群中到处使用( Role 是 namespace 一级的) 
RoloBinding:把角色映射到用户,从而让这些用户继承角色在 namespace 中的权限。 
ClusterRoleBinding: 让用户继承 ClusterRole 在整个集群中的权限。

4 部署traefik

首先部署一个 ingress controller 类型的 pod,pod中运行的容器是traefik,然后选择是通过基于不同的域名或者不同的 URL,关联到不同的 service。而后通过 ingress 来监控 service 的变化,最终形成相应的配置文件。
4.1 创建ingress-rbac.yaml
将用于service account验证。部署之前先创建 clusterRoleBinding,它的作用是将下面列出的权限授予给 traefik-ingress-controller 这个 ServiceAccount,然后 pod 就由这个用户启动。这样一来,pod 就拥有这些权限了。
$ cat ingress-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress
  namespace: kube-system

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: ingress
subjects:
  - kind: ServiceAccount
    name: ingress
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

4.2  创建DaemonSet

traefik 可以部署为 Deployment 和 DaemonSet 两种模式,如果使用官方提供的 deployment 的安装方式,traefik pod 的 80/443/8080 端口会通过 NodePort 的方式暴露出来,也就是说你无法通过节点的 ip + 80 端口进行访问,因此你前面还得给它加上一个负载均衡器。
而官方的 DaemonSet 就不存在这样的问题了,它使用的是 NET_BIND_SERVICE 这样一个 capabilities,意思是可以直接使用宿主机网络名称空间的端口。使用它之后,你在宿主机上看不到它监听了 80 端口,但是你却可以直接访问,而不是 Deployment 这样的 NodePort 方式。但是 DaemonSet 的缺点也很明显,你的 node 节点数量越多,就越消耗资源。
有没有两全其美的方法呢?那肯定是有的,无论你使用 Deployment 还是 DaemonSet,只要都使用 NET_BIND_SERVICE,然后定义好节点的亲和性或者污点就能让 pod 只运行在特定的节点上,然后域名解析指向这些节点就行。当然前提是这些节点的 80/443 都没有被占用。
我们使用DaemonSet类型来部署Traefik,并使用nodeSelector来限定Traefik所部署的主机。

4.2.1 创建节点标签

选择节点来调度traefik-ingress-lb运行在它上面
kubectl label nodes 172.19.8.114 edgenode=true
kubectl label nodes 172.19.8.115 edgenode=true

4.2.2 部署traefik-ingress-controller.yml

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: traefik-ingress-lb
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      restartPolicy: Always
      serviceAccountName: ingress
      containers:
      - image: traefik
        name: traefik-ingress-lb
        resources:
          limits:
            cpu: 200m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: admin
          containerPort: 8080
          hostPort: 8080
        args:
        - --web
        - --web.address=:8080
        - --kubernetes
        - --insecureskipverify
      nodeSelector:
        edgenode: "true"

4.3 配置traefik UI

使用下面的yaml配置来创建Traefik的Web UI。
$ cat ui.yaml
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8580
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  rules:
  - host: traefikui.test.com
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web

配置完成后就可以启动treafik ui ingress了。

这其中的backend中要配置default namespace中启动的service名字,如果你没有配置namespace名字,默认使用default namespace,如果你在其他namespace中创建服务想要暴露到kubernetes集群外部,可以创建新的ingress.yaml文件,同时在文件中指定该namespace,其他配置与上面的文件格式相同。。path就是URL地址后的路径,如traefik.frontend.io/path,service将会接受path这个路径,host最好使用service-name.filed1.filed2.domain-name这种类似主机名称的命名方式,方便区分服务。
根据你自己环境中部署的service的名字和端口自行修改,有新service增加时,修改该文件后可以使用kubectl replace -f ui.yaml来更新。

4.3.1 访问测试

修改本地hosts    172.19.8.115  traefikui.test.com

5 配置traefik支持https

5.1 https介绍

设置https有2种方式
1、 client与traefik间采用https加密通信,但traefik与svc间则是明文的http通信
client --- (via https) ---> traefik ---- (via http) ---->  services

2、 client与traefik间采用https加密通信,但traefik与svc也是采用https通信

client --- (via https) ---> traefik ---- (via https) ---->  services

5.2 配置证书

证书本地存放路径 /etc/kubernetes/ssl/
# mkdir /etc/kubernetes/ssl/
# cd /etc/kubernetes/ssl/
# 将证书存放在此目录
# ls 
pt1.20201216.key  pt1.20201216.pem

5.3  创建secret,保存https证书,注意操作目录,如果不是在此目录下操作,须指定绝对路径,同时配置traefik多域名证书形式,traefik能够自动识别证书

[root@node-01 ssl]# kubectl create secret generic traefik-cert --from-file=/etc/kubernetes/ssl/pt1.20201216.key --from-file=/etc/kubernetes/ssl/pt1.20201216.pem
--from-file=/etc/kubernetes/ssl/pt2.20201216.key --from-file=/etc/kubernetes/ssl/pt2.20201216.pem 
-n kube-system

5.4  创建configmap,保存traefik配置

说明:此处两种配置方式,1.所有http请求全部rewrite为https。2.区分http和https两种请求。此处使用第二种

# 1.所有http请求全部rewrite为https
[root@node-01 traefik]# cat traefik.toml
defaultEntryPoints = ["http","https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/etc/kubernetes/ssl/pt1.20201216.pem"
      keyFile = "/etc/kubernetes/ssl/pt1.20201216.key"
      [[entryPoints.https.tls.certificates]]
      certFile = "/etc/kubernetes/ssl/pt2.20201216.pem"
      keyFile = "/etc/kubernetes/ssl/pt2.20201216.key"
 
# 2.区分http和https两种请求
[root@node-01 traefik]# cat traefik.toml
defaultEntryPoints = ["http","https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/etc/kubernetes/ssl/pt1.20201216.pem"
      keyFile = "/etc/kubernetes/ssl/pt1.20201216.key"
      [[entryPoints.https.tls.certificates]]
      certFile = "/etc/kubernetes/ssl/pt2.20201216.pem"
      keyFile = "/etc/kubernetes/ssl/pt2.20201216.key"
 

注意:部署traefik.toml的节点,需要根据traefik.toml中指明的位置存放证书文件
[root@node-01 traefik]# kubectl create configmap traefik-conf --from-file=traefik.toml -n kube-system
  [root@node-01 traefik]# kubectl get cm -n kube-system
  NAME                                 DATA   AGE
  traefik-conf                         1      13s

查看创建的configmap

[root@node-01 traefik]# kubectl get configmap traefik-conf -o yaml -n kube-system
apiVersion: v1
data:
  traefik.toml: |
    defaultEntryPoints = ["http","https"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
        entryPoint = "https"
      [entryPoints.https]
      address = ":443"
        [entryPoints.https.tls]
          [[entryPoints.https.tls.certificates]]
          certFile = "/etc/kubernetes/ssl/pt1.20201216.pem"
          keyFile = "/etc/kubernetes/ssl/pt1.20201216.key"
          [[entryPoints.https.tls.certificates]]
          certFile = "/etc/kubernetes/ssl/pt2.20201216.pem"
          keyFile = "/etc/kubernetes/ssl/pt2.20201216.key"

kind: ConfigMap metadata: creationTimestamp: "2019-05-07T11:31:07Z" name: traefik-conf namespace: kube-system resourceVersion: "222740" selfLink: /api/v1/namespaces/kube-system/configmaps/traefik-conf uid: 9b315499-70bb-11e9-9a10-fa0caed1a100

5.5  部署traefik,这里主要是要关联创建的secret和configmap,并挂载到pod中的目录。

5.5.1  为主机打标签,确定traefik部署在哪些node上面

kubectl label nodes 172.19.8.114 edgenode=true
kubectl label nodes 172.19.8.115 edgenode=true

5.5.2  由于k8s启用了rbac,因此需要创建权限

[root@node-01 traefik]# cat ingress-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

我们使用DaemonSet类型来部署Traefik,并使用nodeSelector来限定Traefik所部署的主机。

[root@node-01 traefik]# cat traefik-deployment.yaml
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      volumes:
      - name: ssl
        secret:
          secretName: traefik-cert
      - name: config
        configMap:
          name: traefik-conf
      containers:
      - image: traefik
        name: traefik-ingress-lb
        volumeMounts:
        - mountPath: "/etc/kubernetes/ssl"
          name: "ssl"
        - mountPath: "/config"
          name: "config"
        resources:
          limits:
            cpu: 200m
            memory: 300Mi
          requests:
            cpu: 100m
            memory: 200Mi
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443
        - name: admin
          containerPort: 8080
          hostPort: 8080
        args:
        - --api
        - --kubernetes
        - --configfile=/config/traefik.toml
        - --insecureskipverify      
      nodeSelector:
        edgenode: "true"  #指定部署traefik的主机标签
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 443
      name: https
    - protocol: TCP
      port: 8080
      name: admin
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  rules:
  - host: traefikui.test.com
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-ingress-service
          servicePort: admin

volumes用来引用创建的secret和configmap,volumeMounts 指定将引用的secret和configmap挂载到什么位置。

5.5.3 检查创建的traefik ui

[root@node-01 traefik]# kubectl get pods -A |grep trae
kube-system   traefik-ingress-controller-6q7hx        1/1     Running   0          14h
kube-system   traefik-ingress-controller-wgr7d        1/1     Running   0          14h

查看日志,检查有无报错
[root@node-01 traefik]# kubectl logs -f traefik-ingress-controller-wgr7d -n kube-system
测试traefik ui能否正常访问: 

通过本地host测试, 172.19.8.115 traefikui.test.com , 访问 http://traefikui.test.com   和 https://traefikui.test.com

6 实战小测

1、client --- (via https) ---> traefik ---- (via http) ---->  services
[root@node-01 testnginx]# cat test-nginx.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 443
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    app: my-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: my-nginx
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-nginx
spec:
  rules:
  - host: test.test.com
    http:
      paths:
      - backend:
          serviceName: my-nginx
          servicePort: 80
        path: /
  tls:
    - hosts:
      - test.test.com

2、client --- (via https) ---> traefik ---- (via https) ---->  services
此处使用k8s dashboard测试,因为dashboard里面只开启了443端口
假设dashboard已部署完成。此处只需添加ingress文件即可。
[root@node-01 ~]# cat kubernetes-dashboard-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kube-ui
  namespace: kube-system
spec:
  rules:
  - host: k8sui.ptengine.jp
    http:
      paths:
      - path: '/'
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 443

测试完成。
此处需要注意traefik开启 --insecureskipverify 否则会出现403报错。
posted @ 2019-05-23 14:57  cptao  阅读(3823)  评论(0编辑  收藏  举报