一、安装k8s高可用集群
| 主机名 | IP | 配置 | 网络 |
| master控制节点 | 192.168.10.10 | centos 7.9 4核4G | 桥接 |
| node1工作节点 | 192.168.10.11 | centos 7.9 2核2G | 桥接 |
初始化环境步骤到：
https://www.cnblogs.com/yangmeichong/p/16452316.html
docker镜像加速文件
cat > /etc/docker/daemon.json <<EOF { "registry-mirrors":["https://pft7f97f.mirror.aliyuncs.com","https://rsbud4vc.mirror.aliyuncs.com","https://registry.docker-cn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hub-mirror.c.163.com","http://qtid6917.mirror.aliyuncs.com", "https://rncxm540.mirror.aliyuncs.com"], "exec-opts":["native.cgroupdriver=systemd"], "log-driver":"json-file", "log-opts": { "max-size": "100m" }, "storage-driver":"overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ] } EOF systemctl daemon-reload && systemctl restart docker && systemctl status docker
k8s如果是离线安装：
https://github.com/kubernetes/kubernetes 选择对应下载的版本导入镜像，然后使用kubeadm init安装
calico镜像：
Search · Quay需要calico-cni和calico-node,也可以去github下载 calico，以及calicao.yaml
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 12d v1.20.7
node1 Ready worker 12d v1.20.7
[root@master calico]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused
scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused
etcd-0 Healthy {"health":"true"}
看到 cotroller-manager 和 scheduler 是 Unhealthy，按如下方面解决：
vim /etc/kubernetes/manifests/kube-scheduler.yaml
修改如下内容：
把--bind-address=127.0.0.1 变成--bind-address=192.168.10.10
把—port=0 删除
vim /etc/kubernetes/manifests/kube-controller-manager.yaml
修改如下内容：
把--bind-address=127.0.0.1 变成--bind-address=192.168.10.10
把—port=0 删除
修改之后在 k8s 各个节点执行
systemctl restart kubelet
[root@master calico]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
二、Ingress 和 Ingress Controller(七层)概述
2.1 回顾四层负载均衡器Service
（1）Pod漂移问题
Kubernetes 具有强大的副本控制能力，能保证在任意副本（Pod）挂掉时自动从其他机器启动一个新的，还可以动态扩容等，通俗地说，这个 Pod 可能在任何时刻出现在任何节点上，也可能在任何时刻死在任何节点上；那么自然随着 Pod 的创建和销毁，Pod IP 肯定会动态变化；那么如何把这个动态的 Pod IP 暴露出去？
这里借助于 Kubernetes 的 Service 机制，Service 可以以标签的形式选定一组带有指定标签的 Pod，并监控和自动负载他们的 Pod IP，那么我们向外暴露只暴露 Service IP 就行了；这就是 NodePort 模式：即在每个节点上开起一个端口，然后转发到内部 Pod IP 上，如下图所示
此时的访问方式：http://nodeip:nodeport/ ，即数据包流向如下：
客户端请求-->node 节点的 ip:端口--->service 的 ip:端口--->pod 的 ip:端口
（2）端口管理问题
采用 NodePort 方式暴露服务面临的问题是，服务一旦多起来，NodePort 在每个节点上开启的端口会及其庞大，而且难以维护；这时，我们能否使用一个 Nginx 直接对内进行转发呢？
众所周知的是，Pod 与 Pod 之间是可以互相通信的，而 Pod 是可以共享宿主机的网络名称空间的，也就是说当在共享网络名称 空间时，Pod 上所监听的就是 Node 上的端口。那么这又该如何实现呢？
简单的实现就是使用 DaemonSet 在每个 Node 上监听 80，然后写好规则，因为 Nginx 外面绑定了宿主机 80 端口（就像 NodePort），本身又在集群内，那么向后直接转发到相应 Service IP 就行了，
如下图所示：
（3）域名分配及动态更新问题
从上面的方法，采用 Nginx-Pod 似乎已经解决了问题，但是其实这里面有一个很大缺陷：当每次有新服务加入又该如何修改 Nginx 配置呢？
我们知道使用 Nginx 可以通过虚拟主机域名进行区分不同的服务， 而每个服务通过 upstream 进行定义不同的负载均衡池，再加上 location 进行负载均衡的反向代理，在 日常使用中只需要修改 nginx.conf 即可实现，那在 K8S 中又该如何实现这种方式的调度呢？
假设后端的 服务初始服务只有 ecshop，后面增加了 bbs 和 member 服务，那么又该如何将这 2 个服务加入到 Nginx-Pod 进行调度呢？
总不能每次手动改或者 Rolling Update 前端 Nginx Pod 吧！此时 Ingress 出 现了，如果不算上面的 Nginx，Ingress 包含两大组件：Ingress Controller 和 Ingress。
2.2 Ingress介绍
Ingress官网定义:Ingress可以把进入到集群内部的请求转发到集群中的一些服务上，从而可以把服务映射到集群外部。Ingress能把集群内Service 配置成外网能够访问的URL，流量负载均衡，提供基于域名访问的虚拟主机等。
Ingress 简单的理解就是你原来需要改 Nginx 配置，然后配置各种域名对应哪个 Service，现在把这个动作抽象出来，变成一个 Ingress对象，你可以用 yaml 创建，每次不要去改 Nginx 了，直接改 yaml 然后创建/更新就行了；那么问题来了：”Nginx 该怎么处理？”
Ingress Controller 这东西就是解决 “Nginx 的处理方式” 的；Ingress Controller 通过与 Kubernetes API 交互，动态的去感知集群中 Ingress 规则变化，然后读取他，按照他自己模板生成一段 Nginx 配置，再写到 Nginx Pod 里，最后 reload 一下，工作流程如下图：
实际上 Ingress 也是 Kubernetes API 的标准资源类型之一，它其实就是一组基于 DNS 名称（host）或 URL 路径把请求转发到指定的 Service 资源的规则。用于将集群外部的请求流量转发到集群内部完成的服务发布。我们需要明白的是，Ingress 资源自身不能进行“流量穿透”，仅仅是一组规则的集合，这些集合规则还需要其他功能的辅助，比如监听某套接字，然后根据这些规则的匹配进行路由转发，这些能够为 Ingress 资源监听套接字并将流量转发的组件就是 Ingress Controller。
注：Ingress 控制器不同于 Deployment 控制器的是，Ingress 控制器不直接运行为 kube-controller-manager 的一部分，它仅仅是 Kubernetes 集群的一个附件，类似于 CoreDNS，需要在集群上单独部署。
2.3 Ingress Controller概述
Ingress Controller 是一个七层负载均衡调度器，客户端的请求先到达这个七层负载均衡调度器，由七层负载均衡器在反向代理到后端 pod，常见的七层负载均衡器有 nginx、traefik，以我们熟悉的 nginx 为例，假如请求到达 nginx，会通过 upstream 反向代理到后端 pod 应用，但是后端 pod 的 ip 地址是一 直在变化的，因此在后端 pod 前需要加一个 service，这个 service 只是起到分组的作用，那么我们 upstream 只需要填写 service 地址即可
2.4 Ingress 和Ingress Controller 总结
Ingress Controller 可以理解为控制器，它通过不断的跟 Kubernetes API 交互，实时获取后端 Service、Pod 的变化，比如新增、删除等，结合 Ingress 定义的规则生成配置，然后动态更新上边的 Nginx 或者 trafik 负载均衡器，并刷新使配置生效，来达到服务自动发现的作用
Ingress 则是定义规则，通过它定义某个域名的请求过来之后转发到集群中指定的 Service。它可以通过 Yaml 文件定义，可以给一个或多个 Service 定义一个或多个 Ingress 规则。
2.5 使用Ingress Controller代理k8s内部应用的流程
（1）部署 Ingress controller，我们 ingress controller 使用的是 nginx
（2）创建 Service，用来分组 pod
（3）创建 Pod 应用，可以通过控制器创建 pod
（4）创建 Ingress http，测试通过 http 访问应用
（5）创建 Ingress https，测试通过 https 访问应用（域名访问或者ip端口访问）
客户端通过七层调度器访问后端 pod 的方式
使用七层负载均衡调度器 ingress controller 时，当客户端访问 kubernetes 集群内部的应用时，数据包走向如下图流程所示
2.6 安装Nginx Ingress Controller
把 yaml 文件上传到对应的 K8s 的 master1 节点上，把 defaultbackend.tar.gz 和 nginx-ingresscontroller.tar.gz 镜像上传到 k8s 的 node1 节点，手动解压镜像：
[root@node1 ingress]# docker load -i defaultbackend.tar.gz
Loaded image: registry.cn-hangzhou.aliyuncs.com/hachikou/defaultbackend:1.0
[root@node1 ingress]# docker load -i nginx-ingress-controller.tar.gz
Loaded image: registry.cn-hangzhou.aliyuncs.com/peter1009/nginx-ingress-controller:0.20.0
更新yaml文件
[root@master ingress]# kubectl apply -f nginx-ingress-controller-rbac.yml
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
# 注意default-backend.yaml中设置了node工作节点
[root@master ingress]# kubectl apply -f default-backend.yaml
deployment.apps/default-http-backend created
service/default-http-backend created
# nodeName设置要和default-backend.yaml中设置一样，需要调度到统一个基点，否则就会报错
[root@master ingress]# kubectl apply -f nginx-ingress-controller.yaml
deployment.apps/nginx-ingress-controller created
[root@master ingress]# kubectl get pods -n kube-system | grep nginx
nginx-ingress-controller-74cf657846-f6k94 1/1 Running 0 41s
[root@master ingress]# cat nginx-ingress-controller-rbac.yml #apiVersion: v1 #kind: Namespace #metadata: #这里是创建一个namespace，因为此namespace早有了就不用再创建了 # name: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount #创建一个serveerAcount namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: nginx-ingress-clusterrole #这个ServiceAcount所绑定的集群角色 rules: - apiGroups: - "" resources: #此集群角色的权限，它能操作的API资源 - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "extensions" resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: nginx-ingress-role #这是一个角色，而非集群角色 namespace: kube-system rules: #角色的权限 - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get - create - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding #角色绑定 metadata: name: nginx-ingress-role-nisa-binding namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount #绑定在这个用户 namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding #集群绑定 metadata: name: nginx-ingress-clusterrole-nisa-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount #集群绑定到这个serviceacount namespace: kube-system #集群角色是可以跨namespace，但是这里只指明给这个namespce来使用
[root@master ingress]# cat default-backend.yaml apiVersion: apps/v1 kind: Deployment metadata: name: default-http-backend labels: k8s-app: default-http-backend namespace: kube-system spec: replicas: 1 selector: matchLabels: k8s-app: default-http-backend template: metadata: labels: k8s-app: default-http-backend spec: terminationGracePeriodSeconds: 60 containers: - name: default-http-backend # Any image is permissable as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint image: registry.cn-hangzhou.aliyuncs.com/hachikou/defaultbackend:1.0 livenessProbe: httpGet: path: /healthz #这个URI是 nginx-ingress-controller中nginx里配置好的localtion port: 8080 scheme: HTTP initialDelaySeconds: 30 #30s检测一次/healthz timeoutSeconds: 5 ports: - containerPort: 8080 # resources: # limits: # cpu: 10m # memory: 20Mi # requests: # cpu: 10m # memory: 20Mi nodeName: node1 --- apiVersion: v1 kind: Service #为default backend 创建一个service metadata: name: default-http-backend namespace: kube-system labels: k8s-app: default-http-backend spec: ports: - port: 80 targetPort: 8080 selector: k8s-app: default-http-backend
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-ingress-controller labels: k8s-app: nginx-ingress-controller namespace: kube-system spec: replicas: 1 selector: matchLabels: k8s-app: nginx-ingress-controller template: metadata: labels: k8s-app: nginx-ingress-controller spec: # hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration # however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host # that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used # like with kubeadm # hostNetwork: true #注释表示不使用宿主机的80口， terminationGracePeriodSeconds: 60 hostNetwork: true #表示容器使用和宿主机一样的网络 serviceAccountName: nginx-ingress-serviceaccount #引用前面创建的serviceacount containers: - image: registry.cn-hangzhou.aliyuncs.com/peter1009/nginx-ingress-controller:0.20.0 #容器使用的镜像 name: nginx-ingress-controller #容器名 readinessProbe: #启动这个服务时要验证/healthz 端口10254会在运行的node上监听。 httpGet: path: /healthz port: 10254 scheme: HTTP livenessProbe: httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 #每隔10做健康检查 timeoutSeconds: 1 ports: - containerPort: 80 hostPort: 80 #80映射到80 # - containerPort: 443 # hostPort: 443 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend # - --default-ssl-certificate=$(POD_NAMESPACE)/ingress-secret #这是启用Https时用的 # nodeSelector: #指明运行在哪，此IP要和default backend是同一个IP # kubernetes.io/hostname: 10.3.1.17 #上面映射到了hostport80，确保此IP80，443没有占用. # nodeName: node1
# ingress-nginx-controller:v1.9.1版本 # deploy，记得添加hostnetwork:true apiVersion: v1 kind: Namespace metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx name: ingress-nginx --- apiVersion: v1 automountServiceAccountToken: true kind: ServiceAccount metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx namespace: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx namespace: ingress-nginx rules: - apiGroups: - "" resources: - namespaces verbs: - get - apiGroups: - "" resources: - configmaps - pods - secrets - endpoints verbs: - get - list - watch - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses/status verbs: - update - apiGroups: - networking.k8s.io resources: - ingressclasses verbs: - get - list - watch - apiGroups: - coordination.k8s.io resourceNames: - ingress-nginx-leader resources: - leases verbs: - get - update - apiGroups: - coordination.k8s.io resources: - leases verbs: - create - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-admission namespace: ingress-nginx rules: - apiGroups: - "" resources: - secrets verbs: - get - create --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets - namespaces verbs: - list - watch - apiGroups: - coordination.k8s.io resources: - leases verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - networking.k8s.io resources: - ingresses/status verbs: - update - apiGroups: - networking.k8s.io resources: - ingressclasses verbs: - get - list - watch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-admission rules: - apiGroups: - admissionregistration.k8s.io resources: - validatingwebhookconfigurations verbs: - get - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx namespace: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: ingress-nginx subjects: - kind: ServiceAccount name: ingress-nginx namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-admission namespace: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: ingress-nginx-admission subjects: - kind: ServiceAccount name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-nginx subjects: - kind: ServiceAccount name: ingress-nginx namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-admission roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-nginx-admission subjects: - kind: ServiceAccount name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: v1 data: allow-snippet-annotations: "false" kind: ConfigMap metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-controller namespace: ingress-nginx --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-controller namespace: ingress-nginx spec: externalTrafficPolicy: Local ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - appProtocol: http name: http port: 80 protocol: TCP targetPort: http - appProtocol: https name: https port: 443 protocol: TCP targetPort: https selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx type: LoadBalancer --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-controller-admission namespace: ingress-nginx spec: ports: - appProtocol: https name: https-webhook port: 443 targetPort: webhook selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx type: ClusterIP --- apiVersion: apps/v1 kind: Deployment metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-controller namespace: ingress-nginx spec: minReadySeconds: 0 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx strategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate template: metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 spec: hostNetwork: true containers: - args: - /nginx-ingress-controller - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller - --election-id=ingress-nginx-leader - --controller-class=k8s.io/ingress-nginx - --ingress-class=nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: LD_PRELOAD value: /usr/local/lib/libmimalloc.so #image: registry.k8s.io/ingress-nginx/controller:v1.9.1@sha256:605a737877de78969493a4b1213b21de4ee425d2926906857b98050f57a95b25 image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.9.1 imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /wait-shutdown livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: controller ports: - containerPort: 80 name: http protocol: TCP - containerPort: 443 name: https protocol: TCP - containerPort: 8443 name: webhook protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: requests: cpu: 100m memory: 90Mi securityContext: allowPrivilegeEscalation: true capabilities: add: - NET_BIND_SERVICE drop: - ALL runAsUser: 101 volumeMounts: - mountPath: /usr/local/certificates/ name: webhook-cert readOnly: true dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/os: linux serviceAccountName: ingress-nginx terminationGracePeriodSeconds: 300 volumes: - name: webhook-cert secret: secretName: ingress-nginx-admission --- apiVersion: batch/v1 kind: Job metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-admission-create namespace: ingress-nginx spec: template: metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-admission-create spec: containers: - args: - create - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc - --namespace=$(POD_NAMESPACE) - --secret-name=ingress-nginx-admission env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20230407 imagePullPolicy: IfNotPresent name: create securityContext: allowPrivilegeEscalation: false nodeSelector: kubernetes.io/os: linux restartPolicy: OnFailure securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 2000 serviceAccountName: ingress-nginx-admission --- apiVersion: batch/v1 kind: Job metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-admission-patch namespace: ingress-nginx spec: template: metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-admission-patch spec: containers: - args: - patch - --webhook-name=ingress-nginx-admission - --namespace=$(POD_NAMESPACE) - --patch-mutating=false - --secret-name=ingress-nginx-admission - --patch-failure-policy=Fail env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace # image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20230407 imagePullPolicy: IfNotPresent name: patch securityContext: allowPrivilegeEscalation: false nodeSelector: kubernetes.io/os: linux restartPolicy: OnFailure securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 2000 serviceAccountName: ingress-nginx-admission --- apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: nginx spec: controller: k8s.io/ingress-nginx --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-admission namespace: ingress-nginx spec: egress: - {} podSelector: matchLabels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx policyTypes: - Ingress - Egress --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.9.1 name: ingress-nginx-admission webhooks: - admissionReviewVersions: - v1 clientConfig: service: name: ingress-nginx-controller-admission namespace: ingress-nginx path: /networking/v1/ingresses failurePolicy: Fail matchPolicy: Equivalent name: validate.nginx.ingress.kubernetes.io rules: - apiGroups: - networking.k8s.io apiVersions: - v1 operations: - CREATE - UPDATE resources: - ingresses sideEffects: None # ingress-myapp.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myapp-v1 namespace: ingress-nginx spec: ingressClassName: nginx # classname在ingress-nginx-controller中有设置 rules: - host: tomcat.lucky.com http: paths: - path: / pathType: Prefix backend: service: name: tomcat port: number: 8080 # tomcat.yaml修改下namespace # 访问的时候请查看下kubectl get pods -n ingress-nginx | grep nginx 的IP
ingress-nginx部署文件下载：https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.9.1/deploy/static/provider/cloud/deploy.yaml
https://github.com/kubernetes/ingress-nginx/blob/controller-v1.9.1/deploy/static/provider/cloud/deploy.yaml
2.7 测试Ingress HTTP代理k8s内部的tomcat站点
apiVersion: v1 kind: Service metadata: name: tomcat namespace: default spec: selector: app: tomcat release: canary ports: - name: http targetPort: 8080 port: 8080 - name: ajp targetPort: 8009 port: 8009 --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-deploy namespace: default spec: replicas: 2 selector: matchLabels: app: tomcat release: canary template: metadata: labels: app: tomcat release: canary spec: containers: - name: tomcat image: tomcat:8.5.34-jre8-alpine imagePullPolicy: IfNotPresent ports: - name: http containerPort: 8080 name: ajp containerPort: 8009
[root@master tomcat]# kubectl apply -f deploy.yaml service/tomcat created deployment.apps/tomcat-deploy created
[root@master tomcat]# kubectl get pods NAME READY STATUS RESTARTS AGE tomcat-deploy-66b67fcf7b-cwnhq 1/1 Running 0 53s tomcat-deploy-66b67fcf7b-kt4q7 1/1 Running 0 53s
[root@master tomcat]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 12d tomcat ClusterIP 10.110.211.147 <none> 8080/TCP,8009/TCP 111s
# 新版写法 https://kubernetes.io/zh-cn/docs/concepts/services-networking/ingress/#deprecated-annotation [root@master tomcat]# cat ingress-myapp.yaml apiVersion: extensions/v1beta1 #api 版本，新版本换名字了 kind: Ingress #清单类型 metadata: #元数据 name: ingress-myapp #ingress 的名称 namespace: default #所属名称空间 annotations: #注解信息 kubernetes.io/ingress.class: "nginx" spec: #规格 rules: #定义后端转发的规则 - host: tomcat.lucky.com #浏览器访问域名进行转发 http: paths: - path: #配置访问路径，如果通过 url 进行转发，需要修改；空默认为访问的路径为"/" backend: #配置后端服务 serviceName: tomcat servicePort: 8080 # warning忽略 [root@master tomcat]# kubectl apply -f ingress-myapp.yaml Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress ingress.extensions/ingress-myapp created [root@master tomcat]# kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE ingress-myapp <none> tomcat.lucky.com 80 11m [root@master tomcat]# kubectl describe ingress ingress-myapp Name: ingress-myapp Namespace: default Address: Default backend: default-http-backend:80 (10.244.166.129:8080) Rules: Host Path Backends ---- ---- -------- tomcat.lucky.com tomcat:8080 (10.244.166.130:8080,10.244.166.131:8080) Annotations: kubernetes.io/ingress.class: nginx Events: <none> # 应该有东西，但是没出来 [root@master tomcat]# kubectl exec -it nginx-ingress-controller-74cf657846-f6k94 -n kube-system -- /bin/bash www-data@node1:/etc/nginx$ cat nginx.conf server { server_name tomcat.lucky.com ; listen 80; listen [::]:80; set $proxy_upstream_name "-"; location / { set $namespace "default"; set $ingress_name "ingress-myapp"; set $service_name "tomcat"; set $service_port "8080"; set $location_path "/"; proxy_pass http://upstream_balancer;
[root@master tomcat]# kubectl get pods -n kube-system -o wide | grep nginx nginx-ingress-controller-74cf657846-f6k94 1/1 Running 0 62m 192.168.10.11 node1 <none> <none> default-http-backend-d868478df-6fp7f 1/1 Running 0 66m 10.244.166.129 node1 <none> <none> # http访问绑定hosts,本地绑定IP为 192.168.10.11 tomcat.lucky.com
访问：http://tomcat.lucky.com
2.8 测试Ingress HTTPS 代理k8s内部的tomcat
构建TLS站点
#1.准备证书，在k8s控制节点操作 openssl genrsa -out tls.key 2048 openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.lucky.com # 2.生成secret,在控制节点操作 kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key # 3.查看secret [root@master https]# kubectl get secret NAME TYPE DATA AGE default-token-cj9pm kubernetes.io/service-account-token 3 12d tomcat-ingress-secret kubernetes.io/tls 2 37s # 4.创建Ingress [root@master https]# cat ingress-tomcat-tls.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-tomcat-tls namespace: default annotations: kubernetes.io/ingress.class: "nginx" spec: tls: - hosts: - tomcat.lucky.com secretName: tomcat-ingress-secret rules: - host: tomcat.lucky.com http: paths: - path: backend: serviceName: tomcat servicePort: 8080 [root@master https]# kubectl apply -f ingress-tomcat-tls.yaml Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress ingress.extensions/ingress-tomcat-tls created [root@master https]# kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE ingress-myapp <none> tomcat.lucky.com 80 41m ingress-tomcat-tls <none> tomcat.lucky.com 80, 443 34s # tomcat后端服务还是用之前http的,我们只写了ingress规则
访问：https://tomcat.lucky.com
