Kubernetes入门与实战：01
1、Kubernetes基本概览
由于Kubernetes每年三四个版本发布，官网永远是最权威的；话不多说，先提供官网地址：https://kubernetes.io/zh/docs/tutorials/kubernetes-basics/
1.1、集群与节点
- 查看master主键角色：
1 [root@master1 ~]# kubectl get componentstatuses # 也可以使用简写： kubectl get cs 2 Warning: v1 ComponentStatus is deprecated in v1.19+ 3 NAME STATUS MESSAGE ERROR 4 scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused 5 controller-manager Healthy ok 6 etcd-0 Healthy {"health":"true","reason":""} 7 [root@master1 ~]#
- 查看node节点列表：
1 [root@master1 ~]# kubectl get nodes -o wide # 也可以使用简写： kubectl get no 2 NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 3 master1 Ready control-plane,master 44h v1.22.2 192.168.1.66 <none> CentOS Linux 8 4.18.0-240.el8.x86_64 docker://20.10.8 4 node1 Ready <none> 44h v1.22.2 192.168.1.77 <none> CentOS Linux 8 4.18.0-240.el8.x86_64 docker://20.10.8 5 node2 Ready <none> 44h v1.22.2 192.168.1.88 <none> CentOS Linux 8 4.18.0-240.el8.x86_64 docker://20.10.8 6 [root@master1 ~]#
- 查看node节点详情：
1 [root@master1 ~]# kubectl describe nodes node2 2 Name: node2 3 Roles: <none> 4 Labels: beta.kubernetes.io/arch=amd64 5 beta.kubernetes.io/os=linux 6 kubernetes.io/arch=amd64 7 kubernetes.io/hostname=node2 8 kubernetes.io/os=linux 9 Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"be:68:4a:f9:ac:de"} 10 flannel.alpha.coreos.com/backend-type: vxlan 11 flannel.alpha.coreos.com/kube-subnet-manager: true 12 flannel.alpha.coreos.com/public-ip: 192.168.1.88 13 kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock 14 node.alpha.kubernetes.io/ttl: 0 15 volumes.kubernetes.io/controller-managed-attach-detach: true 16 CreationTimestamp: Fri, 01 Oct 2021 06:59:33 -0400 17 Taints: <none> 18 Unschedulable: false 19 Lease: 20 HolderIdentity: node2 21 AcquireTime: <unset> 22 RenewTime: Sun, 03 Oct 2021 03:45:59 -0400 23 Conditions: 24 Type Status LastHeartbeatTime LastTransitionTime Reason Message 25 ---- ------ ----------------- ------------------ ------ ------- 26 NetworkUnavailable False Sun, 03 Oct 2021 02:47:43 -0400 Sun, 03 Oct 2021 02:47:43 -0400 FlannelIsUp Flannel is running on this node 27 MemoryPressure False Sun, 03 Oct 2021 03:42:49 -0400 Fri, 01 Oct 2021 06:59:33 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available 28 DiskPressure False Sun, 03 Oct 2021 03:42:49 -0400 Fri, 01 Oct 2021 06:59:33 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure 29 PIDPressure False Sun, 03 Oct 2021 03:42:49 -0400 Fri, 01 Oct 2021 06:59:33 -0400 KubeletHasSufficientPID kubelet has sufficient PID available 30 Ready True Sun, 03 Oct 2021 03:42:49 -0400 Fri, 01 Oct 2021 07:03:54 -0400 KubeletReady kubelet is posting ready status 31 Addresses: # 节点的IP地址和主机名 32 InternalIP: 192.168.1.88 33 Hostname: node2 34 Capacity: # 节点总资源 35 cpu: 2 36 ephemeral-storage: 17398Mi 37 hugepages-1Gi: 0 38 hugepages-2Mi: 0 39 memory: 1996236Ki 40 pods: 110 41 Allocatable: # 节点可以分配的资源 42 cpu: 2 43 ephemeral-storage: 16418812697 44 hugepages-1Gi: 0 45 hugepages-2Mi: 0 46 memory: 1893836Ki 47 pods: 110 48 System Info: # 系统的相关信息 49 Machine ID: fd49b99f63534bc9a6544ff2f1d68ef5 50 System UUID: 2a454d56-033c-c61e-c4f8-37b171f12218 51 Boot ID: c66bb70c-b358-4d60-b6f2-9f2cb49dad2e 52 Kernel Version: 4.18.0-240.el8.x86_64 53 OS Image: CentOS Linux 8 54 Operating System: linux 55 Architecture: amd64 56 Container Runtime Version: docker://20.10.8 57 Kubelet Version: v1.22.2 # kubelet版本 58 Kube-Proxy Version: v1.22.2 # kube-proxy版本 59 PodCIDR: 172.16.2.0/24 # pod网络 60 PodCIDRs: 172.16.2.0/24 61 Non-terminated Pods: (2 in total) # 当前节点已经运行pod情况，及每个pod占用的资源情况 62 Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age 63 --------- ---- ------------ ---------- --------------- ------------- --- 64 kube-system kube-flannel-ds-zp785 100m (5%) 100m (5%) 50Mi (2%) 50Mi (2%) 44h 65 kube-system kube-proxy-s6ndp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 53m 66 Allocated resources: # 已分配资源情况 67 (Total limits may be over 100 percent, i.e., overcommitted.) 68 Resource Requests Limits 69 -------- -------- ------ 70 cpu 100m (5%) 100m (5%) 71 memory 50Mi (2%) 50Mi (2%) 72 ephemeral-storage 0 (0%) 0 (0%) 73 hugepages-1Gi 0 (0%) 0 (0%) 74 hugepages-2Mi 0 (0%) 0 (0%) 75 Events: 76 Type Reason Age From Message 77 ---- ------ ---- ---- ------- 78 Normal Starting 44h kubelet Starting kubelet. 79 Normal NodeHasSufficientMemory 44h (x2 over 44h) kubelet Node node2 status is now: NodeHasSufficientMemory 80 Normal NodeHasNoDiskPressure 44h (x2 over 44h) kubelet Node node2 status is now: NodeHasNoDiskPressure 81 Normal NodeHasSufficientPID 44h (x2 over 44h) kubelet Node node2 status is now: NodeHasSufficientPID 82 Normal NodeAllocatableEnforced 44h kubelet Updated Node Allocatable limit across pods 83 Normal NodeReady 44h kubelet Node node2 status is now: NodeReady 84 [root@master1 ~]#
1.2、容器与应用
kubernetes是容器编排引擎，其负责容器的调度，管理和容器的运行，但kubernetes调度最小单位并非是container，而是pod，pod中可包含多个container，通常集群中不会直接运行pod，而是通过各种工作负载的控制器如Deployments，ReplicaSets，DaemonSets的方式运行，为啥？因为控制器能够保证pod状态的一致性，正如官方所描述的一样“make sure the current state match to the desire state”，确保当前状态和预期的一致，简单来说就是pod异常了，控制器会在其他节点重建，确保集群当前运行的pod和预期设定的一致。
- pod是kubernetes中运行的最小单元
- pod中包含一个容器或者多个容器
- pod不会单独使用，需要有工作负载来控制，如Deployments，StatefulSets，DaemonSets，CronJobs等
Pod概览：
1.3、服务访问
kubernetes中pod是实际运行的载体，pod依附于node中，node可能会出现故障，kubernetes的控制器如replicasets会在其他node上重新拉起一个pod，新的pod会分配一个新的IP；再者，应用部署时会包含多个副本replicas，如同个应用deployments部署了3个pod副本，pod相当于后端的Real Server，如何实现这三个应用访问呢？对于这种情况，我们一般会在Real Server前面加一个负载均衡Load Balancer，service就是pod的负载均衡调度器，service将动态的pod抽象为一个服务，应用程序直接访问service即可，service会自动将请求转发到后端的pod。负责service转发规则有两种机制：iptables和ipvs，iptables通过设置DNAT等规则实现负载均衡，ipvs通过ipvsadm设置转发规。
Service 和 Label：
根据服务不同的访问方式，service分为如下几种类型：ClusterIP，NodePort，LoadBalancer和_ExternalName，可通过type设置：
- ClusterIP，集群内部互访，与DNS结合实现集群内部的服务发现；
- NodePort，通过NAT将每个node节点暴露一个端口实现外部访问；
- LoadBalancer，实现云厂商外部接入方式的接口，需要依赖云服务提供商实现具体技术细节，如腾讯云实现与CLB集成；
- ExternalName，通过服务名字暴露服务名，当前可由ingress实现，将外部的请求以域名转发的形式转发到集群，需要依附具体的外部实现，如nginx，traefik,各大云计算厂商实现接入细节；
pod是动态变化的，ip地址可能会变化（如node故障），副本数可能会变化，如应用扩展scale up，应用锁容scale down等，service如何识别到pod的动态变化呢？答案是labels，通过labels自动会过滤出某个应用的Endpoints，当pod变化时会自动更新Endpoints，不同的应用会有由不同的label组成。labels相关可以参考下 https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
2、创建应用
下面开始部署一个应用即deployments，kubernetes中包含各种workload如无状态话的Deployments，有状态化的StatefulSets，守护进程的DaemonSets，每种workload对应不同的应用场景，我们先以Deployments为例入门，其他workload均以此类似，一般而言，在kubernetes中部署应用均以yaml文件方式部署，对于初学者而言，编写yaml文件太冗长，不适合初学，我们先kubectl命令行方式实现API的接入。
2.1、创建两个副本的nginx的应用并查看是否处于：AVAILABLE状态：
1 [root@master1 ~]# kubectl create deployment demo-nginx --image=nginx:1.8.1 --replicas=2 2 deployment.apps/demo-nginx created 3 [root@master1 ~]# 4 [root@master1 ~]# kubectl get deployments.apps 5 NAME READY UP-TO-DATE AVAILABLE AGE 6 demo-nginx 2/2 2 2 59s 7 [root@master1 ~]#
2.2、查看应用的详细信息,我们可以知道Deployments是通过ReplicaSets控制副本数的，由Replicaset控制pod数：
1 [root@master1 ~]# kubectl describe deployments.apps demo-nginx 2 Name: demo-nginx 3 Namespace: default 4 CreationTimestamp: Sun, 03 Oct 2021 04:13:57 -0400 5 Labels: app=demo-nginx # 标签，后面的service选择的依据 6 Annotations: deployment.kubernetes.io/revision: 1 7 Selector: app=demo-nginx 8 Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable 9 StrategyType: RollingUpdate # 滚动升级的策略 10 MinReadySeconds: 0 11 RollingUpdateStrategy: 25% max unavailable, 25% max surge 12 Pod Template: # 容器应用模版，包含镜像，port，存储等 13 Labels: app=demo-nginx 14 Containers: 15 nginx: 16 Image: nginx:1.8.1 17 Port: <none> 18 Host Port: <none> 19 Environment: <none> 20 Mounts: <none> 21 Volumes: <none> 22 Conditions: 23 Type Status Reason 24 ---- ------ ------ 25 Available True MinimumReplicasAvailable 26 Progressing True NewReplicaSetAvailable 27 OldReplicaSets: <none> 28 NewReplicaSet: demo-nginx-55ccc46db6 (2/2 replicas created) # 控制器的名称 29 Events: 30 Type Reason Age From Message 31 ---- ------ ---- ---- ------- 32 Normal ScalingReplicaSet 78s deployment-controller Scaled up replica set demo-nginx-55ccc46db6 to 2 33 [root@master1 ~]#
2.3、查看replicasets情况，通过查看可知replicasets副本控制器生成了两个pod：
1 [root@master1 ~]# kubectl get replicasets.apps 2 NAME DESIRED CURRENT READY AGE 3 demo-nginx-55ccc46db6 2 2 2 112s 4 [root@master1 ~]# 5 [root@master1 ~]# kubectl describe replicasets.apps demo-nginx-55ccc46db6 6 Name: demo-nginx-55ccc46db6 7 Namespace: default 8 Selector: app=demo-nginx,pod-template-hash=55ccc46db6 9 Labels: app=demo-nginx 10 pod-template-hash=55ccc46db6 11 Annotations: deployment.kubernetes.io/desired-replicas: 2 # 滚动升级的信息，副本树，最大数，应用版本 12 deployment.kubernetes.io/max-replicas: 3 13 deployment.kubernetes.io/revision: 1 14 Controlled By: Deployment/demo-nginx # 控制器类型 15 Replicas: 2 current / 2 desired # 当前和期望的pod数量 16 Pods Status: 2 Running / 0 Waiting / 0 Succeeded / 0 Failed 17 Pod Template: 18 Labels: app=demo-nginx 19 pod-template-hash=55ccc46db6 20 Containers: 21 nginx: 22 Image: nginx:1.8.1 23 Port: <none> 24 Host Port: <none> 25 Environment: <none> 26 Mounts: <none> 27 Volumes: <none> 28 Events: # 日志事件：生成了三个不同的pod 29 Type Reason Age From Message 30 ---- ------ ---- ---- ------- 31 Normal SuccessfulCreate 2m7s replicaset-controller Created pod: demo-nginx-55ccc46db6-lg4nz 32 Normal SuccessfulCreate 2m7s replicaset-controller Created pod: demo-nginx-55ccc46db6-jmjkx 33 [root@master1 ~]#
2.4、查看pod的情况，实际应用部署的载体，pod中部署了一个nginx的容器并分配了一个ip，可通过该ip直接访问应用：
1 [root@master1 ~]# kubectl get pods -o wide # 每个pod的名称前缀和上面的replicasets一样 2 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 3 demo-nginx-55ccc46db6-jmjkx 1/1 Running 0 20m 172.16.2.4 node2 <none> <none> 4 demo-nginx-55ccc46db6-lg4nz 1/1 Running 0 20m 172.16.1.5 node1 <none> <none> 5 [root@master1 ~]# 6 [root@master1 ~]# kubectl describe pods demo-nginx-55ccc46db6-jmjkx 7 Name: demo-nginx-55ccc46db6-jmjkx 8 Namespace: default 9 Priority: 0 10 Node: node2/192.168.1.88 11 Start Time: Sun, 03 Oct 2021 04:13:57 -0400 12 Labels: app=demo-nginx 13 pod-template-hash=55ccc46db6 14 Annotations: <none> 15 Status: Running 16 IP: 172.16.2.4 # pod的IP地址 17 IPs: 18 IP: 172.16.2.4 19 Controlled By: ReplicaSet/demo-nginx-55ccc46db6 20 Containers: 21 nginx: 22 Container ID: docker://430d6e35fb2bb8e4cce6d498591da05748592ad254182c19dc87e41fca951dfd 23 Image: nginx:1.8.1 24 Image ID: docker-pullable://nginx@sha256:9b3e9f189890ef9d6713c3384da3809731bdb0bff84e7b68da330ebadf533085 25 Port: <none> 26 Host Port: <none> 27 State: Running 28 Started: Sun, 03 Oct 2021 04:14:28 -0400 29 Ready: True 30 Restart Count: 0 31 Environment: <none> 32 Mounts: 33 /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n2r59 (ro) 34 Conditions: 35 Type Status 36 Initialized True 37 Ready True 38 ContainersReady True 39 PodScheduled True 40 Volumes: 41 kube-api-access-n2r59: 42 Type: Projected (a volume that contains injected data from multiple sources) 43 TokenExpirationSeconds: 3607 44 ConfigMapName: kube-root-ca.crt 45 ConfigMapOptional: <nil> 46 DownwardAPI: true 47 QoS Class: BestEffort 48 Node-Selectors: <none> 49 Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s 50 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s 51 Events: 52 Type Reason Age From Message 53 ---- ------ ---- ---- ------- 54 Normal Scheduled 21m default-scheduler Successfully assigned default/demo-nginx-55ccc46db6-jmjkx to node2 55 Normal Pulling 21m kubelet Pulling image "nginx:1.8.1" 56 Normal Pulled 21m kubelet Successfully pulled image "nginx:1.8.1" in 30.161859239s 57 Normal Created 21m kubelet Created container nginx 58 Normal Started 21m kubelet Started container nginx 59 [root@master1 ~]# 60 [root@master1 ~]# curl 172.16.2.4 # 通过pod的IP地址直接访问 61 <!DOCTYPE html> 62 <html> 63 <head> 64 <title>Welcome to nginx!</title> 65 <style> 66 body { 67 width: 35em; 68 margin: 0 auto; 69 font-family: Tahoma, Verdana, Arial, sans-serif; 70 } 71 </style> 72 </head> 73 <body> 74 <h1>Welcome to nginx!</h1> 75 <p>If you see this page, the nginx web server is successfully installed and 76 working. Further configuration is required.</p> 77 78 <p>For online documentation and support please refer to 79 <a href="http://nginx.org/">nginx.org</a>.<br/> 80 Commercial support is available at 81 <a href="http://nginx.com/">nginx.com</a>.</p> 82 83 <p><em>Thank you for using nginx.</em></p> 84 </body> 85 </html> 86 [root@master1 ~]#
3、访问应用
kubernetes为每个pod都分配了一个ip地址，可通过该地址直接访问应用，相当于访问RS，但一个应用是一个整体，由多个副本数组成，需要依赖于service来实现应用的负载均衡，service我们探讨ClusterIP和NodePort的访问方式。
3.1、设置pod的内容，为了方便区分，我们将两个pod的nginx站点内容设置为不同，以观察通过各自的IP的访问效果：
1 [root@master1 ~]# kubectl get pods -o wide 2 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 3 demo-nginx-55ccc46db6-jmjkx 1/1 Running 0 35m 172.16.2.4 node2 <none> <none> 4 demo-nginx-55ccc46db6-lg4nz 1/1 Running 0 35m 172.16.1.5 node1 <none> <none> 5 [root@master1 ~]# 6 [root@master1 ~]# kubectl exec -it demo-nginx-55ccc46db6-jmjkx /bin/bash # 修改网页内容为：nginx-web1 7 kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. 8 root@demo-nginx-55ccc46db6-jmjkx:/# echo 'nginx-web1' > /usr/share/nginx/html/index.html 9 root@demo-nginx-55ccc46db6-jmjkx:/# exit 10 exit 11 [root@master1 ~]# kubectl exec -it demo-nginx-55ccc46db6-lg4nz /bin/bash # 修改网页内容为：nginx-web2 12 kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. 13 root@demo-nginx-55ccc46db6-lg4nz:/# echo 'nginx-web2' > /usr/share/nginx/html/index.html 14 root@demo-nginx-55ccc46db6-lg4nz:/# exit 15 exit 16 [root@master1 ~]# # 验证访问结果 17 [root@master1 ~]# curl http://172.16.2.4 18 nginx-web1 19 [root@master1 ~]# curl http://172.16.1.5 20 nginx-web2 21 [root@master1 ~]#
3.2、通过ClusterIP访问
通过pod的ip直接访问应用，对于单个pod的应用可以实现，对于多个副本replicas的应用则不符合要求，需要通过service来实现负载均衡，service需要设置不同的type，默认为ClusterIP即集群内部访问，如下通过expose子命令将服务暴露到service：
- 暴露service,其中port表示代理监听端口，target-port代表是容器的端口，type设置的是service的类型，查看service的详情，可以看到service通过labels选择器selector自动将pod的ip生成endpoints
1 [root@master1 ~]# kubectl expose deployment demo-nginx --port=80 --protocol=TCP --type ClusterIP 2 service/demo-nginx exposed 3 [root@master1 ~]# 4 [root@master1 ~]# kubectl get svc # 查看service列表，显示有两个，kubernetes为默认集群创建的service 5 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 6 demo-nginx ClusterIP 10.96.199.219 <none> 80/TCP 15s 7 kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 46h 8 [root@master1 ~]# 9 [root@master1 ~]# kubectl describe service demo-nginx # 查看service详情，可以看到Labels的Seletor和前面Deployments设置一致，Endpoints将pod组成一个列表 10 Name: demo-nginx 11 Namespace: default 12 Labels: app=demo-nginx 13 Annotations: <none> 14 Selector: app=demo-nginx 15 Type: ClusterIP # service类型为ClusterIP 16 IP Family Policy: SingleStack 17 IP Families: IPv4 18 IP: 10.96.199.219 19 IPs: 10.96.199.219 20 Port: <unset> 80/TCP # 服务端口，即ClusterIP对外访问的端口 21 TargetPort: 80/TCP 22 Endpoints: 172.16.1.5:80,172.16.2.4:80 # 访问后端地址列表 23 Session Affinity: None # 默认调度算法为：rr 24 Events: <none> 25 [root@master1 ~]# 26 [root@master1 ~]# kubectl get endpoints 27 NAME ENDPOINTS AGE 28 demo-nginx 172.16.1.5:80,172.16.2.4:80 71s 29 kubernetes 192.168.1.66:6443 46h 30 [root@master1 ~]# 31 [root@master1 ~]# kubectl describe endpoints demo-nginx 32 Name: demo-nginx 33 Namespace: default 34 Labels: app=demo-nginx 35 Annotations: endpoints.kubernetes.io/last-change-trigger-time: 2021-10-03T08:59:17Z 36 Subsets: 37 Addresses: 172.16.1.5,172.16.2.4 38 NotReadyAddresses: <none> 39 Ports: 40 Name Port Protocol 41 ---- ---- -------- 42 <unset> 80 TCP 43 44 Events: <none> 45 [root@master1 ~]#
- 通过service访问服务，默认的调度算法是：rr；同时可以通过ipvsadm命令检查相关调度的详情：
1 [root@master1 ~]# kubectl get service 2 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 3 demo-nginx ClusterIP 10.96.199.219 <none> 80/TCP 6m52s 4 kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 46h 5 [root@master1 ~]# 6 [root@master1 ~]# curl http://10.96.199.219 7 nginx-web1 8 [root@master1 ~]# curl http://10.96.199.219 9 nginx-web2 10 [root@master1 ~]# curl http://10.96.199.219 11 nginx-web1 12 [root@master1 ~]# curl http://10.96.199.219 13 nginx-web2 14 [root@master1 ~]# 15 [root@master1 ~]# kubectl get pods -o wide 16 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 17 demo-nginx-55ccc46db6-jmjkx 1/1 Running 0 52m 172.16.2.4 node2 <none> <none> 18 demo-nginx-55ccc46db6-lg4nz 1/1 Running 0 52m 172.16.1.5 node1 <none> <none> 19 [root@master1 ~]# 20 [root@master1 ~]# ipvsadm -ln 21 IP Virtual Server version 1.2.1 (size=4096) 22 Prot LocalAddress:Port Scheduler Flags 23 -> RemoteAddress:Port Forward Weight ActiveConn InActConn 24 TCP 10.96.0.1:443 rr 25 -> 192.168.1.66:6443 Masq 1 0 0 26 TCP 10.96.0.10:53 rr 27 -> 172.16.0.4:53 Masq 1 0 0 28 -> 172.16.0.5:53 Masq 1 0 0 29 TCP 10.96.0.10:9153 rr 30 -> 172.16.0.4:9153 Masq 1 0 0 31 -> 172.16.0.5:9153 Masq 1 0 0 32 TCP 10.96.199.219:80 rr # 通过ipvsadm命令可以很清晰的看到想情况，包括上面访问了四次，两台服务器上各有两次访问 33 -> 172.16.1.5:80 Masq 1 0 2 34 -> 172.16.2.4:80 Masq 1 0 2 35 UDP 10.96.0.10:53 rr 36 -> 172.16.0.4:53 Masq 1 0 0 37 -> 172.16.0.5:53 Masq 1 0 0 38 [root@master1 ~]#
3.3、通过NodePort访问
Service通过ClusterIP只能提供集群内部的应用访问，外部无法直接访问应用，如果需要外部访问有如下几种方式：NodePort，LoadBalancer和Ingress，其中LoadBalancer需要由云服务提供商实现，Ingress需要安装单独的Ingress Controller，日常测试可以通过NodePort的方式实现，NodePort可以将node的某个端口暴露给外部网络访问。
- 修改type的类型由ClusterIP修改为NodePort类型（或者重新创建，指定type的类型为NodePort），并验证访问结果（包括原来的service的地址是否可以访问）
1 [root@master1 ~]# kubectl get service demo-nginx -o yaml 2 apiVersion: v1 3 kind: Service 4 metadata: 5 creationTimestamp: "2021-10-03T08:59:17Z" 6 labels: 7 app: demo-nginx 8 name: demo-nginx 9 namespace: default 10 resourceVersion: "13905" 11 uid: 148f6bc4-963c-4751-a733-02fd8a88c0bc 12 spec: 13 clusterIP: 10.96.199.219 14 clusterIPs: 15 - 10.96.199.219 16 internalTrafficPolicy: Cluster 17 ipFamilies: 18 - IPv4 19 ipFamilyPolicy: SingleStack 20 ports: 21 - port: 80 22 protocol: TCP 23 targetPort: 80 24 selector: 25 app: demo-nginx 26 sessionAffinity: None 27 type: ClusterIP # 修改前 28 status: 29 loadBalancer: {} 30 [root@master1 ~]# 31 [root@master1 ~]# kubectl patch service demo-nginx -p '{"spec":{"type":"NodePort"}}' # 你也可以直接使用：kubectl edit service demo-nginx 直接修改 32 service/demo-nginx patched 33 [root@master1 ~]# 34 [root@master1 ~]# kubectl get service demo-nginx -o yaml 35 apiVersion: v1 36 kind: Service 37 metadata: 38 creationTimestamp: "2021-10-03T08:59:17Z" 39 labels: 40 app: demo-nginx 41 name: demo-nginx 42 namespace: default 43 resourceVersion: "15278" 44 uid: 148f6bc4-963c-4751-a733-02fd8a88c0bc 45 spec: 46 clusterIP: 10.96.199.219 47 clusterIPs: 48 - 10.96.199.219 49 externalTrafficPolicy: Cluster 50 internalTrafficPolicy: Cluster 51 ipFamilies: 52 - IPv4 53 ipFamilyPolicy: SingleStack 54 ports: 55 - nodePort: 32638 56 port: 80 57 protocol: TCP 58 targetPort: 80 59 selector: 60 app: demo-nginx 61 sessionAffinity: None 62 type: NodePort # 修改后 63 status: 64 loadBalancer: {} 65 [root@master1 ~]# 66 [root@master1 ~]# kubectl get service 67 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 68 demo-nginx NodePort 10.96.199.219 <none> 80:32638/TCP 16m 69 kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 46h 70 [root@master1 ~]# 71 [root@master1 ~]# cat /etc/hosts 72 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 73 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 74 192.168.1.66 master1 75 192.168.1.77 node1 76 192.168.1.88 node2 77 [root@master1 ~]# # 验证访问结果：通过NodePort访问应用程序,每个node的地址相当于vip，可以实现相同的负载均衡效果，同时CluserIP功能依可用 78 [root@master1 ~]# curl http://master1:32638 79 nginx-web1 80 [root@master1 ~]# curl http://node1:32638 81 nginx-web1 82 [root@master1 ~]# curl http://node2:32638 83 nginx-web1 84 [root@master1 ~]# curl http://node2:32638 85 nginx-web2 86 [root@master1 ~]# # 验证访问结果：通过NodePort访问应用程序,同时CluserIP功能依可用 87 [root@master1 ~]# curl http://10.96.199.219 88 nginx-web1 89 [root@master1 ~]# curl http://10.96.199.219 90 nginx-web2 91 [root@master1 ~]#
4、扩展应用
当应用程序的负载比较高无法满足应用请求时，一般我们会通过扩展RS的数量来实现，在kubernetes中，扩展RS实际上通过扩展副本数replicas来实现，扩展RS非常便利，快速实现弹性伸缩。kubernets能提供两种方式的伸缩能力：
- 手动伸缩能力scale up和scale down；
- 动态的弹性伸缩horizontalpodautoscalers,基于CPU的利用率实现自动的弹性伸缩，需要依赖与监控组件如metrics server，当前未实现，后续再做深入探讨，本文以手动的scale的方式扩展应用的副本数。
- 手动扩展副本数到四副本并查看：
1 [root@master1 ~]# kubectl get deployments.apps -o wide 2 NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR 3 demo-nginx 2/2 2 2 92m nginx nginx:1.8.1 app=demo-nginx 4 [root@master1 ~]# 5 [root@master1 ~]# kubectl get deployments.apps -o wide 6 NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR 7 demo-nginx 2/2 2 2 92m nginx nginx:1.8.1 app=demo-nginx 8 [root@master1 ~]# 9 [root@master1 ~]# kubectl scale deployment demo-nginx --replicas=4 # 扩容到4副本 10 deployment.apps/demo-nginx scaled 11 [root@master1 ~]# 12 [root@master1 ~]# kubectl get deployments.apps -o wide 13 NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR 14 demo-nginx 4/4 4 4 93m nginx nginx:1.8.1 app=demo-nginx 15 [root@master1 ~]#
- 此时再次查看service的情况会怎样呢？查看service详情,新扩展的pod会自动更新到service的endpoints中，自动服务发现：
1 [root@master1 ~]# kubectl get service 2 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 3 demo-nginx NodePort 10.96.199.219 <none> 80:32638/TCP 51m 4 kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 46h 5 [root@master1 ~]# 6 [root@master1 ~]# kubectl describe service demo-nginx 7 Name: demo-nginx 8 Namespace: default 9 Labels: app=demo-nginx 10 Annotations: <none> 11 Selector: app=demo-nginx 12 Type: NodePort 13 IP Family Policy: SingleStack 14 IP Families: IPv4 15 IP: 10.96.199.219 16 IPs: 10.96.199.219 17 Port: <unset> 80/TCP 18 TargetPort: 80/TCP 19 NodePort: <unset> 32638/TCP 20 Endpoints: 172.16.1.5:80,172.16.1.6:80,172.16.2.4:80 + 1 more... # 新生成的两个pod地址已经加进来了 21 Session Affinity: None 22 External Traffic Policy: Cluster 23 Events: <none> 24 [root@master1 ~]# 25 [root@master1 ~]# kubectl describe endpoints demo-nginx 26 Name: demo-nginx 27 Namespace: default 28 Labels: app=demo-nginx 29 Annotations: endpoints.kubernetes.io/last-change-trigger-time: 2021-10-03T09:47:26Z 30 Subsets: 31 Addresses: 172.16.1.5,172.16.1.6,172.16.2.4,172.16.2.5 # 新生成的两个pod地址已经加进来了 32 NotReadyAddresses: <none> 33 Ports: 34 Name Port Protocol 35 ---- ---- -------- 36 <unset> 80 TCP 37 38 Events: <none> 39 [root@master1 ~]#
- 测试验证，将新加入的两个pod站点内容设置为：web3，web4，参考前面的设置方法，测试service的ip，查看负载均衡效果：
1 [root@master1 ~]# kubectl get pods -o wide 2 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 3 demo-nginx-55ccc46db6-jmjkx 1/1 Running 0 93m 172.16.2.4 node2 <none> <none> 4 demo-nginx-55ccc46db6-l2vj5 1/1 Running 0 26s 172.16.2.5 node2 <none> <none> 5 demo-nginx-55ccc46db6-lg4nz 1/1 Running 0 93m 172.16.1.5 node1 <none> <none> 6 demo-nginx-55ccc46db6-vrlhw 1/1 Running 0 26s 172.16.1.6 node1 <none> <none> 7 [root@master1 ~]# # 修改新加入的两个pod的网页内容： 8 [root@master1 ~]# kubectl exec -it demo-nginx-55ccc46db6-l2vj5 /bin/bash 9 kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. 10 root@demo-nginx-55ccc46db6-l2vj5:/# echo 'nginx-web3' > /usr/share/nginx/html/index.html 11 root@demo-nginx-55ccc46db6-l2vj5:/# exit 12 exit 13 [root@master1 ~]# kubectl exec -it demo-nginx-55ccc46db6-vrlhw /bin/bash 14 kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. 15 root@demo-nginx-55ccc46db6-vrlhw:/# echo 'nginx-web4' > /usr/share/nginx/html/index.html 16 root@demo-nginx-55ccc46db6-vrlhw:/# exit 17 exit 18 [root@master1 ~]# 19 [root@master1 ~]# kubectl get service 20 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 21 demo-nginx NodePort 10.96.199.219 <none> 80:32638/TCP 51m 22 kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 46h 23 [root@master1 ~]# # 验证情况如下： 24 [root@master1 ~]# curl http://10.96.199.219 25 nginx-web3 26 [root@master1 ~]# curl http://10.96.199.219 27 nginx-web4 28 [root@master1 ~]# curl http://10.96.199.219 29 nginx-web1 30 [root@master1 ~]# curl http://10.96.199.219 31 nginx-web2 32 [root@master1 ~]#
由此可知，弹性伸缩会自动自动加入到service中实现服务自动发现和负载均衡，应用的扩展相比于传统应用快速非常多。此外，kubernetes还支持自动弹性扩展的能力，即Horizontal Pod AutoScaler，自动横向伸缩能力，配合监控系统根据CPU的利用率弹性扩展Pod个数。
5、滚动升级
- 使用最新版本的nginx替换当前nginx:1.8.1的版本并检查验证相关结果：
1 [root@master1 ~]# kubectl get deployments.apps demo-nginx -o wide 2 NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR 3 demo-nginx 4/4 4 4 106m nginx nginx:1.8.1 app=demo-nginx 4 [root@master1 ~]# curl -I http://10.96.199.219 5 HTTP/1.1 200 OK 6 Server: nginx/1.8.1 # 升级前的nginx版本 7 Date: Sun, 03 Oct 2021 10:01:58 GMT 8 Content-Type: text/html 9 Content-Length: 11 10 Last-Modified: Sun, 03 Oct 2021 09:49:12 GMT 11 Connection: keep-alive 12 ETag: "61597c98-b" 13 Accept-Ranges: bytes 14 15 [root@master1 ~]# 16 [root@master1 ~]# kubectl set image deployments.apps demo-nginx nginx=nginx:latest # 具体的升级命令 17 deployment.apps/demo-nginx image updated 18 [root@master1 ~]# 19 [root@master1 ~]# kubectl get deployments.apps demo-nginx -o wide 20 NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR 21 demo-nginx 4/4 4 4 110m nginx nginx:latest app=demo-nginx 22 [root@master1 ~]# curl -I http://10.96.199.219 23 HTTP/1.1 200 OK 24 Server: nginx/1.21.3 # 升级后的nginx版本 25 Date: Sun, 03 Oct 2021 10:04:44 GMT 26 Content-Type: text/html 27 Content-Length: 615 28 Last-Modified: Tue, 07 Sep 2021 15:21:03 GMT 29 Connection: keep-alive 30 ETag: "6137835f-267" 31 Accept-Ranges: bytes 32 33 [root@master1 ~]#
- 观察升级过程，通过查看可知，升级过程中是通过新建+删除的方式逐个替换pod的方式：
1 [root@master1 ~]# kubectl get pods -w 2 NAME READY STATUS RESTARTS AGE 3 demo-nginx-55ccc46db6-jmjkx 1/1 Running 0 107m 4 demo-nginx-55ccc46db6-l2vj5 1/1 Running 0 13m 5 demo-nginx-55ccc46db6-lg4nz 1/1 Running 0 107m 6 demo-nginx-55ccc46db6-vrlhw 1/1 Running 0 13m 7 # 这里是升级前后的分界…… 8 demo-nginx-6b99c5bdf5-qxdkh 0/1 Pending 0 0s 9 demo-nginx-6b99c5bdf5-qxdkh 0/1 Pending 0 0s 10 demo-nginx-55ccc46db6-vrlhw 1/1 Terminating 0 16m 11 demo-nginx-6b99c5bdf5-qxdkh 0/1 ContainerCreating 0 0s 12 demo-nginx-6b99c5bdf5-vsrc8 0/1 Pending 0 0s 13 demo-nginx-6b99c5bdf5-vsrc8 0/1 Pending 0 0s 14 demo-nginx-6b99c5bdf5-vsrc8 0/1 ContainerCreating 0 0s 15 demo-nginx-55ccc46db6-vrlhw 0/1 Terminating 0 16m 16 demo-nginx-55ccc46db6-vrlhw 0/1 Terminating 0 16m 17 demo-nginx-55ccc46db6-vrlhw 0/1 Terminating 0 16m 18 demo-nginx-6b99c5bdf5-qxdkh 1/1 Running 0 2s 19 demo-nginx-55ccc46db6-l2vj5 1/1 Terminating 0 16m 20 demo-nginx-6b99c5bdf5-kwkzs 0/1 Pending 0 0s 21 demo-nginx-6b99c5bdf5-kwkzs 0/1 Pending 0 0s 22 demo-nginx-6b99c5bdf5-kwkzs 0/1 ContainerCreating 0 0s 23 demo-nginx-6b99c5bdf5-vsrc8 1/1 Running 0 2s 24 demo-nginx-55ccc46db6-lg4nz 1/1 Terminating 0 110m 25 demo-nginx-6b99c5bdf5-cpb2t 0/1 Pending 0 0s 26 demo-nginx-6b99c5bdf5-cpb2t 0/1 Pending 0 0s 27 demo-nginx-6b99c5bdf5-cpb2t 0/1 ContainerCreating 0 1s 28 demo-nginx-55ccc46db6-lg4nz 0/1 Terminating 0 110m 29 demo-nginx-55ccc46db6-lg4nz 0/1 Terminating 0 110m 30 demo-nginx-55ccc46db6-lg4nz 0/1 Terminating 0 110m 31 demo-nginx-55ccc46db6-l2vj5 0/1 Terminating 0 16m 32 demo-nginx-55ccc46db6-l2vj5 0/1 Terminating 0 16m 33 demo-nginx-55ccc46db6-l2vj5 0/1 Terminating 0 16m 34 demo-nginx-6b99c5bdf5-cpb2t 1/1 Running 0 3s 35 demo-nginx-55ccc46db6-jmjkx 1/1 Terminating 0 110m 36 demo-nginx-6b99c5bdf5-kwkzs 1/1 Running 0 3s 37 demo-nginx-55ccc46db6-jmjkx 0/1 Terminating 0 110m 38 demo-nginx-55ccc46db6-jmjkx 0/1 Terminating 0 110m 39 demo-nginx-55ccc46db6-jmjkx 0/1 Terminating 0 110m
- 再次查看deployments的详情可知道，deployments已经更换了新的replicasets，原来的replicasets的版本为1，可用于回滚；查看滚动升级的版本，可以看到有两个版本，分别对应的两个不同的replicasets：
1 [root@master1 ~]# kubectl describe deployments.apps demo-nginx 2 Name: demo-nginx 3 Namespace: default 4 CreationTimestamp: Sun, 03 Oct 2021 04:13:57 -0400 5 Labels: app=demo-nginx 6 Annotations: deployment.kubernetes.io/revision: 2 # 新的版本号，用于回滚 7 Selector: app=demo-nginx 8 Replicas: 4 desired | 4 updated | 4 total | 4 available | 0 unavailable 9 StrategyType: RollingUpdate 10 MinReadySeconds: 0 11 RollingUpdateStrategy: 25% max unavailable, 25% max surge 12 Pod Template: 13 Labels: app=demo-nginx 14 Containers: 15 nginx: 16 Image: nginx:latest 17 Port: <none> 18 Host Port: <none> 19 Environment: <none> 20 Mounts: <none> 21 Volumes: <none> 22 Conditions: 23 Type Status Reason 24 ---- ------ ------ 25 Available True MinimumReplicasAvailable 26 Progressing True NewReplicaSetAvailable 27 OldReplicaSets: <none> 28 NewReplicaSet: demo-nginx-6b99c5bdf5 (4/4 replicas created) # 和下面的命令：kubectl get replicasets.apps查看结果对应上 29 Events: # replicasets 之间替换的过程 30 Type Reason Age From Message 31 ---- ------ ---- ---- ------- 32 Normal ScalingReplicaSet 23m deployment-controller Scaled up replica set demo-nginx-55ccc46db6 to 4 33 Normal ScalingReplicaSet 6m45s deployment-controller Scaled up replica set demo-nginx-6b99c5bdf5 to 1 34 Normal ScalingReplicaSet 6m45s deployment-controller Scaled down replica set demo-nginx-55ccc46db6 to 3 35 Normal ScalingReplicaSet 6m45s deployment-controller Scaled up replica set demo-nginx-6b99c5bdf5 to 2 36 Normal ScalingReplicaSet 6m43s deployment-controller Scaled down replica set demo-nginx-55ccc46db6 to 2 37 Normal ScalingReplicaSet 6m43s deployment-controller Scaled up replica set demo-nginx-6b99c5bdf5 to 3 38 Normal ScalingReplicaSet 6m43s deployment-controller Scaled down replica set demo-nginx-55ccc46db6 to 1 39 Normal ScalingReplicaSet 6m43s deployment-controller Scaled up replica set demo-nginx-6b99c5bdf5 to 4 40 Normal ScalingReplicaSet 6m40s deployment-controller Scaled down replica set demo-nginx-55ccc46db6 to 0 41 [root@master1 ~]# 42 [root@master1 ~]# 43 [root@master1 ~]# kubectl rollout history deployment demo-nginx 44 deployment.apps/demo-nginx 45 REVISION CHANGE-CAUSE 46 1 <none> 47 2 <none> 48 49 [root@master1 ~]# ： # 查看replicasets列表,旧的包含pod为0 50 NAME DESIRED CURRENT READY AGE 51 demo-nginx-55ccc46db6 0 0 0 127m 52 demo-nginx-6b99c5bdf5 4 4 4 17m 53 [root@master1 ~]#
- 回滚到旧的版本：
1 [root@master1 ~]# kubectl rollout undo deployment demo-nginx --to-revision=1 # 回退到指定的版本 2 deployment.apps/demo-nginx rolled back 3 [root@master1 ~]# # 相关查看验证 4 [root@master1 ~]# curl -I http://10.96.199.219 5 HTTP/1.1 200 OK 6 Server: nginx/1.8.1 7 Date: Sun, 03 Oct 2021 10:27:31 GMT 8 Content-Type: text/html 9 Content-Length: 612 10 Last-Modified: Tue, 26 Jan 2016 15:24:47 GMT 11 Connection: keep-alive 12 ETag: "56a78fbf-264" 13 Accept-Ranges: bytes 14 15 [root@master1 ~]# 16 [root@master1 ~]# kubectl get replicasets.apps 17 NAME DESIRED CURRENT READY AGE 18 demo-nginx-55ccc46db6 4 4 4 133m 19 demo-nginx-6b99c5bdf5 0 0 0 23m 20 [root@master1 ~]# 21 [root@master1 ~]# kubectl describe deployments.apps demo-nginx 22 Name: demo-nginx 23 Namespace: default 24 CreationTimestamp: Sun, 03 Oct 2021 04:13:57 -0400 25 Labels: app=demo-nginx 26 Annotations: deployment.kubernetes.io/revision: 3 27 Selector: app=demo-nginx 28 Replicas: 4 desired | 4 updated | 4 total | 4 available | 0 unavailable 29 StrategyType: RollingUpdate 30 MinReadySeconds: 0 31 RollingUpdateStrategy: 25% max unavailable, 25% max surge 32 Pod Template: 33 Labels: app=demo-nginx 34 Containers: 35 nginx: 36 Image: nginx:1.8.1 37 Port: <none> 38 Host Port: <none> 39 Environment: <none> 40 Mounts: <none> 41 Volumes: <none> 42 Conditions: 43 Type Status Reason 44 ---- ------ ------ 45 Available True MinimumReplicasAvailable 46 Progressing True NewReplicaSetAvailable 47 OldReplicaSets: <none> 48 NewReplicaSet: demo-nginx-55ccc46db6 (4/4 replicas created) 49 Events: 50 Type Reason Age From Message 51 ---- ------ ---- ---- ------- 52 Normal ScalingReplicaSet 24m deployment-controller Scaled up replica set demo-nginx-6b99c5bdf5 to 1 53 Normal ScalingReplicaSet 24m deployment-controller Scaled down replica set demo-nginx-55ccc46db6 to 3 54 Normal ScalingReplicaSet 24m deployment-controller Scaled up replica set demo-nginx-6b99c5bdf5 to 2 55 Normal ScalingReplicaSet 23m deployment-controller Scaled up replica set demo-nginx-6b99c5bdf5 to 3 56 Normal ScalingReplicaSet 23m deployment-controller Scaled down replica set demo-nginx-55ccc46db6 to 1 57 Normal ScalingReplicaSet 23m deployment-controller Scaled down replica set demo-nginx-55ccc46db6 to 2 58 Normal ScalingReplicaSet 23m deployment-controller Scaled up replica set demo-nginx-6b99c5bdf5 to 4 59 Normal ScalingReplicaSet 23m deployment-controller Scaled down replica set demo-nginx-55ccc46db6 to 0 60 Normal ScalingReplicaSet 46s deployment-controller Scaled up replica set demo-nginx-55ccc46db6 to 1 61 Normal ScalingReplicaSet 46s deployment-controller Scaled down replica set demo-nginx-6b99c5bdf5 to 3 62 Normal ScalingReplicaSet 46s (x2 over 133m) deployment-controller Scaled up replica set demo-nginx-55ccc46db6 to 2 63 Normal ScalingReplicaSet 43s (x2 over 40m) deployment-controller Scaled up replica set demo-nginx-55ccc46db6 to 4 64 Normal ScalingReplicaSet 43s deployment-controller Scaled down replica set demo-nginx-6b99c5bdf5 to 2 65 Normal ScalingReplicaSet 43s deployment-controller Scaled up replica set demo-nginx-55ccc46db6 to 3 66 Normal ScalingReplicaSet 43s deployment-controller Scaled down replica set demo-nginx-6b99c5bdf5 to 1 67 Normal ScalingReplicaSet 41s deployment-controller Scaled down replica set demo-nginx-6b99c5bdf5 to 0 68 [root@master1 ~]# kubectl rollout history deployment demo-nginx 69 deployment.apps/demo-nginx 70 REVISION CHANGE-CAUSE 71 2 <none> 72 3 <none> 73 74 [root@master1 ~]#
