Pod亲和与反亲和、污点与容忍、驱逐
一、Pod调度
1.1、Pod调度流程
1.2、nodeSelector简介
nodeSelector 基于node标签选择器， 将pod调度的指定的目的节点上。
https://kubernetes.io/zh/docs/concepts/scheduling-eviction/assign-pod-node/
可用于基于服务类型干预Pod调度结果， 如对磁盘I/O要求高的pod调度到SSD节点， 对内存要求比较高的pod调度的内存较高的节点。
也可以用于区分不同项目的pod， 如将node添加不同项目的标签， 然后区分调度。
[root@easzlab-deploy Affinit-case]# kubectl describe node 172.16.88.157 Name: 172.16.88.157 Roles: master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=172.16.88.157 kubernetes.io/os=linux kubernetes.io/role=master Annotations: node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Tue, 18 Oct 2022 17:08:10 +0800 Taints: node.kubernetes.io/unschedulable:NoSchedule Unschedulable: true Lease: HolderIdentity: 172.16.88.157 AcquireTime: <unset> RenewTime: Thu, 27 Oct 2022 10:47:06 +0800 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Wed, 19 Oct 2022 22:11:11 +0800 Wed, 19 Oct 2022 22:11:11 +0800 CalicoIsUp Calico is running on this node MemoryPressure False Thu, 27 Oct 2022 10:47:08 +0800 Tue, 18 Oct 2022 17:08:10 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 27 Oct 2022 10:47:08 +0800 Tue, 18 Oct 2022 17:08:10 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 27 Oct 2022 10:47:08 +0800 Tue, 18 Oct 2022 17:08:10 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Thu, 27 Oct 2022 10:47:08 +0800 Wed, 19 Oct 2022 22:09:58 +0800 KubeletReady kubelet is posting ready status. AppArmor enabled Addresses: InternalIP: 172.16.88.157 Hostname: 172.16.88.157 Capacity: cpu: 2 ephemeral-storage: 49869532Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 4026076Ki pods: 500 Allocatable: cpu: 2 ephemeral-storage: 45959760616 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3718876Ki pods: 500 System Info: Machine ID: 1a19d04d6cc141e58368dc5dcc39f1a9 System UUID: 8f09cdfe-374c-4faf-b9f2-78f0d9d3bd9a Boot ID: d7c79d7f-c664-44a6-9d03-0bf4ad6db51b Kernel Version: 5.4.0-128-generic OS Image: Ubuntu 20.04.4 LTS Operating System: linux Architecture: amd64 Container Runtime Version: containerd://1.6.4 Kubelet Version: v1.24.2 Kube-Proxy Version: v1.24.2 PodCIDR: 10.200.1.0/24 PodCIDRs: 10.200.1.0/24 Non-terminated Pods: (1 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-system calico-node-xh79g 250m (12%) 0 (0%) 0 (0%) 0 (0%) 8d Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 250m (12%) 0 (0%) memory 0 (0%) 0 (0%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: <none> [root@easzlab-deploy Affinit-case]#
1.3、nodeSelector调度
#为node节点打标签： [root@easzlab-deploy Affinit-case]# kubectl label node 172.16.88.160 project="magedu" node/172.16.88.160 labeled [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl label node 172.16.88.160 disktype="ssd" node/172.16.88.161 labeled [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl get node --show-labels=true NAME STATUS ROLES AGE VERSION LABELS 172.16.88.157 Ready,SchedulingDisabled master 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.157,kubernetes.io/os=linux,kubernetes.io/role=master 172.16.88.158 Ready,SchedulingDisabled master 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.158,kubernetes.io/os=linux,kubernetes.io/role=master 172.16.88.159 Ready,SchedulingDisabled master 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.159,kubernetes.io/os=linux,kubernetes.io/role=master 172.16.88.160 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=ssd,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.160,kubernetes.io/os=linux,kubernetes.io/role=node,project=magedu 172.16.88.161 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.161,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.162 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.162,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.163 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.163,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.164 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.164,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.165 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.165,kubernetes.io/os=linux,kubernetes.io/role=node [root@easzlab-deploy Affinit-case]# #将pod调度到目的node， yaml文件中指定的key与value必须精确匹配： [root@easzlab-deploy Affinit-case]# vi case1-nodeSelector.yaml [root@easzlab-deploy Affinit-case]# cat case1-nodeSelector.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app2-deployment-label name: magedu-tomcat-app2-deployment namespace: magedu spec: replicas: 4 selector: matchLabels: app: magedu-tomcat-app2-selector template: metadata: labels: app: magedu-tomcat-app2-selector spec: containers: - name: magedu-tomcat-app2-container image: tomcat:7.0.94-alpine imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http env: - name: "password" value: "123456" - name: "age" value: "18" resources: limits: cpu: 1 memory: "512Mi" requests: cpu: 500m memory: "512Mi" nodeSelector: project: magedu disktype: ssd [root@easzlab-deploy Affinit-case]# kubectl apply -f case1-nodeSelector.yaml deployment.apps/magedu-tomcat-app2-deployment created [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d20h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 44h 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-75d7bb596c-n2c2s 1/1 Running 0 11m 10.200.40.212 172.16.88.160 <none> <none> magedu-tomcat-app2-deployment-75d7bb596c-pk2c2 1/1 Running 0 11m 10.200.40.214 172.16.88.160 <none> <none> magedu-tomcat-app2-deployment-75d7bb596c-pvf8c 1/1 Running 0 11m 10.200.40.215 172.16.88.160 <none> <none> magedu-tomcat-app2-deployment-75d7bb596c-z7klc 1/1 Running 0 11m 10.200.40.213 172.16.88.160 <none> <none> [root@easzlab-deploy Affinit-case]#
指定节点创建
[root@easzlab-deploy Affinit-case]# vi case2-nodename.yaml [root@easzlab-deploy Affinit-case]# cat case2-nodename.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app2-deployment-label name: magedu-tomcat-app2-deployment namespace: magedu spec: replicas: 1 selector: matchLabels: app: magedu-tomcat-app2-selector template: metadata: labels: app: magedu-tomcat-app2-selector spec: nodeName: 172.16.88.161 containers: - name: magedu-tomcat-app2-container image: tomcat:7.0.94-alpine imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http env: - name: "password" value: "123456" - name: "age" value: "18" resources: limits: cpu: 1 memory: "512Mi" requests: cpu: 500m memory: "512Mi" [root@easzlab-deploy Affinit-case]# kubectl apply -f case2-nodename.yaml deployment.apps/magedu-tomcat-app2-deployment configured [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d20h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 44h 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-756bb8b58b-xcd5f 1/1 Running 0 49s 10.200.233.116 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]#
二、node节点亲和性
affinity是Kubernetes 1.2版本后引入的新特性， 类似于nodeSelector， 允许使用者指定一些Pod在Node间调度的约束， 目前支持两种形式：
- requiredDuringSchedulingIgnoredDuringExecution #必须满足pod调度匹配条件， 如果不满足则不进行调度
- preferredDuringSchedulingIgnoredDuringExecution #倾向满足pod调度匹配条件， 不满足的情况下会调度的不符合条件的Node上
- IgnoreDuringExecution表示如果在Pod运行期间Node的标签发生变化， 导致亲和性策略不能满足， 也会继续运行当前的Pod。
- Affinity与anti-affinity的目的也是控制pod的调度结果， 但是相对于nodeSelector， Affinity(亲和)与anti-affinity(反亲和)的功能更加强大
affinity与nodeSelector对比：
1、亲和与反亲和对目的标签的选择匹配不仅仅支持and， 还支持In、 NotIn、 Exists、 DoesNotExist、 Gt、 Lt。
- In： 标签的值存在匹配列表中(匹配成功就调度到目的node， 实现node亲和)
- NotIn： 标签的值不存在指定的匹配列表中(不会调度到目的node， 实现反亲和)
- Gt： 标签的值大于某个值(字符串)
- Lt： 标签的值小于某个值(字符串)
- Exists： 指定的标签存在
2、可以设置软匹配和硬匹配， 在软匹配下， 如果调度器无法匹配节点， 仍然将pod调度到其它不符合条件的节点。
3、还可以对pod定义亲和策略， 比如允许哪些pod可以或者不可以被调度至同一台node。
注：
- 如果定义一个nodeSelectorTerms(条件)中通过一个matchExpressions基于列表指定了多个operator条件， 则只要满足其中一个条件， 就会被调度到相应的节点上， 即or的关系， 即如果nodeSelectorTerms下面有多个条件的话， 只要满足任何一个条件就可以了。
- 如果定义一个nodeSelectorTerms中都通过一个matchExpressions(匹配表达式)指定key匹配多个条件， 则所有的目的条件都必须满足才会调度到对应的节点， 即and的关系， 即果matchExpressions有多个选项的话， 则必须同时满足所有这些条件才能正常调度。
2.1、硬亲和-nodeAffinity-requiredDuring-matchExpressions-1
affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: #匹配条件1,有一个key但是有多个values、 则只要匹配成功一个value就可以调度 - key: disktype operator: In values: - ssd #只有一个value是匹配成功也可以调度 - xxx - matchExpressions: #匹配条件2,多个matchExpressions加上以及每个matchExpressions 有一个key及多个values只有其中一个value匹配成功就可以调度 - key: project operator: In values: - mmm #即使这俩条件都匹配不上也可以调度， 即多个matchExpressions只要有任意一个能匹配任何一个value就可以调用。 - nnn
验证测试
[root@easzlab-deploy Affinit-case]# vi case3-1.1-nodeAffinity-requiredDuring-matchExpressions.yaml [root@easzlab-deploy Affinit-case]# cat case3-1.1-nodeAffinity-requiredDuring-matchExpressions.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app2-deployment-label name: magedu-tomcat-app2-deployment namespace: magedu spec: replicas: 5 selector: matchLabels: app: magedu-tomcat-app2-selector template: metadata: labels: app: magedu-tomcat-app2-selector spec: containers: - name: magedu-tomcat-app2-container image: tomcat:7.0.94-alpine imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: #匹配条件1,有一个key但是有多个values、则只要匹配成功一个value就可以调度 - key: disktype operator: In values: - ssd # 只有一个value是匹配成功也可以调度 - xxx - matchExpressions: #匹配条件1,有一个key但是有多个values、则只要匹配成功一个value就可以调度 - key: project operator: In values: - mmm #即使这俩条件都匹配不上也可以调度，即多个matchExpressions只要有任意一个能匹配任何一个value就可以调用。 - nnn [root@easzlab-deploy Affinit-case]# kubectl apply -f case3-1.1-nodeAffinity-requiredDuring-matchExpressions.yaml deployment.apps/magedu-tomcat-app2-deployment configured [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d20h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 45h 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-555df55d49-4qfkc 1/1 Running 0 25s 10.200.233.122 172.16.88.161 <none> <none> magedu-tomcat-app2-deployment-555df55d49-89psp 1/1 Running 0 28s 10.200.40.216 172.16.88.160 <none> <none> magedu-tomcat-app2-deployment-555df55d49-8trjm 1/1 Running 0 25s 10.200.40.218 172.16.88.160 <none> <none> magedu-tomcat-app2-deployment-555df55d49-h6rrn 1/1 Running 0 28s 10.200.233.120 172.16.88.161 <none> <none> magedu-tomcat-app2-deployment-555df55d49-slkm8 1/1 Running 0 28s 10.200.40.217 172.16.88.160 <none> <none> [root@easzlab-deploy Affinit-case]#
2.2、硬亲和-nodeAffinity-requiredDuring-matchExpressions-2
affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: #匹配条件1， 同一个key的多个value只有有一个匹配成功就认为当前key匹配成功 - key: disktype operator: In values: - ssd - hdd - key: project #匹配条件2， 当前key也要匹配成功一个value， 即条件1和条件2必须同时每个key匹配成功一个value,否则不调度 operator: In values: - magedu
验证测试
[root@easzlab-deploy Affinit-case]# vi case3-1.2-nodeAffinity-requiredDuring-matchExpressions.yaml [root@easzlab-deploy Affinit-case]# cat case3-1.2-nodeAffinity-requiredDuring-matchExpressions.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app2-deployment-label name: magedu-tomcat-app2-deployment namespace: magedu spec: replicas: 1 selector: matchLabels: app: magedu-tomcat-app2-selector template: metadata: labels: app: magedu-tomcat-app2-selector spec: containers: - name: magedu-tomcat-app2-container image: tomcat:7.0.94-alpine imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: #匹配条件1，同一个key的多个value只有有一个匹配成功就认为当前key匹配成功 - key: disktype operator: In values: - ssd - hddx - key: project #匹配条件2，当前key也要匹配成功一个value，即条件1和条件2必须同时每个key匹配成功一个value,否则不调度 operator: In values: - magedu [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl delete -f case3-1.1-nodeAffinity-requiredDuring-matchExpressions.yaml #删除旧的pod deployment.apps "magedu-tomcat-app2-deployment" deleted [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu #确认是否删除 NAME READY STATUS RESTARTS AGE magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d21h magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 45h [root@easzlab-deploy Affinit-case]# kubectl apply -f case3-1.2-nodeAffinity-requiredDuring-matchExpressions.yaml deployment.apps/magedu-tomcat-app2-deployment created [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d21h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 45h 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-585f4864c4-brw6j 1/1 Running 0 15s 10.200.40.219 172.16.88.160 <none> <none> [root@easzlab-deploy Affinit-case]#
2.3、软亲和-nodeAffinity-preferredDuring
affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 #软亲和条件1， weight值越大优先级越高， 越优先匹配调度 preference: matchExpressions: - key: project operator: In values: - magedu - weight: 60 #软亲和条件2， 在条件1不满足时匹配条件2 preference: matchExpressions: - key: disktype operator: In values: - ssd
验证测试
[root@easzlab-deploy Affinit-case]# kubectl delete -f case3-1.2-nodeAffinity-requiredDuring-matchExpressions.yaml #删除旧的pod deployment.apps "magedu-tomcat-app2-deployment" deleted [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu #确实是否彻底删除 NAME READY STATUS RESTARTS AGE magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d21h magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 45h [root@easzlab-deploy Affinit-case]#
#修改node label， 测试pod能否调度软策略优先级教高的节点 [root@easzlab-deploy Affinit-case]# kubectl label node 172.16.88.160 project- #取消160节点 project标签 node/172.16.88.160 unlabeled [root@easzlab-deploy Affinit-case]# kubectl label node 172.16.88.161 project=magedu #指定161节点 project标签 node/172.16.88.161 labeled [root@easzlab-deploy Affinit-case]# kubectl get node --show-labels=true NAME STATUS ROLES AGE VERSION LABELS 172.16.88.157 Ready,SchedulingDisabled master 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.157,kubernetes.io/os=linux,kubernetes.io/role=master 172.16.88.158 Ready,SchedulingDisabled master 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.158,kubernetes.io/os=linux,kubernetes.io/role=master 172.16.88.159 Ready,SchedulingDisabled master 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.159,kubernetes.io/os=linux,kubernetes.io/role=master 172.16.88.160 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=ssd,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.160,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.161 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=ssd,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.161,kubernetes.io/os=linux,kubernetes.io/role=node,project=magedu 172.16.88.162 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.162,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.163 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.163,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.164 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.164,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.165 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.165,kubernetes.io/os=linux,kubernetes.io/role=node [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# vi case3-2.1-nodeAffinity-preferredDuring.yaml [root@easzlab-deploy Affinit-case]# cat case3-2.1-nodeAffinity-preferredDuring.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app2-deployment-label name: magedu-tomcat-app2-deployment namespace: magedu spec: replicas: 1 selector: matchLabels: app: magedu-tomcat-app2-selector template: metadata: labels: app: magedu-tomcat-app2-selector spec: containers: - name: magedu-tomcat-app2-container image: tomcat:7.0.94-alpine imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 #软亲和条件1，weight值越大优先级越高，越优先匹配调度 preference: matchExpressions: - key: project operator: In values: - magedu - weight: 60 #软亲和条件2，在条件1不满足时匹配条件2 preference: matchExpressions: - key: disktype operator: In values: - ssd [root@easzlab-deploy Affinit-case]# kubectl apply -f case3-2.1-nodeAffinity-preferredDuring.yaml deployment.apps/magedu-tomcat-app2-deployment created [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d21h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 45h 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-56c9bdffdd-sl62q 1/1 Running 0 18s 10.200.233.123 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]#
硬亲和与软亲和结合使用-nodeAffinity-requiredDuring-preferredDuring
基于硬亲和和软亲和实现pod调度： 1.绝对不会调度在哪些node 2.倾向于调度到哪些node affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: #硬策略 nodeSelectorTerms: - matchExpressions: #硬匹配条件1 - key: "kubernetes.io/role" operator: NotIn values: - "master" #硬性匹配key 的值kubernetes.io/role不包含master的节点,即绝对不会调度到master节点(node反亲和) preferredDuringSchedulingIgnoredDuringExecution: #软策略 - weight: 80 preference: matchExpressions: - key: project operator: In values: - magedu - weight: 60 preference: matchExpressions: - key: disktype operator: In values: - ssd
验证测试
[root@easzlab-deploy Affinit-case]# vi case3-2.2-nodeAffinity-requiredDuring-preferredDuring.yaml [root@easzlab-deploy Affinit-case]# cat case3-2.2-nodeAffinity-requiredDuring-preferredDuring.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app2-deployment-label name: magedu-tomcat-app2-deployment namespace: magedu spec: replicas: 1 selector: matchLabels: app: magedu-tomcat-app2-selector template: metadata: labels: app: magedu-tomcat-app2-selector spec: containers: - name: magedu-tomcat-app2-container image: tomcat:7.0.94-alpine imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: #硬亲和 nodeSelectorTerms: - matchExpressions: #硬匹配条件1 - key: "kubernetes.io/role" operator: NotIn values: - "master" #硬性匹配key 的值kubernetes.io/role不包含master的节点,即绝对不会调度到master节点(node反亲和) preferredDuringSchedulingIgnoredDuringExecution: #软亲和 - weight: 80 preference: matchExpressions: - key: project operator: In values: - magedu - weight: 60 preference: matchExpressions: - key: disktype operator: In values: - hdd [root@easzlab-deploy Affinit-case]# kubectl delete -f case3-2.1-nodeAffinity-preferredDuring.yaml deployment.apps "magedu-tomcat-app2-deployment" deleted [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d21h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 45h 10.200.105.177 172.16.88.164 <none> <none> [root@easzlab-deploy Affinit-case]# kubectl apply -f case3-2.2-nodeAffinity-requiredDuring-preferredDuring.yaml deployment.apps/magedu-tomcat-app2-deployment created [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d21h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 45h 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-7564998d64-6565f 1/1 Running 0 4s 10.200.233.125 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]#
2.4、反亲和-nodeantiaffinity
affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: #匹配条件1 - key: disktype operator: NotIn #调度的目的节点没有key为disktype且值为hdd的标签 values: - hdd #绝对不会调度到含有label的key为disktype且值为hdd的hdd的节点,即会调度到没有key为disktype且值为hdd的hdd的节点
验证测试
[root@easzlab-deploy Affinit-case]# kubectl delete -f case3-2.2-nodeAffinity-requiredDuring-preferredDuring.yaml #删除旧的pod deployment.apps "magedu-tomcat-app2-deployment" deleted [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d23h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 47h 10.200.105.177 172.16.88.164 <none> <none> [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# vi case3-3.1-nodeantiaffinity.yaml [root@easzlab-deploy Affinit-case]# cat case3-3.1-nodeantiaffinity.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app2-deployment-label name: magedu-tomcat-app2-deployment namespace: magedu spec: replicas: 5 selector: matchLabels: app: magedu-tomcat-app2-selector template: metadata: labels: app: magedu-tomcat-app2-selector spec: containers: - name: magedu-tomcat-app2-container image: tomcat:7.0.94-alpine imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: #匹配条件1 - key: disktype operator: NotIn #调度的目的节点没有key为disktype且值为hdd的标签 values: - hdd #绝对不会调度到含有label的key为disktype且值为hdd的hdd的节点,即会调度到没有key为disktype且值为hdd的hdd的节点 [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl label node 172.16.88.162 disktype="hdd" node/172.16.88.162 labeled [root@easzlab-deploy Affinit-case]# kubectl get node --show-labels=true NAME STATUS ROLES AGE VERSION LABELS 172.16.88.157 Ready,SchedulingDisabled master 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.157,kubernetes.io/os=linux,kubernetes.io/role=master 172.16.88.158 Ready,SchedulingDisabled master 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.158,kubernetes.io/os=linux,kubernetes.io/role=master 172.16.88.159 Ready,SchedulingDisabled master 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.159,kubernetes.io/os=linux,kubernetes.io/role=master 172.16.88.160 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=ssd,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.160,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.161 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=ssd,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.161,kubernetes.io/os=linux,kubernetes.io/role=node,project=magedu 172.16.88.162 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=hdd,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.162,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.163 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.163,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.164 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.164,kubernetes.io/os=linux,kubernetes.io/role=node 172.16.88.165 Ready node 8d v1.24.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.88.165,kubernetes.io/os=linux,kubernetes.io/role=node [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl apply -f case3-3.1-nodeantiaffinity.yaml deployment.apps/magedu-tomcat-app2-deployment created [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d23h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 47h 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-5c5fc87946-5mcrk 1/1 Running 0 96s 10.200.105.186 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-5c5fc87946-629nd 1/1 Running 0 96s 10.200.233.127 172.16.88.161 <none> <none> magedu-tomcat-app2-deployment-5c5fc87946-k9gv7 1/1 Running 0 96s 10.200.245.38 172.16.88.165 <none> <none> magedu-tomcat-app2-deployment-5c5fc87946-ls5p7 1/1 Running 0 96s 10.200.40.220 172.16.88.160 <none> <none> magedu-tomcat-app2-deployment-5c5fc87946-sklbb 1/1 Running 0 96s 10.200.104.201 172.16.88.163 <none> <none> [root@easzlab-deploy Affinit-case]#
三、Pod Affinity与anti-affinity简介
3.1、Pod Affinity与anti-affinity简介
- Pod亲和性与反亲和性可以基于已经在node节点上运行的Pod的标签来约束新创建的Pod可以调度到的目的节点， 注意不是基于node上的标签而是使用的已经运行在node上的pod标签匹配。
- 其规则的格式为如果 node节点 A已经运行了一个或多个满足调度新创建的Pod B的规则， 那么新的Pod B在亲和的条件下会调度到A节点之上， 而在反亲和性的情况下则不会调度到A节点至上。
- 其中规则表示一个具有可选的关联命名空间列表的LabelSelector， 只所以Pod亲和与反亲和需可以通过LabelSelector选择namespace， 是因为Pod是命名空间限定的而node不属于任何nemespace所以node的亲和与反亲和不需要namespace， 因此作用于Pod标签的标签选择算符必须指定选择算符应用在哪个命名空间。
- 从概念上讲， node节点是一个拓扑域(具有拓扑结构的域)， 比如k8s集群中的单台node节点、 一个机架、云供应商可用区、 云供应商地理区域等， 可以使用topologyKey来定义亲和或者反亲和的颗粒度是node级别还是可用区级别， 以便kubernetes调度系统用来识别并选择正确的目的拓扑域。
- Pod 亲和性与反亲和性的合法操作符(operator)有 In、 NotIn、 Exists、 DoesNotExist。
- 在Pod亲和性配置中， 在requiredDuringSchedulingIgnoredDuringExecution和preferredDuringSchedulingIgnoredDuringExecution中， topologyKey不允许为空(Empty topologyKey is not allowed.)。
- 在Pod反亲和性中配置中， requiredDuringSchedulingIgnoredDuringExecution和preferredDuringSchedulingIgnoredDuringExecution 中， topologyKey也不可以为空(Empty topologyKey is not allowed.)。
- 对于requiredDuringSchedulingIgnoredDuringExecution要求的Pod反亲和性， 准入控制器LimitPodHardAntiAffinityTopology被引入以确保topologyKey只能是 kubernetes.io/hostname， 如果希望 topologyKey 也可用于其他定制拓扑逻辑， 可以更改准入控制器或者禁用。
- 除上述情况外， topologyKey 可以是任何合法的标签键。
3.2、部署web服务测试
编写yaml文件，在magedu anmespace部署一个nginx服务，nginx pod将用于后续的pod 亲和及反亲和测试，且pod的label如下：
- app: python-nginx-selector
- project: python
[root@easzlab-deploy Affinit-case]# kubectl delete -f case3-3.1-nodeantiaffinity.yaml #删除旧的pod deployment.apps "magedu-tomcat-app2-deployment" deleted [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d23h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 47h 10.200.105.177 172.16.88.164 <none> <none> [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# vi case4-4.1-nginx.yaml [root@easzlab-deploy Affinit-case]# cat case4-4.1-nginx.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: python-nginx-deployment-label name: python-nginx-deployment namespace: magedu spec: replicas: 1 selector: matchLabels: app: python-nginx-selector template: metadata: labels: app: python-nginx-selector project: python spec: containers: - name: python-nginx-container image: nginx:1.20.2-alpine #command: ["/apps/tomcat/bin/run_tomcat.sh"] #imagePullPolicy: IfNotPresent imagePullPolicy: Always ports: - containerPort: 80 protocol: TCP name: http - containerPort: 443 protocol: TCP name: https env: - name: "password" value: "123456" - name: "age" value: "18" # resources: # limits: # cpu: 2 # memory: 2Gi # requests: # cpu: 500m # memory: 1Gi --- kind: Service apiVersion: v1 metadata: labels: app: python-nginx-service-label name: python-nginx-service namespace: magedu spec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: 80 nodePort: 30014 - name: https port: 443 protocol: TCP targetPort: 443 nodePort: 30453 selector: app: python-nginx-selector project: python #一个或多个selector，至少能匹配目标pod的一个标签 [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl apply -f case4-4.1-nginx.yaml deployment.apps/python-nginx-deployment created service/python-nginx-service created [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 3d23h 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 2d 10.200.105.177 172.16.88.164 <none> <none> python-nginx-deployment-5c658bf86b-qvzpn 1/1 Running 0 86s 10.200.233.65 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]#
3.3、Pod Affinity-软亲和
基于软亲和实现多pod在一个node：先创建一个nginx服务,后续的服务和nginx运行在同一个node节点 [root@easzlab-deploy Affinit-case]# vi case4-4.1-nginx.yaml [root@easzlab-deploy Affinit-case]# cat case4-4.1-nginx.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: python-nginx-deployment-label name: python-nginx-deployment namespace: magedu spec: replicas: 1 selector: matchLabels: app: python-nginx-selector template: metadata: labels: app: python-nginx-selector project: python spec: containers: - name: python-nginx-container image: nginx:1.20.2-alpine #command: ["/apps/tomcat/bin/run_tomcat.sh"] #imagePullPolicy: IfNotPresent imagePullPolicy: Always ports: - containerPort: 80 protocol: TCP name: http - containerPort: 443 protocol: TCP name: https env: - name: "password" value: "123456" - name: "age" value: "18" # resources: # limits: # cpu: 2 # memory: 2Gi # requests: # cpu: 500m # memory: 1Gi --- kind: Service apiVersion: v1 metadata: labels: app: python-nginx-service-label name: python-nginx-service namespace: magedu spec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: 80 nodePort: 30014 - name: https port: 443 protocol: TCP targetPort: 443 nodePort: 30453 selector: app: python-nginx-selector project: python #一个或多个selector，至少能匹配目标pod的一个标签 [root@easzlab-deploy Affinit-case]# kubectl apply -f case4-4.1-nginx.yaml deployment.apps/python-nginx-deployment created service/python-nginx-service created [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 4d 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 2d 10.200.105.177 172.16.88.164 <none> <none> python-nginx-deployment-5c658bf86b-qvzpn 1/1 Running 0 86s 10.200.233.67 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]# #实现软亲和pod调度 [root@easzlab-deploy Affinit-case]# vi case4-4.2-podaffinity-preferredDuring.yaml [root@easzlab-deploy Affinit-case]# cat case4-4.2-podaffinity-preferredDuring.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app2-deployment-label name: magedu-tomcat-app2-deployment namespace: magedu spec: replicas: 1 selector: matchLabels: app: magedu-tomcat-app2-selector template: metadata: labels: app: magedu-tomcat-app2-selector spec: containers: - name: magedu-tomcat-app2-container image: tomcat:7.0.94-alpine imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http affinity: podAffinity: #Pod亲和 #requiredDuringSchedulingIgnoredDuringExecution: #硬亲和，必须匹配成功才调度，如果匹配失败则拒绝调度。 preferredDuringSchedulingIgnoredDuringExecution: #软亲和，能匹配成功就调度到一个topology，匹配不成功会由kubernetes自行调度。 - weight: 100 podAffinityTerm: labelSelector: #标签选择 matchExpressions: #正则匹配 - key: project operator: In values: - python topologyKey: kubernetes.io/hostname namespaces: - magedu [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl apply -f case4-4.2-podaffinity-preferredDuring.yaml deployment.apps/magedu-tomcat-app2-deployment created [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 4d 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 2d 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-74d68bdfcf-xsxs6 1/1 Running 0 12s 10.200.233.69 172.16.88.161 <none> <none> python-nginx-deployment-5c658bf86b-qvzpn 1/1 Running 0 96s 10.200.233.67 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]#
3.4、Pod Affinity-硬亲和
[root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 4d 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 2d 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-74d68bdfcf-xsxs6 1/1 Running 0 5m52s 10.200.233.69 172.16.88.161 <none> <none> python-nginx-deployment-5c658bf86b-qvzpn 1/1 Running 0 7m16s 10.200.233.67 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# vi case4-4.3-podaffinity-requiredDuring.yaml [root@easzlab-deploy Affinit-case]# cat case4-4.3-podaffinity-requiredDuring.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app2-deployment-label name: magedu-tomcat-app2-deployment namespace: magedu spec: replicas: 3 selector: matchLabels: app: magedu-tomcat-app2-selector template: metadata: labels: app: magedu-tomcat-app2-selector spec: containers: - name: magedu-tomcat-app2-container image: tomcat:7.0.94-alpine imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: #硬亲和 - labelSelector: matchExpressions: - key: project operator: In values: - python topologyKey: "kubernetes.io/hostname" namespaces: - magedu [root@easzlab-deploy Affinit-case]# kubectl appy -f case4-4.3-podaffinity-requiredDuring.yaml error: unknown command "appy" for "kubectl" Did you mean this? apply [root@easzlab-deploy Affinit-case]# kubectl apply -f case4-4.3-podaffinity-requiredDuring.yaml deployment.apps/magedu-tomcat-app2-deployment configured [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 4d 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 2d 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-68c5977495-48m5g 1/1 Running 0 28s 10.200.233.73 172.16.88.161 <none> <none> magedu-tomcat-app2-deployment-68c5977495-nldc2 1/1 Running 0 23s 10.200.233.72 172.16.88.161 <none> <none> magedu-tomcat-app2-deployment-68c5977495-pjgck 1/1 Running 0 25s 10.200.233.74 172.16.88.161 <none> <none> python-nginx-deployment-5c658bf86b-qvzpn 1/1 Running 0 8m18s 10.200.233.67 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]#
3.5、podAntiAffinity-硬反亲和
[root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 4d 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 2d 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-68c5977495-48m5g 1/1 Running 0 28s 10.200.233.73 172.16.88.161 <none> <none> magedu-tomcat-app2-deployment-68c5977495-nldc2 1/1 Running 0 23s 10.200.233.72 172.16.88.161 <none> <none> magedu-tomcat-app2-deployment-68c5977495-pjgck 1/1 Running 0 25s 10.200.233.74 172.16.88.161 <none> <none> python-nginx-deployment-5c658bf86b-qvzpn 1/1 Running 0 8m18s 10.200.233.67 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]# vi case4-4.4-podAntiAffinity-requiredDuring.yaml [root@easzlab-deploy Affinit-case]# cat case4-4.4-podAntiAffinity-requiredDuring.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app2-deployment-label name: magedu-tomcat-app2-deployment namespace: magedu spec: replicas: 1 selector: matchLabels: app: magedu-tomcat-app2-selector template: metadata: labels: app: magedu-tomcat-app2-selector spec: containers: - name: magedu-tomcat-app2-container image: tomcat:7.0.94-alpine imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: project operator: In values: - python topologyKey: "kubernetes.io/hostname" namespaces: - magedu [root@easzlab-deploy Affinit-case]# kubectl apply -f case4-4.4-podAntiAffinity-requiredDuring.yaml deployment.apps/magedu-tomcat-app2-deployment configured [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 4d 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 2d 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-85f7f8f6d9-cl4ll 1/1 Running 0 46s 10.200.2.42 172.16.88.162 <none> <none> python-nginx-deployment-5c658bf86b-qvzpn 1/1 Running 0 11m 10.200.233.67 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]#
3.6、podAntiAffinity-软反亲和
[root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 4d 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 2d 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-85f7f8f6d9-cl4ll 1/1 Running 0 46s 10.200.2.42 172.16.88.162 <none> <none> python-nginx-deployment-5c658bf86b-qvzpn 1/1 Running 0 11m 10.200.233.67 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# vi case4-4.5-podAntiAffinity-preferredDuring.yaml [root@easzlab-deploy Affinit-case]# cat case4-4.5-podAntiAffinity-preferredDuring.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app2-deployment-label name: magedu-tomcat-app2-deployment namespace: magedu spec: replicas: 1 selector: matchLabels: app: magedu-tomcat-app2-selector template: metadata: labels: app: magedu-tomcat-app2-selector spec: containers: - name: magedu-tomcat-app2-container image: tomcat:7.0.94-alpine imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: project operator: In values: - pythonx topologyKey: kubernetes.io/hostname namespaces: - magedu [root@easzlab-deploy Affinit-case]# kubectl apply -f case4-4.5-podAntiAffinity-preferredDuring.yaml deployment.apps/magedu-tomcat-app2-deployment configured [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 4d 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 2d 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app2-deployment-5b4596845-xfhs4 1/1 Running 0 18s 10.200.40.221 172.16.88.160 <none> <none> python-nginx-deployment-5c658bf86b-qvzpn 1/1 Running 0 13m 10.200.233.67 172.16.88.161 <none> <none> [root@easzlab-deploy Affinit-case]#
四、污点与容忍
4.1、污点简介
污点(taints),用于node节点排斥 Pod调度， 与亲和的作用是完全相反的,即taint的node和pod是排斥调度关系。
容忍(toleration),用于Pod容忍node节点的污点信息， 即node有污点信息也会将新的pod调度到node
- https://kubernetes.io/zh/docs/concepts/scheduling-eviction/taint-and-toleration/
污点的三种类型：
- NoSchedule: 表示k8s将不会将Pod调度到具有该污点的Node上
- PreferNoSchedule: 表示k8s将尽量避免将Pod调度到具有该污点的Node上
- NoExecute: 表示k8s将不会将Pod调度到具有该污点的Node上， 同时会将Node上已经存在的Pod强制驱逐出去
[root@easzlab-deploy Affinit-case]# kubectl taint nodes 172.16.88.160 key1=value1:NoSchedule #设置污点 node/172.16.88.160 tainted [root@easzlab-deploy Affinit-case]# kubectl describe node 172.16.88.160 #查看污点 Name: 172.16.88.160 Roles: node Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux disktype=ssd kubernetes.io/arch=amd64 kubernetes.io/hostname=172.16.88.160 kubernetes.io/os=linux kubernetes.io/role=node Annotations: node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Tue, 18 Oct 2022 17:09:40 +0800 Taints: key1=value1:NoSchedule Unschedulable: false Lease: HolderIdentity: 172.16.88.160 AcquireTime: <unset> RenewTime: Thu, 27 Oct 2022 14:56:20 +0800 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Wed, 19 Oct 2022 22:12:21 +0800 Wed, 19 Oct 2022 22:12:21 +0800 CalicoIsUp Calico is running on this node MemoryPressure False Thu, 27 Oct 2022 14:55:59 +0800 Wed, 19 Oct 2022 14:30:48 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 27 Oct 2022 14:55:59 +0800 Wed, 19 Oct 2022 14:30:48 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 27 Oct 2022 14:55:59 +0800 Wed, 19 Oct 2022 14:30:48 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Thu, 27 Oct 2022 14:55:59 +0800 Wed, 19 Oct 2022 22:10:59 +0800 KubeletReady kubelet is posting ready status. AppArmor enabled Addresses: InternalIP: 172.16.88.160 Hostname: 172.16.88.160 Capacity: cpu: 4 ephemeral-storage: 49869532Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 8148220Ki pods: 500 Allocatable: cpu: 4 ephemeral-storage: 45959760616 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 7841020Ki pods: 500 System Info: Machine ID: 1a19d04d6cc141e58368dc5dcc39f1a9 System UUID: 591ea877-a39d-4aab-81e2-76cf3a9b08cd Boot ID: e7ce258c-2b9a-4a2a-8bc8-e91af3888bd4 Kernel Version: 5.4.0-128-generic OS Image: Ubuntu 20.04.4 LTS Operating System: linux Architecture: amd64 Container Runtime Version: containerd://1.6.4 Kubelet Version: v1.24.2 Kube-Proxy Version: v1.24.2 PodCIDR: 10.200.3.0/24 PodCIDRs: 10.200.3.0/24 Non-terminated Pods: (4 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-system calico-kube-controllers-5c8bb696bb-5nrbl 0 (0%) 0 (0%) 0 (0%) 0 (0%) 8d kube-system calico-node-68k76 250m (6%) 0 (0%) 0 (0%) 0 (0%) 8d magedu magedu-tomcat-app2-deployment-5b4596845-xfhs4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 8m55s velero-system velero-858b9459f9-4jbns 500m (12%) 1 (25%) 128Mi (1%) 512Mi (6%) 8d Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 750m (18%) 1 (25%) memory 128Mi (1%) 512Mi (6%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: <none> [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl taint node 172.16.88.160 key1:NoSchedule- #取消污点 node/172.16.88.160 untainted [root@easzlab-deploy Affinit-case]# kubectl describe node 172.16.88.160 |grep Taints Taints: <none> [root@easzlab-deploy Affinit-case]#
4.2、容忍简介
tolerations容忍：
- 定义 Pod 的容忍度(可以接受node的哪些污点)， 容忍后可以将Pod调度至含有该污点的node。
基于operator的污点匹配：
- 如果operator是Exists， 则容忍度不需要value而是直接匹配污点类型。
- 如果operator是Equal， 则需要指定value并且value的值需要等于tolerations的key。
验证测试
[root@easzlab-deploy Affinit-case]# kubectl delete -f case4-4.1-nginx.yaml #删除旧的Pod deployment.apps "python-nginx-deployment" deleted service "python-nginx-service" deleted [root@easzlab-deploy Affinit-case]# kubectl delete -f case5.1-taint-tolerations.yaml #删除旧得pod deployment.apps "magedu-tomcat-app1-deployment" deleted service "magedu-tomcat-app1-service" deleted [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl taint nodes 172.16.88.160 key1=value1:NoExecute #节点设置不可调度 node/172.16.88.160 tainted [root@easzlab-deploy Affinit-case]# kubectl taint nodes 172.16.88.163 key1=value1:NoSchedule #节点打上污点信息 node/172.16.88.163 tainted [root@easzlab-deploy Affinit-case]# kubectl describe node 172.16.88.160 |grep Taints Taints: key1=value1:NoExecute [root@easzlab-deploy Affinit-case]# kubectl describe node 172.16.88.163 |grep Taints Taints: key1=value1:NoSchedule [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# vi case5.1-taint-tolerations.yaml [root@easzlab-deploy Affinit-case]# cat case5.1-taint-tolerations.yaml kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app1-deployment-label name: magedu-tomcat-app1-deployment namespace: magedu spec: replicas: 5 selector: matchLabels: app: magedu-tomcat-app1-selector template: metadata: labels: app: magedu-tomcat-app1-selector spec: containers: - name: magedu-tomcat-app1-container #image: harbor.magedu.net/magedu/tomcat-app1:v7 image: tomcat:7.0.93-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 8080 protocol: TCP name: http tolerations: - key: "key1" operator: "Equal" value: "value1" effect: "NoSchedule" --- kind: Service apiVersion: v1 metadata: labels: app: magedu-tomcat-app1-service-label name: magedu-tomcat-app1-service namespace: magedu spec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: 8080 #nodePort: 40003 selector: app: magedu-tomcat-app1-selector [root@easzlab-deploy Affinit-case]# kubectl apply -f case5.1-taint-tolerations.yaml #测试对污点容忍和不容忍分别能否调度 deployment.apps/magedu-tomcat-app1-deployment created service/magedu-tomcat-app1-service created [root@easzlab-deploy Affinit-case]# [root@easzlab-deploy Affinit-case]# kubectl get pod -n magedu -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES magedu-jenkins-deployment-79bbd88cb7-84999 1/1 Running 0 4d 10.200.104.250 172.16.88.163 <none> <none> magedu-jenkins-deployment-79bbd88cb7-pmd9z 1/1 Running 0 2d 10.200.105.177 172.16.88.164 <none> <none> magedu-tomcat-app1-deployment-785c799fc9-5f8gw 1/1 Running 0 22s 10.200.233.81 172.16.88.161 <none> <none> magedu-tomcat-app1-deployment-785c799fc9-cps9r 1/1 Running 0 22s 10.200.105.189 172.16.88.164 <none> <none> magedu-tomcat-app1-deployment-785c799fc9-npthx 1/1 Running 0 22s 10.200.245.41 172.16.88.165 <none> <none> magedu-tomcat-app1-deployment-785c799fc9-rmqxl 1/1 Running 0 22s 10.200.104.206 172.16.88.163 <none> <none> magedu-tomcat-app1-deployment-785c799fc9-rxf5g 1/1 Running 0 22s 10.200.2.46 172.16.88.162 <none> <none> [root@easzlab-deploy Affinit-case]#
删除node节点限制规则
[root@easzlab-deploy Affinit-case]# kubectl taint nodes 172.16.88.163 key1:NoSchedule- node/172.16.88.163 untainted
[root@easzlab-deploy Affinit-case]# kubectl taint nodes 172.16.88.160 key1:NoExecute- node/172.16.88.160 untainted [root@easzlab-deploy Affinit-case]# kubectl describe node 172.16.88.160 |grep Taints Taints: <none> [root@easzlab-deploy Affinit-case]# kubectl describe node 172.16.88.163 |grep Taints Taints: <none> [root@easzlab-deploy Affinit-case]#
五、kubernetes pod驱逐
5.1、驱逐简介
驱逐是由各kubelet进程主动终止Pod， 以回收节点上的内存、 磁盘空间等资源的过程， kubelet监控当前node节点的CPU、 内存、 磁盘空间和文件系统的inode等资源， 当这些资源中的一个或者多个达到特定的消耗水平， kubelet就会主动地将节点上一个或者多个Pod强制驱逐， 以防止当前node节点资源无法正常分配而引发的OOM(OutOfMemory)。
https://kubernetes.io/zh/docs/concepts/scheduling-eviction/node-pressure-eviction/
宿主机内存：
- memory.available #node节点可用内存， 默认 <100Mi
nodefs是节点的主要文件系统,用于保存本地磁盘卷、 emptyDir、 日志存储等数据， 默认是/var/lib/kubelet/， 或者是通过kubelet通过--root-dir所指定的磁盘挂载目录
- nodefs.inodesFree #nodefs的可用inode， 默认<5%
- nodefs.available #nodefs的可用空间,默认<10%
imagefs是可选文件系统， 用于给容器提供运行时存储容器镜像和容器可写层。
- imagefs.inodesFree #imagefs的inode可用百分比
- imagefs.available #imagefs的磁盘空间可用百分比， 默认<15%
- pid.available #可用pid百分比
示例：
evictionHard:
- imagefs.inodesFree: 5%
- imagefs.available: 15%
- memory.available: 300Mi
- nodefs.available: 10%
- nodefs.inodesFree: 5%
- pid.available: 5%
kube-controller-manager实现 eviction:
- node宕机后的驱逐
kubelet实现的eviction：
- 基于node负载、 资源利用率等进行pod驱逐。
驱逐(eviction， 节点驱逐)， 用于当node节点资源不足的时候自动将pod进行强制驱逐， 以保证当前node节点的正常运行。
Kubernetes基于是QoS(服务质量等级)驱逐Pod , Qos等级包括目前包括以下三个:
Guaranteed: #limits和request的值相等， 等级最高、 最后被驱逐
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 500m
memory: 256Mi
Burstable: #limit和request不相等， 等级折中、 中间被驱逐
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 256m
memory: 128Mi
BestEffort: #没有限制， 即resources为空， 等级最低、 最先被驱逐
5.2、驱逐条件
eviction-signal：kubelet捕获node节点驱逐触发信号，进行判断是否驱逐，比如通过cgroupfs获取memory.available的值来进行下一步匹配。
operator：操作符，通过操作符对比条件是否匹配资源量是否触发驱逐。
quantity：使用量，即基于指定的资源使用值进行判断，如memory.available: 300Mi、nodefs.available: 10%等。
- 比如：nodefs.available<10% #当node节点磁盘空间可用率低于10%就会触发驱逐信号
软驱逐
软驱逐不会立即驱逐pod， 可以自定义宽限期， 在条件持续到宽限期还没有恢复， kubelet再强制杀死pod并触发驱逐：
软驱逐条件：
- eviction-soft： 软驱逐触发条件， 比如memory.available < 1.5Gi， 如果驱逐条件持续时长超过指定的宽限期， 可以触发Pod驱逐。
- eviction-soft-grace-period： 软驱逐宽限期， 如 memory.available=1m30s， 定义软驱逐条件在触发Pod驱逐之前必须保持多长时间。
- eviction-max-pod-grace-period： 终止pod的宽限期， 即在满足软驱逐条件而终止Pod时使用的最大允许宽限期（以秒为单位） 。
硬驱逐
硬驱逐条件没有宽限期， 当达到硬驱逐条件时， kubelet 会强制立即杀死 pod并驱逐：
kubelet 具有以下默认硬驱逐条件(可以自行调整)：
- memory.available<100Mi
- nodefs.available<10%
- imagefs.available<15%
- nodefs.inodesFree<5%（Linux 节点）
kubelet service文件：
- # vim /etc/systemd/system/kubelet.service
kubelet配置文件：
- # vim /var/lib/kubelet/config.yaml
evictionHard:
- imagefs.available: 15%
- memory.available: 300Mi
- nodefs.available: 10%
- nodefs.inodesFree: 5%
