k8s资源编排
在 k8s 中，一般使用 YAML 格式的文件来创建符合我们预期期望的 pod,这样的 YAML 文件称为资源清单。
1. 常用字段
必须存在的属性
更全的配置如下：(大的理解可以分为版本apiVersion\类型kind\metadata元数据\spec详细描述)
apiVersion: v1 #指定api版本，此值必须在kubectl api-versions中 kind: Pod #指定创建资源的角色/类型 ，值在 kubectl api-resources 中 metadata: #资源的元数据/属性 name: web04-pod #资源的名字，在同一个namespace中必须唯一 labels: #设定资源的标签 k8s-app: apache version: v1 kubernetes.io/cluster-service: "true" annotations: #自定义注解列表 - name: String #自定义注解名字 spec: #specification of the resource content 指定该资源的内容 restartPolicy: Always #表明该容器一直运行，默认k8s的策略，在此容器退出后，会立即创建一个相同的容器 nodeSelector: #节点选择，先给主机打标签kubectl label nodes kube-node1 zone=node1 zone: node1 containers: - name: web04-pod #容器的名字 image: web:apache #容器使用的镜像地址 imagePullPolicy: Never #三个选择Always、Never、IfNotPresent，每次启动时检查和更新（从registery）images的策略， # Always，每次都检查 # Never，每次都不检查（不管本地是否有） # IfNotPresent，如果本地有就不检查，如果没有就拉取 command: ['sh'] #启动容器的运行命令，将覆盖容器中的Entrypoint,对应Dockefile中的ENTRYPOINT args: ["$(str)"] #启动容器的命令参数，对应Dockerfile中CMD参数 env: #指定容器中的环境变量 - name: str #变量的名字 value: "/etc/run.sh" #变量的值 resources: #资源管理 requests: #容器运行时，最低资源需求，也就是说最少需要多少资源容器才能正常运行 cpu: 0.1 #CPU资源（核数），两种方式，浮点数或者是整数+m，0.1=100m，最少值为0.001核（1m） memory: 32Mi #内存使用量 limits: #资源限制 cpu: 0.5 memory: 32Mi ports: - containerPort: 80 #容器开发对外的端口 name: httpd #名称 protocol: TCP livenessProbe: #pod内容器健康检查的设置 httpGet: #通过httpget检查健康，返回200-399之间，则认为容器正常 path: / #URI地址 port: 80 #host: 127.0.0.1 #主机地址 scheme: HTTP initialDelaySeconds: 180 #表明第一次检测在容器启动后多长时间后开始 timeoutSeconds: 5 #检测的超时时间 periodSeconds: 15 #检查间隔时间 #也可以用这种方法 #exec: 执行命令的方法进行监测，如果其退出码不为0，则认为容器正常 # command: # - cat # - /tmp/health #也可以用这种方法 #tcpSocket: //通过tcpSocket检查健康 # port: number lifecycle: #生命周期管理 postStart: #容器运行之前运行的任务 exec: command: - 'sh' - 'yum upgrade -y' preStop: #容器关闭之前运行的任务 exec: command: ['service httpd stop'] volumeMounts: #详情请见http://blog.csdn.net/liyingke112/article/details/76577520 - name: volume #挂载设备的名字，与volumes[*].name 需要对应 mountPath: /data #挂载到容器的某个路径下 readOnly: True volumes: #定义一组挂载设备 - name: volume #定义一个挂载设备的名字 #meptyDir: {} hostPath: path: /opt #挂载设备类型为hostPath，路径为宿主机下的/opt,这里设备类型支持很多种
比如之前我们安装flannel 的时候就用的yml， 网址： https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml ， 内容如下：
--- apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: psp.flannel.unprivileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default spec: privileged: false volumes: - configMap - secret - emptyDir - hostPath allowedHostPaths: - pathPrefix: "/etc/cni/net.d" - pathPrefix: "/etc/kube-flannel" - pathPrefix: "/run/flannel" readOnlyRootFilesystem: false # Users and groups runAsUser: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny # Privilege Escalation allowPrivilegeEscalation: false defaultAllowPrivilegeEscalation: false # Capabilities allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] defaultAddCapabilities: [] requiredDropCapabilities: [] # Host namespaces hostPID: false hostIPC: false hostNetwork: true hostPorts: - min: 0 max: 65535 # SELinux seLinux: # SELinux is unused in CaaSP rule: 'RunAsAny' --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: flannel rules: - apiGroups: ['extensions'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: ['psp.flannel.unprivileged'] - apiGroups: - "" resources: - pods verbs: - get - apiGroups: - "" resources: - nodes verbs: - list - watch - apiGroups: - "" resources: - nodes/status verbs: - patch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- kind: ConfigMap apiVersion: v1 metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node app: flannel data: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } } --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni-plugin image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.0 command: - cp args: - -f - /flannel - /opt/cni/bin/flannel volumeMounts: - name: cni-plugin mountPath: /opt/cni/bin - name: install-cni image: rancher/mirrored-flannelcni-flannel:v0.16.1 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: rancher/mirrored-flannelcni-flannel:v0.16.1 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni-plugin hostPath: path: /opt/cni/bin - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg
--- 标识多个yml 文件，也可以认为是多个yml 配置合并到一个文件。
2. 快速编写yml
有两种方式。
第一种：使用kubectl create 命令快速生成yml 文件， --dry-run 是尝试运行但是不真正运行
kubectl create deployment web --image=nginx -o yaml --dry-run
也可以输出到本地：
kubectl create deployment web --image=nginx -o yaml --dry-run > my.yml
查看内容如下：
[root@k8smaster1 ~]# cat my.yml apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: web name: web spec: replicas: 1 selector: matchLabels: app: web strategy: {} template: metadata: creationTimestamp: null labels: app: web spec: containers: - image: nginx name: nginx resources: {} status: {}
第二种方式： 使用现有的deployment 部署
(1) 获取deployment
[root@k8smaster1 ~]# kubectl get deploy -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR nginx 1/1 1 1 10h nginx nginx app=nginx
(2) 导出yaml 文件
[root@k8smaster1 ~]# kubectl get deploy nginx -o=yaml --export > my2.yml Flag --export has been deprecated, This flag is deprecated and will be removed in future.
(3) 查看文件内容
apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" creationTimestamp: null generation: 1 labels: app: nginx managedFields: - apiVersion: apps/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:labels: .: {} f:app: {} f:spec: f:progressDeadlineSeconds: {} f:replicas: {} f:revisionHistoryLimit: {} f:selector: f:matchLabels: .: {} f:app: {} f:strategy: f:rollingUpdate: .: {} f:maxSurge: {} f:maxUnavailable: {} f:type: {} f:template: f:metadata: f:labels: .: {} f:app: {} f:spec: f:containers: k:{"name":"nginx"}: .: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:dnsPolicy: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: {} f:terminationGracePeriodSeconds: {} manager: kubectl operation: Update time: "2022-01-09T03:49:49Z" - apiVersion: apps/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:deployment.kubernetes.io/revision: {} f:status: f:availableReplicas: {} f:conditions: .: {} k:{"type":"Available"}: .: {} f:lastTransitionTime: {} f:lastUpdateTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} k:{"type":"Progressing"}: .: {} f:lastTransitionTime: {} f:lastUpdateTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} f:observedGeneration: {} f:readyReplicas: {} f:replicas: {} f:updatedReplicas: {} manager: kube-controller-manager operation: Update time: "2022-01-09T03:52:10Z" name: nginx selfLink: /apis/apps/v1/namespaces/default/deployments/nginx spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: nginx strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: app: nginx spec: containers: - image: nginx imagePullPolicy: Always name: nginx resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 status: {}
3. 测试：
1. 清除历史资源
(1) 查看历史资源
[root@k8smaster1 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d2h nginx NodePort 10.96.201.24 <none> 80:30951/TCP 3d [root@k8smaster1 ~]# kubectl get deployments NAME READY UP-TO-DATE AVAILABLE AGE nginx 1/1 1 1 3d [root@k8smaster1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-f89759699-cnj62 1/1 Running 0 3d
(2) 删除历史资源
[root@k8smaster1 ~]# kubectl delete deployments nginx # 删除deplyments， 自动删除pod deployment.apps "nginx" deleted [root@k8smaster1 ~]# kubectl get deployments No resources found in default namespace. [root@k8smaster1 ~]# kubectl get pods No resources found in default namespace. [root@k8smaster1 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d2h nginx NodePort 10.96.201.24 <none> 80:30951/TCP 3d [root@k8smaster1 ~]# kubectl delete svc nginx # 删除service，访问规则 service "nginx" deleted [root@k8smaster1 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d2h
2. 导出yaml
[root@k8smaster1 ~]# kubectl create deployment web --image=nginx -o yaml --dry-run > web.yml W0111 23:38:16.116036 115485 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
3. 查看yml 默认内容
[root@k8smaster1 ~]# cat web.yml apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: web name: web spec: replicas: 1 selector: matchLabels: app: web strategy: {} template: metadata: creationTimestamp: null labels: app: web spec: containers: - image: nginx name: nginx resources: {} status: {}
4. 修改yml 内容为如下：
[root@k8smaster1 ~]# cat web.yml apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: web name: web spec: replicas: 2 selector: matchLabels: app: web strategy: {} template: metadata: creationTimestamp: null labels: app: web spec: containers: - image: nginx:1.14 name: nginx resources: {} status: {}
修改其副本数量为2， 并且修改nginx 镜像版本为1.14
5. 创建nginx 服务
[root@k8smaster1 ~]# kubectl apply -f web.yml deployment.apps/web created [root@k8smaster1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE web-65b7447c7-6d6jx 0/1 ContainerCreating 0 11s web-65b7447c7-9rcbc 0/1 ContainerCreating 0 11s
等待其拉取镜像完成之后再次查看：
[root@k8smaster1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-65b7447c7-6d6jx 1/1 Running 0 57s 10.244.2.5 k8snode2 <none> <none> web-65b7447c7-9rcbc 1/1 Running 0 57s 10.244.1.6 k8snode1 <none> <none> [root@k8smaster1 ~]# curl http://10.244.2.5 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>
6. 直接从k8smaster 节点进入到pod 中查看nginx 版本信息
[root@k8smaster1 ~]# kubectl exec -it web-65b7447c7-6d6jx bash kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead. root@web-65b7447c7-6d6jx:/# nginx -v nginx version: nginx/1.14.2
kubectl exec -it podName bash 等价于用docker 进入容器。
7. 使用yml 删除创建的资源
[root@k8smaster1 ~]# kubectl delete -f web.yml deployment.apps "web" deleted [root@k8smaster1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE web-65b7447c7-6d6jx 0/1 Terminating 0 21h web-65b7447c7-9rcbc 0/1 Terminating 0 21h [root@k8smaster1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE web-65b7447c7-6d6jx 0/1 Terminating 0 21h web-65b7447c7-9rcbc 0/1 Terminating 0 21h [root@k8smaster1 ~]# kubectl get deployments No resources found in default namespace. [root@k8smaster1 ~]# kubectl get pods No resources found in default namespace. [root@k8smaster1 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d
可以使用kubectl delete -f yml 指定根据yml 配置文件删除指定的资源。然后等待k8s 停止删除后再次查看发现会被删除掉。
【当你用心写完每一篇博客之后,你会发现它比你用代码实现功能更有成就感!】
