linux运维、架构之路-kubernetes集群维护

一、Etcd数据库备份与恢复

       Kubernetes 使用Etcd 数据库实时存储集群中的数据，安全起见，一定要备份！

1、kubeadm部署方式备份

①备份

ETCDCTL_API=3 etcdctl \
snapshot save snap.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/peer.crt \
--key=/etc/kubernetes/pki/etcd/peer.key

②恢复

暂停kube-apiserver和etcd容器

mv /etc/kubernetes/manifests /etc/kubernetes/manifests.bak
mv /var/lib/etcd/ /var/lib/etcd.bak

执行恢复命令

ETCDCTL_API=3 etcdctl \
snapshot restore snap.db \
--data-dir=/var/lib/etcd

启动kube-apiserver和etcd容器

mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests

2、二进制部署方式备份

①备份

ETCDCTL_API=3 etcdctl \
snapshot save snap.db \
--endpoints=https://192.168.56.61:2379 \
--cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/server.pem \
--key=/opt/etcd/ssl/server-key.pem

②恢复

暂停kube-apiserver和etcd

systemctl stop kube-apiserver
systemctl stop etcd
mv /var/lib/etcd/default.etcd /var/lib/etcd/default.etcd.bak

在每个节点上恢复

ETCDCTL_API=3 etcdctl snapshot restore snap.db \
--name etcd-1 \
--initial-cluster="etcd-1=https://192.168.56.61:2380,etcd-2=https://192.168.56.62:2380,etcd-3=https://192.168.56.63:2380" \
--initial-cluster-token=etcd-cluster \
--initial-advertise-peer-urls=https://192.168.56.61:2380 \
--data-dir=/var/lib/etcd/default.etcd

启动kube-apiserver和etcd服务

systemctl start kube-apiserver
systemctl start etcd

 二、Node节点扩容

1、Bootstrap Token 方式增加Node节点

        在kubernetes集群中，Node上组件kubelet和kube-proxy都需要与kube-apiserver进行通信，为了增加传输安全性，采用https方式。这就涉及到Node组件需要具备kube-apiserver用的证书颁发机构（CA）签发客户端证书，当规模较大时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。
为了简化流程，Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，所以强烈建议在Node上使用这种方式。

①kube-apiserver配置文件中是否启用Bootstrap Token

参与官方文档

https://kubernetes.io/zh/docs/reference/access-authn-authz/bootstrap-tokens/

https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/

[root@k8s-node1 ~]# cat /app/kubernetes/cfg/kube-apiserver.conf
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/app/kubernetes/logs \
--etcd-servers=https://192.168.29.15:2379,https://192.168.29.16:2379,https://192.168.29.17:2379 \
--bind-address=192.168.29.15 \
--secure-port=6443 \
--advertise-address=192.168.29.15 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \ 
--token-auth-file=/app/kubernetes/cfg/token.csv \

②使用Secret存储Bootstrap Token

Bootstrap Token值格式：07401b.f395accd246ae52d (点左边是Token ID，右边Token Secret)

apiVersion: v1
kind: Secret
metadata:
  # name 必须是 "bootstrap-token-<token id>" 格式的
  name: bootstrap-token-07401b
  namespace: kube-system

# type 必须是 'bootstrap.kubernetes.io/token'
type: bootstrap.kubernetes.io/token
stringData:
  # 供人阅读的描述，可选。
  description: "The default bootstrap token generated by 'kubeadm init'."

  # 令牌 ID 和秘密信息，必需。
  token-id: 07401b
  token-secret: base64(f395accd246ae52d)

  # 可选的过期时间字段
  expiration: "2025-10-10T03:22:11Z"   #只修改此处即可
  # 允许的用法
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"

  # 令牌要认证为的额外组，必须以 "system:bootstrappers:" 开头
  auth-extra-groups: system:bootstrappers:worker,system:bootstrappers:ingress

创建bootstrap-token.yaml即可

③创建RBAC角色绑定，允许kubelet tls bootstrap创建CSR请求

# Approve all CSRs for the group "system:bootstrappers"
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: auto-approve-csrs-for-group
subjects:
- kind: Group
  name: system:bootstrappers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: approve-node-client-csr
  apiGroup: rbac.authorization.k8s.io

创建bootstrap-rbac.yaml

④node节点安装组件，Docker、kubelet、kube-proxy、cni插件等

修改kubelet、kube-proxy配置文件中主机名

⑤kubelet配置Bootstrap kubeconfig文件

[root@k8s-node1 ~]# cat /app/kubernetes/cfg/kubelet.conf 
KUBELET_OPTS="--logtostderr=false \
--v=4 \
--log-dir=/app/kubernetes/logs \
--hostname-override=k8s-node1 \
--network-plugin=cni \
--kubeconfig=/app/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/app/kubernetes/cfg/bootstrap.kubeconfig \
--config=/app/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/app/kubernetes/ssl \
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"