kubernetes-v1.20.8二进制安装(八)-kubelet

kubelet 是负责容器真正运行的核心组件，主要的职责如下所示：

负责 Node 节点上 Pod 的创建、修改、监控、删除等全生命周期的管理
定时上报本地 Node 的状态信息给 API Server
kubelet 是 Master 和 Node 之间的桥梁，接收 API Server 分配给它的任务并执行
kubelet 通过 API Server 间接与 Etcd 集群交互来读取集群配置信息

kubelet 启动时自动向 kube-apiserver 注册节点信息，内置的 cadvisor 统计和监控节点的资源使用情况。

1. 部署

1.1. 生成bootstarp-kubeconfig

kubelet 启动时查找 --kubeletconfig 参数对应的文件是否存在，如果不存在则使用 --bootstrap-kubeconfig 指定的 kubeconfig 文件向 kube-apiserver 发送证书签名请求 (CSR)。

kube-apiserver 收到 CSR 请求后，对其中的 Token 进行认证，认证通过后将请求的 user 设置为 system:bootstrap:，group 设置为 system:bootstrappers，这一过程称为 Bootstrap Token Auth。

获取token.csv 随机字符串 token.csv文件在kube-apiserver篇章生成

BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /opt/software/kubernetes/master/config/token.csv)

设置集群参数

cd /opt/software/kubernetes/certs/

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://islb.k8s-host.com:6443 --kubeconfig=/opt/software/kubernetes/node/kubeconfig/kubelet-bootstrap.kubeconfig

设置客户端认证参数

kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=/opt/software/kubernetes/node/kubeconfig/kubelet-bootstrap.kubeconfig

设置上下文参数

kubectl config set-context kubelet-bootstrap --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=/opt/software/kubernetes/node/kubeconfig/kubelet-bootstrap.kubeconfig

设置默认上下文

kubectl config use-context kubelet-bootstrap --kubeconfig=/opt/software/kubernetes/node/kubeconfig/kubelet-bootstrap.kubeconfig

角色权限绑定

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

如无配置则出现如下报错:

E0315 16:35:25.834798   14131 certificate_manager.go:434] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:9jcwib" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope

1.2. 编写kubelet配置文件

不同node节点修改配置 address healthzBindAddress

cat >  /opt/software/kubernetes/node/config/kubelet-config.yaml<<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: "10.0.0.40"
port: 10250
healthzBindAddress: "10.0.0.40"
healthzPort: 10248
readOnlyPort: 0
cgroupDriver: "cgroupfs"
clusterDomain: "cluster.local."
clusterDNS: ["10.254.0.2"]
authentication:
    x509:
        clientCAFile: "/opt/software/kubernetes/certs/ca.pem"
    webhook:
        enabled: true
        cacheTTL: "2m0s"
    anonymous:
        enabled: false
authorization:
    mode: Webhook
    webhook:
        cacheAuthorizedTTL: "5m0s"
        cacheUnauthorizedTTL: "30s"
hairpinMode: "promiscuous-bridge"
serializeImagePulls: false
featureGates:
    RotateKubeletClientCertificate: true
    RotateKubeletServerCertificate: true
EOF

address：kubelet 安全端口（https，10250）监听的地址，不能为 127.0.0.1，否则 kube-apiserver、heapster 等不能调用 kubelet 的 API；
readOnlyPort=0：关闭只读端口(默认 10255)，等效为未指定；
authentication.anonymous.enabled：设置为 false，不允许匿名访问 10250 端口；
authentication.x509.clientCAFile：指定签名客户端证书的 CA 证书，开启 HTTP 证书认证；
authentication.webhook.enabled=true：开启 HTTPs bearer token 认证；
对于未通过 x509 证书和 webhook 认证的请求(kube-apiserver 或其他客户端)，将被拒绝，提示 Unauthorized；
authroization.mode=Webhook：kubelet 使用 SubjectAccessReview API 查询 kube-apiserver 某 user、group 是否具有操作资源的权限(RBAC)；

注意不同节点修改--hostname-override

cat >  /opt/software/kubernetes/node/config/kubelet-config.conf<<EOF
KUBE_KUBELET_OPTS="--root-dir=/data/kubelet \\
  --kubeconfig=/opt/software/kubernetes/kubeconfig/kubelet.kubeconfig \\
  --bootstrap-kubeconfig=/opt/software/kubernetes/kubeconfig/kubelet-bootstrap.kubeconfig \\
  --cert-dir=/opt/software/kubernetes/certs/ \\
  --network-plugin=cni \\
  --cni-bin-dir=/opt/software/kubernetes/cni/bin \\
  --cni-conf-dir=/opt/software/kubernetes/cni/net.d \\
  --container-runtime=remote \\
  --container-runtime-endpoint=unix:///run/containerd/containerd.sock \\
  --config=/opt/software/kubernetes/config/kubelet-config.yaml \\
  --hostname-override=node01.k8s-host.com \\
  --image-pull-progress-deadline=15m \\
  --logtostderr=true \\
  --v=4"
EOF

--kubeconfig 当前使用bootstrap机制, 该文件可为空, 验证成功后会自动生成
--bootstrap-kubeconfig bootstrap机制会发送csr请求
如果设置了 --hostname-override 选项，则 kube-proxy 也需要设置该选项，否则会出现找不到 Node 的情况；

1.3. 编写service启动文件

cat > /opt/software/kubernetes/node/service/kubelet.service<<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
WorkingDirectory=/opt/software/kubernetes
EnvironmentFile=-/opt/software/kubernetes/config/kubelet-config.conf
ExecStart=/opt/software/kubernetes/bin/kubelet \$KUBE_KUBELET_OPTS

Restart=always
RestartSec=5
StartLimitInterval=0

[Install]
WantedBy=multi-user.target
EOF

1.4. 复制文件至节点

hosts=(node01 node02 node03)
domain='k8s-host.com'
config_files=('config/kubelet-config.yaml' 'config/kubelet-config.conf')
cd /opt/software/kubernetes
for host in ${hosts[*]}
do
    scp -r node/{bin,certs,kubeconfig,service} ${host}.${domain}:/opt/software/kubernetes/
done

# 不可重复执行,否则覆盖修改的配置
for host in ${hosts[*]}
do
	for file in ${config_files[*]}
	do
    	scp -r node/${file} ${host}.${domain}:/opt/software/kubernetes/${file}
    done
done

1.5. 启动服务

服务启动后kubelet会向apiserver发送csr请求, 需要手动允许才能将节点添加至集群中. 如有需要, 可以在node节点启动前配置自动允许csr请求功能, 请看1.7章节

hosts=(node01 node02 node03)
domain='k8s-host.com'
for host in ${hosts[*]}
do
	# 软链接
    ssh root@${host}.${domain} "ln -s /opt/software/kubernetes/service/kubelet.service  /usr/lib/systemd/system/kubelet.service "
    # 开机启动并启动服务
    ssh root@${host}.${domain} "systemctl daemon-reload && systemctl enable kubelet --now "
done

1.6. 手动允许csr请求

查看

$ kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-7jpNjn_RuhiLCQmo4yLDK0OB5YHmnt_lKwF3VR7k_8g   23s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending
node-csr-hJyBfpLLXldZ6QU2jsOTGODfATIDUyd4E6Sb9P5eZ2s   23s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

允许node节点csr请求

kubectl certificate approve node-csr-7jpNjn_RuhiLCQmo4yLDK0OB5YHmnt_lKwF3VR7k_8g
kubectl certificate approve node-csr-hJyBfpLLXldZ6QU2jsOTGODfATIDUyd4E6Sb9P5eZ2s

# 等待几分钟后查看
kubectl get nodes

NAME                  STATUS   ROLES    AGE   VERSION
node01.k8s-host.com   Ready    <none>   28s   v1.20.8
node02.k8s-host.com   Ready    <none>   25m   v1.20.8

token 有效期为 1 天，超期后将不能再被用来 boostrap kubelet，且会被 kube-controller-manager 的 tokencleaner 清理；

测试中 node节点的STATUS状态有时候为NoReady 有时候为Ready, 如果为NoReady , 则尝试将kube-proxy启动后再次查看

csr请求通过后, 如正常, 会在节点中生成kubeconfig文件, 路径由配置--kubeconfig指定

1.7. 扩展: 自动允许csr请求(可选)

kubelet 创建 CSR 请求后，下一步需要创建被 approve，有两种方式：

kube-controller-manager 自动 aprrove；
手动使用命令 kubectl certificate approve；

CSR 被 approve 后，kubelet 向 kube-controller-manager 请求创建 client 证书，kube-controller-manager 中的 csrapproving controller 使用 SubjectAccessReview API 来检查 kubelet 请求（对应的 group 是 system:bootstrappers）是否具有相应的权限。

创建三个 ClusterRoleBinding，分别授予 group system:bootstrappers 和 group system:nodes 进行 approve client、renew client、renew server 证书的权限（server csr 是手动 approve 的）：

cat >> /opt/software/kubernetes/yaml/approve_csr.yaml <<EOF
# Approve all CSRs for the group "system:bootstrappers"
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: auto-approve-csrs-for-group
subjects:
- kind: Group
  name: system:kubelet-bootstrap
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
  apiGroup: rbac.authorization.k8s.io
---
# To let a node of the group "system:nodes" renew its own credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: node-client-cert-renewal
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/selfnodeserver"]
  verbs: ["create"]
---
# To let a node of the group "system:nodes" renew its own server credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: node-server-cert-renewal
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: approve-node-server-renewal-csr
  apiGroup: rbac.authorization.k8s.io
EOF

auto-approve-csrs-for-group：自动 approve node 的第一次 CSR；注意第一次 CSR 时，请求的 Group 为 system:bootstrappers；
node-client-cert-renewal：自动 approve node 后续过期的 client 证书，自动生成的证书 Group 为 system:nodes;
node-server-cert-renewal：自动 approve node 后续过期的 server 证书，自动生成的证书 Group 为 system:nodes;

执行

kubectl apply -f  /opt/software/kubernetes/yaml/approve_csr.yaml

问题

问题一

# kubectl certificate approve  node-csr-Mf03MjWjIOyKnWy0-OqT3CrXOJ-DQtmISfIVhmlZkvQ
No resources found
error: no kind "CertificateSigningRequest" is registered for version "certificates.k8s.io/v1" in scheme "k8s.io/kubectl/pkg/scheme/scheme.go:28"

kubectl客户端版本不够高, 需要重新部署kubectl客户端

参考链接

posted @ 2021-09-03 18:04 风吹蛋生丶阅读(1207) 评论(0) 编辑收藏举报

刷新页面返回顶部

风吹蛋生丶