010.Kubernetes二进制master节点部署kube-apiserver

一 获取Kubernetes二进制文件

1.1 master节点服务

kubernetes master 节点运行如下组件:

  • kube-apiserver
  • kube-scheduler
  • kube-controller-manager
  • kube-nginx

kube-apiserver、kube-scheduler 和 kube-controller-manager 均以多实例模式运行:

kube-scheduler 和 kube-controller-manager 会自动选举产生一个 leader 实例,其它实例处于阻塞模式,当 leader 挂了后,重新选举产生新的 leader,从而保证服务可用性。

kube-apiserver 是无状态的,需要通过 kube-nginx 进行代理访问,从而保证服务可用性。

1.2 安装Kubernetes

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# wget https://storage.googleapis.com/kubernetes-release/release/v1.18.3/kubernetes-server-linux-amd64.tar.gz
  3 [root@master01 work]# tar -xzvf kubernetes-server-linux-amd64.tar.gz
  4 [root@master01 work]# cd kubernetes
  5 [root@master01 kubernetes]# tar -xzvf kubernetes-src.tar.gz

提示:本步骤操作仅需要在master01节点操作。

1.3 分发Kubernetes

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     scp -rp kubernetes/server/bin/{apiextensions-apiserver,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@${master_ip}:/opt/k8s/bin/
  7     ssh root@${master_ip} "chmod +x /opt/k8s/bin/*"
  8   done

提示:本步骤操作仅需要在master01节点操作。

二 创建kube-apiserver证书

2.1 高可用apiserver介绍

本实验部署一个三实例 kube-apiserver 集群的步骤,它们通过 kube-nginx 进行代理访问,从而保证服务可用性。

2.2 创建Kubernetes证书和私钥

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# cat > kubernetes-csr.json <<EOF
  4 {
  5   "CN": "kubernetes",
  6   "hosts": [
  7     "127.0.0.1",
  8     "172.24.8.71",
  9     "172.24.8.72",
 10     "172.24.8.73",
 11     "172.24.8.100",
 12     "${CLUSTER_KUBERNETES_SVC_IP}",
 13     "kubernetes",
 14     "kubernetes.default",
 15     "kubernetes.default.svc",
 16     "kubernetes.default.svc.cluster",
 17     "kubernetes.default.svc.cluster.local."
 18   ],
 19   "key": {
 20     "algo": "rsa",
 21     "size": 2048
 22   },
 23   "names": [
 24     {
 25       "C": "CN",
 26       "ST": "Shanghai",
 27       "L": "Shanghai",
 28       "O": "k8s",
 29       "OU": "System"
 30     }
 31   ]
 32 }
 33 EOF		#创建Kubernetes的CA证书请求文件

提示:本步骤操作仅需要在master01节点操作。

解释:

hosts 字段指定授权使用该证书的 IP 和域名列表,这里列出了 master 节点 IP、kubernetes 服务的 IP 和域名;

kubernetes 服务 IP 是 apiserver 自动创建的,一般是 --service-cluster-ip-range 参数指定的网段的第一个IP,后续可以通过下面命令获取:

# kubectl get svc kubernetes

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
  3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \
  4 -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes	#生成密钥和证书

提示:本步骤操作仅需要在master01节点操作。

2.3 分发证书和私钥

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "mkdir -p /etc/kubernetes/cert"
  7     scp kubernetes*.pem root@${master_ip}:/etc/kubernetes/cert/
  8   done

提示:本步骤操作仅需要在master01节点操作。

三 配置kube-apiserver审计

3.1 创建加密配置文件

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# cat > encryption-config.yaml <<EOF
  4 kind: EncryptionConfig
  5 apiVersion: v1
  6 resources:
  7   - resources:
  8       - secrets
  9     providers:
 10       - aescbc:
 11           keys:
 12             - name: key1
 13               secret: ${ENCRYPTION_KEY}
 14       - identity: {}
 15 EOF

提示:本步骤操作仅需要在master01节点操作。

3.2 分发加密配置文件

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     scp encryption-config.yaml root@${master_ip}:/etc/kubernetes/
  7   done

提示:本步骤操作仅需要在master01节点操作。

3.3 创建审计策略文件

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# cat > audit-policy.yaml <<EOF
  4 apiVersion: audit.k8s.io/v1beta1
  5 kind: Policy
  6 rules:
  7   # The following requests were manually identified as high-volume and low-risk, so drop them.
  8   - level: None
  9     resources:
 10       - group: ""
 11         resources:
 12           - endpoints
 13           - services
 14           - services/status
 15     users:
 16       - 'system:kube-proxy'
 17     verbs:
 18       - watch
 19 
 20   - level: None
 21     resources:
 22       - group: ""
 23         resources:
 24           - nodes
 25           - nodes/status
 26     userGroups:
 27       - 'system:nodes'
 28     verbs:
 29       - get
 30 
 31   - level: None
 32     namespaces:
 33       - kube-system
 34     resources:
 35       - group: ""
 36         resources:
 37           - endpoints
 38     users:
 39       - 'system:kube-controller-manager'
 40       - 'system:kube-scheduler'
 41       - 'system:serviceaccount:kube-system:endpoint-controller'
 42     verbs:
 43       - get
 44       - update
 45 
 46   - level: None
 47     resources:
 48       - group: ""
 49         resources:
 50           - namespaces
 51           - namespaces/status
 52           - namespaces/finalize
 53     users:
 54       - 'system:apiserver'
 55     verbs:
 56       - get
 57 
 58   # Don't log HPA fetching metrics.
 59   - level: None
 60     resources:
 61       - group: metrics.k8s.io
 62     users:
 63       - 'system:kube-controller-manager'
 64     verbs:
 65       - get
 66       - list
 67 
 68   # Don't log these read-only URLs.
 69   - level: None
 70     nonResourceURLs:
 71       - '/healthz*'
 72       - /version
 73       - '/swagger*'
 74 
 75   # Don't log events requests.
 76   - level: None
 77     resources:
 78       - group: ""
 79         resources:
 80           - events
 81 
 82   # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
 83   - level: Request
 84     omitStages:
 85       - RequestReceived
 86     resources:
 87       - group: ""
 88         resources:
 89           - nodes/status
 90           - pods/status
 91     users:
 92       - kubelet
 93       - 'system:node-problem-detector'
 94       - 'system:serviceaccount:kube-system:node-problem-detector'
 95     verbs:
 96       - update
 97       - patch
 98 
 99   - level: Request
100     omitStages:
101       - RequestReceived
102     resources:
103       - group: ""
104         resources:
105           - nodes/status
106           - pods/status
107     userGroups:
108       - 'system:nodes'
109     verbs:
110       - update
111       - patch
112 
113   # deletecollection calls can be large, don't log responses for expected namespace deletions
114   - level: Request
115     omitStages:
116       - RequestReceived
117     users:
118       - 'system:serviceaccount:kube-system:namespace-controller'
119     verbs:
120       - deletecollection
121 
122   # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
123   # so only log at the Metadata level.
124   - level: Metadata
125     omitStages:
126       - RequestReceived
127     resources:
128       - group: ""
129         resources:
130           - secrets
131           - configmaps
132       - group: authentication.k8s.io
133         resources:
134           - tokenreviews
135   # Get repsonses can be large; skip them.
136   - level: Request
137     omitStages:
138       - RequestReceived
139     resources:
140       - group: ""
141       - group: admissionregistration.k8s.io
142       - group: apiextensions.k8s.io
143       - group: apiregistration.k8s.io
144       - group: apps
145       - group: authentication.k8s.io
146       - group: authorization.k8s.io
147       - group: autoscaling
148       - group: batch
149       - group: certificates.k8s.io
150       - group: extensions
151       - group: metrics.k8s.io
152       - group: networking.k8s.io
153       - group: policy
154       - group: rbac.authorization.k8s.io
155       - group: scheduling.k8s.io
156       - group: settings.k8s.io
157       - group: storage.k8s.io
158     verbs:
159       - get
160       - list
161       - watch
162 
163   # Default level for known APIs
164   - level: RequestResponse
165     omitStages:
166       - RequestReceived
167     resources:
168       - group: ""
169       - group: admissionregistration.k8s.io
170       - group: apiextensions.k8s.io
171       - group: apiregistration.k8s.io
172       - group: apps
173       - group: authentication.k8s.io
174       - group: authorization.k8s.io
175       - group: autoscaling
176       - group: batch
177       - group: certificates.k8s.io
178       - group: extensions
179       - group: metrics.k8s.io
180       - group: networking.k8s.io
181       - group: policy
182       - group: rbac.authorization.k8s.io
183       - group: scheduling.k8s.io
184       - group: settings.k8s.io
185       - group: storage.k8s.io
186 
187   # Default level for all other requests.
188   - level: Metadata
189     omitStages:
190       - RequestReceived
191 EOF

提示:本步骤操作仅需要在master01节点操作。

3.4 分发策略文件

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     scp audit-policy.yaml root@${master_ip}:/etc/kubernetes/audit-policy.yaml
  7   done

提示:有关审计知识参考:https://www.cnblogs.com/liabio/p/11696081.html。

提示:本步骤操作仅需要在master01节点操作。

四 配置metrics-server

4.1 创建访问 metrics-server的证书和密钥

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# cat > proxy-client-csr.json <<EOF
  3 {
  4   "CN": "system:metrics-server",
  5   "hosts": [],
  6   "key": {
  7     "algo": "rsa",
  8     "size": 2048
  9   },
 10   "names": [
 11     {
 12       "C": "CN",
 13       "ST": "Shanghai",
 14       "L": "Shanghai",
 15       "O": "k8s",
 16       "OU": "System"
 17     }
 18   ]
 19 }
 20 EOF		#创建metrics-server的CA证书请求文件

提示:本步骤操作仅需要在master01节点操作。

解释:

CN 名称需要位于 kube-apiserver 的 --requestheader-allowed-names 参数中,否则后续访问 metrics 时会提示权限不足。

  1 [root@master01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
  2 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \
  3 -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client #生成密钥和证书

提示:本步骤操作仅需要在master01节点操作。

4.2 分发证书和私钥

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     scp proxy-client*.pem root@${master_ip}:/etc/kubernetes/cert/
  7   done

提示:本步骤操作仅需要在master01节点操作。

五 配置kube-apiserver

5.1 创建kube-apiserver的systemd

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# cat > kube-apiserver.service.template <<EOF
  4 [Unit]
  5 Description=Kubernetes API Server
  6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
  7 After=network.target
  8 
  9 [Service]
 10 WorkingDirectory=${K8S_DIR}/kube-apiserver
 11 ExecStart=/opt/k8s/bin/kube-apiserver \\
 12   --insecure-port=0 \\
 13   --secure-port=6443 \\
 14   --bind-address=##MASTER_IP## \\
 15   --advertise-address=##MASTER_IP## \\
 16   --default-not-ready-toleration-seconds=360 \\
 17   --default-unreachable-toleration-seconds=360 \\
 18   --feature-gates=DynamicAuditing=true \\
 19   --max-mutating-requests-inflight=2000 \\
 20   --max-requests-inflight=4000 \\
 21   --default-watch-cache-size=200 \\
 22   --delete-collection-workers=2 \\
 23   --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\
 24   --etcd-cafile=/etc/kubernetes/cert/ca.pem \\
 25   --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\
 26   --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\
 27   --etcd-servers=${ETCD_ENDPOINTS} \\
 28   --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\
 29   --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\
 30   --audit-dynamic-configuration \\
 31   --audit-log-maxage=30 \\
 32   --audit-log-maxbackup=3 \\
 33   --audit-log-maxsize=100 \\
 34   --audit-log-truncate-enabled=true \\
 35   --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\
 36   --audit-policy-file=/etc/kubernetes/audit-policy.yaml \\
 37   --profiling \\
 38   --anonymous-auth=false \\
 39   --client-ca-file=/etc/kubernetes/cert/ca.pem \\
 40   --enable-bootstrap-token-auth=true \\
 41   --requestheader-allowed-names="system:metrics-server" \\
 42   --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
 43   --requestheader-extra-headers-prefix=X-Remote-Extra- \\
 44   --requestheader-group-headers=X-Remote-Group \\
 45   --requestheader-username-headers=X-Remote-User \\
 46   --service-account-key-file=/etc/kubernetes/cert/ca.pem \\
 47   --authorization-mode=Node,RBAC \\
 48   --runtime-config=api/all=true \\
 49   --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction \\
 50   --allow-privileged=true \\
 51   --apiserver-count=3 \\
 52   --event-ttl=168h \\
 53   --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\
 54   --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\
 55   --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\
 56   --kubelet-https=true \\
 57   --kubelet-timeout=10s \\
 58   --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\
 59   --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\
 60   --service-cluster-ip-range=${SERVICE_CIDR} \\
 61   --service-node-port-range=${NODE_PORT_RANGE} \\
 62   --logtostderr=true \\
 63   --v=2
 64 Restart=on-failure
 65 RestartSec=10
 66 Type=notify
 67 LimitNOFILE=65536
 68 
 69 [Install]
 70 WantedBy=multi-user.target
 71 EOF

提示:本步骤操作仅需要在master01节点操作。

使用 --audit-policy-file 标志将包含策略的文件传递给 kube-apiserver。如果不设置该标志,则不记录事件。

解释:

  • --advertise-address:apiserver 对外通告的 IP(kubernetes 服务后端节点 IP);
  • --default-*-toleration-seconds:设置节点异常相关的阈值;
  • --max-*-requests-inflight:请求相关的最大阈值;
  • --etcd-*:访问 etcd 的证书和 etcd 服务器地址;
  • --experimental-encryption-provider-config:指定用于加密 etcd 中 secret 的配置;
  • --bind-address: https 监听的 IP,不能为 127.0.0.1,否则外界不能访问它的安全端口 6443;
  • --secret-port:https 监听端口;
  • --insecure-port=0:关闭监听 http 非安全端口(8080);
  • --tls-*-file:指定 apiserver 使用的证书、私钥和 CA 文件;
  • --audit-*:配置审计策略和审计日志文件相关的参数;
  • --client-ca-file:验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书;
  • --enable-bootstrap-token-auth:启用 kubelet bootstrap 的 token 认证;
  • --requestheader-*:kube-apiserver 的 aggregator layer 相关的配置参数,proxy-client & HPA 需要使用;
  • --requestheader-client-ca-file:用于签名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的证书;在启用了 metric aggregator 时使用;
  • --requestheader-allowed-names:不能为空,值为逗号分割的 --proxy-client-cert-file 证书的 CN 名称,这里设置为 "aggregator";
  • --service-account-key-file:签名 ServiceAccount Token 的公钥文件,kube-controller-manager 的 --service-account-private-key-file 指定私钥文件,两者配对使用;
  • --runtime-config=api/all=true: 启用所有版本的 APIs,如 autoscaling/v2alpha1;
  • --authorization-mode=Node,RBAC、--anonymous-auth=false: 开启 Node 和 RBAC 授权模式,拒绝未授权的请求;
  • --enable-admission-plugins:启用一些默认关闭的 plugins;
  • --allow-privileged:运行执行 privileged 权限的容器;
  • --apiserver-count=3:指定 apiserver 实例的数量;
  • --event-ttl:指定 events 的保存时间;
  • --kubelet-*:如果指定,则使用 https 访问 kubelet APIs;需要为证书对应的用户(上面 kubernetes*.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则,否则访问 kubelet API 时提示未授权;
  • --proxy-client-*:apiserver 访问 metrics-server 使用的证书;
  • --service-cluster-ip-range: 指定 Service Cluster IP 地址段;
  • --service-node-port-range: 指定 NodePort 的端口范围。

提示:如果 kube-apiserver 机器没有运行 kube-proxy,则还需要添加 --enable-aggregator-routing=true 参数。

注意:requestheader-client-ca-file 指定的 CA 证书,必须具有 client auth and server auth;

如果 --requestheader-allowed-names 为空,或者 --proxy-client-cert-file 证书的 CN 名称不在 allowed-names 中,则后续查看 node 或 pods 的 metrics 失败,提示:

  1 [root@master01 ~]# kubectl top nodes
  2 Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "aggregator" cannot list resource "nodes" in API group "metrics.k8s.io" at the cluster scope

5.2 分发systemd

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for (( i=0; i < 3; i++ ))
  4   do
  5     sed -e "s/##MASTER_NAME##/${MASTER_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${MASTER_IPS[i]}.service
  6   done
  7 [root@master01 work]# ls kube-apiserver*.service			#替换相应的IP
  8 
  9 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
 10   do
 11     echo ">>> ${master_ip}"
 12     scp kube-apiserver-${master_ip}.service root@${master_ip}:/etc/systemd/system/kube-apiserver.service
 13   done						#分发systemd

提示:本步骤操作仅需要在master01节点操作。

六 启动并验证

6.1 启动kube-apiserver服务

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "mkdir -p ${K8S_DIR}/kube-apiserver"
  7     ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"
  8   done

提示:本步骤操作仅需要在master01节点操作。

6.2 检查kube-apiserver服务

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "systemctl status kube-apiserver |grep 'Active:'"
  7   done

提示:本步骤操作仅需要在master01节点操作。

clipboard

6.3 查看kube-apiserver写入etcd的数据

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# ETCDCTL_API=3 etcdctl \
  4     --endpoints=${ETCD_ENDPOINTS} \
  5     --cacert=/opt/k8s/work/ca.pem \
  6     --cert=/opt/k8s/work/etcd.pem \
  7     --key=/opt/k8s/work/etcd-key.pem \
  8     get /registry/ --prefix --keys-only

提示:本步骤操作仅需要在master01节点操作。

6.4 检查集群信息

  1 [root@master01 work]# kubectl cluster-info
  2 [root@master01 work]# kubectl get all --all-namespaces
  3 [root@master01 work]# kubectl get componentstatuses
  4 [root@master01 work]# netstat -lnpt|grep 6443		#检查 kube-apiserver 监听的端口

clipboard

提示:

如果执行 kubectl 命令式时输出如下错误信息,则说明使用的 ~/.kube/config 文件不对,检查《009.Kubernetes二进制master节点部署kubectl》是否正确完成,然后再执行该命令:

The connection to the server localhost:8080 was refused - did you specify the right host or port?

执行 kubectl get componentstatuses 命令时,apiserver 默认向 127.0.0.1 发送请求。目前controller-manager和scheduler还未部署,因此显示Unhealthy;

6443: 接收 https 请求的安全端口,对所有请求做认证和授权;

16443:Nginx反向代理监听端口;

由于关闭了非安全端口,故没有监听 8080。

6.5 授权

授予 kube-apiserver 访问 kubelet API 的权限。

在执行 kubectl exec、run、logs 等命令时,apiserver 会将请求转发到 kubelet 的 https 端口。本实验定义 RBAC 规则,授权 apiserver 使用的证书(kubernetes.pem)用户名(CN:kuberntes)访问 kubelet API 的权限:

  1 [root@master01 ~]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

提示:本步骤操作仅需要在master01节点操作。

posted @ 2019-11-16 21:03  木二  阅读(1213)  评论(0编辑  收藏  举报