2-3.3 部署kube-apiserver集群
3.3.1 集群规划#
| 主机名 | 角色 | IP地址 |
|---|---|---|
| pg60-21.k8s.host.com | kube-apiserver | 10.20.60.21 |
| pg60-22.k8s.host.com | kube-apiserver | 10.20.60.22 |
| pg60-23.k8s.host.com | kube-apiserver | 10.20.60.23 |
注意：通过nginx upstream反向代理两个 kube-apiserver，实现高可用。nginx配置将在下节讲解。 |
3.3.2 下载并安装kubernetes软件#
在 pg60-21.k8s.host.com 虚机实例上操作，另外一台运算节点安装部署方法类似。
shell> wget http://dlsw.91donkey.com/software/source/k8s/v1.18.2/kubernetes-server-linux-amd64.tar.gz
shell> tar zxf kubernetes-server-linux-amd64.tar.gz -C /opt/
shell> mv /opt/kubernetes /opt/kubernetes-v1.18.2-linux-amd64
shell> ln -s /opt/kubernetes-v1.18.2-linux-amd64 /opt/kubernetes
shell> mkdir /opt/kubernetes/server/sbin
shell> mkdir -p /export/kubernetes/logs/kube-apiserver
3.3.3 签发kube-apiserver证书#
在 pg60-200.k8s.host.com 虚机实例上操作
- 创建 kubernetes-master 证书和私钥
shell> cat > apiserver-csr.json << EOF
{
"CN": "apiserver",
"hosts": [
"127.0.0.1",
"192.168.0.1",
"10.20.60.10",
"10.20.60.11",
"10.20.60.12",
"10.20.60.21",
"10.20.60.22",
"10.20.60.23",
"10.20.60.31",
"10.20.60.32",
"10.20.60.33",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local."
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "91donkey",
"OU": "ops"
}
]
}
EOF
- 生成kube-apiserver证书和私钥
shell> cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json | cfssl-json -bare apiserver
2020/05/09 12:57:50 [INFO] generate received request
2020/05/09 12:57:50 [INFO] received CSR
2020/05/09 12:57:50 [INFO] generating key: rsa-2048
2020/05/09 12:57:51 [INFO] encoded CSR
2020/05/09 12:57:51 [INFO] signed certificate with serial number 704123904604325511866815694194275404583444068714
2020/05/09 12:57:51 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
- 将生成的证书和私钥文件拷贝到所有 master 节点：
shell> scp apiserver*.pem root@${master_ip}:/etc/kubernetes/pki/
3.3.4 创建加密和审计配置文件#
在 pg60-21.k8s.host.com 虚机实例上操作
- 创建加密配置文件
shell> cat > /etc/kubernetes/encryption-config.yaml << EOF
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: $(head -c 32 /dev/urandom | base64)
- identity: {}
EOF
- 创建审计策略文件
# https://kubernetes.io/zh/docs/tasks/debug-application-cluster/audit/
shell> cat > /etc/kubernetes/audit-policy.yaml << "EOF"
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
# The following requests were manually identified as high-volume and low-risk, so drop them.
- level: None
resources:
- group: ""
resources:
- endpoints
- services
- services/status
users:
- 'system:kube-proxy'
verbs:
- watch
- level: None
resources:
- group: ""
resources:
- nodes
- nodes/status
userGroups:
- 'system:nodes'
verbs:
- get
- level: None
namespaces:
- kube-system
resources:
- group: ""
resources:
- endpoints
users:
- 'system:kube-controller-manager'
- 'system:kube-scheduler'
- 'system:serviceaccount:kube-system:endpoint-controller'
verbs:
- get
- update
- level: None
resources:
- group: ""
resources:
- namespaces
- namespaces/status
- namespaces/finalize
users:
- 'system:apiserver'
verbs:
- get
# Don't log HPA fetching metrics.
- level: None
resources:
- group: metrics.k8s.io
users:
- 'system:kube-controller-manager'
verbs:
- get
- list
# Don't log these read-only URLs.
- level: None
nonResourceURLs:
- '/healthz*'
- /version
- '/swagger*'
# Don't log events requests.
- level: None
resources:
- group: ""
resources:
- events
# node and pod status calls from nodes are high-volume and can be large, don't log responses
# for expected updates from nodes
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- nodes/status
- pods/status
users:
- kubelet
- 'system:node-problem-detector'
- 'system:serviceaccount:kube-system:node-problem-detector'
verbs:
- update
- patch
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- nodes/status
- pods/status
userGroups:
- 'system:nodes'
verbs:
- update
- patch
# deletecollection calls can be large, don't log responses for expected namespace deletions
- level: Request
omitStages:
- RequestReceived
users:
- 'system:serviceaccount:kube-system:namespace-controller'
verbs:
- deletecollection
# Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
# so only log at the Metadata level.
- level: Metadata
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- secrets
- configmaps
- group: authentication.k8s.io
resources:
- tokenreviews
# Get repsonses can be large; skip them.
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
- group: admissionregistration.k8s.io
- group: apiextensions.k8s.io
- group: apiregistration.k8s.io
- group: apps
- group: authentication.k8s.io
- group: authorization.k8s.io
- group: autoscaling
- group: batch
- group: certificates.k8s.io
- group: extensions
- group: metrics.k8s.io
- group: networking.k8s.io
- group: policy
- group: rbac.authorization.k8s.io
- group: scheduling.k8s.io
- group: settings.k8s.io
- group: storage.k8s.io
verbs:
- get
- list
- watch
# Default level for known APIs
- level: RequestResponse
omitStages:
- RequestReceived
resources:
- group: ""
- group: admissionregistration.k8s.io
- group: apiextensions.k8s.io
- group: apiregistration.k8s.io
- group: apps
- group: authentication.k8s.io
- group: authorization.k8s.io
- group: autoscaling
- group: batch
- group: certificates.k8s.io
- group: extensions
- group: metrics.k8s.io
- group: networking.k8s.io
- group: policy
- group: rbac.authorization.k8s.io
- group: scheduling.k8s.io
- group: settings.k8s.io
- group: storage.k8s.io
# Default level for all other requests.
- level: Metadata
omitStages:
- RequestReceived
EOF
- 分发加密配置和审计策略文件到 master 节点的 /etc/kubernetes 目录下：
shell> scp encryption-config.yaml audit-policy.yaml root@${master_ip}:/etc/kubernetes/
3.3.5 创建 kube-apiserver systemd unit 模板文件#
在 pg60-21.k8s.host.com 虚机实例上操作
shell> cat > kube-apiserver.service.template << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
ExecStart=/opt/kubernetes/server/bin/kube-apiserver \\
--advertise-address=10.20.60.21 \\
--default-not-ready-toleration-seconds=360 \\
--default-unreachable-toleration-seconds=360 \\
--feature-gates=DynamicAuditing=true \\
--max-mutating-requests-inflight=2000 \\
--max-requests-inflight=4000 \\
--default-watch-cache-size=200 \\
--delete-collection-workers=2 \\
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\
--etcd-cafile=/etc/kubernetes/pki/ca.pem \\
--etcd-certfile=/etc/kubernetes/pki/admin.pem \\
--etcd-keyfile=/etc/kubernetes/pki/admin-key.pem \\
--etcd-servers=https://10.20.60.21:2379,https://10.20.60.22:2379,https://10.20.60.23:2379 \\
--tls-cert-file=/etc/kubernetes/pki/apiserver.pem \\
--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \\
--audit-dynamic-configuration \\
--audit-log-maxage=15 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-truncate-enabled \\
--audit-log-path=/export/kubernetes/logs/kube-apiserver/audit.log \\
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \\
--profiling \\
--anonymous-auth=false \\
--client-ca-file=/etc/kubernetes/pki/ca.pem \\
--enable-bootstrap-token-auth \\
--requestheader-allowed-names="aggregator" \\
--requestheader-client-ca-file=/etc/kubernetes/pki/ca.pem \\
--requestheader-extra-headers-prefix="X-Remote-Extra-" \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--service-account-key-file=/etc/kubernetes/pki/ca-key.pem \\
--authorization-mode=Node,RBAC \\
--runtime-config=api/all=true \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \\
--allow-privileged=true \\
--apiserver-count=3 \\
--event-ttl=168h \\
--kubelet-certificate-authority=/etc/kubernetes/pki/ca.pem \\
--kubelet-client-certificate=/etc/kubernetes/pki/admin.pem \\
--kubelet-client-key=/etc/kubernetes/pki/admin-key.pem \\
--kubelet-https=true \\
--kubelet-timeout=10s \\
--proxy-client-cert-file=/etc/kubernetes/pki/proxy-client.pem \\
--proxy-client-key-file=/etc/kubernetes/pki/proxy-client-key.pem \\
--service-cluster-ip-range=10.254.0.0/16 \\
--service-node-port-range=30000-32767 \\
--logtostderr=true \\
--log-dir=/export/kubernetes/logs/kube-apiserver \\
--v=2
Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
shell> cp kube-apiserver.service.template /etc/systemd/system/kube-apiserver.service
shell> systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver
3.3.6 检查集群状态#
分类:
Kubernetes
【推荐】国内首个AI IDE，深度理解中文开发场景，立即下载体验Trae
【推荐】编程新体验，更懂你的AI，立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包，你的智能百科全书，全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell：AI 加持，快人一步
· AI与.NET技术实操系列（二）：开始使用ML.NET
· 记一次.NET内存居高不下排查解决与启示
· 探究高空视频全景AR技术的实现原理
· 理解Rust引用及其生命周期标识（上）
· 浏览器原生「磁吸」效果！Anchor Positioning 锚点定位神器解析
· DeepSeek 开源周回顾「GitHub 热点速览」
· 物流快递公司核心技术能力-地址解析分单基础技术分享
· .NET 10首个预览版发布：重大改进与新特性概览！
· AI与.NET技术实操系列（二）：开始使用ML.NET
· 单线程的Redis速度为什么快？