k8s服务端二进制部署-kube-apiserver
本文章是 k8s二进制高可用集群部署 的分支。详细步骤请参考目录。
二进制下载地址
压缩包下载:https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG
注:打开链接你会发现里面有很多包,下载一个Server包就够了,里面包含了Server和Client以及Node二进制文件。
单个下载:https://www.downloadkubernetes.com/
具体下载那个,自由发挥。
1.使用自签CA签发kube-apiserver证书
1.申请自签CA【略,直接复用etcd流程中的CA,如果k8s要使用不同的CA请参考etcd3.5.0版本集群部署流程中的自签CA流程】
2.签发
# 创建证书请求文件 cat > kube-apiserver-csr.json << EOF { "CN": "kubernetes", "hosts": [ "10.0.0.1", "127.0.0.1", "192.168.64.130", "192.168.64.131", "192.168.64.132", "192.168.64.133", "192.168.64.134", "10.255.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Tianjin", "ST": "Tianjin", "O": "k8s", "OU": "System" } ] } EOF # 生成证书 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
mkdir -p /opt/kubernetes/{cfg,ssl,logs} tar -zxvf kubernetes-server-linux-amd64.tar.gz cd kubernetes/server/bin cp kube-apiserver kubectl kube-scheduler kube-controller-manager /usr/local/bin
3.将CA以及kube-apiserver HTTPS证书移动到 /opt/kubernetes/ssl
4.创建token文件
#生成 cat > /opt/kubernetes/cfg/token.csv << EOF $(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF #格式:token,用户名,UID,用户组。Token也可自行生成替换 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' #写死的 cat > /opt/kubernetes/cfg/token.csv << EOF c47ffb939f5ca36231d9e3121a232940,kubelet-bootstrap,10001,"system:node-bootstrapper" EOF
5.创建配置文件[https://kubernetes.io/zh-cn/docs/reference/command-line-tools-reference/kube-apiserver/]
#很多都被移除了,看下面的配置
cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF KUBE_APISERVER_OPTS=" \\ --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\ --anonymous-auth=false \\ #不允许匿名请求到 kubelet 服务(默认 true ) --bind-address=192.168.64.130 \\ #当前机器的ip地址 --secure-port=6443 \\ --advertise-address=192.168.64.130 \\ #当前机器的ip地址 --insecure-port=0 \\ #非安全端口 --authorization-mode=RBAC,Node \\ --runtime-config=api/all=true \\ #开启/关闭某个指定的 API 版本 --enable-bootstrap-token-auth=true \\ --service-cluster-ip-range=10.0.0.0/24 \\ --token-auth-file=/opt/kubernetes/cfg/token.csv \\ --service-node-port-range=30000-50000 \\ --kubelet-client-certificate=/opt/kubernetes/ssl/kube-apiserver.pem \\ --kubelet-client-key=/opt/kubernetes/ssl/kube-apiserver-key.pem \\ --tls-cert-file=/opt/kubernetes/ssl/kube-apiserver.pem \\ --tls-private-key-file=/opt/kubernetes/ssl/kube-apiserver-key.pem \\ --client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --service-account-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\ # 1.20以上版本必须有此参数 --service-account-issuer=https://kubernetes.default.svc.cluster.local \\ # 1.20以上版本必须有此参数 --etcd-servers=https://192.168.64.130:2379,https://192.168.64.131:2379,https://192.168.64.132:2379,https://192.168.64.133:2379,https://192.168.64.134:2379 \\ --etcd-cafile=/opt/etcd/ssl/ca.pem \\ --etcd-certfile=/opt/etcd/ssl/server.pem \\ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \\ --allow-privileged=true \\ --audit-log-maxage=30 \\ --audit-log-maxbackup=3 \\ --audit-log-maxsize=100 \\ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log \\ --logtostderr=false \\ --v=4 \\ --log-dir=/opt/kubernetes/logs \\ --enable-swagger-ui=true \\ --apiserver-count=3 \\ --event-ttl=1h \\ --alsologtostderr=true \\ --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\ --proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\ --requestheader-allowed-names=kubernetes \\ --requestheader-extra-headers-prefix=X-Remote-Extra- \\ --requestheader-group-headers=X-Remote-Group \\ --requestheader-username-headers=X-Remote-User \\ --enable-aggregator-routing=true" EOF #上面两个 \\ 第一个是转义符,第二个是换行符,使用转义符是为了使用EOF保留换行符。 -logtostderr: 启用日志 -v:日志等级 -log-dir: 日志目录 -etcd-servers: etcd集群地址 -bind-address: 监听地址 -serure-port: https安全端口 -advertise-address: 集群通告地址 -addlow-privileged: 启用授权 -service-cluster-ip-range: Service虚拟ip地址段 -enable-admission-plugins: 准入控制模块 -authorization-mode: 认证授权,启用RBAC授权和节点自管理 -enable-bootstrap-token-auth: 启用TLS bootstrap机制 -token-auth-file: bootstrap token 文件 -service-node-port-range: Service nodeport类型默认分配端口范围 -kubelet-client-xxx: apiserver访问kubelet客户端证书 -tls-xxx-file: apiserver https证书 -etcd-xxxfile: 连接etcd集群证书 -audit-log-xxx: 审计日志 1.20版本必须加的参数:--service-account-issuer,--service-account-signing-key-file --etcd-xxxfile:连接Etcd集群证书 --audit-log-xxx:审计日志 启动聚合层相关配置:--requestheader-client-ca-file,--proxy-client-cert-file,--proxy-client-key-file,--requestheader-allowed-names,--requestheader-extra-headers-prefix,--requestheader-group-headers,--requestheader-username-headers,--enable-aggregator-routing
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
Flag --enable-swagger-ui has been deprecated, swagger 1.2 support has been removed
Flag --apiserver-count has been deprecated, apiserver-count is deprecated and will be removed in a future version.
cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ --anonymous-auth=false \ --bind-address=192.168.64.130 \ --secure-port=6443 \ --advertise-address=192.168.64.130 \ --authorization-mode=RBAC,Node \ --runtime-config=api/all=true \ --enable-bootstrap-token-auth=true \ --service-cluster-ip-range=10.0.0.0/24 \ --token-auth-file=/opt/kubernetes/cfg/token.csv \ --service-node-port-range=30000-50000 \ --kubelet-client-certificate=/opt/kubernetes/ssl/kube-apiserver.pem \ --kubelet-client-key=/opt/kubernetes/ssl/kube-apiserver-key.pem \ --tls-cert-file=/opt/kubernetes/ssl/kube-apiserver.pem \ --tls-private-key-file=/opt/kubernetes/ssl/kube-apiserver-key.pem \ --client-ca-file=/opt/kubernetes/ssl/ca.pem \ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \ --service-account-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --etcd-servers=https://192.168.64.130:2379,https://192.168.64.131:2379,https://192.168.64.132:2379,https://192.168.64.133:2379,https://192.168.64.134:2379 \ --etcd-cafile=/opt/etcd/ssl/ca.pem \ --etcd-certfile=/opt/etcd/ssl/server.pem \ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \ --allow-privileged=true \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log \ --v=4 \ --event-ttl=1h \ --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \ --proxy-client-cert-file=/opt/kubernetes/ssl/kube-apiserver.pem \ --proxy-client-key-file=/opt/kubernetes/ssl/kube-apiserver-key.pem \ --requestheader-allowed-names=kubernetes \ --requestheader-extra-headers-prefix=X-Remote-Extra- \ --requestheader-group-headers=X-Remote-Group \ --requestheader-username-headers=X-Remote-User \ --enable-aggregator-routing=true" EOF
6.启用 TLS Bootstrapping 机制【-enable-bootstrap-token-auth配置的说明,阅读】
TLS Bootstraping:Master apiserver启用TLS认证后,Node节点kubelet和kube-proxy要与kube-apiserver进行通信,必须使用CA签发的有效证书才可以,当Node节点很多时,这种客户端证书颁发需要大量工作,同样也会增加集群扩展复杂度。为了简化流程,Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书,kubelet会以一个低权限用户自动向apiserver申请证书,kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式,目前主要用于kubelet,kube-proxy还是由我们统一颁发一个证书。
7.使用systemd管理apiserver
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf ExecStart=kube-apiserver \$KUBE_APISERVER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF
8.启动并设置开机启动
systemctl daemon-reload systemctl start kube-apiserver systemctl enable kube-apiserver