k8s服务端二进制部署-kube-apiserver

本文章是 k8s二进制高可用集群部署 的分支。详细步骤请参考目录。

二进制下载地址

  压缩包下载:https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG

  注:打开链接你会发现里面有很多包,下载一个Server包就够了,里面包含了Server和Client以及Node二进制文件。

  单个下载:https://www.downloadkubernetes.com/

具体下载那个,自由发挥。

1.使用自签CA签发kube-apiserver证书

  1.申请自签CA【略,直接复用etcd流程中的CA,如果k8s要使用不同的CA请参考etcd3.5.0版本集群部署流程中的自签CA流程】

  2.签发

# 创建证书请求文件
cat > kube-apiserver-csr.json << EOF
{
  "CN": "kubernetes",
  "hosts": [
      "10.0.0.1",
    "127.0.0.1",
    "192.168.64.130",
    "192.168.64.131",
    "192.168.64.132",
    "192.168.64.133",
    "192.168.64.134",
    "10.255.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Tianjin",
      "ST": "Tianjin",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
2.解压二进制包并移动到可执行目录
mkdir -p /opt/kubernetes/{cfg,ssl,logs} 
tar -zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver kubectl kube-scheduler kube-controller-manager /usr/local/bin

3.将CA以及kube-apiserver HTTPS证书移动到  /opt/kubernetes/ssl

4.创建token文件

#生成
cat > /opt/kubernetes/cfg/token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

#格式:token,用户名,UID,用户组。Token也可自行生成替换
head -c 16 /dev/urandom | od -An -t x | tr -d ' '

#写死的
cat > /opt/kubernetes/cfg/token.csv << EOF
c47ffb939f5ca36231d9e3121a232940,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

5.创建配置文件[https://kubernetes.io/zh-cn/docs/reference/command-line-tools-reference/kube-apiserver/]

#很多都被移除了,看下面的配置
cat
> /opt/kubernetes/cfg/kube-apiserver.conf << EOF KUBE_APISERVER_OPTS=" \\ --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\ --anonymous-auth=false \\ #不允许匿名请求到 kubelet 服务(默认 true--bind-address=192.168.64.130 \\ #当前机器的ip地址 --secure-port=6443 \\ --advertise-address=192.168.64.130 \\ #当前机器的ip地址 --insecure-port=0 \\ #非安全端口 --authorization-mode=RBAC,Node \\ --runtime-config=api/all=true \\ #开启/关闭某个指定的 API 版本 --enable-bootstrap-token-auth=true \\ --service-cluster-ip-range=10.0.0.0/24 \\ --token-auth-file=/opt/kubernetes/cfg/token.csv \\ --service-node-port-range=30000-50000 \\ --kubelet-client-certificate=/opt/kubernetes/ssl/kube-apiserver.pem \\ --kubelet-client-key=/opt/kubernetes/ssl/kube-apiserver-key.pem \\ --tls-cert-file=/opt/kubernetes/ssl/kube-apiserver.pem \\ --tls-private-key-file=/opt/kubernetes/ssl/kube-apiserver-key.pem \\ --client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --service-account-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\ # 1.20以上版本必须有此参数 --service-account-issuer=https://kubernetes.default.svc.cluster.local \\ # 1.20以上版本必须有此参数 --etcd-servers=https://192.168.64.130:2379,https://192.168.64.131:2379,https://192.168.64.132:2379,https://192.168.64.133:2379,https://192.168.64.134:2379 \\ --etcd-cafile=/opt/etcd/ssl/ca.pem \\ --etcd-certfile=/opt/etcd/ssl/server.pem \\ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \\ --allow-privileged=true \\ --audit-log-maxage=30 \\ --audit-log-maxbackup=3 \\ --audit-log-maxsize=100 \\ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log \\ --logtostderr=false \\ --v=4 \\ --log-dir=/opt/kubernetes/logs \\ --enable-swagger-ui=true \\ --apiserver-count=3 \\ --event-ttl=1h \\ --alsologtostderr=true \\ --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\ --proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\ --requestheader-allowed-names=kubernetes \\ --requestheader-extra-headers-prefix=X-Remote-Extra- \\ --requestheader-group-headers=X-Remote-Group \\ --requestheader-username-headers=X-Remote-User \\ --enable-aggregator-routing=true" EOF #上面两个 \\ 第一个是转义符,第二个是换行符,使用转义符是为了使用EOF保留换行符。 -logtostderr: 启用日志 -v:日志等级 -log-dir: 日志目录 -etcd-servers: etcd集群地址 -bind-address: 监听地址 -serure-port: https安全端口 -advertise-address: 集群通告地址 -addlow-privileged: 启用授权 -service-cluster-ip-range: Service虚拟ip地址段 -enable-admission-plugins: 准入控制模块 -authorization-mode: 认证授权,启用RBAC授权和节点自管理 -enable-bootstrap-token-auth: 启用TLS bootstrap机制 -token-auth-file: bootstrap token 文件 -service-node-port-range: Service nodeport类型默认分配端口范围 -kubelet-client-xxx: apiserver访问kubelet客户端证书 -tls-xxx-file: apiserver https证书 -etcd-xxxfile: 连接etcd集群证书 -audit-log-xxx: 审计日志 1.20版本必须加的参数:--service-account-issuer,--service-account-signing-key-file --etcd-xxxfile:连接Etcd集群证书 --audit-log-xxx:审计日志 启动聚合层相关配置:--requestheader-client-ca-file,--proxy-client-cert-file,--proxy-client-key-file,--requestheader-allowed-names,--requestheader-extra-headers-prefix,--requestheader-group-headers,--requestheader-username-headers,--enable-aggregator-routing

    Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
    Flag --enable-swagger-ui has been deprecated, swagger 1.2 support has been removed

    Flag --apiserver-count has been deprecated, apiserver-count is deprecated and will be removed in a future version.

cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
    --anonymous-auth=false \
    --bind-address=192.168.64.130 \
    --secure-port=6443 \
    --advertise-address=192.168.64.130 \
    --authorization-mode=RBAC,Node \
    --runtime-config=api/all=true \
    --enable-bootstrap-token-auth=true \
    --service-cluster-ip-range=10.0.0.0/24 \
    --token-auth-file=/opt/kubernetes/cfg/token.csv \
    --service-node-port-range=30000-50000 \
    --kubelet-client-certificate=/opt/kubernetes/ssl/kube-apiserver.pem \
    --kubelet-client-key=/opt/kubernetes/ssl/kube-apiserver-key.pem \
    --tls-cert-file=/opt/kubernetes/ssl/kube-apiserver.pem \
    --tls-private-key-file=/opt/kubernetes/ssl/kube-apiserver-key.pem \
    --client-ca-file=/opt/kubernetes/ssl/ca.pem \
    --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
    --service-account-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
    --service-account-issuer=https://kubernetes.default.svc.cluster.local \
    --etcd-servers=https://192.168.64.130:2379,https://192.168.64.131:2379,https://192.168.64.132:2379,https://192.168.64.133:2379,https://192.168.64.134:2379 \
    --etcd-cafile=/opt/etcd/ssl/ca.pem \
    --etcd-certfile=/opt/etcd/ssl/server.pem \
    --etcd-keyfile=/opt/etcd/ssl/server-key.pem \
    --allow-privileged=true \
    --audit-log-maxage=30 \
    --audit-log-maxbackup=3 \
    --audit-log-maxsize=100 \
    --audit-log-path=/opt/kubernetes/logs/k8s-audit.log \
    --v=4 \
    --event-ttl=1h \
    --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
    --proxy-client-cert-file=/opt/kubernetes/ssl/kube-apiserver.pem \
    --proxy-client-key-file=/opt/kubernetes/ssl/kube-apiserver-key.pem \
    --requestheader-allowed-names=kubernetes \
    --requestheader-extra-headers-prefix=X-Remote-Extra- \
    --requestheader-group-headers=X-Remote-Group \
    --requestheader-username-headers=X-Remote-User \
    --enable-aggregator-routing=true"
EOF

 

6.启用 TLS Bootstrapping 机制【-enable-bootstrap-token-auth配置的说明,阅读】

  TLS Bootstraping:Master apiserver启用TLS认证后,Node节点kubelet和kube-proxy要与kube-apiserver进行通信,必须使用CA签发的有效证书才可以,当Node节点很多时,这种客户端证书颁发需要大量工作,同样也会增加集群扩展复杂度。为了简化流程,Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书,kubelet会以一个低权限用户自动向apiserver申请证书,kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式,目前主要用于kubelet,kube-proxy还是由我们统一颁发一个证书。

7.使用systemd管理apiserver

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

8.启动并设置开机启动

systemctl daemon-reload
systemctl start kube-apiserver 
systemctl enable kube-apiserver

 

posted @ 2023-02-20 16:25  SpringCore  阅读(419)  评论(0编辑  收藏  举报