kubernetes-v1.20.8二进制安装(三)-apiserver

API Server 提供了资源对象的唯一操作入口，其它所有组件都必须通过它提供的 API 来操作资源数据。

只有 API Server 会与 etcd 进行通信，其它模块都必须通过 API Server 访问集群状态

1. apiserver部署

1.1. 下载kubernetes二进制包

当前使用kubernetes版本为 1.20.8

获取安装包(需要FQ下载)

wget https://dl.k8s.io/v1.20.8/kubernetes-server-linux-amd64.tar.gz -P /opt/software/kubernetes/tools

解压相关文件至对应目录

cd /opt/software/kubernetes/bin
tar --strip-components=3 -xvpf  /opt/software/kubernetes/tools/kubernetes-server-linux-amd64.tar.gz kubernetes/server/bin/kubectl

cd /opt/software/kubernetes/master/bin
tar --strip-components=3 -xvpf  /opt/software/kubernetes/tools/kubernetes-server-linux-amd64.tar.gz kubernetes/server/bin/{kube-apiserver,kube-controller-manager,kubectl,kube-scheduler}

cd /opt/software/kubernetes/node/bin
tar --strip-components=3 -xvpf  /opt/software/kubernetes/tools/kubernetes-server-linux-amd64.tar.gz kubernetes/server/bin/{kubelet,kube-proxy}

--strip-components忽略目录层数,只解压出文件

1.2. 生成apiserver证书

cfssl支持SAN（Subject Alternative Name），它是X.509中定义的一个扩展，使用了SAN字段的SSL证书，可以扩展此证书支持的域名，即一个证书可以支持多个不同域名的解析，即下面的 *.k8s-host.com，有利于节点的扩展，不用和以前的部署中提前将需要的IP写入到证书hosts中。

cat > /opt/software/kubernetes/certs/kube-apiserver-csr.json<<EOF
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "*.k8s-host.com",
    "10.254.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GuangZhou",
      "L": "TainHe",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF

10.254.0.1 为 k8s内部kubernetes clusterip
生成证书

cd /opt/software/kubernetes/certs/

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssl-json -bare kube-apiserver

ll kube-apiserver*
-rw-r--r-- 1 root root 1269 Jul  2 18:45 /opt/software/kubernetes/certs/kube-apiserver.csr
-rw-r--r-- 1 root root  448 Jul  2 18:45 /opt/software/kubernetes/certs/kube-apiserver-csr.json
-rw------- 1 root root 1679 Jul  2 18:45 /opt/software/kubernetes/certs/kube-apiserver-key.pem
-rw-r--r-- 1 root root 1643 Jul  2 18:45 /opt/software/kubernetes/certs/kube-apiserver.pem

# 复制
cp kube-apiserver-key.pem kube-apiserver.pem /opt/software/kubernetes/master/certs/

1.3. 生成bootstarp的token

# 执行命令
cat > /opt/software/kubernetes/master/config/token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

# 查看
$ cat /opt/software/kubernetes/master/config/token.csv
cbd3297c1ab1e46ce5fcd0e22cce0133,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

1.4. 生成audit审计配置

cat > /opt/software/kubernetes/master/config/audit-policy.yaml<<EOF
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # The following requests were manually identified as high-volume and low-risk, so drop them.
  - level: None
    resources:
      - group: ""
        resources:
          - endpoints
          - services
          - services/status
    users:
      - 'system:kube-proxy'
    verbs:
      - watch

  - level: None
    resources:
      - group: ""
        resources:
          - nodes
          - nodes/status
    userGroups:
      - 'system:nodes'
    verbs:
      - get

  - level: None
    namespaces:
      - kube-system
    resources:
      - group: ""
        resources:
          - endpoints
    users:
      - 'system:kube-controller-manager'
      - 'system:kube-scheduler'
      - 'system:serviceaccount:kube-system:endpoint-controller'
    verbs:
      - get
      - update

  - level: None
    resources:
      - group: ""
        resources:
          - namespaces
          - namespaces/status
          - namespaces/finalize
    users:
      - 'system:apiserver'
    verbs:
      - get

  # Don't log HPA fetching metrics.
  - level: None
    resources:
      - group: metrics.k8s.io
    users:
      - 'system:kube-controller-manager'
    verbs:
      - get
      - list

  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - '/healthz*'
      - /version
      - '/swagger*'

  # Don't log events requests.
  - level: None
    resources:
      - group: ""
        resources:
          - events

  # node and pod status calls from nodes are high-volume and can be large, don't log responses
  # for expected updates from nodes
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    users:
      - kubelet
      - 'system:node-problem-detector'
      - 'system:serviceaccount:kube-system:node-problem-detector'
    verbs:
      - update
      - patch

  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    userGroups:
      - 'system:nodes'
    verbs:
      - update
      - patch

  # deletecollection calls can be large, don't log responses for expected namespace deletions
  - level: Request
    omitStages:
      - RequestReceived
    users:
      - 'system:serviceaccount:kube-system:namespace-controller'
    verbs:
      - deletecollection

  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - secrets
          - configmaps
      - group: authentication.k8s.io
        resources:
          - tokenreviews
  # Get repsonses can be large; skip them.
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
    verbs:
      - get
      - list
      - watch

  # Default level for known APIs
  - level: RequestResponse
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
      
  # Default level for all other requests.
  - level: Metadata
    omitStages:
      - RequestReceived
EOF

1.5. 编写apiserver配置文件

以systemd环境变量形式添加命令参数

\\ 第一个\为转移符，第二个在文本为换行符

--bind-address --advertise-address必须是IP地址，不能为域名形式，否则报错。

当前0.0.0.0是因为controller-manager和scheduler通过访问自身127.0.0.1。如无阿里云SLB影响，则写对应的IP地址.

配置中无需根据master节点不同进行修改,配置是一致的

cat > /opt/software/kubernetes/master/config/kube-apiserver.conf<<EOF
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
  --anonymous-auth=false \\
  --bind-address=0.0.0.0 \\
  --secure-port=6443 \\
  --advertise-address=0.0.0.0 \\
  --insecure-port=0 \\
  --authorization-mode=Node,RBAC \\
  --enable-bootstrap-token-auth \\
  --service-cluster-ip-range=10.254.0.0/16 \\
  --service-node-port-range=30000-50000 \\
  --token-auth-file=/opt/software/kubernetes/config/token.csv \\
  --tls-cert-file=/opt/software/kubernetes/certs/kube-apiserver.pem  \\
  --tls-private-key-file=/opt/software/kubernetes/certs/kube-apiserver-key.pem \\
  --client-ca-file=/opt/software/kubernetes/certs/ca.pem \\
  --kubelet-client-certificate=/opt/software/kubernetes/certs/kube-apiserver.pem \\
  --kubelet-client-key=/opt/software/kubernetes/certs/kube-apiserver-key.pem \\
  --kubelet-https=true \\
  --kubelet-timeout=10s \\
  --service-account-key-file=/opt/software/kubernetes/certs/ca-key.pem \\
  --service-account-signing-key-file=/opt/software/kubernetes/certs/ca-key.pem  \\
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
  --etcd-cafile=/opt/software/kubernetes/certs/ca.pem \\
  --etcd-certfile=/opt/software/kubernetes/certs/etcd.pem \\
  --etcd-keyfile=/opt/software/kubernetes/certs/etcd-key.pem \\
  --etcd-servers=https://etcd01.k8s-host.com:2379,https://etcd02.k8s-host.com:2379,https://etcd03.k8s-host.com:2379 \\
  --runtime-config=api/all=true \\
  --enable-swagger-ui=true \\
  --allow-privileged=true \\
  --apiserver-count=3 \\
  --audit-log-maxage=30 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-path=/opt/software/kubernetes/logs/kube-apiserver-audit.log \\
  --audit-policy-file=/opt/software/kubernetes/config/audit-policy.yaml \\
  --event-ttl=1h \\
  --alsologtostderr=true \\
  --logtostderr=false \\
  --log-dir=/opt/software/kubernetes/logs \\
  --v=2"
EOF

--advertise-address：apiserver 对外通告的 IP（kubernetes 服务后端节点 IP）；
--default-*-toleration-seconds：设置节点异常相关的阈值；
--max-*-requests-inflight：请求相关的最大阈值；
--etcd-*：访问 etcd 的证书和 etcd 服务器地址；
--bind-address： https 监听的 IP，不能为 127.0.0.1，否则外界不能访问它的安全端口 6443；
--secret-port：https 监听端口；
--insecure-port=0：关闭监听 http 非安全端口(8080)；
--tls-*-file：指定 apiserver 使用的证书、私钥和 CA 文件；
--audit-*：配置审计策略和审计日志文件相关的参数；
--client-ca-file：验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书；
--enable-bootstrap-token-auth：启用 kubelet bootstrap 的 token 认证；
--requestheader-*：kube-apiserver 的 aggregator layer 相关的配置参数，proxy-client & HPA 需要使用；
--requestheader-client-ca-file：用于签名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的证书；在启用了 metric aggregator 时使用；
--requestheader-allowed-names：不能为空，值为逗号分割的 --proxy-client-cert-file 证书的 CN 名称，这里设置为 "aggregator"；
--service-account-key-file：签名 ServiceAccount Token 的公钥文件，kube-controller-manager 的 --service-account-private-key-file 指定私钥文件，两者配对使用；
--runtime-config=api/all=true：启用所有版本的 APIs，如 autoscaling/v2alpha1；
--authorization-mode=Node,RBAC、--anonymous-auth=false：开启 Node 和 RBAC 授权模式，拒绝未授权的请求；
--enable-admission-plugins：启用一些默认关闭的 plugins；
--allow-privileged：运行执行 privileged 权限的容器；
--apiserver-count=3：指定 apiserver 实例的数量；
--event-ttl：指定 events 的保存时间；
--kubelet-*：如果指定，则使用 https 访问 kubelet APIs；需要为证书对应的用户(上面 kubernetes*.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则，否则访问 kubelet API 时提示未授权；
--proxy-client-*：apiserver 访问 metrics-server 使用的证书；
--service-cluster-ip-range：指定 Service Cluster IP 地址段；
--service-node-port-range：指定 NodePort 的端口范围；

如果 kube-apiserver 机器没有运行 kube-proxy，则还需要添加 --enable-aggregator-routing=true 参数；

关于 --requestheader-XXX 相关参数，参考：

1.6. 编写service启动文件

注意$KUBE_APISERVER_OPTS不能写成${KUBE_APISERVER_OPTS} 否则无法识别

cat >/opt/software/kubernetes/master/service/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service

[Service]
Type=notify
EnvironmentFile=-/opt/software/kubernetes/config/kube-apiserver.conf
ExecStart= /opt/software/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF

1.7. 分发

这里使用for循环目录进行拷贝操作,是因为config目录中某部分配置文件在各个节点存在差异,后续的服务如果在拷贝config目录会覆盖掉修改后的配置, 所以需要单独区分出来.

hosts=(master01 master02 master03)
domain='k8s-host.com'
config_files=('config/kube-apiserver.conf' 'config/audit-policy.yaml' 'config/token.csv')
cd /opt/software/kubernetes
for host in ${hosts[*]}
do
    scp -r master/{bin,certs,service} ${host}.${domain}:/opt/software/kubernetes/
done


# 不可重复执行,否则覆盖修改的配置
for host in ${hosts[*]}
do
	for file in ${config_files[*]}
	do
    	scp -r master/${file} ${host}.${domain}:/opt/software/kubernetes/${file}
    done
done

1.8. 节点启动

将kube-apiserver.service软链接至/usr/lib/systemd/system/中, 方便配置文件的维护管理

hosts=(master01 master02 master03)
domain='k8s-host.com'
for host in ${hosts[*]}
do
	ssh root@${host}.${domain} "ln -s /opt/software/kubernetes/service/kube-apiserver.service /usr/lib/systemd/system/kube-apiserver.service"
done

启动

hosts=(master01 master02 master03)
domain='k8s-host.com'
for host in ${hosts[*]}
do
    ssh root@${host}.${domain} "systemctl daemon-reload" 
    ssh root@${host}.${domain} "systemctl  enable kube-apiserver --now"
done

1.9. 验证集群信息

cd /opt/software/kubernetes/certs/

apiserver_cluster=(master01 master02 master03)
domain_name="k8s-host.com"

for i in ${apiserver_cluster[*]};
do
  apiserver_url="${i}.${domain_name}:6443"
  result=$(curl -s --cacert ./ca.pem --cert ./kube-apiserver.pem  --key ./kube-apiserver-key.pem https://${apiserver_url}/healthz)
  echo  "${apiserver_url}: ${result}"
done

正常输出

master01.k8s-host.com:6443: ok
master02.k8s-host.comt:6443: ok
master03.k8s-host.com:6443: ok

2. 问题

(待补充)

参考链接

kubernetes-v1.20.8 githup

kubernetes-v1.20官方文档

posted @ 2021-09-01 15:43 风吹蛋生丶阅读(261) 评论(0) 编辑收藏举报

刷新页面返回顶部

风吹蛋生丶