2-3.3 部署kube-apiserver集群

3.3.1 集群规划

主机名 角色 IP地址
pg60-21.k8s.host.com kube-apiserver 10.20.60.21
pg60-22.k8s.host.com kube-apiserver 10.20.60.22
pg60-23.k8s.host.com kube-apiserver 10.20.60.23
注意：通过nginx upstream反向代理两个 kube-apiserver，实现高可用。nginx配置将在下节讲解。

3.3.2 下载并安装kubernetes软件

在 pg60-21.k8s.host.com 虚机实例上操作，另外一台运算节点安装部署方法类似。

shell> wget http://dlsw.91donkey.com/software/source/k8s/v1.18.2/kubernetes-server-linux-amd64.tar.gz
shell> tar zxf kubernetes-server-linux-amd64.tar.gz -C /opt/
shell> mv /opt/kubernetes /opt/kubernetes-v1.18.2-linux-amd64
shell> ln -s /opt/kubernetes-v1.18.2-linux-amd64 /opt/kubernetes
shell> mkdir /opt/kubernetes/server/sbin
shell> mkdir -p /export/kubernetes/logs/kube-apiserver

3.3.3 签发kube-apiserver证书

在 pg60-200.k8s.host.com 虚机实例上操作

  • 创建 kubernetes-master 证书和私钥
shell> cat > apiserver-csr.json << EOF
{
    "CN": "apiserver",
    "hosts": [
        "127.0.0.1",
        "192.168.0.1",
        "10.20.60.10",
        "10.20.60.11",
        "10.20.60.12",
        "10.20.60.21",
        "10.20.60.22",
        "10.20.60.23",
        "10.20.60.31",
        "10.20.60.32",
        "10.20.60.33",
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local."
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "91donkey",
            "OU": "ops"
        }
    ]
}
EOF
  • 生成kube-apiserver证书和私钥
shell> cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json | cfssl-json -bare apiserver
2020/05/09 12:57:50 [INFO] generate received request
2020/05/09 12:57:50 [INFO] received CSR
2020/05/09 12:57:50 [INFO] generating key: rsa-2048
2020/05/09 12:57:51 [INFO] encoded CSR
2020/05/09 12:57:51 [INFO] signed certificate with serial number 704123904604325511866815694194275404583444068714
2020/05/09 12:57:51 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
  • 将生成的证书和私钥文件拷贝到所有 master 节点：
shell> scp apiserver*.pem root@${master_ip}:/etc/kubernetes/pki/

3.3.4 创建加密和审计配置文件

在 pg60-21.k8s.host.com 虚机实例上操作

  • 创建加密配置文件
shell> cat > /etc/kubernetes/encryption-config.yaml << EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: $(head -c 32 /dev/urandom | base64)
      - identity: {}
EOF
  • 创建审计策略文件
# https://kubernetes.io/zh/docs/tasks/debug-application-cluster/audit/

shell> cat > /etc/kubernetes/audit-policy.yaml << "EOF"
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # The following requests were manually identified as high-volume and low-risk, so drop them.
  - level: None
    resources:
      - group: ""
        resources:
          - endpoints
          - services
          - services/status
    users:
      - 'system:kube-proxy'
    verbs:
      - watch

  - level: None
    resources:
      - group: ""
        resources:
          - nodes
          - nodes/status
    userGroups:
      - 'system:nodes'
    verbs:
      - get

  - level: None
    namespaces:
      - kube-system
    resources:
      - group: ""
        resources:
          - endpoints
    users:
      - 'system:kube-controller-manager'
      - 'system:kube-scheduler'
      - 'system:serviceaccount:kube-system:endpoint-controller'
    verbs:
      - get
      - update

  - level: None
    resources:
      - group: ""
        resources:
          - namespaces
          - namespaces/status
          - namespaces/finalize
    users:
      - 'system:apiserver'
    verbs:
      - get

  # Don't log HPA fetching metrics.
  - level: None
    resources:
      - group: metrics.k8s.io
    users:
      - 'system:kube-controller-manager'
    verbs:
      - get
      - list

  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - '/healthz*'
      - /version
      - '/swagger*'

  # Don't log events requests.
  - level: None
    resources:
      - group: ""
        resources:
          - events

  # node and pod status calls from nodes are high-volume and can be large, don't log responses
  # for expected updates from nodes
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    users:
      - kubelet
      - 'system:node-problem-detector'
      - 'system:serviceaccount:kube-system:node-problem-detector'
    verbs:
      - update
      - patch

  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    userGroups:
      - 'system:nodes'
    verbs:
      - update
      - patch

  # deletecollection calls can be large, don't log responses for expected namespace deletions
  - level: Request
    omitStages:
      - RequestReceived
    users:
      - 'system:serviceaccount:kube-system:namespace-controller'
    verbs:
      - deletecollection

  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - secrets
          - configmaps
      - group: authentication.k8s.io
        resources:
          - tokenreviews
  # Get repsonses can be large; skip them.
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
    verbs:
      - get
      - list
      - watch

  # Default level for known APIs
  - level: RequestResponse
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
      
  # Default level for all other requests.
  - level: Metadata
    omitStages:
      - RequestReceived
EOF
  • 分发加密配置和审计策略文件到 master 节点的 /etc/kubernetes 目录下：
shell> scp encryption-config.yaml audit-policy.yaml root@${master_ip}:/etc/kubernetes/

3.3.5 创建 kube-apiserver systemd unit 模板文件

在 pg60-21.k8s.host.com 虚机实例上操作

shell> cat > kube-apiserver.service.template << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
ExecStart=/opt/kubernetes/server/bin/kube-apiserver \\
  --advertise-address=10.20.60.21 \\
  --default-not-ready-toleration-seconds=360 \\
  --default-unreachable-toleration-seconds=360 \\
  --feature-gates=DynamicAuditing=true \\
  --max-mutating-requests-inflight=2000 \\
  --max-requests-inflight=4000 \\
  --default-watch-cache-size=200 \\
  --delete-collection-workers=2 \\
  --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\
  --etcd-cafile=/etc/kubernetes/pki/ca.pem \\
  --etcd-certfile=/etc/kubernetes/pki/admin.pem \\
  --etcd-keyfile=/etc/kubernetes/pki/admin-key.pem \\
  --etcd-servers=https://10.20.60.21:2379,https://10.20.60.22:2379,https://10.20.60.23:2379 \\
  --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \\
  --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \\
  --audit-dynamic-configuration \\
  --audit-log-maxage=15 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-truncate-enabled \\
  --audit-log-path=/export/kubernetes/logs/kube-apiserver/audit.log \\
  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \\
  --profiling \\
  --anonymous-auth=false \\
  --client-ca-file=/etc/kubernetes/pki/ca.pem \\
  --enable-bootstrap-token-auth \\
  --requestheader-allowed-names="aggregator" \\
  --requestheader-client-ca-file=/etc/kubernetes/pki/ca.pem \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User \\
  --service-account-key-file=/etc/kubernetes/pki/ca-key.pem \\
  --authorization-mode=Node,RBAC \\
  --runtime-config=api/all=true \\
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \\
  --allow-privileged=true \\
  --apiserver-count=3 \\
  --event-ttl=168h \\
  --kubelet-certificate-authority=/etc/kubernetes/pki/ca.pem \\
  --kubelet-client-certificate=/etc/kubernetes/pki/admin.pem \\
  --kubelet-client-key=/etc/kubernetes/pki/admin-key.pem \\
  --kubelet-https=true \\
  --kubelet-timeout=10s \\
  --proxy-client-cert-file=/etc/kubernetes/pki/proxy-client.pem \\
  --proxy-client-key-file=/etc/kubernetes/pki/proxy-client-key.pem \\
  --service-cluster-ip-range=10.254.0.0/16 \\
  --service-node-port-range=30000-32767 \\
  --logtostderr=true \\
  --log-dir=/export/kubernetes/logs/kube-apiserver \\
  --v=2
Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

shell> cp kube-apiserver.service.template /etc/systemd/system/kube-apiserver.service
shell> systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver

3.3.6 检查集群状态

posted @ 2020-06-21 09:06  龍龍小宝  阅读(127)  评论(0)    收藏  举报