K8S从入门到放弃系列-(6)kubernetes集群之kube-controller-manager部署

摘要：

1、Kubernetes控制器管理器是一个守护进程它通过apiserver监视集群的共享状态，并进行更改以尝试将当前状态移向所需状态。

2、kube-controller-manager是有状态的服务，会修改集群的状态信息。如果多个master节点上的相关服务同时生效，则会有同步与一致性问题，所以多master节点中的kube-controller-manager服务只能是主备的关系，kukubernetes采用租赁锁（lease-lock）实现leader的选举，具体到kube-controller-manager，设置启动参数"--leader-elect=true"。

1）创建kube-conftroller-manager证书签名请求

1、kube-controller-mamager连接 apiserver 需要使用的证书，同时本身 10257 端口也会使用此证书

2、kube-controller-mamager与kubei-apiserver通信采用双向TLS认证 

[root@k8s-master01 ~]# vim /opt/k8s/certs/kube-controller-manager-csr.json
{
    "CN": "system:kube-controller-manager",
    "hosts": [
      "127.0.0.1",
      "10.10.0.18",
      "10.10.0.19",
      "10.10.0.20",
      "localhost"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "ShangHai",
            "L": "ShangHai",
            "O": "system:kube-controller-manager",
            "OU": "System"
        }
    ]
}

1、hosts 列表包含所有 kube-controller-manager 节点 IP;

2、CN 为 system:kube-controller-manager;O 为 system:kube-controller-manager;kube-apiserver预定义的 RBAC使用的ClusterRoleBindings system:kube-controller-manager将用户system:kube-controller-manager与ClusterRole system:kube-controller-manager绑定。

2）生成kube-controller-manager证书与私钥

[root@k8s-master01 ~]# cd /opt/k8s/certs/
[root@k8s-master01 certs]# cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
     -ca-key=/etc/kubernetes/ssl/ca-key.pem \
     -config=/opt/k8s/certs/ca-config.json \
     -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
2019/04/24 13:03:36 [INFO] generate received request
2019/04/24 13:03:36 [INFO] received CSR
2019/04/24 13:03:36 [INFO] generating key: rsa-2048
2019/04/24 13:03:36 [INFO] encoded CSR
2019/04/24 13:03:36 [INFO] signed certificate with serial number 461545639209226313174106252389263020486388400892
2019/04/24 13:03:36 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

3）查看证书

[root@k8s-master01 certs]# ll kube-controller-manager*
-rw-r--r-- 1 root root 1155 Apr 24 13:03 kube-controller-manager.csr
-rw-r--r-- 1 root root  432 Apr 24 13:00 kube-controller-manager-csr.json
-rw------- 1 root root 1679 Apr 24 13:03 kube-controller-manager-key.pem
-rw-r--r-- 1 root root 1529 Apr 24 13:03 kube-controller-manager.pem

4）分发证书

[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/kube-controller-manager-key.pem dest=/etc/kubernetes/ssl/'
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/kube-controller-manager.pem dest=/etc/kubernetes/ssl/'

5）生成配置文件kube-controller-manager.kubeconfig

 kube-controller-manager 组件开启安全端口及RBAC认证所需配置

## 配置集群参数
### --kubeconfig：指定kubeconfig文件路径与文件名；如果不设置，默认生成在~/.kube/config文件。
### 后面需要用到此文件，所以我们把配置信息单独指向到指定文件中
[root@k8s-master01 ~]# kubectl config set-cluster kubernetes \
   --certificate-authority=/etc/kubernetes/ssl/ca.pem \
   --embed-certs=true \
   --server=https://127.0.0.1:6443 \
   --kubeconfig=kube-controller-manager.kubeconfig
Cluster "kubernetes" set.
## 配置客户端认证参数
### --server：指定api-server，若不指定，后面脚本中，可以指定master
### 认证用户为前文签名中的"system:kube-controller-manager";
[root@k8s-master01 ~]# kubectl config set-credentials system:kube-controller-manager \
   --client-certificate=/etc/kubernetes/ssl/kube-controller-manager.pem \
   --embed-certs=true \
   --client-key=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
   --kubeconfig=kube-controller-manager.kubeconfig
User "system:kube-controller-manager" set
## 配置上下文参数
[root@k8s-master01 ~]# kubectl config set-context system:kube-controller-manager@kubernetes \
   --cluster=kubernetes \
   --user=system:kube-controller-manager \
   --kubeconfig=kube-controller-manager.kubeconfig
Context "system:kube-controller-manager@kubernetes" created.
## 配置默认上下文
[root@k8s-master01 ~]# kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=kube-controller-manager.kubeconfig
Switched to context "system:kube-controller-manager@kubernetes".

## 分发生成的配置文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/root/kube-controller-manager.kubeconfig dest=/etc/kubernetes/config/'

6）编辑kube-controller-manager核心文件

controller manager 将不安全端口 10252 绑定到 127.0.0.1 确保 kuebctl get cs 有正确返回;将安全端口 10257 绑定到 0.0.0.0 公开，提供服务调用;由于controller manager开始连接apiserver的6443认证端口,所以需要 --use-service-account-credentials 选项来让 controller manager 创建单独的 service account(默认 system:kube-controller-manager 用户没有那么高权限)

[root@k8s-master01 ~]# vim /opt/k8s/cfg/kube-controller-manager.conf
###
# The following values are used to configure the kubernetes controller-manager

# defaults from config and apiserver should be adequate

# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 \
                             --authentication-kubeconfig=/etc/kubernetes/config/kube-controller-manager.kubeconfig \
                             --authorization-kubeconfig=/etc/kubernetes/config/kube-controller-manager.kubeconfig \
                             --bind-address=0.0.0.0 \
                             --cluster-name=kubernetes \
                             --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
                             --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
                             --client-ca-file=/etc/kubernetes/ssl/ca.pem \
                             --controllers=*,bootstrapsigner,tokencleaner \
                             --deployment-controller-sync-period=10s \
                             --experimental-cluster-signing-duration=87600h0m0s \
                             --enable-garbage-collector=true \
                             --kubeconfig=/etc/kubernetes/config/kube-controller-manager.kubeconfig \
                             --leader-elect=true \
                             --node-monitor-grace-period=20s \
                             --node-monitor-period=5s \
                             --port=10252 \
                             --pod-eviction-timeout=2m0s \
                             --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem \
                             --terminated-pod-gc-threshold=50 \
                             --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \
                             --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
                             --root-ca-file=/etc/kubernetes/ssl/ca.pem \
                             --secure-port=10257 \
                             --service-cluster-ip-range=10.254.0.0/16 \
                             --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
                             --use-service-account-credentials=true \
                             --v=2"
## 分发kube-controller-manager配置文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/kube-controller-manager.conf dest=/etc/kubernetes/config'

参数说明：

• address/bind-address：默认值：0.0.0.0，监听--secure-port端口的IP地址。关联的接口必须由集群的其他部分和CLI/web客户端访问。
• cluster-name：集群名称
• cluster-signing-cert-file/cluster-signing-key-file：用于集群范围认证
• controllers：启动的contrller列表，默认为”*”，启用所有的controller，但不包含” bootstrapsigner”与”tokencleaner”;
• kubeconfig：带有授权和master位置信息的kubeconfig文件路径
• leader-elect：在执行主逻辑之前，启动leader选举，并获得leader权
• service-cluster-ip-range：集群service的IP地址范围

8）启动脚本

[root@k8s-master01 ~]# vim /opt/k8s/unit/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/config/kube-controller-manager.conf
User=kube
ExecStart=/usr/local/bin/kube-controller-manager \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_MASTER \
        $KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
## 分发启动脚本
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/unit/kube-controller-manager.service dest=/usr/lib/systemd/system/'

9）启动服务

[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl daemon-reload'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl enable kube-controller-manager'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl start kube-controller-manager'

10）查看leader主机

[root@k8s-master01 ~]# kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master01_aef1b777-6658-11e9-beb0-000c295aa452","leaseDurationSeconds":15,"acquireTime":"2019-04-24T06:18:04Z","renewTime":"2019-04-24T06:20:43Z","leaderTransitions":2}'
  creationTimestamp: "2019-04-24T05:55:13Z"
  name: kube-controller-manager
  namespace: kube-system
  resourceVersion: "4733"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
  uid: 870148c4-6655-11e9-bb69-000c29180723
## 可看到当前k8s-master01为leader节点