1、kube-apiserver、kube-controller-manager启动文件里面的公钥文件、私钥文件竟然是一样的，还怎么配对，这是源文档的大坑
查看源文档kube-apiserver启动文件中公钥
cat etc/systemd/system/kube-apiserver.service
--service-account-key-file=/etc/kubernetes/cert/ca-key.pem
查看源文档kube-controller-manager启动文件中私钥
cat /etc/systemd/system/kube-controller-manager.service
--service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem
发现公钥和私钥都是用的一个私钥，这是不可能配对的，而且原文中已经说明需要配对，说明如下：
原版文件在apiserver启动文字中已经说明 >
--service-account-key-file：签名 ServiceAccount Token 的公钥文件，kube-controller-manager 的 --service-account-private-key-file 指定私钥文件，两者配对使用；
但是原版中apiserver、controller-manager这两个启动文字里面用的私有、公钥都是一个，所以怀疑出错，
按照另一篇对照修改的：https://www.cnblogs.com/effortsing/p/10312081.html，需要修改如下：
生成 service account key
cd /etc/kubernetes/
openssl genrsa -out /etc/kubernetes/sa.key 2048
openssl rsa -in /etc/kubernetes/cert/sa.key -pubout -out /etc/kubernetes/cert/sa.pub
ls /etc/kubernetes/pki/sa.*
cd $HOME
分发service account key到所有master节点
subprocess.call(["ansible k8s -m copy -a 'src=/etc/kubernetes/sa.key dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
subprocess.call(["ansible k8s -m copy -a 'src=/etc/kubernetes/sa.pub dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
修改kube-apiserver启动文件中公钥为sa.pub
cat etc/systemd/system/kube-apiserver.service
--service-account-private-key-file=/etc/kubernetes/cert/sa.pub
修改kube-controller-manager启动文件中私钥为sa.key
cat /etc/systemd/system/kube-controller-manager.service
--service-account-private-key-file=/etc/kubernetes/cert/sa.key
2、源文档kube-controller-manager启动文件里面少两个致命参数，导致flannel启动失败，缺少 --allocate-node-cidrs=true --cluster-cidr=172.30.0.0/16
flannel启动失败，报错如下
Error registering network: failed to acquire lease: node "test4" pod cidr not assigned
查看pod
[root@test4 profile]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-gzvrh 0/1 Error 0 <invalid>
通过docker查看flannel日志
[root@test4 profile]# docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f7be3ebe77fd b949a39093d6 "/opt/bin/flanneld -…" 1 second ago Created k8s_kube-flannel_kube-flannel-ds-7cqww_kube-system_26fab004-2b88-11e9-9085-000c2935f634_0
[root@test4 profile]# docker logs f7be3ebe77fd
I0208 09:58:34.068723 1 main.go:488] Using interface with name ens33 and address 192.168.0.94
I0208 09:58:34.069094 1 main.go:505] Defaulting external address to interface address (192.168.0.94)
I0208 09:58:34.376952 1 kube.go:131] Waiting 10m0s for node controller to sync
I0208 09:58:34.466001 1 kube.go:294] Starting kube subnet manager
I0208 09:58:35.481478 1 kube.go:138] Node controller sync successful
I0208 09:58:35.481666 1 main.go:235] Created subnet manager: Kubernetes Subnet Manager - test4
I0208 09:58:35.481694 1 main.go:238] Installing signal handlers
I0208 09:58:35.482001 1 main.go:353] Found network config - Backend type: vxlan
I0208 09:58:35.482255 1 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
E0208 09:58:35.483159 1 main.go:280] Error registering network: failed to acquire lease: node "test4" pod cidr not assigned
I0208 09:58:35.483433 1 main.go:333] Stopping shutdownHandler...
看到：Error registering network: failed to acquire lease: node "test4" pod cidr not assigned
原因：是因为按照二进制高可用文档安装配置kube-controller-manager启动文件的时候，有坑，当时没有发现，
解决：
启动文件加上下面两句话，那篇文档没有加，所以报错；下面这个cluster-cidr要和kube-flannel.yml里面的地址一致，要和kube-proxy.config.yaml里面的clusterCIDR一致
--allocate-node-cidrs=true \
--cluster-cidr=172.30.0.0/16 \
4、源文档中的kubelet启动参数中都没有带cadvisor监控服务参数，源文档中就可以访问cadvisor监控了。明摆着安装完把cadvisor参数去掉了，这是大坑
5、执行查看资源报错： unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)
[root@test4 ~]# kubectl exec -it http-test-dm2-6dbd76c7dd-cv9qf sh
error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)
解决：创建apiserver到kubelet的权限
注意：user=kubernetes ，这个user要替换掉下面yaml文件里面的用户名
cat > apiserver-to-kubelet.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kubernetes-to-kubelet
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- nodes/stats
- nodes/log
- nodes/spec
- nodes/metrics
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kubernetes
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kubernetes-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes
EOF
创建授权:
kubectl create -f apiserver-to-kubelet.yaml
[root@test4 ~]# kubectl create -f apiserver-to-kubelet.yaml
clusterrole.rbac.authorization.k8s.io/system:kubernetes-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kubernetes created
重新进到容器查看资源
[root@test4 ~]# kubectl exec -it http-test-dm2-6dbd76c7dd-cv9qf sh
/ # exit
现在可以进到容器里面查看资源了
参照文档：https://www.jianshu.com/p/b3d8e8b8fd7e
6、源文档中kube-apiserver启动参数中没有这句话 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
这句话是必须要加的否则用kubectl查看资源、创建dnstools工具时候会报如下错误：
[root@test4 profile]# kubectl run -it --rm --image=infoblox/dnstools dns-client
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
Error attaching, falling back to logs: error dialing backend: dial tcp 0.0.0.0:10250: connect: connection refused
deployment.apps "dns-client" deleted
Error from server: Get https://test4:10250/containerLogs/default/dns-client-86c6d59f7-tzh5c/dns-client: dial tcp 0.0.0.0:10250: connect: connection refused
[root@test4 ~]# kubectl exec -it http-test-dm2-6dbd76c7dd-cv9qf sh
error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)
7、源文档中kube-apiserver启动参数中--enable-admission-plugins= 这个选项里面的参数很少，导致各种错误，必须填写全，如下：
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \
