kukubeadm 1.6.1 + docker1.2.6 安装问题
kubeadm init --apiserver-advertise-address=192.168.20.229 --pod-network-cidr=10.244.0.0/16
kubelet: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"
docker相比1.10增加了KernelMemory变量和CgroupDriver变量，KernelMemory变量表示是否设置linux内核内存限制，CgroupDriver变量表示使用哪个Cgroup驱动，有两种驱动，分别是cgroupfs和systemd，默认使用cgroupfs
由 systemd 变更成 cgroupfs
############################################
或者 --cgroup-driver=systemd \
kubelet的服务配置文件加上这么一行
使用kubeadm 安装 kubernetes1.6.1
环境准备
master 192.168.20.229 node 192.168.20.223
软件版本：
docker使用 1.12.6
查看版本
yum list kubeadm --showduplicates |sort -r kubeadm.x86_64 1.6.1-0 kubernetes kubeadm.x86_64 1.6.0-0 kubernetes
yum list kubelet --showduplicates |sort -r kubelet.x86_64 1.6.1-0 kubernetes kubelet.x86_64 1.6.0-0 kubernetes kubelet.x86_64 1.5.4-0 kubernetes
yum list kubectl --showduplicates |sort -r kubectl.x86_64 1.6.1-0 kubernetes kubectl.x86_64 1.6.0-0 kubernetes kubectl.x86_64 1.5.4-0 kubernetes
yum list kubernets-cni --showduplicates |sort -r kubernetes-cni x86_64 0.5.1-0 kubernetes
系统配置
根据官方文档Installing Kubernetes on Linux with kubeadm中的Limitations小节中的内容，对各节点系统做如下设置:
创建/etc/sysctl.d/k8s.conf文件，添加如下内容：
net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1
初始化集群
kubeadm init --kubernetes-version=v1.6.1 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.20.229
kubeadm init执行成功后输出下面的信息：
kubeadm init --kubernetes-version=v1.6.1 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.61.41 [kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters. [init] Using Kubernetes version: v1.6.1 [init] Using Authorization mode: RBAC [preflight] Running pre-flight checks [preflight] Starting the kubelet service [certificates] Generated CA certificate and key. [certificates] Generated API server certificate and key. [certificates] API Server serving cert is signed for DNS names [node0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.61.41] [certificates] Generated API server kubelet client certificate and key. [certificates] Generated service account token signing key and public key. [certificates] Generated front-proxy CA certificate and key. [certificates] Generated front-proxy client certificate and key. [certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf" [apiclient] Created API client, waiting for the control plane to become ready [apiclient] All control plane components are healthy after 14.583864 seconds [apiclient] Waiting for at least one node to register [apiclient] First node has registered after 6.008990 seconds [token] Using token: e7986d.e440de5882342711 [apiconfig] Created RBAC rules [addons] Created essential addon: kube-proxy [addons] Created essential addon: kube-dns Your Kubernetes master has initialized successfully! To start using your cluster, you need to run (as a regular user): sudo cp /etc/kubernetes/admin.conf $HOME/ sudo chown $(id -u):$(id -g) $HOME/admin.conf export KUBECONFIG=$HOME/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: http://kubernetes.io/docs/admin/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join --token 881f96.aaf02f1f8dc53889 192.168.20.229:6443
Master Node初始化完成，使用kubeadm初始化的Kubernetes集群在Master节点上的核心组件：kube-apiserver,kube-scheduler, kube-controller-manager是以静态Pod的形式运行的。
ls /etc/kubernetes/manifests/
etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml
在/etc/kubernetes/manifests/目录里可以看到kube-apiserver,kube-scheduler, kube-controller-manager的定义文件。另外集群持久化存储etcd也是以单点静态Pod的形式运行的，对于etcd后边我们会把它切换成etcd集群。
查看一下kube-apiserver.yaml的内容：
apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver ....... - --insecure-port=0
注意到kube-apiserver的选项--insecure-port=0，也就是说kubeadm 1.6.0初始化的集群，kube-apiserver没有监听默认的http 8080端口。
所以我们使用kubectl get nodes会报The connection to the server localhost:8080 was refused - did you specify the right host or port?。
查看kube-apiserver的监听端口可以看到只监听了https的6443端口
netstat -nltp | grep apiserver tcp6 0 0 :::6443 :::* LISTEN 9831/kube-apiserver
为了使用kubectl访问apiserver，在~/.bash_profile中追加下面的环境变量：
export KUBECONFIG=/etc/kubernetes/admin.conf
source ~/.bash_profile
此时kubectl命令在master node上就好用了，查看一下当前机器中的Node：
kubectl get nodes NAME STATUS AGE VERSION k8s1 NotReady 3m v1.6.1
安装Pod Network
接下来安装flannel network add-on：
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml serviceaccount "flannel" created configmap "kube-flannel-cfg" created daemonset "kube-flannel-ds" created
如果Node有多个网卡的话，参考flannel issues 39701，目前需要在kube-flannel.yml中使用--iface参数指定集群主机内网网卡的名称，否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地，flanneld启动参数加上--iface=<iface-name>
...... apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: kube-flannel-ds ...... containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.7.0-amd64 command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=eth1" ] ......
使用kubectl get pod --all-namespaces -o wide确保所有的Pod都处于Running状态
kubectl get pod --all-namespaces -o wide 或者 kubectl --kubeconfig=/etc/kubernetes/admin.conf get pod --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system etcd-k8s1 1/1 Running 0 10m 192.168.20.229 k8s1 kube-system kube-apiserver-k8s1 1/1 Running 0 10m 192.168.20.229 k8s1 kube-system kube-controller-manager-k8s1 1/1 Running 0 10m 192.168.20.229 k8s1 kube-system kube-dns-3913472980-g97bm 3/3 Running 0 10m 10.244.1.2 k8s5 kube-system kube-flannel-ds-k87tt 2/2 Running 0 2m 192.168.20.233 k8s5 kube-system kube-flannel-ds-lq62q 2/2 Running 0 2m 192.168.20.229 k8s1 kube-system kube-proxy-0nrp0 1/1 Running 0 10m 192.168.20.229 k8s1 kube-system kube-proxy-qcds5 1/1 Running 0 10m 192.168.20.233 k8s5 kube-system kube-scheduler-k8s1 1/1 Running 0 10m 192.168.20.229 k8s1
使master node参与工作负载
使用kubeadm初始化的集群，出于安全考虑Pod不会被调度到Master Node上，也就是说Master Node不参与工作负载。
这里搭建的是测试环境可以使用下面的命令使Master Node参与工作负载：
kubectl taint nodes --all node-role.kubernetes.io/master-
测试DNS
[root@k8s1 ~]# kubectl --kubeconfig=/etc/kubernetes/admin.conf run curl --image=radial/busyboxplus:curl -i --tty If you don't see a command prompt, try pressing enter. [ root@curl-57077659-s2l5v:/ ]$ nslookup BusyBox v1.22.1 (2014-09-13 22:15:30 PDT) multi-call binary. Usage: nslookup [HOST] [SERVER] Query the nameserver for the IP address of the given HOST optionally using a specified DNS server [ root@curl-57077659-s2l5v:/ ]$ nslookup kube-dns.kube-system Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kube-dns.kube-system Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local [ root@curl-57077659-s2l5v:/ ]$ nslookup kubernetes.default Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
测试OK后，删除掉curl这个Pod。
kubectl delete deploy curl
向集群中添加节点
kubeadm join --token 881f96.aaf02f1f8dc53889 192.168.20.229:6443
查看集群中节点：
[root@k8s1 ~]# kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes NAME STATUS AGE VERSION k8s1 Ready 54m v1.6.1 k8s5 Ready 54m v1.6.1
安装Dashboard插件
wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml kubectl create -f kubernetes-dashboard.yaml
从http://NodeIp:NodePort访问dashboard，浏览器显示下面的错误
User "system:serviceaccount:kube-system:default" cannot list statefulsets.apps in the namespace "default". (get statefulsets.apps)
这是因为Kubernetes 1.6开始API Server启用了RBAC授权，当前的kubernetes-dashboard.yaml没有定义授权的ServiceAccount，所以访问API Server时被拒绝了。
根据https://github.com/kubernetes/dashboard/issues/1803中的内容临时授予system:serviceaccount:kube-system:default cluster_admin的角色，临时解决一下。
创建dashboard-rbac.yaml，定义system:serviceaccount:kube-system:default和ClusterRole cluster-admin绑定：
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: dashboard-admin roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: default namespace: kube-system kubectl --kubeconfig=/etc/kubernetes/admin.conf create -f dashboard-rbac.yml
在集群中运行Heapster
下面安装Heapster为集群添加使用统计和监控功能，为Dashboard添加仪表盘。
下载最新的Heapster到集群中的某个Node上
wget https://github.com/kubernetes/heapster/archive/v1.3.0.tar.gz
使用InfluxDB做为Heapster的后端存储，开始部署，中间会pull相关镜像，包含gcr.io/google_containers/heapster_grafana:v2.6.0-2
tar -zxvf v1.3.0.tar.gz cd heapster-1.3.0/deploy/kube-config/influxdb
添加了RBAC授权
[root@k8s1 influxdb]# cat heapster-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: heapster namespace: kube-system --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1alpha1 metadata: name: heapster subjects: - kind: ServiceAccount name: heapster namespace: kube-system roleRef: kind: ClusterRole name: system:heapster apiGroup: rbac.authorization.k8s.io
[root@k8s1 influxdb]# vim heapster-deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: heapster namespace: kube-system spec: replicas: 1 template: metadata: labels: task: monitoring k8s-app: heapster spec: serviceAccountName: heapster containers: - name: heapster image: gcr.io/google_containers/heapster-amd64:v1.3.0-beta.1 imagePullPolicy: IfNotPresent command: - /heapster - --source=kubernetes:https://kubernetes.default - --sink=influxdb:http://monitoring-influxdb:8086
参考
http://blog.frognew.com/2017/04/kubeadm-install-kubernetes-1.6.html
https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/10-%E9%83%A8%E7%BD%B2Heapster%E6%8F%92%E4%BB%B6.md
