kubenetes 环境的塔建
最近听我朋友说他们公司准备上云,全线把服务迁到 k8s 上面,一下感觉,我们就 lower 了不少,之前服务器一直跑的就是 docker ,想想弄到 k8s 应该还是没有啥,于是我们也开始改造了
参考了不少文档,有兴趣的可以读原文 https://kubernetes.io/docs/setup/independent/install-kubeadm/
https://blog.csdn.net/networken/article/details/84991940
https://jimmysong.io/kubernetes-handbook/practice/install-kubernetes-with-kubeadm.html
| 主机名 | ip | 角色 |
| kmaster | 192.168.9.88 | master |
| knode1 | 192.168.9.81 | node |
| konde2 | 192.168.9.82 | node |
分别在三台机器 中执行
cat >> /etc/hosts <<EOF
192.168.9.88 kmaster
192.168.9.81 knode1
192.168.9.82 knode2
EOF
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
[root@knode1 ~]# cat >> /etc/hosts <<EOF
> 192.168.9.88 kmaster
> 192.168.9.81 knode1
> 192.168.9.82 knode2
> EOF
[root@knode1 ~]# sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config && setenforce 0
[root@knode1 ~]# swapoff -a
[root@knode1 ~]# yes | cp /etc/fstab /etc/fstab_bak
[root@knode1 ~]# cat /etc/fstab_bak |grep -v swap > /etc/fstab
cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system
cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF #执行脚本 chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
#kube-proxy开启ipvs
yum install ipset ipvsadm -y
设置 docker 的源
#配置docker yum源 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
#安装指定版本，这里安装18.06
yum list docker-ce --showduplicates | sort -r
yum install -y docker-ce-18.06.1.ce-3.el7
systemctl start docker && systemctl enable docker
安装 kubeadmin kube kubctl
#配置kubernetes.repo的源，由于官方源国内无法访问，这里使用阿里云yum源 cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF #在所有节点上安装指定版本 kubelet、kubeadm 和 kubectl yum install -y kubelet-1.13.1 kubeadm-1.13.1 kubectl-1.13.1 #启动kubelet服务 systemctl enable kubelet && systemctl start kubelet
部署 master
kubeadm init \
--apiserver-advertise-address=192.168.9.88 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.13.1 \
--pod-network-cidr=10.244.0.0/16
注意这里执行初始化用到了- -image-repository选项，指定初始化需要的镜像源从阿里云镜像仓库拉取
这里有点慢,如果 输出下面这些,就可以了
[preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kmaster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.9.88] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [kmaster localhost] and IPs [192.168.9.88 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [kmaster localhost] and IPs [192.168.9.88 127.0.0.1 ::1] [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 22.007105 seconds [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kmaster" as an annotation [mark-control-plane] Marking the node kmaster as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node kmaster as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: hpvjuo.divmu5zdcqb7oysy [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes master has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join 192.168.9.88:6443 --token hpvjuo.divmu5zdcqb7oysy --discovery-token-ca-cert-hash sha256:a5e36c51c68ad1f1e07286c8c9c58bf5b8794c25182b18b15c1dcb6e99462eb2
#创建普通用户并设置密码123456 useradd k8s && echo "k8s:123456" | chpasswd k8s #追加sudo权限,并配置sudo免密 sed -i '/^root/a\k8s ALL=(ALL) NOPASSWD:ALL' /etc/sudoers
[root@kmaster ~]# su - k8s [k8s@kmaster ~]$ [k8s@kmaster ~]$ [k8s@kmaster ~]$ [k8s@kmaster ~]$ [k8s@kmaster ~]$ mkdir -p $HOME/.kube [k8s@kmaster ~]$ [k8s@kmaster ~]$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [k8s@kmaster ~]$ [k8s@kmaster ~]$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
[k8s@kmaster ~]$ #启用 kubectl 命令自动补全功能（注销重新登录生效）
[k8s@kmaster ~]$ echo "source <(kubectl completion bash)" >> ~/.bashrc
[k8s@kmaster ~]$ kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health": "true"}
[k8s@kmaster ~]$ kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
[k8s@kmaster ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kmaster NotReady master 6m51s v1.13.1
[k8s@kmaster ~]$ kubectl describe node kmaster
Name: kmaster
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=kmaster
node-role.kubernetes.io/master=
查看 pod的情况
[k8s@kmaster ~]$ kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-78d4cf999f-l9f7v 0/1 Pending 0 2m32s <none> <none> <none> <none> coredns-78d4cf999f-n8g4g 0/1 Pending 0 2m32s <none> <none> <none> <none> etcd-kmaster 1/1 Running 0 6m51s 192.168.9.88 kmaster <none> <none> kube-apiserver-kmaster 1/1 Running 0 6m48s 192.168.9.88 kmaster <none> <none> kube-controller-manager-kmaster 1/1 Running 0 6m54s 192.168.9.88 kmaster <none> <none> kube-proxy-57lvg 1/1 Running 0 7m32s 192.168.9.88 kmaster <none> <none> kube-scheduler-kmaster 1/1 Running 0 6m45s 192.168.9.88 kmaster <none> <none>
部署网络插件
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
[k8s@kmaster ~]$ kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-78d4cf999f-l9f7v 1/1 Running 0 9m18s 10.244.0.3 kmaster <none> <none> coredns-78d4cf999f-n8g4g 1/1 Running 0 9m18s 10.244.0.2 kmaster <none> <none> etcd-kmaster 1/1 Running 0 13m 192.168.9.88 kmaster <none> <none> kube-apiserver-kmaster 1/1 Running 0 13m 192.168.9.88 kmaster <none> <none> kube-controller-manager-kmaster 1/1 Running 0 13m 192.168.9.88 kmaster <none> <none> kube-flannel-ds-amd64-dkb2t 1/1 Running 0 2m44s 192.168.9.88 kmaster <none> <none> kube-proxy-57lvg 1/1 Running 0 14m 192.168.9.88 kmaster <none> <none> kube-scheduler-kmaster 1/1 Running 0 13m 192.168.9.88 kmaster <none> <none>
至此，Kubernetes 的 Master 节点就部署完成了。如果你只需要一个单节点的 Kubernetes，现在你就可以使用了。
部署worker节点
[root@knode1 ~]# kubeadm join 192.168.9.88:6443 --token hpvjuo.divmu5zdcqb7oysy --discovery-token-ca-cert-hash sha256:a5e36c51c68ad1f1e07286c8c9c58bf5b8794c25182b18b15c1dcb6e99462eb2 [preflight] Running pre-flight checks [discovery] Trying to connect to API Server "192.168.9.88:6443" [discovery] Created cluster-info discovery client, requesting info from "https://192.168.9.88:6443" [discovery] Requesting info from "https://192.168.9.88:6443" again to validate TLS against the pinned public key [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.9.88:6443" [discovery] Successfully established connection with API Server "192.168.9.88:6443" [join] Reading configuration from the cluster... [join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap... [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "knode1" as an annotation This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the master to see this node join the cluster.
#执行以下命令将节点接入集群
kubeadm join 192.168.92.56:6443 --token 67kq55.8hxoga556caxty7s --discovery-token-ca-cert-hash sha256:7d50e704bbfe69661e37c5f3ad13b1b88032b6b2b703ebd4899e259477b5be69
#如果执行kubeadm init时没有记录下加入集群的命令，可以通过以下命令重新创建 kubeadm token create --print-join-command
查看节点的状态
[k8s@kmaster ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION kmaster Ready master 19m v1.13.1 knode1 NotReady <none> 2m5s v1.13.1 knode2 NotReady <none> 2m9s v1.13.1
稍等处刻
[k8s@kmaster ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION kmaster Ready master 24m v1.13.1 knode1 Ready <none> 7m46s v1.13.1 knode2 Ready <none> 7m50s v1.13.1 [k8s@kmaster ~]$ kubectl get pod --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-78d4cf999f-l9f7v 1/1 Running 0 20m 10.244.0.3 kmaster <none> <none> kube-system coredns-78d4cf999f-n8g4g 1/1 Running 0 20m 10.244.0.2 kmaster <none> <none> kube-system etcd-kmaster 1/1 Running 0 24m 192.168.9.88 kmaster <none> <none> kube-system kube-apiserver-kmaster 1/1 Running 0 24m 192.168.9.88 kmaster <none> <none> kube-system kube-controller-manager-kmaster 1/1 Running 0 24m 192.168.9.88 kmaster <none> <none> kube-system kube-flannel-ds-amd64-44x4d 1/1 Running 0 8m48s 192.168.9.81 knode1 <none> <none> kube-system kube-flannel-ds-amd64-465pk 1/1 Running 2 8m51s 192.168.9.82 knode2 <none> <none> kube-system kube-flannel-ds-amd64-dkb2t 1/1 Running 0 13m 192.168.9.88 kmaster <none> <none> kube-system kube-proxy-4rgz9 1/1 Running 0 8m48s 192.168.9.81 knode1 <none> <none> kube-system kube-proxy-57lvg 1/1 Running 0 25m 192.168.9.88 kmaster <none> <none> kube-system kube-proxy-hbbqj 1/1 Running 0 8m51s 192.168.9.82 knode2 <none> <none> kube-system kube-scheduler-kmaster 1/1 Running 0 24m 192.168.9.88 kmaster <none> <none>
Pod调度到Master节点
[k8s@kmaster ~]$ kubectl taint node kmaster node-role.kubernetes.io/master-
node/kmaster untainted
如果要恢复Master Only状态，执行如下命令：
kubectl taint node k8s-master node-role.kubernetes.io/master=""
kube-proxy开启ipvs
修改ConfigMap的kube-system/kube-proxy中的config.conf，mode: “ipvs”：
[k8s@kmaster ~]$ kubectl edit cm kube-proxy -n kube-system
configmap/kube-proxy edited
之后重启各个节点上的kube-proxy pod：
kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'
[k8s@kmaster ~]$ kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'
pod "kube-proxy-4rgz9" deleted
pod "kube-proxy-57lvg" deleted
pod "kube-proxy-hbbqj" deleted
[k8s@kmaster ~]$ kubectl logs kube-proxy-6btv9 -n kube-system I0125 07:52:50.004289 1 server_others.go:189] Using ipvs Proxier. W0125 07:52:50.004834 1 proxier.go:365] IPVS scheduler not specified, use rr by default I0125 07:52:50.004997 1 server_others.go:216] Tearing down inactive rules. I0125 07:52:50.052950 1 server.go:464] Version: v1.13.1 I0125 07:52:50.067533 1 conntrack.go:52] Setting nf_conntrack_max to 131072 I0125 07:52:50.067821 1 config.go:102] Starting endpoints config controller I0125 07:52:50.069525 1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller I0125 07:52:50.069231 1 config.go:202] Starting service config controller I0125 07:52:50.070363 1 controller_utils.go:1027] Waiting for caches to sync for service config controller I0125 07:52:50.169786 1 controller_utils.go:1034] Caches are synced for endpoints config controller I0125 07:52:50.170565 1 controller_utils.go:1034] Caches are synced for service config controller
