本地搭建3节点kubernetes

kubernetes本地搭建版本选择

• CentOS Linux release 7.7.1908
• kubernetesVersion: v1.17.0
• weave-kube:2.6.0
• ceph/ceph:v14.2.6

CentOS 环境准备

3台机器环境检查

• master内存至少4GiB

• 禁用swap分区 master

sudo swapoff -a

• 永久禁用

sudo vi /etc/fstab
把/dev/mapper/centos-swap swap这行注释掉

• sysctl 配置

cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF

sysctl --system

• key免密

参见网络其他文章

• hostname

hostnamectl set-hostname xx

• hostname及ip映射

cat >> /etc/hosts <<EOF 
172.16.225.129 k8s-02
172.16.225.130 k8s-03
172.16.225.200 k8s-01
EOF

yum源及安装

• 配置kubernetes yum源

# k8s yum源
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=0
enable=1
EOF
>>

• 配置docker yum源

cd /etc/yum.repos.d

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

yum clean all

yum repolist

• 安装docker, kubeadm, kubelet and kubectl

yum install -y docker-ce kubeadm kubelet kubectl

• 检查docker及kubelet

systemctl enable --now docker && systemctl enable --now kubelet

• 检查日志

systemctl status kubelet

journalctl -xeu kubelet

kubernetes环境搭建

安装k8s master

• 安装master，法一：
  查询列表
  kubeadm config images list

• 安装master，法二：

1. 生成kubaeadm.yaml

kubeadm config print init-defaults > kubeadm.yaml

2. 使用kubeadm.yaml 文件（1.15.X版本）

cat <<EOF > kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration: # 去掉master污点
  taints:
  - effect: PreferNoSchedule
    key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
imageRepository: registry.aliyuncs.com/google_containers # images地址
kubernetesVersion: v1.17.0 # version
EOF

2. 初始化集群
   kubeadm init --config kubeadm.yaml | tee kubeadmin.log

• 再次创建tocken（选）
  token expired
  kubeadm token create --print-join-command

return a token
kubeadm join 172.17.55.202:6443 --token 4xv6eq.5kygve4g6uy1smat \ --discovery-token-ca-cert-hash sha256:3ab287132d40f511925d0e416a6a0bd1fba47697c447b6c05f421b12029b8766

To start using your cluster, you need to run the following as a regular user:

• 卸载方法 （选）

kubeadm reset
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
rm -rf $HOME/.kube/config
systemctl stop docker
systemctl stop kubelet
yum remove kubelet
iptables -tnat --flush # iptables 一定要刷新

1. 调整master服务器环境

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

初始化网络插件

• weave:
  kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

启动worker节点

1. 系统设置

2. 配置k8s,docker yum源

3. 下载docker kubelet kubeadm

4. 通过tocken加入集群。 注意tocken的有效期为24小时

补充其他插件

添加dashboard

• yaml文件
  https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/#deploying-the-dashboard-ui

• 修改nodePort

kubectl patch svc -n kube-system kubernetes-dashboard -p '{"spec":{"type":"NodePort"}}'

• 创建dashboard管理用户

kubectl create serviceaccount dashboard-admin -n kube-system

• 绑定用户为集群管理用户

kubectl create clusterrolebinding dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin

• 获取tocken

kubectl describe secret -n kube-system dashboard-admin

• ref: https://cloud.tencent.com/developer/article/1444812

添加存储

• rook: https://github.com/rook/rook/blob/master/Documentation/ceph-quickstart.md#deploy-the-rook-operator

查看安装

kubectl get pods -n rook-ceph-system
kubectl get pods -n rook-ceph

搭建过程中问题

# coredns readiness 一直unhealthy
coredns dial tcp 10.96.0.1:443: connect: no route to host

# 问题
iptables混乱导致

# 解决办法
systemctl stop kubelet
systemctl stop docker
iptables --flush
iptables -tnat --flush
systemctl start kubelet
systemctl start docker
kubectl delete pod weave -n kube-system