k8s 集群1.18.3安装--100%成功

K8S--安装配置--kubeadm安装方式---成功--K8S--dashboard--
安装--配置--重置--卸载--20210914
基本流程为：
基础环境配置--安装docker--安装k8s主件--安装flannel（k8s本身没包括）--安装dashboard--完成
========================================================
本文所有链接，以后可能用不了，还需要自行去找方法，没办法
这个教程里，flannel下载安装就是在网上找的，多次实验安装才成功，网上的教程就不行
整个安装步骤太多了，感觉还是vmware vsphere套件安装好，至少是可视化的。
数据库 etcd，
调度器 scheduler，
集群入口 API Server，
控制器 Controller，
服务代理 kube-proxy
直接管理具体业务容器的 kubelet
flannel 网络连接插件
===================================================
成功---参考以下教程，这个教程k8s主件安装是完全OK的。
https://blog.csdn.net/caiyqn/article/details/107801349?utm_medium=distribute.pc_relevant.none-task-blog-2~default~baidujs_title~default-0.no_search_link&spm=1001.2101.3001.4242
https://www.cnblogs.com/guoxiaobo/p/15025308.html        这个也详细

https://www.cnblogs.com/xiao987334176/p/12696740.html    这个也详细
===================================================
硬件要求，建议至少2 cpu ,2G,20G：3台机器，

192.168.9.3   master

192.168.9.4   node1

192.168.9.5  node2

不然会安装报错，后面就安装不了了
软件要求，版本要匹配，节点的时间要一致
===本次安装==软件版本=======
centos7.8
docker-ce-18.06.1.ce-3.el7
kubelet-1.18.3 kubeadm-1.18.3 kubectl-1.18.3
检查所有节点配置
cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
uname -a
Linux master 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
free -h
===docker与k8s的版本对应版本是有要求的，最新的--并不是匹配的=========================
软件安装中，版本冲突，就换版本，注意系统提示的版本信息
file /usr/bin/kubectl from install of kubectl-1.22.1-0.x86_64 conflicts with file from package kubernetes-client-1.5.2-0.7.git269f928.el7.x86_64
====安装结果如下==master上检查==========================
[root@master ~]# docker -v
Docker version 18.06.1-ce, build e68fc7a
[root@master ~]# yum list installed |grep kub
cri-tools.x86_64 1.13.0-0 @kubernetes
kubeadm.x86_64 1.18.3-0 @kubernetes
kubectl.x86_64 1.18.3-0 @kubernetes
kubelet.x86_64 1.18.3-0 @kubernetes
kubernetes-cni.x86_64 0.8.7-0 @kubernetes
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 4h13m v1.18.3
node1 Ready <none> 4h10m v1.18.3
node2 Ready <none> 4h10m v1.18.3
===安装开始===============================================
1.===基础环境配置==所有节点操作=============================
==root用户操作==所有节点==当然也可不用========================
1.1关闭防火墙==所有节点操作============
systemctl stop firewalld
systemctl disable firewalld
1.2关闭selinux==所有节点操作============
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
1.3关闭swap==所有节点操作============
临时关闭：
swapoff -a
永久关闭：注释掉/etc/fstab文件中的swap行
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
1.4添加hosts==所有节点操作============
vi /etc/hosts
192.168.9.3 master
192.168.9.4 node1
192.168.9.5 node2
1.5内核修改相关==所有节点操作============
加载内核模块==所有节点操作============
modprobe ip_vs_rr
modprobe br_netfilter
1.6优化内核参数==所有节点操作============
vi /etc/sysctl.d/kubernetes.conf
---复制吧---------------
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
#由于tcp_tw_recycle与kubernetes的NAT冲突，必须关闭！否则会导致服务不通。4.1x内核已经废弃这项了
#net.ipv4.tcp_tw_recycle=0
#禁止使用 swap 空间，只有当系统 OOM 时才允许使用它
vm.swappiness=0
#不检查物理内存是否够用
vm.overcommit_memory=1
#开启 OOM
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
#关闭不使用的ipv6协议栈，防止触发docker BUG.
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
--再执行，所有节点----------------------
sysctl -p /etc/sysctl.d/kubernetes.conf
sysctl --system
============================
#必须关闭 tcp_tw_recycle，否则和 NAT 冲突，会导致服务不通；
#关闭 IPV6，防止触发 docker BUG；
=============================
1.7开启IP转发功能==所有节点操作============
echo "1" > /proc/sys/net/ipv4/ip_forward
开启时间同步==所有节点操作===如果时间正常，可以不配置=========
echo "*/3 * * * * /usr/sbin/ntpdate ntp3.aliyun.com &> /dev/null" > /tmp/crontab
crontab /tmp/crontab
1.8创建免密访问===master上配置============
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.9.4
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.9.5
1.9修改PATH环境变量==所有节点操作============
echo 'PATH=/opt/k8s/bin:$PATH' >>/root/.bashrc
source /root/.bashrc
1.10安装依赖包==如果系统已经有了，可以不安装========
yum install -y epel-release
yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget lsof telnet
1.11安装docker==所有节点操作============
1.12#安装docker需要的工具==所有节点操作============
yum install -y yum-utils device-mapper-persistent-data lvm2
1.13#添加docker镜像包==所有节点操作============
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
1.13#安装并更新docker-ce镜像包==所有节点操作============
1.14#更新缓存
yum makecache fast
1.15#安装docker默认安装最新，可指定版本安装
yum -y install docker-ce
#查看版本
docker -v
1.16#开启docker服务==所有节点操作============
systemctl enable docker
systemctl start docker
1.17#附：docker镜像加速方法==所有节点操作============
#添加指定加速地址，这里指定阿里云的
vim /etc/docker/daemon.json
---复制吧-----------
{
"registry-mirrors": [
"https://registry.cn-hangzhou.aliyuncs.com"
]
}
------------------
1.18#重新加载服务==所有节点操作============
systemctl daemon-reload
systemctl restart docker
#查看 Registry Mirrors是否变成阿里云加速地址
docker info
1.19添加kubenetes软件包==所有节点操作============
#这里直接使用阿里云的软件包
vi /etc/yum.repos.d/kubernetes.repo
----复制吧------------------
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
-----------------------
1.20#更新软件包==所有节点操作============
yum makecache fast
==================================
基础环境配置完成
==================================
1.21==开始安装k8s主件==所有节点操作============
#安装kubelet、kubeadm、kubectl==所有节点操作============（由于这套环境后续还有其他用处，因此指定了版本安装）
yum install -y kubelet-1.18.3 kubeadm-1.18.3 kubectl-1.18.3
systemctl enable kubelet
systemctl start kubelet

2.0==开始部署k8s集群==Master上执行=================
#只能在matser上面部署
----复制吧---------------
kubeadm init \
--apiserver-advertise-address=192.168.9.3 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.18.3 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.244.0.0/16
--------------------
====安装完成 显示以下内容，表示安装成功========================
==安装过程简要如下=========
。。省略前面显示过程。。。。。
Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:
2.1#==只在Master上执行，不论是不是用root用户执行整个的安装，都要执行，（虽然这里说明是只要，非root用户执行）
-----复制吧-------
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
-------------------
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:
2.2#下面这个很重要，是node节点加入集群里的命令，直接执行，就好，2个小时有效，2小时后，再用createa命令生成吧
kubeadm join 192.168.9.3:6443 --token rv7mmx.nkpsf0u426eajiwl --discovery-token-ca-cert-hash sha256:4293225b17a544f912e9218da98dabdaf8b4e5c250a63288dcd4a8867b3ac262
===================================================================
2.3===安装后，会自动下载k8s的相关docker镜像，查下有哪些====#只在Master上执行=============================
[root@master ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-apiserver v1.22.1 f30469a2491a 2 weeks ago 128MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.22.1 6e002eb89a88 2 weeks ago 122MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.22.1 aca5ededae9c 2 weeks ago 52.7MB
registry.aliyuncs.com/google_containers/kube-proxy v1.22.1 36c4ebbc9d97 2 weeks ago 104MB
registry.aliyuncs.com/google_containers/etcd 3.5.0-0 004811815584 2 months ago 295MB
registry.aliyuncs.com/google_containers/pause 3.5 ed210e3e4a5b 5 months ago 683kB
registry.aliyuncs.com/google_containers/kube-proxy v1.13.1 fdb321fd30a0 2 years ago 80.2MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.13.1 26e6f1db2a52 2 years ago 146MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.13.1 ab81d7360408 2 years ago 79.6MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.13.1 40a63db91ef8 2 years ago 181MB
registry.aliyuncs.com/google_containers/coredns 1.2.6 f59dcacceff4 2 years ago 40MB
registry.aliyuncs.com/google_containers/etcd 3.2.24 3cab8e1b9802 2 years ago 220MB
registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 3 years ago 742kB
2.4==按照上面提示，操作===#只在Master上执行==========================
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
==下面就可以直接使用kubectl命令了========================================
2.5===查看节点状态=======================================
kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady master 5m47s v1.18.3
2.6===上面notready,是因为flannel插件没安装，k8s节点的专用网络不通=======================================
#由于github官网原因，这2个链接目前不能下载了，只能手动下载flannel.docker安装包，手动load
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml
kubectl apply -f https://github.com/caoran/kube-flannel.yml/blob/master/kube-flannel.yml
2.7#只在Master上执行，先上传到master,再执行=====flannel版本不确定，我只下载了0.12.0，当然也用其它网络插件，其它自行测试吧，
docker load < flanneld-v0.12.0-amd64.docker
#kube-flannel.yml，文件名可自定，内容只能在网上去复制了，再执行
kubectl apply -f kube-flannel.yml
2.8再查看k8s集群状态
#查看所有节点是否ready
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 5h7m v1.18.3
node1 Ready <none> 5h5m v1.18.3
node2 Ready <none> 5h5m v1.18.3
kubectl get nodes
2.9#查看flannel节点是否运行正常
kubectl get pod -n kube-system | grep flannel
kube-flannel-ds-amd64-2dd6b 1/1 Running 0 5h5m
kube-flannel-ds-amd64-7dg9q 1/1 Running 0 5h5m
kube-flannel-ds-amd64-swspw 1/1 Running 0 5h5m
2.10#查看kube-system启动的所有pod
kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-d7gjx 1/1 Running 0 5h11m
coredns-7ff77c879f-tpr2t 1/1 Running 0 5h11m
etcd-master 1/1 Running 0 5h11m
kube-apiserver-master 1/1 Running 0 5h11m
kube-controller-manager-master 1/1 Running 0 5h11m
kube-flannel-ds-amd64-2dd6b 1/1 Running 0 5h6m
kube-flannel-ds-amd64-7dg9q 1/1 Running 0 5h6m
kube-flannel-ds-amd64-swspw 1/1 Running 0 5h6m
kube-proxy-52xxt 1/1 Running 0 5h9m
kube-proxy-hnn4b 1/1 Running 0 5h8m
kube-proxy-t7zl8 1/1 Running 0 5h11m
kube-scheduler-master 1/1 Running 0 5h11m
2.11====查看集群配置信息======================================
kubectl -n kube-system get cm kubeadm-config -oyaml
==========================================
2.22//查看节点信息
kubectl get pod -n kube-system
kubectl get pod -n kube-system -w //监视
kubectl get pod -n kube-system -o wide // 详细信息
========================================================
*****到这里，k8s集群后台主件安装，基本完成*****************
========================================================
3.0===重建集群==#只在Master上执行===
kubeadm reset
再执行2.0步骤
========================================================
4.0==卸载K8S=====#只在Master上执行=====可保存成脚本，自动跑，更好=================
yum remove -y kubelet kubeadm kubectl
kubeadm reset -f
modprobe -r ipip
lsmod
rm -rf ~/.kube/
rm -rf /etc/kubernetes/
rm -rf /etc/systemd/system/kubelet.service.d
rm -rf /etc/systemd/system/kubelet.service
rm -rf /usr/bin/kube*
rm -rf /etc/cni
rm -rf /opt/cni
rm -rf /var/lib/etcd
rm -rf /var/etcd
====以下为可选安装==================================================================================
5.0 dashboard安装---K8S的web管理界面==专业术语叫--仪表盘
===master上执行====查看对应版本的Dashboard======= https://github.com/kubernetes/dashboard/releases
5.11 下载dashboard配置文件
5.12 初始化安装
5.13 创建admin-user.yaml文件
5.14 查看token
5.15 网页访问集群
========================================================
5.11 下载dashboard配置文件
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.1/aio/deploy/recommended.yaml
上面链接失效了，用下面的吧，亲测OK,手动复制吧
https://www.cnblogs.com/zjz20/p/12588234.html
5.12初始化安装
kubectl apply -f recommended.yaml
#本例中，recommended.yaml配置了访问端口30002，端口可自定，随便。
检查dashboard服务状态
kubectl get service -n kubernetes-dashboard | grep dashboard
5.13 创建admin-user.yaml文件，复制或命令授权，二选其一就行==================
5.1301 创建admin-user.yaml，或者dashboard-rbac.yaml,文件名可自定，
#执行，加载到系统里
kubectl apply -f admin-user.yaml
#本例中，admin-user.yaml配置了用户名admin-user-toen，并放在kube-system命名空间下，并将cluster-admin角色绑定到admin-user-toen账户，
这样admin-user-toen账户就有了管理员的权限。默认情况下，kubeadm创建集群时已经创建了cluster-admin角色，我们直接绑定即可.
#查看secrets
kubectl get secrets -n kube-system | grep admin

5.1302 也可用命令进行授权，复制或命令，二选其一就行===本步骤命令未作测试，自行修改测试========================。
#创建账号
kubectl create serviceaccount admin-user-toen -n kube-system
#授权
kubectl create clusterrolebinding admin-user-toen-rb --clusterrole=cluster-admin --serviceaccount=kube-system:admin-user-toen

#本例中，admin-user.yaml配置了用户名admin-user-toen，并放在kube-system命名空间下，并将cluster-admin角色绑定到admin-user-toen账户，
这样admin-user-toen账户就有了管理员的权限。默认情况下，kubeadm创建集群时已经创建了cluster-admin角色，我们直接绑定即可.

5.14 查看token
kubectl describe secret -n kube-system $(kubectl get secret -n kube-system |grep admin-user-token| awk '{print $1}')
----------------------------------------------------------------------------------------------------
admin-user-token-token-5mfmr kubernetes.io/service-account-token 3 3h30m
[root@master ~]# kubectl describe secret -n kube-system $(kubectl get secret -n kube-system |grep admin-user-token| awk '{print $1}')
Name: admin-user-token-token-5mfmr
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: admin-user-token
kubernetes.io/service-account.uid: 9534f4f9-d503-40b0-8981-125fa932c05c

Type: kubernetes.io/service-account-token

Data
====
ca.crt: 1025 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IngyYnAyajdxLVZUb0puZ0RmY19QMlp1c05YTkdiREhReUJyQzNzQU9ITHMifQ.
eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOi
JrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXRva
2VuLTVtZm1yIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXItdG9r
ZW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI5NTM0ZjRmOS1kNTAzLTQwYjAtODk
4MS0xMjVmYTkzMmMwNWMiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlci10b2tlbiJ9.
A-g2llxoqBzQiD4YFeTkrZ_5Q0b7ayO1CorBLu13qclT4myQpoOazLkCdWiKucr2keHGcZbKaZUjqAgDOesXnZSNcU-fUvpwM1wZCiwBF_
mtLeZ3QIre8qjsOV62mXhVQ6aXxYlB27GFlE8LBQMsx-nJm1zc0RtZRW_3V5sTQKmJZF-DUt-HzbjnAA-WV0A5x_owQZ0RpCA-hqdcXh8J2W_
wZ985lEDJ1EmjNFj1yEiDXjbEWzbCr-7uQ2wanRc3U1t9SZWaXfI3QKc9uL0uTK8cdWO8DEO0kiellwRez5NeIwkqsngTHuayiHVCuL-1davQ1Zea-yhC3KAzYHKqOA
--------------------------------------------------------------------------------------------
5.15 网页访问集群---master的IP
https://192.168.9.3:30002
选择---token登录
输入上面查到的token
===dashboard安装完成======已经累成狗了====================================================
==============================================================================
===以下为扩展安装===网上教程，自行测试========================================
==============================================================================
一个大神的教程：以下都是他的操作，只能说，世界太复杂了
https://www.cnblogs.com/bigberg/
部署一个简单的例子
https://www.cnblogs.com/bigberg/p/13469637.html

6.0 部署GlusterFS
https://www.cnblogs.com/bigberg/p/13489592.html
7.0 部署Deployment
https://www.cnblogs.com/bigberg/p/13489676.html
8.0 部署RBAC
https://www.cnblogs.com/bigberg/p/13494546.html
9.0 部署StatefulSet
https://www.cnblogs.com/bigberg/p/13494583.html
10. 部署Init容器
https://www.cnblogs.com/bigberg/p/13500297.html
11.0 部署ConfigMap
https://www.cnblogs.com/bigberg/p/13500428.html
12.0 部署Secret
https://www.cnblogs.com/bigberg/p/13553373.html
13.0 在容器内获取Pod的信息
https://www.cnblogs.com/bigberg/p/13559228.html
14.0 部署Metrics Server
https://www.cnblogs.com/bigberg/p/13559251.html
15.0 Pod健康检查机制
https://www.cnblogs.com/bigberg/p/13559308.html
16.0 HPA Pod自动扩缩容
https://www.cnblogs.com/bigberg/p/13559410.html
17.0 Job and CronJob
https://www.cnblogs.com/bigberg/p/13563069.html
18.0 部署Ingress
https://www.cnblogs.com/bigberg/p/13563079.html
19.0 部署EFK日志系统
https://www.cnblogs.com/bigberg/p/13571127.html
20.0 部署SonarQube
https://www.cnblogs.com/bigberg/p/13575685.html
21.0 Elasticsearch + Kibana + Filebeat 日志收集
https://www.cnblogs.com/bigberg/p/13551964.html
22.0 Filebeat定义index
https://www.cnblogs.com/bigberg/p/13551987.html