k8s升级，HA集群1.12.0～HA集群1.13.2

k8s升级，此次升级是1.12.0 至1.13.2

准备

# 首先升级master节点的基础组件kubeadm、kubelet、kubectl
apt policy kubeadm
找到相应的版本，如果没有，可以去github上面下载binary包进行安装：
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#changelog-since-v1131
然后替换之前的包，如果之前是用apt-get  install安装的，都是在/usr/bin下面

apt-get update && apt-get upgrade  kubeadm

# 更新到1.13.2后，在master执行,执行一次

kubeadm upgrade plan

[upgrade/versions] Latest version in the v1.12 series: v1.13.2

External components that should be upgraded manually before you upgrade the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Etcd 3.2.22 3.2.24

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.12.0 v1.13.2

Upgrade to the latest version in the v1.12 series:

COMPONENT CURRENT AVAILABLE
API Server v1.12.0 v1.13.2
Controller Manager v1.12.0 v1.13.2
Scheduler v1.12.0 v1.13.2
Kube Proxy v1.12.0 v1.13.2
CoreDNS 1.2.2 1.2.6

You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.13.2

需要先升级etcd服务，我们的版本是3.2.22，需要升级到3.2.24

# 更新
 kubeadm upgrade apply v1.13.2
# 备份etcd数据(以防万一)
 etcdctl snapshot save backup.db
 
# 查看备份的状态
 etcdctl --write-out=table snapshot status backup.db
 
# 下载新的etcd，这里我们部署1.13.2的kubernetes需要的是3.2.24
# (https://github.com/etcd-io/etcd/releases/download/v3.2.24/etcd-v3.2.24-linux-amd64.tar.gz)

  wget https://github.com/etcd-io/etcd/releases/download/v3.2.24/etcd-v3.2.24-linux-amd64.tar.gz

#由于之前部署的是三节点的etcd集群，所以可以做到无缝升级，一个节点一个节点进行升级，每个节点恢复正常后在升级下一个节点；
# 停掉当前节点
 systemctl stop etcd

# 将下载好的文件解压，然后替换之前的etcd和etcdctl，替换完成后启动etcd

 systemctl start etcd

# 每个节点替换完成后检查服务状态

 etcdctl endpoint health

Upgrade master and node packages

kubectl drain node_name --ignore-daemonsets

#master节点需要加上--ignore-daemonsets，node节点不用

#更新
apt-get update
apt-get upgrade -y  kubeadm

#更新过程中要保证服务不中断，轮询替换更新

Upgrade kubelet on each node

# On each node except the master node, upgrade the kubelet config:

kubeadm upgrade node config --kubelet-version $(kubelet --version | cut -d  ' '  -f 2) 

# Restart the kubelet process:

systemctl restart kubelet

# Verify that the new version of the kubelet is running on the node:

systemctl status kubelet

# Bring the node back online by marking it schedulable:

kubectl uncordon $NODE

# verify that all nodes are available again by running the following command from anywhere kubectl can access the cluster:

kubectl get nodes


#The STATUS column should show Ready for all your nodes, and the version number should be updated.

QA

使用calico+ipip的模式，升级到1.13.1的时候出现一个请求长度的限制，超过1400就会被拒绝掉.
解决方式有两种：
1、修改mtu的值，ipip模式回家一个ip头部，20字节。所以calico的mtu+20 必须小于 物理机网卡的mtu，也要小于1500.
2、切换到BGP模式就解决了这个问题