kubeadm部署kubernetes-1.12.0 HA集群-ipvs
一、概述
主要介绍搭建流程及使用注意事项，如果线上使用的话，请务必做好相关测试及压测。
1、基础环境准备
系统：ubuntu TLS 16.04 5台
docker-ce：17.06.2
kubeadm、kubelet、kubectl：1.12.0
keepalived、haproxy
etcd-3.2.22
2、安装前准备
1）k8s各节点SSH免密登录。
2）各Node必须关闭swap：swapoff -a，否则kubelet启动失败。
vim /etc/sysctl.conf
vm.swappiness = 0
sysctl -p
3）各节点主机名和IP加入/etc/hosts解析
etcd采用单独部署的集群。
二、部署准备
1、安装docker：
apt-get update apt-get install \ apt-transport-https \ ca-certificates \ curl \ software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - apt-key fingerprint 0EBFCD88 add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" apt-get update apt policy docker-ce 找到自己需要的版本，然后安装 apt install -y docker-ce=17.06.2~ce-0~ubuntu
修改docker的一些设置
vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --graph=/data/docker --storage-driver=overlay2
配置docker的proxy，否则无法直接下载镜像。
mkdir /etc/systemd/system/docker.service.d
cat http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://10.42.3.110:8118" "NO_PROXY=localhost,127.0.0.1,0.0.0.0,10.42.73.110,10.10.25.49,10.2.140.154,10.42.104.113,10.2.177.142,10.2.68.77,172.11.0.0,172.10.0.0,172.11.0.0/16,172.10.0.0/16,10.,172.,.evo.get.com,.kube.hp.com,charts.gitlab.io,.mirror.ucloud.cn"
cat https-proxy.conf
[Service]
Environment="HTTPS_PROXY=http://10.42.3.110:8118" "NO_PROXY=localhost,127.0.0.1,0.0.0.0,10.42.73.110,10.10.25.49,10.2.140.154,10.42.104.113,10.2.177.142,10.2.68.77,172.11.0.0,172.10.0.0,172.11.0.0/16,172.10.0.0/16,10.,172.,.evo.get.com,.kube.hp.com,charts.gitlab.io,.mirror.ucloud.cn"
systemctl daemon-reload
systemctl restart docker
iptables设置，17.06.2默认的FORWARD是DROP，这样pod之间的通信就会收到影响，所以需要设置成ACCEPT
iptables -P FORWARD ACCEPT
iptables-save
2、安装CA证书
1）安装cfssl证书管理工具
mkdir /opt/bin
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /opt/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /opt/bin/cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /opt/bin/cfssl-certinfo
echo "export PATH=/opt/bin:$PATH" > /etc/profile.d/k8s.sh
source /etc/profile.d/k8s.sh
2)创建CA配置文件
mkdir ssl;cd ssl
cfssl print-defaults config >config.json
cfssl print-defaults csr >csr.json
根据生成的文件创建ca-config.json文件，过期时间设置为87600h
# cat ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
创建CA证书签名请求
# cat ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GD",
"L": "SZ",
"O": "k8s",
"OU": "System"
}
]
}
生成证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
copy证书到相应节点
mkdir /etc/kubernetes/ssl -p
cp ~/ssl/ca* /etc/kubernetes/ssl
scp -r /etc/kubernetes/ km12-02:/etc
scp -r /etc/kubernetes/ km12-03:/etc
3、下载etcd，新版本的k8s要求etcd版本3.2.18以上，这里使用3.2.22
wget https://github.com/coreos/etcd/releases/download/v3.2.22/etcd-v3.2.22-linux-amd64.tar.gz tar xf etcd-v3.2.22-linux-amd64.tar.gz cd etcd-v3.2.22-linux-amd64 cp etcd etcdctl /opt/bin
source /etc/profile.d/k8s.sh
制作etcd证书：
创建签名文件
# cat etcd-csr.json
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"10.42.13.230",
"10.42.43.147",
"10.42.150.212"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GD",
"L": "SZ",
"O": "k8s",
"OU": "System"
}
]
}
生成证书和私钥
cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubernetes/ssl/ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
mkdir /etc/etcd/ssl -p
cp etcd*.pem /etc/etcd/ssl
scp -r /etc/etcd/ km12-02:/etc
scp -r /etc/etcd/ km12-03:/etc
部署etcd集群：
创建etcd的systemd unit文件
# mkdir -p /var/lib/etcd
# cat /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/opt/bin/etcd \
--name=etcd-host0 \
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem \
--peer-cert-file=/etc/etcd/ssl/etcd.pem \
--peer-key-file=/etc/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--initial-advertise-peer-urls=https://10.42.13.230:2380 \
--listen-peer-urls=https://10.42.13.230:2380 \
--listen-client-urls=https://10.42.13.230:2379,http://127.0.0.1:2379 \
--advertise-client-urls=https://10.42.13.230:2379 \
--initial-cluster-token=etcd-cluster-1 \
--initial-cluster=etcd-host0=https://10.42.13.230:2380,etcd-host1=https://10.42.43.147:2380,etcd-host2=https://10.42.150.212:2380 \
--initial-cluster-state=new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
将创建好的启动文件和之前解压的etcd文件都copy到其他master节点上
tar xf etcd-v3.2.22-linux-amd64.tar.gz
cd etcd-v3.2.22-linux-amd64
cp etcd etcdctl /opt/bin
scp -r /opt/bin km12-02:/opt
scp -r /opt/bin km12-03:/opt
source /etc/profile.d/k8s.sh
copy到其他节点 ，注意修改几个地方，修改为相应的名字和地址：
--name=etcd-host0
--initial-advertise-peer-urls=https://10.42.13.230:2380
--listen-peer-urls=https://10.42.13.230:2380
--listen-client-urls=https://10.42.13.230:2379,http://127.0.0.1:2379
--advertise-client-urls=https://10.42.13.230:2379
启动etcd
systemctl enable etcd
systemctl start etcd
systemctl status etcd
查看集群状态
etcdctl --key-file /etc/etcd/ssl/etcd-key.pem --cert-file /etc/etcd/ssl/etcd.pem --ca-file /etc/kubernetes/ssl/ca.pem member list
etcdctl --key-file /etc/etcd/ssl/etcd-key.pem --cert-file /etc/etcd/ssl/etcd.pem --ca-file /etc/kubernetes/ssl/ca.pem cluster-health
为了方便使用etcdctl我们可以做一个alias,因为kubernetes默认使用etcdv3，而etcd部署默认使用etcdv2，所以在etcd里面查看不到kubernetes的数据，需要添加一个全局变量export ETCDCTL_API=3,
在所有master节点上添加如下内容：
# vim ~/.bashrc
#etcd
host1='10.42.13.230:2379'
host2='10.42.43.147:2379'
host3='10.42.150.212:2379'
endpoints=$host1,$host2,$host3
alias etcdctl='etcdctl --endpoints=$endpoints --key /etc/etcd/ssl/etcd-key.pem --cert /etc/etcd/ssl/etcd.pem --cacert /etc/kubernetes/ssl/ca.pem'
export ETCDCTL_API=3
是内容生效：
#source ~/.bashrc
4、在另外的两台服务器上面部署keepalived、haproxy
apt install -y keepalived haproxy
1)创建keepalived配置文件
master:
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost #发送邮箱
}
notification_email_from keepalived@localhost #邮箱地址
smtp_server 127.0.0.1 #邮件服务器地址
smtp_connect_timeout 30
router_id km12-01 #主机名，每个节点不同即可
}
vrrp_instance VI_1 {
state MASTER #在另一个节点上为BACKUP
interface eth0 #IP地址漂移到的网卡
virtual_router_id 6 #多个节点必须相同
priority 100 #优先级，备用节点的值必须低于主节点的值
advert_int 1 #通告间隔1秒
authentication {
auth_type PASS #预共享密钥认证
auth_pass 571f97b2 #密钥
}
unicast_src_ip 10.42.13.23
unicast_peer {
10.42.43.14
}
virtual_ipaddress {
10.42.79.210/16 #VIP地址
}
}
slave1:
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost #发送邮箱
}
notification_email_from keepalived@localhost #邮箱地址
smtp_server 127.0.0.1 #邮件服务器地址
smtp_connect_timeout 30
router_id km12-02 #主机名，每个节点不同即可
}
vrrp_instance VI_1 {
state BACKUP #在另一个节点上为MASTER
interface eth0 #IP地址漂移到的网卡
virtual_router_id 6 #多个节点必须相同
priority 80 #优先级，备用节点的值必须低于主节点的值
advert_int 1 #通告间隔1秒
authentication {
auth_type PASS #预共享密钥认证
auth_pass 571f97b2 #密钥
}
unicast_src_ip 10.42.43.14
unicast_peer {
10.42.13.23
}
virtual_ipaddress {
10.42.79.210/16 #漂移过来的IP地址
}
}
启动keepalived
systemctl enable keepalived
systemctl start keepalived
keepalived采用的是单播。
2)haproxy配置文件
cat /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
defaults
mode tcp
log global
retries 3
timeout connect 10s
timeout client 1m
timeout server 1m
frontend kubernetes
bind *:6443
mode tcp
default_backend kubernetes-master
backend kubernetes-master
balance roundrobin
server master 10.42.13.230:6443 check maxconn 2000
server master2 10.42.43.147:6443 check maxconn 2000
两台配置文件是一样的。
启动haproxy
systemctl enable haproxy
systemctl start haproxy
5、安装指定版本的kubeadm、kubectl、kubelet
由于服务器不FQ，下载会有问题，所以我们需要换一下源：
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
EOF
安装
apt-get update
apt policy kubeadm
apt-get install kubeadm=1.12.0-00 kubectl=1.12.0-00 kubelet=1.12.0-00
升级软件
apt-get upgrade kubeadm
回退版本
apt-get install aptitude
比如升到了kubeadm1.13.0，想退回到1.12.0
aptitude install kubeadm=1.12.0-00
卸载软件
apt-get --purge remove 包名称
如果不记得包的全部名称，则可以使用如下命令查找
dpkg --get-selections | grep kube
创建初始化的kubeadm-config.yaml
# cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1alpha3
controlPlaneEndpoint: 10.42.9.210:6443
kind: ClusterConfiguration
kubernetesVersion: v1.12.0
networking:
podSubnet: 192.244.0.0/16
serviceSubnet: 192.96.0.0/16
apiServerCertSANs:
- 10.42.13.230
- 10.42.43.147
- 10.42.150.212
- km12-01
- km12-02
- km12-03
- 127.0.0.1
- localhost
- 10.42.9.210
etcd:
external:
endpoints:
- https://10.42.13.230:2379
- https://10.42.43.147:2379
- https://10.42.150.212:2379
caFile: /etc/kubernetes/ssl/ca.pem
certFile: /etc/etcd/ssl/etcd.pem
keyFile: /etc/etcd/ssl/etcd-key.pem
dataDir: /var/lib/etcd
clusterDNS:
- 10.96.0.2
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
controlPlaneEndpoint是haproxy的地址，或vip
检查是否加载了ipvs：
lsmod | grep ip_vs
如果没有输出，则证明没有加载ipvs，需要手动加载，这里为了方便写了一个脚本：
# cat ipvs.sh
#!/bin/bash
list="ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4"
for i in $list
do
modprobe $i
done
初始化
kubeadm init --config kubeadm-config.yaml
如果是已经初始化完成的集群想要修改为ipvs，做如下修改：
kubectl edit configmap kube-proxy -n kube-system
ipvs:
excludeCIDRs: null
minSyncPeriod: 0s
scheduler: ""
syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
重新部署kube-proxy:
kubectl delete pod kube-proxy-ppsk90sd -n kube-system
将生成的pki拷贝到其他节点
scp -r /etc/kuberbetes/pki km12-02:/etc/kubernetes
scp -r /etc/kubernetes/pki km12-03:/etc/kubernetes
注意，到其他服务器上面删掉里面的两个文件：
rm -rf apiserver.crt apiserver.key
将初始化的文件以及/etc/profile.d/k8s.conf拷贝到其他节点
scp -r kubeadm-config.yaml km12-02:
scp -r kubeadm-config.yaml km12-03:
scp -r /etc/profile.d/k8s.sh km12-02:/etc/profile.d
scp -r /etc/profile.d/k8s.sh km12-03:/etc/profile.d
有与所有的节点都设置了taints，所以都不能部署，此时会看到coredns一直处于pending的状态，我们暂时把taints去掉，使其能正常部署
kubectl taint nodes --all node-role.kubernetes.io/master-
6、部署网络
网络有几种选择flannel+vxlan、flannel+host-gw、calico+BGP,经过测试calico+BGP这种方式网络性能是比较好的，这里给出两种方式部署网络flannel+host-gw和calico+BGP。推荐使用calico+BGP。
第一种calico+BGP：
calico的datastore有两种方式：kubernetes和etcd，这里建议使用etcd进行数据存储。calico默认使用ipip模式，如果想关闭ipip模式只需要将CALICO_IPV4POOL_IPIP设置为Off，然后就会使用BGP。
下载文件：
wget https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/hosted/calico.yaml
etcd使用前面部署好的etcd集群，所以需要修改calico.yaml文件：
主要修改几个地方：一个是etcd的相关配置，一个是CALICO_IPV4POOL_CIDR
# vim calico.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: calico-config
namespace: kube-system
data:
# Configure this with the location of your etcd cluster.
etcd_endpoints: "https://10.42.13.230:2379,https://10.42.43.147:2379,https://10.42.150.212:2379"
# If you're using TLS enabled etcd uncomment the following.
# You must also populate the Secret below with these files.
etcd_ca: "/calico-secrets/etcd-ca"
etcd_cert: "/calico-secrets/etcd-cert"
etcd_key: "/calico-secrets/etcd-key"
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: calico-etcd-secrets
namespace: kube-system
data:
# Populate the following files with etcd TLS configuration if desired, but leave blank if
# not using TLS for etcd.
# This self-hosted install expects three files with the following names. The values
# should be base64 encoded strings of the entire contents of each file.
etcd-key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFB0NBUUVBd2F3ZHNKcHB5WTR1UjFtZXJocHl5dWlPU1B3N0lWS1Zxc3dXaFV6ZG1YVWRzN2swCmlMaVNGbDhIUjRaVDcvcEIzOGRuenJINE1pQit4TTRDVHRKVkZjTFJBK2tkNUwzZWdodC9KOHlBS3UyVnE2M2wKTWpBazlIb3NqUU96a2dLSnN1RlhyRFdaZG5FeGlvNkZrQXBncDFySmZJdFh1UkpFdDFRMGplblVTM0xLaFZVWQpkdTdTV0thVkc5T3RmS3BzZUsvc2tYcDJlVmJIdGpiRE9lQURodnhyNk1FcDV4SVJ2bmZlWXppWmpUaUJXMkUrCkdFOCtjTGVCTmVLMVYxWFYvR1pmR0FTN2orWFVLOEIxVHFEQ2ZzU3QrRTEzTGMvQWhnTTF2QTZjaDIzMm16RlUKWWw0L21VaU5ZeUFHZmNkTW9XMkR4ck96M28yQ0JoMm50R1FIbndJREFRQUJBb0lCQVFDdUF2dGFKaU5PbVpVegpQbTY0ZDhNVVZRY2l4SjVNUG11RFhvUU5tUmF5UHV4RkF1OFV1T0ZwZ0ZDR0QwZHpveU4wOFRGd0lhK2pPbGNzCmRQelYzVjNkZzRGUlJpODV5dThWYjZjOEZ4NUJjOTY2dGkvc2ZCMTRIQjNUUmJRZVNIRjRhOGZlVHZwbjFTUnkKSGNRYnUzZEVadW9LSVJqY2pIRjVWd3N4UnhzYi9hSGE3MzUzR2pwTDhsZThQVGtielVyVktGYkZDN1U2cE1IYwo3MGxTeW5uUUVmNC9QdjJFcGNWa1hjQVFvWk44MkJUNWNEbTFheWhaWWhaUk5wdEhmV0w5VWtHaUZma3h5TXRnCjllSFJPNFpoMnZ6RW12QWRGbTZKcjB3aDlOQVVMUDhkNlh5OStnUFJ0Q1I4K2p0WHd3SzgwazZZMlpTWXhSTS8KYlAxYytyc0JBb0dCQU1xdk12cHVjTTVrc04vdkl0NE9Uc09lNE5QMzhSQVdaWktycmt4TjEwTXFrSEM5V0cxNQpwYmcyWGVyN2ZYTUFycmtpZUhiUU5ySzB6OTVVN3VnMzMrZ3VvVzRMNUJrYlNYVHY0eDJqMFBuQ3lhOFlMNkwrCjhuTi95SFBmUW1scnZWazJOd1FPSzgyaHQwL0RuOHFQeENNZVY1RmJUOWNjVVN2ZEI1czZqMHcvQW9HQkFQU2UKQzFTRnhJYThFSzVobGxZdXhQSUFHd29BNEIxV2R3RXRLMUF2WjRaamdCdmJqRjEySi9yMGNoQ08zbnllbThGdQovTU51WnczM1hsTThLTmJJWGpja3Y5RG9zQ2lOUXVtRE1ZUVY0SGFCL3U2SHo0MkNiaW1IMUlOSmtacDg5eS9nClRsQmRzb3puMUZiMG9aSFBsamcxR3BlM1hlUW9IZjJnRFFtSFJleWhBb0dBQjhXYURkSklUeEc3V29yZjZtWGcKRFU0OVRzTjJJZWpKOGtyVS95WUkrQkZjd29Yd0t4cDhDVWxFWnF0SUR5M2FoVXpXRTdBK09MSUlPbjFGUC9CTQpqS21sRlFRdHRleSs3MVgzZm41MmwrMHBNS2FieFFORHByd2lvcGJRQkJ6V2dPSThUOUovU1g1RytpOEZKSTVJCnRoUUd4Wk1ieDZMVlZmbyt2V3dNYko4Q2dZQkVEUk9wMSt6c2JyVlZUZUM4NlJYeEhRWm9xZ1d2STdiSHBRRS8dFhCZmVwN1JJU3JUZFdONTlUY21WQmloSXA3Q0dWWklLQmFUVkJYeG9mTGFqYk5vTTlrSkRUSzBsVmZnRHBkZgozNVlxWWMvQ0hCWXBqL2VGcGp3QXFoN1BrNlJRdFY0VURYejJwaWYwYU9ucDNvNHo1TklaRXZJVDN2VTQ4YVd1CjJPc2pBUUtCZ0c0OXZiNjNyc1p0eHZ3ZkhSaUVPUWdZZE9uWkhET2I0K2MxU2hpQWluN0hpOGRjdFR3cnl4a2UKdzBwZzV0R09xa1d1QUJ5VTFSQWR1c0pnUnZQZDBFc2l2YVBVTXhmd0cyWjBEVFpIZGVjS0tjWkNrdXkxQ3NKdgpkVEJLbnRjNmhwUUUrRnl3eHpBc1VTSCtpYUJ4YzczRlp3SWEwWEw2T1BuOFpnTnl5Qi9lCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg
etcd-cert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ0akNDQXNxZ0UJBZ0lVWFdsazdGY3JJR005T3Nxc08vOVBqTitjWGlrd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1d6RUxNQWtHQTFVRUJoTUNRMDR4Q3pBSkJnTlZCQWdUQWtkRU1Rc3dDUVlEVlFRSEV3SlRXakVNTUFvRwpBMVVFQ2hNRGF6aHpNUTh3RFFZRFZRUUxFd1pUZVhOMFpXMHhFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13CkhoY05NVGd4TVRBeE1EVTBNVEF3V2hjTk1qZ3hNREk1TURVME1UQXdXakJWTVFzd0NRWURWUVFHRXdKRFRqRUwKTUFrR0ExVUVDQk1DUjBReEN6QUpCZ05WQkFjVEFsTmFNUXd3Q2dZRFZRUUtFd05yT0hNeER6QU5CZ05WQkFzVApCbE41YzNSbGJURU5NQXNHQTFVRUF4TUVaWFJqWkRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDCkFRb0NnZ0VCQU1Hc0hiQ2FhY21PTGtkWm5xNGFjc3JvamtqOE95RlNsYXJNRm9WTTNabDFIYk81TklpNGtoWmYKQjBlR1UrLzZRZC9IWjg2eCtESWdmc1RPQWs3U1ZSWEMwUVBwSGVTOTNvSWJmeWZNZ0NydGxhdXQ1VEl3SlBSNgpMSTBEczVJQ2liTGhWNncxbVhaeE1ZcU9oWkFLWUtkYXlYeUxWN2tTUkxkVU5JM3AxRXR5eW9WVkdIYnUwbGltCmxSdlRyWHlxYkhpdjdKRjZkbmxXeDdZMnd6bmdBNGI4YStqQktlY1NFYjUzM21NNG1ZMDRnVnRoUGhoUFBuQzMKZ1RYaXRWZFYxZnhtWHhnRXU0L2wxQ3ZBZFU2Z3duN0VyZmhOZHkzUHdJWUROYndPbklkdDlwc3hWR0plUDVsSQpqV01nQm4zSFRLRnRnOGF6czk2TmdnWWRwN1JrQjU4Q0F3RUFBYU9Cb3pDQm9EQU9CZ05WSFE4QkFmOEVCQU1DCkJhQXdIUVlEVlIwbEJCWXdGQVlJS3dZQkJRVUhBd0VHQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQ01BQXcKSFFZRFZSME9CQllFRkdDa2lWS0RwcEtWUHp3TVkrU1hWY2NoeFB4OE1COEdBMVVkSXdRWU1CYUFGSlFEaTdGNQp2WlNaSFhFVTlqK0FXVE9Xemx6TU1DRUdBMVVkRVFRYU1CaUhCSDhBQUFHSEJBb3FqSnFIQkFvcWFIR0hCQW9xCnNZNHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQmlkNUh2VmNGUTNlUjhBZXNHekhaRHRsWUZsNGVSUVlsZ0kKd0orbC9XV09QamwxL0lwSDJtanBsWHlnNlJBUmhIeENnZXY1NG1BSGhyK0U4TktBOXMyZFdMNWpnd2twL2p3aApzaDhsUEljeVE2ZHdXdjJNYWZaTU1Xdng5OWIrd2wyRHRGQUpiTzM5bWU4QVFJc2FrRGgrV2ZlZk02dW5QVkdHClhqMjVFbjVwQlB4NHFyRTRUcTVmeURaYUF1ay9tN0NianQ2WklaUUthRjVwYmczMVpCNjZRTVdQbjYvU05lWmkKbEhkWHY3c2xweExvWkZFczdXdEpxM0VUcDE4aXRjTER3S3FDWGZaVzhVZkRWaWNHdlRNYmpBb3NXSG1uaFdnZwpEc0w5MVJhUW9qTTzZnd1d0VaZ3lMaEhrQi96NFR1UUFXT1NaRWU3clJ0ckx0UFNNdVE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
etcd-ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURxakNDQXBLZ0F3SUJBZ0lVR0tCU2dDUTdVU2t6b2E4WnRlNTJ6UERGUk1zd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1d6RUxNQWtHQTFVRUJoTUNRMDR4Q3pBSkJnTlZCQWdUQWtkRU1Rc3dDUVlEVlFRSEV3SlRXakVNTUFvRwpBMVVFQ2hNRGF6aHpNUTh3RFFZRFZRUUxFd1pUZVhOMFpXMHhFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13CkhoY05NVGd4TVRBeE1EVXpPREF3V2hjTk1qTXhNRE14TURVek9EQXdXakJiTVFzd0NRWURWUVFHRXdKRFRqRUwKTUFrR0ExVUVDQk1DUjBReEN6QUpCZ05WQkFjVEFsTmFNUXd3Q2dZRFZRUUtFd05yT0hNeER6QU5CZ05WQkFzVApCbE41YzNSbGJURVRNQkVHQTFVRUF4TUthM1ZpWlhKdVpYUmxjekNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBTjdLd0ZJUlFNL2hTZTdZdTRsSTEyVDRLdzJHYVc4bnA4YU9XZXZzNzhGQU9qYnAKQnBsSEZVQmFrVVcrN3pnU0FHQVMzRnU2Um14ZGk1T2c3a2dZUG9TbitNRDFkWjVIYm1oQjhCY2RtOGVnQnZRcApONUtlS0ovOTJTN09SSm9VeHpmWEhUN3V5bTlZZUhSa0owaTFzSlNJWVB5dXJUbGhOUGs2K3B6aEUzN3Rhd1NkClNHM3l5UlJOY25BOHFyRE5nQlVyUUJBVVRwa0N5SklyVG1oOEpqckl1YTdKMml4YTRidXo3ZWxGRURDYTJpcTcKVmNQZkZ3MVdnM09pMVNmSWVxWVlTUW0zQkpiRnZ2cjBCaUQzYVM5cDNJenNoNEtRYlkyUlhrKzZxTWxYb2ZNaAo4Vmp0MVU2Tmh4RHlkZGwwVDJrRWNwRkx3dWk4ZFRRSEtndU4yQTBDQXdFQUFhTm1NR1F3RGdZRFZSMFBBUUgvCkJBUURBZ0VHTUJJR0ExVWRFd0VCL3dRSU1BWUJBZjhDQVFJd0hRWURWUjBPQkJZRUZKUURpN0Y1dlpTWkhYRVUKOWorQVdUT1d6bHpNTUI4R0ExVWRJd1FZTUJhQUZKUURpN0Y1dlpTWkhYRVU5aitBV1RPV3psek1NQTBHQ1NxRwpTSWIzRFFFQkN3VUFBNElCQVFBamRMakRoMGN2bnNSai9STVhwbWFZRnEwaFowcmZnVmtWcnowK1VDdWRkZnZ3ClVISVEva0p2RFVCdDl5WEtud3ZqQ0JKR3grMWsySTlENVVUcDI4bUpXcUpFbXdWcllqaXo4eUhYamNVbUlNVWkKeHFrZUlwU2lYR0tVZ3NDV1d0U0cwR2Vnbi9telVPMERwVHBVMkdyM0ZjbXJPcGlVNFd0RURZVUZtUm4yMXc0Wgo1S3RnSkpPNWFXaGFMR3c2VU52MFVhRUhTTjZoUk9qWG9pMy8zK09KN1pRajc1YXFnU1k0TURheDZBSFZ4NENSCnk2Q0srNUx0NWhNc0lKVkhETjZkQXYyTlFBU0VxSFVZazJmVFJ1VjJjeExIcE1MQVVTc1JRaThZWmJwYys2TUMKZjkxOHp3MG03NjZ3eTdqYVFJYmJTTXJzbGUvTnhoZnFTQSs0anNFdwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
etcd-key：填写cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n' 的输出
etcd-cert：填写 cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n' 的输出
etcd-ca：填写cat /etc/kubernetes/ssl/ca.pem | base64 | tr -d '\n' 的输出
- name: CALICO_IPV4POOL_CIDR
value: "192.244.0.0/16"
CALICO_IPV4POOL_CIDR的value需要与初始化设置的 podSubnet: 192.244.0.0/16相同，不能和serviceSubnet重复。
# Auto-detect the BGP IP address.
- name: IP
value: "autodetect"
- name: IP_AUTODETECTION_METHOD
value: "can-reach=8.8.8.8"
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
value: "Off"
部署
kubectl create-f calico.yaml
删除calico
kubectl delete -f calico.yaml modprobe -r ipip
ifconfig cali开头的网卡 down
ip link delete cali开头的网卡
etcdctl del --prefix /calico
QA：部署过程中可能会碰到如下错误：
2018-11-19 08:51:49.719 [INFO][8] startup.go 264: Early log level set to info
2018-11-19 08:51:49.719 [INFO][8] startup.go 284: Using HOSTNAME environment (lowercase) for node name
2018-11-19 08:51:49.719 [INFO][8] startup.go 292: Determined node name: ui
2018-11-19 08:51:49.720 [INFO][8] startup.go 105: Skipping datastore connection test
2018-11-19 08:51:49.721 [INFO][8] startup.go 365: Building new node resource Name="ui"
2018-11-19 08:51:49.721 [INFO][8] startup.go 380: Initialize BGP data
2018-11-19 08:51:49.722 [INFO][8] startup.go 582: Using autodetected IPv4 address on interface br-99ef8e85b719: 172.18.0.1/16
2018-11-19 08:51:49.722 [INFO][8] startup.go 450: Node IPv4 changed, will check for conflicts
2018-11-19 08:51:49.722 [WARNING][8] startup.go 879: Calico node 'ku' is already using the IPv4 address 172.18.0.1.
2018-11-19 08:51:49.722 [WARNING][8] startup.go 1104: Terminating
Calico node failed to start
导致node节点的calico-node不能启动，这个主要是ip地址冲突导致，因为calico里面有个参数IP_AUTODETECTION_METHOD默认值是first-found.官方解释：
The method to use to autodetect the IPv4 address for this host. This is only used when the IPv4 address is being autodetected. See IP Autodetection methods for details of the valid methods. [Default: first-found]
https://docs.projectcalico.org/v3.4/reference/node/configuration
由于我们使用的ipv4，如果是ipv6的花还需要设置IP6_AUTODETECTION_METHOD。
这里的解决方式是在calico.yaml文件里面添加如下修改：
# Auto-detect the BGP IP address.
- name: IP
value: "autodetect"
- name: IP_AUTODETECTION_METHOD
value: "can-reach=8.8.8.8"
重新部署calico即可解决。
QA: 之前使用calico+ipip的模式，升级到1.13.0的时候出现一个请求长度的限制，超过1400就会被拒绝掉，切换到BGP模式就解决了这个问题，所以直接使用BGP模式。
安装calicoctl：
https://docs.projectcalico.org/v3.4/usage/calicoctl/install
我这里采用binary的方式安装
curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.3.2/calicoctl
chmod +x calicoctl
我们之前是将数据存储到etcd里面，所以为calicoctl创建如下配置文件：
# cat /etc/calico/calicoctl.cfg
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
etcdEndpoints: https://10.42.140.154:2379,https://10.42.104.113:2379,https://10.42.177.142:2379
etcdKeyFile: /etc/etcd/ssl/etcd-key.pem
etcdCertFile: /etc/etcd/ssl/etcd.pem
etcdCACertFile: /etc/kubernetes/ssl/ca.pem
第二种flannel+host-gw：
下载文件
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
需要作如下修改
vim kube-flannel.yml
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth0
如果是自己定义的 podSubnet: 10.244.0.0/16 和serviceSubnet: 10.96.0.0/16，则还需要修改一个地方，修改为 podSubnet：
net-conf.json: |
{
"Network": "192.244.0.0/16",
"Backend": {
"Type": "host-gw"
}
}
部署
kubectl apply -f kube-flannel.yaml
删除flannel
kubectl delete -f flannel.yaml
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/
三、添加node节点
默认每个节点使用kubeadm初始化的时候都会创建一个token，可以选择任意一个master节点的token来添加节点，但是这样管理起来不是很方便。所一先把节点的所有token都删掉，然后创建一个token
用来添加节点，token默认ttl是24小时，可以讲ttl设置为0，这样token就永久有效，不建议将ttl设置成0，设置成2天轮换一次。具体步骤如下：
1、删掉原油默认的token
kubeadm tokne list
找到默认的token，然后都删掉
kubeadm token delete l36ybs.04f5zkvewgtsd703
kubeadm token delete zvbdfg.9kx9fzozxfz8seko
kubeadm token delete zvfoiw.r7p0rxtvgxc95owu
使用kubeadm token create生成token
kubeadm token create --ttl 72h
为了方便使用，写了一个脚本：
# cat uptk.sh
#!/bin/bash
tk=`kubeadm token list | awk '{print $1}' |grep -v TOKEN`
kubeadm token delete $tk
kubeadm token create --ttl 72h
crontab -e
* * */2 * * /data/scripts/uptk.sh 2>&1 & >>/dev/null
关于kubeadm管理token更多细节参考官方文档：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-token/
2、添加节点
准备一台新的服务器，做好免密交互，安装kubeadm、kubelet、docker等，按照上面准备master那样，只是不执行初始化，而是kubeadm join，注意版本要一致：
在master上操作：
查看集群的token
kubeadm token list | grep authentication,signing | awk '{print $1}'
查看discovery-token-ca-cert-hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
为了方便使用将上述命令做成了一个生成脚本，需要在master节点上执行，当然可以根据服务器情况进行修改：
cat generate_kubeadm_join.sh
#!/bin/bash
#delete the invalid token and create new token
tk_old=`kubeadm token list | awk '{print $1}' | grep -v TOKEN`
kubeadm token delete $tk_old >>/dev/null 2>& 1
kubeadm token create --ttl 72h >>/dev/null 2>& 1
#print option which kubeadm join
tk_new=`kubeadm token list | awk '{print $1}' | grep -v TOKEN`
hs=`openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'`
port=6443
ip=`ip a | grep eth0 | grep inet| awk -F / '{print $1}'| awk '{print $2}'`
op="kubeadm join --token $tk_new $ip:$port --discovery-token-ca-cert-hash sha256:$hs"
echo $op
在node节点上操作：
根据上面获取的token和hash
kubeadm join --token c04f89.b781cdb55d83c1ef 10.42.9.210:6443 --discovery-token-ca-cert-hash sha256:986e83a9cb948368ad0552b95232e31d3b76e2476b595bd1d905d5242ace29af
在master上操作：
为node节点打标签
kubectl label node knode1 node-role.kubernetes.io/node=
删除节点：
在master节点上：
kubectl drain node1 --delete-local-data --force --ignore-daemonsets
kubectl delete node node1
在node节点上：
kubeadm reset
ps：node1 是节点名称
如果部署的有问题，可以使用如下的方法进行重置：
#kubeadm reset ifconfig cni0 down ip link delete cni0 ifconfig flannel.1 down ip link delete flannel.1 rm -rf /var/lib/cni/
rm -rf ~/.kube
ipvsadm --clear
如果使用的etcd是external的方式，也就是外部集群，使用kubeadm reset重置集群的时候并不能删除etcd里面的数据，需要使用etcdctl工具手动删除。
由于前面已经设置了alias，所以可以直接使用etcdctl：
etcdctl del "" --prefix
四、部署监控
1、部署metrics-server
git clone https://github.com/cuishuaigit/k8s-monitor.git
cd k8s-monitor/metrics-server
我都是把这种服务部署在master节点上面，此时需要修改metrics-server-deployment.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-server
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
tolerations:
- effect: NoSchedule
key: node.kubernetes.io/unschedulable
operator: Exists
- key: NoSchedule
operator: Exists
effect: NoSchedule
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server-amd64:v0.3.1
imagePullPolicy: Always
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
volumeMounts:
- name: tmp-dir
mountPath: /tmp
nodeSelector:
metrics: "yes"
为master添加label
kubectl label nodes ku metrics=yes
部署kubectl create -f metrics-server/deploy/1.8+/
如果有报错参考我的另外一片博客：https://www.cnblogs.com/cuishuai/p/9857120.html
接下来稍等几十秒就可以查看状态了
2、部署prometheus
下载相关文件
使用上面clone的k8s-monitor部署
cd k8s-monitor
1.搭建nfs服务动态提供持久化存储（如果已有可以跳过）
1.安装nfs
sudo apt-get install -y nfs-kernel-server
sudo apt-get install -y nfs-common
sudo vi /etc/exports
/data/opv *(rw,sync,no_root_squash,no_subtree_check)
注意将*换成自己的ip段，纯内网的话也可以用*，代替任意
sudo /etc/init.d/rpcbind restart
sudo /etc/init.d/nfs-kernel-server restart
sudo systemctl enable rpcbind nfs-kernel-server
客户端挂在使用
sudo apt-get install -y nfs-common
mount -t nfs ku13-1:/data/opv /data/opv -o proto=tcp -o nolock
为了方便使用将上面的mount命令直接放到.bashrc里面
2.创建namesapce
kubectl creaet -f nfs/monitoring-namepsace.yaml
3.为nfs创建rbac
kubectl create -f nfs/rbac.yaml
4.创建deployment
kubectl create -f nfs/nfs-deployment.yaml
5.创建storageclass
kubectl create -f nfs/storageClass.yaml
2.安装Prometheus
cd k8s-monitor/Promutheus/prometheus
1.创建权限
kubectl create -f rbac.yaml
2.创建 node-exporter
kubectl create -f prometheus-node-exporter-daemonset.yaml
kubectl create -f prometheus-node-exporter-service.yaml
3.创建 kube-state-metrics
kubectl create -f kube-state-metrics-deployment.yaml
kubectl create -f kube-state-metrics-service.yaml
4.创建 node-directory-size-metrics
kubectl create -f node-directory-size-metrics-daemonset.yaml
5.创建 prometheus
kubectl create -f prometheus-pvc.yaml
kubectl create -f prometheus-core-configmap.yaml
kubectl create -f prometheus-core-deployment.yaml
kubectl create -f prometheus-core-service.yaml
kubectl create -f prometheus-rules-configmap.yaml
6.修改core-configmap里的etcd地址
3.安装Grafana
cd k8s-monitor/Promutheus/grafana
1.安装grafana service
kubectl create -f grafana-svc.yaml
2.创建configmap
kubectl create -f grafana-configmap.yaml
3.创建pvc
kubectl create -f grafana-pvc.yaml
4.创建gragana deployment
kubectl create -f grafana-deployment.yaml
5.创建dashboard configmap
kubectl create configmap "grafana-import-dashboards" --from-file=dashboards/ --namespace=monitoring
6.创建job，导入dashboard等数据
kubectl create -f grafana-job.yaml
prometheus和grafana都是采用的nodePort方式暴漏的服务，所以可以直接访问。
grafana默认的用户名密码：admin/admin
