二进制安装kubernetes v1.11.2 (第五章 flannel网络插件部署)

继续部署。

五、部署flannel网络插件

kubernetes要求集群内各节点(包括master节点)能通过pod网段互联互通。flannel使用vxlan技术为各节点创建一个可以互通的pod网络,使用的端口为UDP 8472,需要开放该端口(如公有云aws等)。

flannel第一次启动时,从etcd获取pod网段信息,为本节点分配 一个未使用的/24段地址,然后创建flannel开头的一个接口。

flannel将分配的pod网段信息写入/run/flannel/docker文件,docker后续使用这个文件中的环境变量设置docker0网桥。

5.1 下载二进制文件

下载地址:https://github.com/coreos/flannel/releases

# 下载二进制文件
[k8s@k8s-m1 ~]$ mkdir -p /home/k8s/k8s/flannel-v0.10.0 && cd /home/k8s/k8s
[k8s@k8s-m1 k8s]$ wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
[k8s@k8s-m1 k8s]$ tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz -C flannel-v0.10.0

# 分发到所有节点
[k8s@k8s-m1 k8s]$ source /opt/k8s/bin/environment.sh
[k8s@k8s-m1 k8s]$ for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp  flannel-v0.10.0/{flanneld,mk-docker-opts.sh} k8s@${node_ip}:/opt/k8s/bin/
    ssh k8s@${node_ip} "chmod +x /opt/k8s/bin/*"
done

5.2 创建flannel证书和私钥

flannel从etcd集群存取网段分配信息,而etcd集群启用了双向x509证书认证,所以需要为flannel生成证书和私钥。

# 创建证书签名请求
[k8s@k8s-m1 cert]$ cd /opt/k8s/cert
[k8s@k8s-m1 cert]$ cat > flanneld-csr.json <<EOF
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

# 生成证书和私钥
[k8s@k8s-m1 cert]$ cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
  -ca-key=/etc/kubernetes/cert/ca-key.pem \
  -config=/etc/kubernetes/cert/ca-config.json \
  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
[k8s@k8s-m1 cert]$ ls flanneld*pem


# 分发证书和私钥到各节点
[k8s@k8s-m1 cert]$ for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld"
    scp flanneld*.pem k8s@${node_ip}:/etc/flanneld/cert
done

5.3 向etcd写入集群pod网段信息

source /opt/k8s/bin/environment.sh
etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
  • flannel当前版本(v0.10.0)不支持etcd v3,所以使用etcd v2 API写入配置key和网段数据;
  • 写入的pod网段${CLUSTER_CIDR}必须是/16段地址必须与kube-controller-manager的--cluster-cidr参数值一致;

5.4 创建flanneld的system unit文件

source /opt/k8s/bin/environment.sh
export IFACE=eth0
cat > flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\
  -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\
  -etcd-endpoints=${ETCD_ENDPOINTS} \\
  -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
  -iface=${IFACE}
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF
  • mk-docker-opts.sh脚本将分配给flanneld的pod子网网段信息写入/run/flannel/docker文件,后续docker启动时使用这个文件中的环境变量配置docker0网桥
  • flanneld使用系统默认路由所在的接口与其他节点通信,对于有多个网络接口(如内网和公网)的节点,可以用-iface参数指定通信接口,如上面的enp0s3接口
  • flanneld运行时需要root权限

完整的unit文件如下

[k8s@k8s-m1 template]$ cat flanneld.service 
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \
  -etcd-certfile=/etc/flanneld/cert/flanneld.pem \
  -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \
  -etcd-endpoints=https://192.168.56.20:2379,https://192.168.56.21:2379 \
  -etcd-prefix=/kubernetes/network \
  -iface=
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

5.5 分发flanneld systemd unit文件到所有节点

source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp flanneld.service root@${node_ip}:/etc/systemd/system/
done

5.6 启动flanneld服务

source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"
done

5.7 检查分配给各flanneld的pod网段信息

查看集群pod网段(/16):

source /opt/k8s/bin/environment.sh
etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  get ${FLANNEL_ETCD_PREFIX}/config

输出:

{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}

查看已经分配的pod子网列表(/24):

source /opt/k8s/bin/environment.sh
etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  ls ${FLANNEL_ETCD_PREFIX}/subnets

输出:

/kubernetes/network/subnets/172.30.17.0-24
/kubernetes/network/subnets/172.30.5.0-24
/kubernetes/network/subnets/172.30.75.0-24
/kubernetes/network/subnets/172.30.57.0-24

查看某一pod网段对应节点IP和flannel接口地址:

source /opt/k8s/bin/environment.sh
etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.17.0-24

输出:

{"PublicIP":"192.168.72.68","BackendType":"vxlan","BackendData":{"VtepMAC":"c2:d6:2a:e4:96:ce"}}

5.8 验证各节点能通过pod网段互通

在各节点部署flannel后,检查是否创建了flannel接口

source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet"
done

输出:

>>> 192.168.56.20
    inet 172.30.57.0/32 scope global flannel.1
>>> 192.168.56.21
    inet 172.30.17.0/32 scope global flannel.1
>>> 192.168.56.30
    inet 172.30.5.0/32 scope global flannel.1
>>> 192.168.56.31
    inet 172.30.75.0/32 scope global flannel.1

在各节点上ping所有flannel接口ip,确保能通:

source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh ${node_ip} "ping -c 1 172.30.57.0"
    ssh ${node_ip} "ping -c 1 172.30.17.0"
    ssh ${node_ip} "ping -c 1 172.30.5.0"
    ssh ${node_ip} "ping -c 1 172.30.75.0"
done

 

posted on 2018-10-24 11:44  冰冰爱学习  阅读(360)  评论(0编辑  收藏  举报

导航