Flannel VxLAN 模式
Flannel VxLAN 模式
Flannel VxLAN 模式
一、环境信息
| 主机 | IP |
|---|---|
| ubuntu | 172.16.94.141 |
| 软件 | 版本 |
|---|---|
| docker | 26.1.4 |
| helm | v3.15.0-rc.2 |
| kind | 0.18.0 |
| clab | 0.54.2 |
| kubernetes | 1.23.4 |
| ubuntu os | Ubuntu 20.04.6 LTS |
| kernel | 5.11.5 内核升级文档 |
二、安装服务
kind 配置文件信息
$ cat install.sh
#!/bin/bash
date
set -v
# 1.prep noCNI env
cat <<EOF | kind create cluster --name=flannel-vxlan --image=kindest/node:v1.23.4 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
# kind 默认使用 rancher cni，cni 我们需要自己创建
disableDefaultCNI: true
# 定义节点使用的 pod 网段
podSubnet: "10.244.0.0/16"
nodes:
- role: control-plane
- role: worker
- role: worker
containerdConfigPatches:
- |-
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.evescn.com"]
endpoint = ["https://harbor.evescn.com"]
EOF
# 2.install cni
kubectl apply -f ./flannel.yaml
# 3.install necessary tools
# cd /opt/
# curl -o calicoctl -O -L "https://gh.api.99988866.xyz/https://github.com/containernetworking/plugins/releases/download/v0.9.0/cni-plugins-linux-amd64-v0.9.0.tgz"
# tar -zxvf cni-plugins-linux-amd64-v0.9.0.tgz
for i in $(docker ps -a --format "table {{.Names}}" | grep flannel)
do
echo $i
docker cp /opt/bridge $i:/opt/cni/bin/
docker cp /usr/bin/ping $i:/usr/bin/ping
docker exec -it $i bash -c "sed -i -e 's/jp.archive.ubuntu.com\|archive.ubuntu.com\|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list"
docker exec -it $i bash -c "apt-get -y update >/dev/null && apt-get -y install net-tools tcpdump lrzsz bridge-utils >/dev/null 2>&1"
done
flannel.yaml
配置文件
# flannel.yaml
---
kind: Namespace
apiVersion: v1
metadata:
name: kube-flannel
labels:
pod-security.kubernetes.io/enforce: privileged
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni-plugin
image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
command:
- cp
args:
- -f
- /flannel
- /opt/cni/bin/flannel
volumeMounts:
- name: cni-plugin
mountPath: /opt/cni/bin
- name: install-cni
image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel:v0.19.2
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel:v0.19.2
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: EVENT_QUEUE_DEPTH
value: "5000"
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
- name: xtables-lock
mountPath: /run/xtables.lock
- name: tun
mountPath: /dev/net/tun
volumes:
- name: tun
hostPath:
path: /dev/net/tun
- name: run
hostPath:
path: /run/flannel
- name: cni-plugin
hostPath:
path: /opt/cni/bin
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
- name: xtables-lock
hostPath:
path: /run/xtables.lock
type: FileOrCreate
flannel.yaml 参数解释
Backend.Type- 含义: 用于指定
flannel工作模式。 vxlan:flannel工作在vxlan模式。
- 含义: 用于指定
- 安装
k8s集群和flannel服务
# ./install.sh
Creating cluster "flannel-vxlan" ...
✓ Ensuring node image (kindest/node:v1.23.4) 🖼
✓ Preparing nodes 📦 📦 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
Set kubectl context to "kind-flannel-vxlan"
You can now use your cluster with:
kubectl cluster-info --context kind-flannel-vxlan
Thanks for using kind! 😊
- 查看安装的服务
root@kind:~# kubectl get pods -A -w
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-64897985d-875hz 1/1 Running 0 74s
kube-system coredns-64897985d-kdskq 1/1 Running 0 74s
kube-system etcd-flannel-vxlan-control-plane 1/1 Running 0 90s
kube-system kube-apiserver-flannel-vxlan-control-plane 1/1 Running 0 90s
kube-system kube-controller-manager-flannel-vxlan-control-plane 1/1 Running 0 88s
kube-system kube-flannel-ds-cwb2m 1/1 Running 0 52s
kube-system kube-flannel-ds-k29kt 1/1 Running 0 52s
kube-system kube-flannel-ds-t72t4 1/1 Running 0 52s
kube-system kube-proxy-752hc 1/1 Running 0 54s
kube-system kube-proxy-7fgcg 1/1 Running 0 67s
kube-system kube-proxy-p9vfs 1/1 Running 0 74s
kube-system kube-scheduler-flannel-vxlan-control-plane 1/1 Running 0 88s
local-path-storage local-path-provisioner-5ddd94ff66-4kj2k 1/1 Running 0 74s
k8s 集群安装 Pod 测试网络
root@kind:~# cat cni.yaml
apiVersion: apps/v1
kind: DaemonSet
#kind: Deployment
metadata:
labels:
app: cni
name: cni
spec:
#replicas: 1
selector:
matchLabels:
app: cni
template:
metadata:
labels:
app: cni
spec:
containers:
- image: harbor.dayuan1997.com/devops/nettool:0.9
name: nettoolbox
securityContext:
privileged: true
---
apiVersion: v1
kind: Service
metadata:
name: serversvc
spec:
type: NodePort
selector:
app: cni
ports:
- name: cni
port: 80
targetPort: 80
nodePort: 32000
root@kind:~# kubectl apply -f cni.yaml
daemonset.apps/cni created
service/serversvc created
root@kind:~# kubectl run net --image=harbor.dayuan1997.com/devops/nettool:0.9
pod/net created
- 查看安装服务信息
root@kind:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cni-kmmzm 1/1 Running 0 15s 10.244.1.5 flannel-vxlan-worker2 <none> <none>
cni-wll8q 1/1 Running 0 15s 10.244.2.2 flannel-vxlan-worker <none> <none>
net 1/1 Running 0 9s 10.244.2.3 flannel-vxlan-worker <none> <none>
root@kind:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 98s
serversvc NodePort 10.96.231.56 <none> 80:32000/TCP 11s
三、测试网络
同节点 Pod 网络通讯
可以查看此文档 Flannel UDP 模式 中，同节点网络通讯，数据包转发流程一致
Flannel 同节点通信通过
l2网络通信，2层交换机完成
不同节点 Pod 网络通讯
Pod节点信息
## ip 信息
root@kind:~# kubectl exec -it net -- ip a l
3: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether aa:fd:5d:93:ed:c1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.2.3/24 brd 10.244.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a8fd:5dff:fe93:edc1/64 scope link
valid_lft forever preferred_lft forever
## 路由信息
root@kind:~# kubectl exec -it net -- ip r s
default via 10.244.2.1 dev eth0
10.244.0.0/16 via 10.244.2.1 dev eth0
10.244.2.0/24 dev eth0 proto kernel scope link src 10.244.2.3
Pod节点所在Node节点信息
root@kind:~# docker exec -it flannel-vxlan-worker bash
## ip 信息
root@flannel-vxlan-worker:/# ip a l
2: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether ba:a8:34:05:62:75 brd ff:ff:ff:ff:ff:ff
inet 10.244.2.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::b8a8:34ff:fe05:6275/64 scope link
valid_lft forever preferred_lft forever
3: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 1a:3f:f2:b1:44:d0 brd ff:ff:ff:ff:ff:ff
inet 10.244.2.1/24 brd 10.244.2.255 scope global cni0
valid_lft forever preferred_lft forever
inet6 fe80::183f:f2ff:feb1:44d0/64 scope link
valid_lft forever preferred_lft forever
4: veth3fcfa5b1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether 3a:7e:4f:8b:46:94 brd ff:ff:ff:ff:ff:ff link-netns cni-d1764373-dd43-dd10-2e32-b75e4aec8623
inet6 fe80::387e:4fff:fe8b:4694/64 scope link
valid_lft forever preferred_lft forever
5: veth47587db8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether da:d4:46:04:75:7d brd ff:ff:ff:ff:ff:ff link-netns cni-e7036160-f254-9529-05c0-ceaa90daa64d
inet6 fe80::d8d4:46ff:fe04:757d/64 scope link
valid_lft forever preferred_lft forever
19: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.18.0.3/16 brd 172.18.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fc00:f853:ccd:e793::3/64 scope global nodad
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe12:3/64 scope link
valid_lft forever preferred_lft forever
## 路由信息
root@flannel-vxlan-worker:/# ip r s
default via 172.18.0.1 dev eth0
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.2.0/24 dev cni0 proto kernel scope link src 10.244.2.1
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.3
Pod节点进行ping包测试，访问cni-kmmzmPod节点
root@kind:~# kubectl exec -it net -- ping 10.244.1.5 -c 1
PING 10.244.1.5 (10.244.1.5): 56 data bytes
64 bytes from 10.244.1.5: seq=0 ttl=62 time=1.116 ms
--- 10.244.1.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.116/1.116/1.116 ms
Pod节点eth0网卡抓包
net~$ tcpdump -pne -i eth0
08:25:55.746125 aa:fd:5d:93:ed:c1 > 1a:3f:f2:b1:44:d0, ethertype IPv4 (0x0800), length 98: 10.244.2.3 > 10.244.1.5: ICMP echo request, id 53, seq 0, length 64
08:25:55.746986 1a:3f:f2:b1:44:d0 > aa:fd:5d:93:ed:c1, ethertype IPv4 (0x0800), length 98: 10.244.1.5 > 10.244.2.3: ICMP echo reply, id 53, seq 0, length 64
数据包源 mac 地址: e6:7c:59:58:76:c1 为 eth0 网卡 mac 地址，而目的 mac 地址: ae:13:7e:8b:60:99 为 net Pod 节点 cni0 网卡对应的网卡 mac 地址，cni0
网卡 ip 地址为网络网关地址 10.244.2.1 ， flannel 为 2 层网络模式通过路由送往数据到网关地址
net~$ arp -n
Address HWtype HWaddress Flags Mask Iface
10.244.2.1 ether 1a:3f:f2:b1:44:d0 C eth0
而通过 veth pair 可以确定 Pod 节点 eth0 网卡对应的 veth pair 为 veth47587db8@if3 网卡
flannel-vxlan-worker节点veth47587db8网卡抓包
root@flannel-vxlan-worker:/# tcpdump -pne -i veth47587db8
08:28:17.785088 aa:fd:5d:93:ed:c1 > 1a:3f:f2:b1:44:d0, ethertype IPv4 (0x0800), length 98: 10.244.2.3 > 10.244.1.5: ICMP echo request, id 67, seq 0, length 64
08:28:17.785203 1a:3f:f2:b1:44:d0 > aa:fd:5d:93:ed:c1, ethertype IPv4 (0x0800), length 98: 10.244.1.5 > 10.244.2.3: ICMP echo reply, id 67, seq 0, length 64
因为他们互为 veth pair 所以抓包信息相同
flannel-vxlan-worker节点cni0网卡抓包
root@flannel-vxlan-worker:/# tcpdump -pne -i cni0
08:28:32.748285 aa:fd:5d:93:ed:c1 > 1a:3f:f2:b1:44:d0, ethertype IPv4 (0x0800), length 98: 10.244.2.3 > 10.244.1.5: ICMP echo request, id 73, seq 0, length 64
08:28:32.748373 1a:3f:f2:b1:44:d0 > aa:fd:5d:93:ed:c1, ethertype IPv4 (0x0800), length 98: 10.244.1.5 > 10.244.2.3: ICMP echo reply, id 73, seq 0, length 64
数据包源 mac 地址: e6:7c:59:58:76:c1 为 net Pod 节点 eth0 网卡 mac 地址，而目的 mac 地址: ae:13:7e:8b:60:99 为 cni0 网卡 mac 地址
查看
flannel-vxlan-worker主机路由信息，发现并在数据包会在通过10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink路由信息转发
flannel-vxlan-worker节点flannel.1网卡抓包
root@flannel-vxlan-worker:/# tcpdump -pne -i flannel.1 icmp
08:29:36.894243 ba:a8:34:05:62:75 > be:88:07:e5:f0:55, ethertype IPv4 (0x0800), length 98: 10.244.2.3 > 10.244.1.5: ICMP echo request, id 79, seq 0, length 64
08:29:36.894411 be:88:07:e5:f0:55 > ba:a8:34:05:62:75, ethertype IPv4 (0x0800), length 98: 10.244.1.5 > 10.244.2.3: ICMP echo reply, id 79, seq 0, length 64
数据包源 mac 地址: ba:a8:34:05:62:75 为 flannel-vxlan-worker 节点 flannel.1 网卡 mac 地址，而目的 mac 地址: be:88:07:e5:f0:55 是谁的 mac 地址？查看宿主机 arp 信息，目的 mac 地址: be:88:07:e5:f0:55 是 10.244.1.0 网段 mac 地址，这个地址如何学习到的？可以查看 FDB 自动学习绑定过程检测
root@flannel-vxlan-worker:/# arp -n
Address HWtype HWaddress Flags Mask Iface
10.244.1.0 ether be:88:07:e5:f0:55 CM flannel.1
查看 fdb 信息
root@flannel-vxlan-worker:/# bridge fdb show
be:88:07:e5:f0:55 dev flannel.1 dst 172.18.0.4 self permanent
通过查看 fdb 表信息可以看到 be:88:07:e5:f0:55 dev flannel.1 dst 172.18.0.4 self permanent 标示了 mac 地址 be:88:07:e5:f0:55 所在的主机为 172.18.0.4 ，因此 vxlan 封装的外层数据的目的 ip 是使用 172.18.0.4，而 vxlan 封装的外层数据的源的 ip 是本机 eth0 网卡 ip
flannel-vxlan-worker节点eth0网卡抓包
-
request数据包信息信息icmp包中，外部mac信息中，源mac: 02:42:ac:12:00:03为flannel-vxlan-worker的eth0网卡mac，目的mac: 02:42:ac:12:00:04为对端Pod宿主机flannel-vxlan-worker2的eth0网卡mac。使用udp协议8472端口进行数据传输，vxlan信息中vni为1。icmp包中，内部mac信息中，源mac: ba:a8:34:05:62:75为flannel-vxlan-worker的flannel.1网卡mac，目的mac: be:88:07:e5:f0:55为对端flannel-vxlan-worker2主机的flannel.1网卡mac。
-
flannel-vxlan-worker节点vxlan信息
root@flannel-vxlan-worker:/# ip -d link show
2: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether ba:a8:34:05:62:75 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
vxlan id 1 local 172.18.0.3 dev eth0 srcport 0 0 dstport 8472 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
数据包流向
- 数据从
pod服务发出，通过查看本机路由表，送往10.244.2.1网卡。路由:10.244.0.0/16 via 10.244.2.1 dev eth0 - 通过
veth pair网卡veth47587db8发送数据到flannel-vxlan-worker主机上，在转送到cni0: 10.244.2.1网卡 flannel-vxlan-worker主机查看自身路由后，会送往flannel.1接口，因为目的地址为10.244.1.5。路由:10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlinkflannel.1接口为vxlan模式，会重新封装数据包，封装信息查看arp信息10.244.1.0 ether be:88:07:e5:f0:55 CM flannel.1、fdb信息be:88:07:e5:f0:55 dev flannel.1 dst 172.18.0.4 self permanent- 数据封装完成后，会送往
eth0 接口，并送往对端flannel-vxlan-worker2主机。 - 对端
flannel-vxlan-worker2主机接受到数据包后，发现这个是一个送往UDP 8472接口的vxlan数据包，将数据包交给监听UDP 8472端口的应用程序或内核模块处理。 - 解封装后发现内部的数据包，目的地址为
10.244.1.5，通过查看本机路由表，送往cni0网卡。路由:10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1 - 通过
cni0网卡brctl showmacs cni0mac信息 ，最终会把数据包送到cni-kmmzm主机
Service 网络通讯
可以查看此文档 Flannel UDP 模式 中，Service 网络通讯，数据包转发流程一致
