Flannel IPIP DR 模式

合集 - Flannel(9)

1.Flannel (转载)2024-08-11 2.Flannel UDP 模式2024-08-11 3.Flannel HOST-GW 模式2024-08-14 4.Flannel VxLAN 模式2024-08-15 5.Flannel VxLAN DR 模式2024-08-20 6.Flannel IPIP 模式2024-08-21

7.Flannel IPIP DR 模式2024-08-21

8.Flannel IPsec 模式2024-08-22 9.Flannel Wireguard 模式2024-08-22

Flannel IPIP DR 模式

一、环境信息

主机	IP
ubuntu	172.16.94.141

软件	版本
docker	26.1.4
helm	v3.15.0-rc.2
kind	0.18.0
clab	0.54.2
kubernetes	1.23.4
ubuntu os	Ubuntu 20.04.6 LTS
kernel	5.11.5 内核升级文档

二、安装服务

`kind` 配置文件信息

$ cat install.sh

#!/bin/bash
date
set -v

# 1.prep noCNI env
cat <<EOF | kind create cluster --name=clab-flannel-ipip-directrouting --image=kindest/node:v1.23.4 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  # kind 默认使用 rancher cni，cni 我们需要自己创建
  disableDefaultCNI: true
  # 定义节点使用的 pod 网段
  podSubnet: "10.244.0.0/16"
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker

containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.dayuan1997.com"]
    endpoint = ["https://harbor.dayuan1997.com"]
EOF

# 2.remove taints
controller_node=`kubectl get nodes --no-headers  -o custom-columns=NAME:.metadata.name| grep control-plane`
kubectl taint nodes $controller_node node-role.kubernetes.io/master:NoSchedule-
kubectl get nodes -o wide

# 3.install necessary tools
# cd /opt/
# curl -o calicoctl -O -L "https://gh.api.99988866.xyz/https://github.com/containernetworking/plugins/releases/download/v0.9.0/cni-plugins-linux-amd64-v0.9.0.tgz" 
# tar -zxvf cni-plugins-linux-amd64-v0.9.0.tgz

for i in $(docker ps -a --format "table {{.Names}}" | grep flannel) 
do
    echo $i
    docker cp /opt/bridge $i:/opt/cni/bin/
    docker cp /usr/bin/ping $i:/usr/bin/ping
    docker exec -it $i bash -c "sed -i -e 's/jp.archive.ubuntu.com\|archive.ubuntu.com\|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list"
    docker exec -it $i bash -c "apt-get -y update >/dev/null && apt-get -y install net-tools tcpdump lrzsz bridge-utils >/dev/null 2>&1"
done

安装 k8s 集群

root@kind:~# ./install.sh

Creating cluster "clab-flannel-ipip-directrouting" ...
 ✓ Ensuring node image (kindest/node:v1.23.4) 🖼
 ✓ Preparing nodes 📦 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-clab-flannel-ipip-directrouting"
You can now use your cluster with:

kubectl cluster-info --context kind-clab-flannel-ipip-directrouting

Have a nice day! 👋

root@kind:~# kubectl get node -o wide
NAME                                            STATUS     ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION          CONTAINER-RUNTIME
clab-flannel-ipip-directrouting-control-plane   NotReady   control-plane,master   66s   v1.23.4   172.18.0.4    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-ipip-directrouting-worker          NotReady   <none>                 46s   v1.23.4   172.18.0.2    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-ipip-directrouting-worker2         NotReady   <none>                 33s   v1.23.4   172.18.0.5    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-ipip-directrouting-worker3         NotReady   <none>                 33s   v1.23.4   172.18.0.3    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10

创建 `clab` 容器环境

创建网桥

root@kind:~# brctl addbr br-pool0
root@kind:~# ifconfig br-pool0 up
root@kind:~# brctl addbr br-pool1
root@kind:~# ifconfig br-pool1 up

root@kind:~# ip a l
29: br-pool0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default qlen 1000
    link/ether aa:c1:ab:02:66:b8 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1cab:b8ff:fe40:2b38/64 scope link 
       valid_lft forever preferred_lft forever
30: br-pool1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default qlen 1000
    link/ether aa:c1:ab:1b:40:50 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c830:c5ff:fe84:1258/64 scope link 
       valid_lft forever preferred_lft forever

创建这两个网桥主要是为了让 kind 上节点通过虚拟交换机连接到 containerLab ，为什么不直连接 containerLab ，如果 10.1.5.10/24 使用 vethPair 和 containerLab 进行连接， 10.1.5.11/24 就没有额外的端口进行连接

`clab` 网络拓扑文件

# flannel.ipip.directrouting.clab.yml
name: flannel-ipip-directrouting
topology:
  nodes:
    gw0:
      kind: linux
      image: vyos/vyos:1.2.8
      cmd: /sbin/init
      binds:
        - /lib/modules:/lib/modules
        - ./startup-conf/gw0-boot.cfg:/opt/vyatta/etc/config/config.boot

    br-pool0:
      kind: bridge
  
    br-pool1:
      kind: bridge


    server1:
      kind: linux
      image: harbor.dayuan1997.com/devops/nettool:0.9
      # 复用节点网络，共享网络命名空间
      network-mode: container:clab-flannel-ipip-directrouting-control-plane
      # 配置是为了设置节点上的业务网卡，同时将默认路由的网关进行更改，使用业务网卡为出接口。
      exec:      
      - ip addr add 10.1.5.10/24 dev net0      
      - ip route replace default via 10.1.5.1

    server2:
      kind: linux
      image: harbor.dayuan1997.com/devops/nettool:0.9
      # 复用节点网络，共享网络命名空间
      network-mode: container:clab-flannel-ipip-directrouting-worker
      # 配置是为了设置节点上的业务网卡，同时将默认路由的网关进行更改，使用业务网卡为出接口。
      exec:
      - ip addr add 10.1.5.11/24 dev net0
      - ip route replace default via 10.1.5.1

    server3:
      kind: linux
      image: harbor.dayuan1997.com/devops/nettool:0.9
      # 复用节点网络，共享网络命名空间
      network-mode: container:clab-flannel-ipip-directrouting-worker2
      # 配置是为了设置节点上的业务网卡，同时将默认路由的网关进行更改，使用业务网卡为出接口。
      exec:
      - ip addr add 10.1.8.10/24 dev net0
      - ip route replace default via 10.1.8.1

    server4:
      kind: linux
      image: harbor.dayuan1997.com/devops/nettool:0.9
      # 复用节点网络，共享网络命名空间
      network-mode: container:clab-flannel-ipip-directrouting-worker3
      # 配置是为了设置节点上的业务网卡，同时将默认路由的网关进行更改，使用业务网卡为出接口。
      exec:
      - ip addr add 10.1.8.11/24 dev net0
      - ip route replace default via 10.1.8.1


  links:
    - endpoints: ["br-pool0:br-pool0-net0", "server1:net0"]
    - endpoints: ["br-pool0:br-pool0-net1", "server2:net0"]
    - endpoints: ["br-pool1:br-pool1-net0", "server3:net0"]
    - endpoints: ["br-pool1:br-pool1-net1", "server4:net0"]

    - endpoints: ["gw0:eth1", "br-pool0:br-pool0-net2"]
    - endpoints: ["gw0:eth2", "br-pool1:br-pool1-net2"]

`VyOS` 配置文件

gw0-boot.cfg

配置文件

# ./startup-conf/gw0-boot.cfg
interfaces {
    ethernet eth1 {
        address 10.1.5.1/24
        duplex auto
        smp-affinity auto
        speed auto
    }
    ethernet eth2 {
        address 10.1.8.1/24
        duplex auto
        smp-affinity auto
        speed auto
    }
    loopback lo {
    }
}
# 配置 nat 信息，gw0 网络下的其他服务器可以访问外网
nat {
    source {
        rule 100 {
            outbound-interface eth0
            source {
                address 10.1.0.0/16
            }
            translation {
                address masquerade
            }
        }
    }
}
system {
    config-management {
        commit-revisions 100
    }
    console {
        device ttyS0 {
            speed 9600
        }
    }
    host-name vyos
    login {
        user vyos {
            authentication {
                encrypted-password $6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/
                plaintext-password ""
            }
            level admin
        }
    }
    ntp {
        server 0.pool.ntp.org {
        }
        server 1.pool.ntp.org {
        }
        server 2.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "wanloadbalance@3:l2tp@1:pptp@1:ntp@1:mdns@1:webgui@1:conntrack@1:ipsec@5:cluster@1:dhcp-server@5:nat@4:dhcp-relay@2:webproxy@1:system@10:pppoe-server@2:dns-forwarding@1:ssh@1:quagga@7:broadcast-relay@1:qos@1:snmp@1:firewall@5:zone-policy@1:config-management@1:webproxy@2:vrrp@2:conntrack-sync@1" === */
/* Release version: 1.2.8 */

部署服务

# tree -L 2 ./
./
├── flannel.ipip.directrouting.clab.yml
└── startup-conf
    └── gw0-boot.cfg

# clab deploy -t flannel.ipip.directrouting.clab.yml

INFO[0000] Containerlab v0.54.2 started                 
INFO[0000] Parsing & checking topology file: clab.yaml  
INFO[0000] Creating lab directory: /root/wcni-kind/flannel/6-flannel-ipip-directrouting/clab-flannel-ipip-directrouting 
WARN[0000] node clab-flannel-ipip-directrouting-worker2 referenced in namespace sharing not found in topology definition, considering it an external dependency. 
WARN[0000] node clab-flannel-ipip-directrouting-worker3 referenced in namespace sharing not found in topology definition, considering it an external dependency. 
WARN[0000] node clab-flannel-ipip-directrouting-control-plane referenced in namespace sharing not found in topology definition, considering it an external dependency. 
WARN[0000] node clab-flannel-ipip-directrouting-worker referenced in namespace sharing not found in topology definition, considering it an external dependency. 
INFO[0000] Creating container: "gw0"                    
INFO[0001] Created link: gw0:eth1 <--> br-pool0:br-pool0-net2 
INFO[0001] Created link: gw0:eth2 <--> br-pool1:br-pool1-net2 
INFO[0003] Creating container: "server3"                
INFO[0003] Creating container: "server2"                
INFO[0004] Created link: br-pool0:br-pool0-net1 <--> server2:net0 
INFO[0004] Created link: br-pool1:br-pool1-net0 <--> server3:net0 
INFO[0005] Creating container: "server4"                
INFO[0005] Creating container: "server1"                
INFO[0006] Created link: br-pool0:br-pool0-net0 <--> server1:net0 
INFO[0006] Created link: br-pool1:br-pool1-net1 <--> server4:net0 
INFO[0006] Executed command "ip addr add 10.1.5.11/24 dev net0" on the node "server2". stdout: 
INFO[0006] Executed command "ip route replace default via 10.1.5.1" on the node "server2". stdout: 
INFO[0006] Executed command "ip addr add 10.1.8.10/24 dev net0" on the node "server3". stdout: 
INFO[0006] Executed command "ip route replace default via 10.1.8.1" on the node "server3". stdout: 
INFO[0006] Executed command "ip addr add 10.1.5.10/24 dev net0" on the node "server1". stdout: 
INFO[0006] Executed command "ip route replace default via 10.1.5.1" on the node "server1". stdout: 
INFO[0006] Executed command "ip addr add 10.1.8.11/24 dev net0" on the node "server4". stdout: 
INFO[0006] Executed command "ip route replace default via 10.1.8.1" on the node "server4". stdout: 
INFO[0006] Adding containerlab host entries to /etc/hosts file 
INFO[0006] Adding ssh config for containerlab nodes     
INFO[0006] 🎉 New containerlab version 0.56.0 is available! Release notes: https://containerlab.dev/rn/0.56/
Run 'containerlab version upgrade' to upgrade or go check other installation options at https://containerlab.dev/install/ 
+---+-----------------------------------------+--------------+------------------------------------------+-------+---------+----------------+----------------------+
| # |                  Name                   | Container ID |                  Image                   | Kind  |  State  |  IPv4 Address  |     IPv6 Address     |
+---+-----------------------------------------+--------------+------------------------------------------+-------+---------+----------------+----------------------+
| 1 | clab-flannel-ipip-directrouting-gw0     | 3e133b73fa87 | vyos/vyos:1.2.8                          | linux | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 2 | clab-flannel-ipip-directrouting-server1 | 71cd9cb72a9d | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A            | N/A                  |
| 3 | clab-flannel-ipip-directrouting-server2 | 843eb9b12b31 | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A            | N/A                  |
| 4 | clab-flannel-ipip-directrouting-server3 | e722bfec3127 | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A            | N/A                  |
| 5 | clab-flannel-ipip-directrouting-server4 | 1281563feb4f | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A            | N/A                  |
+---+-----------------------------------------+--------------+------------------------------------------+-------+---------+----------------+----------------------+

检查 k8s 集群信息

root@kind:~# kubectl get node -o wide
NAME                                            STATUS     ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION          CONTAINER-RUNTIME
clab-flannel-ipip-directrouting-control-plane   NotReady   control-plane,master   11m   v1.23.4   172.18.0.4    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-ipip-directrouting-worker          NotReady   <none>                 11m   v1.23.4   172.18.0.2    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-ipip-directrouting-worker2         NotReady   <none>                 11m   v1.23.4   172.18.0.5    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-ipip-directrouting-worker3         NotReady   <none>                 11m   v1.23.4   172.18.0.3    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10

# 查看 node 节点 ip 信息
root@kind:~# docker exec -it clab-flannel-ipip-directrouting-control-plane ip a l
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
25: eth0@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.4/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::4/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:4/64 scope link 
       valid_lft forever preferred_lft forever
33: net0@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default 
    link/ether aa:c1:ab:84:70:e7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.1.5.10/24 scope global net0
       valid_lft forever preferred_lft forever
    inet6 fe80::a8c1:abff:fe84:70e7/64 scope link 
       valid_lft forever preferred_lft forever

# 查看 node 节点路由信息
root@kind:~# docker exec -it clab-flannel-ipip-directrouting-control-plane ip r s
default via 10.1.5.1 dev net0 
10.1.5.0/24 dev net0 proto kernel scope link src 10.1.5.10
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.4

查看 k8s 集群发现 node 节点 ip 地址分配了，登陆容器查看到了新的 ip 地址，并且默认路由信息调整为了 10.1.5.0/24 dev net0 proto kernel scope link src 10.1.5.10

安装 `flannel` 服务

flannel.yaml

配置文件

# flannel.yaml
---
kind: Namespace
apiVersion: v1
metadata:
  name: kube-flannel
  labels:
    pod-security.kubernetes.io/enforce: privileged
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "ipip",
        "DirectRouting": true
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni-plugin
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
        command:
        - cp
        args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        volumeMounts:
        - name: cni-plugin
          mountPath: /opt/cni/bin
      - name: install-cni
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel:v0.19.2
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel:v0.19.2
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: EVENT_QUEUE_DEPTH
          value: "5000"
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
        - name: xtables-lock
          mountPath: /run/xtables.lock
        - name: tun
          mountPath: /dev/net/tun
      volumes:
      - name: tun
        hostPath:
          path: /dev/net/tun
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni-plugin
        hostPath:
          path: /opt/cni/bin
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate

flannel.yaml 参数解释

Backend.Type
- 含义: 用于指定 flannel 工作模式。
- ipip: flannel 工作在 ipip 模式。
Backend.DirectRouting
- 含义: 用于指定 ipip 模式同网段使用 host-gw 模式。
- true: ipip 模式中，同网段的 node 节点之间数据转发使用 host-gw 模式。

root@kind:~# kubectl apply -f flannel.yaml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

查看 k8s 集群和 flannel 服务

root@kind:~# kubectl get node -o wide
NAME                                            STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION          CONTAINER-RUNTIME
clab-flannel-ipip-directrouting-control-plane   Ready    control-plane,master   14m   v1.23.4   172.18.0.4    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-ipip-directrouting-worker          Ready    <none>                 13m   v1.23.4   172.18.0.2    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-ipip-directrouting-worker2         Ready    <none>                 13m   v1.23.4   172.18.0.5    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-ipip-directrouting-worker3         Ready    <none>                 13m   v1.23.4   172.18.0.3    <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10

查看安装的服务

root@kind:~# kubectl get pods -A
NAMESPACE            NAME                                                                    READY   STATUS    RESTARTS   AGE
kube-system          coredns-64897985d-7rmd8                                                 1/1     Running   0          14m
kube-system          coredns-64897985d-9qzts                                                 1/1     Running   0          14m
kube-system          etcd-clab-flannel-ipip-directrouting-control-plane                      1/1     Running   0          14m
kube-system          kube-apiserver-clab-flannel-ipip-directrouting-control-plane            1/1     Running   0          14m
kube-system          kube-controller-manager-clab-flannel-ipip-directrouting-control-plane   1/1     Running   0          14m
kube-system          kube-flannel-ds-c7bnc                                                   1/1     Running   0          34s
kube-system          kube-flannel-ds-ch8r5                                                   1/1     Running   0          34s
kube-system          kube-flannel-ds-gcpmk                                                   1/1     Running   0          34s
kube-system          kube-flannel-ds-nm8mw                                                   1/1     Running   0          34s
kube-system          kube-proxy-cr88l                                                        1/1     Running   0          14m
kube-system          kube-proxy-fs8k7                                                        1/1     Running   0          13m
kube-system          kube-proxy-hb76d                                                        1/1     Running   0          13m
kube-system          kube-proxy-q2d2w                                                        1/1     Running   0          13m
kube-system          kube-scheduler-clab-flannel-ipip-directrouting-control-plane            1/1     Running   0          14m
local-path-storage   local-path-provisioner-5ddd94ff66-kgdw2                                 1/1     Running   0          14m

`k8s` 集群安装 `Pod` 测试网络

root@kind:~# cat cni.yaml

apiVersion: apps/v1
kind: DaemonSet
#kind: Deployment
metadata:
  labels:
    app: cni
  name: cni
spec:
  #replicas: 1
  selector:
    matchLabels:
      app: cni
  template:
    metadata:
      labels:
        app: cni
    spec:
      containers:
      - image: harbor.dayuan1997.com/devops/nettool:0.9
        name: nettoolbox
        securityContext:
          privileged: true

---
apiVersion: v1
kind: Service
metadata:
  name: serversvc
spec:
  type: NodePort
  selector:
    app: cni
  ports:
  - name: cni
    port: 80
    targetPort: 80
    nodePort: 32000

root@kind:~# kubectl apply -f cni.yaml
daemonset.apps/cni created
service/serversvc created

root@kind:~# kubectl run net --image=harbor.dayuan1997.com/devops/nettool:0.9
pod/net created

查看安装服务信息

root@kind:~# kubectl get pods -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP           NODE                                            NOMINATED NODE   READINESS GATES
cni-8vvgl   1/1     Running   0          19s   10.244.3.2   clab-flannel-ipip-directrouting-worker2         <none>           <none>
cni-gqht9   1/1     Running   0          19s   10.244.1.2   clab-flannel-ipip-directrouting-worker          <none>           <none>
cni-rk5hx   1/1     Running   0          19s   10.244.2.2   clab-flannel-ipip-directrouting-worker3         <none>           <none>
cni-wfv4m   1/1     Running   0          19s   10.244.0.5   clab-flannel-ipip-directrouting-control-plane   <none>           <none>
net         1/1     Running   0          18s   10.244.3.3   clab-flannel-ipip-directrouting-worker2         <none>           <none>

root@kind:~# kubectl get svc 
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        14m
serversvc    NodePort    10.96.171.181   <none>        80:32000/TCP   26s

三、测试网络

同节点 `Pod` 网络通讯

可以查看此文档 Flannel UDP 模式中，同节点网络通讯，数据包转发流程一致

Flannel 同节点通信通过 l2 网络， 2 层交换机完成

跨节点同 `Node` 网段 `Pod` 网络通讯

可以查看此文档 Flannel HOST-GW 模式中，不同节点 Pod 网络通讯，数据包转发流程一致

跨节点不同 `Node` 网段 `Pod` 网络通讯

Pod 节点信息

## ip 信息
root@kind:~# kubectl exec -it net -- ip a l
4: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9480 qdisc noqueue state UP group default 
    link/ether 66:1e:0c:01:b0:60 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.244.3.3/24 brd 10.244.3.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::641e:cff:fe01:b060/64 scope link 
       valid_lft forever preferred_lft forever

## 路由信息
root@kind:~# kubectl exec -it net -- ip r s
default via 10.244.3.1 dev eth0 
10.244.0.0/16 via 10.244.3.1 dev eth0
10.244.3.0/24 dev eth0 proto kernel scope link src 10.244.3.3

Pod 节点所在 Node 节点信息

root@kind:~# docker exec -it clab-flannel-ipip-directrouting-worker2 bash

## ip 信息
root@clab-flannel-ipip-directrouting-worker2:/# ip a l 
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: flannel.ipip@NONE: <NOARP,UP,LOWER_UP> mtu 9480 qdisc noqueue state UNKNOWN group default 
    link/ipip 10.1.8.10 brd 0.0.0.0
    inet 10.244.3.0/32 scope global flannel.ipip
       valid_lft forever preferred_lft forever
    inet6 fe80::5efe:a01:80a/64 scope link 
       valid_lft forever preferred_lft forever
4: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9480 qdisc noqueue state UP group default qlen 1000
    link/ether 6a:99:0f:73:74:63 brd ff:ff:ff:ff:ff:ff
    inet 10.244.3.1/24 brd 10.244.3.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::6899:fff:fe73:7463/64 scope link 
       valid_lft forever preferred_lft forever
5: veth16279eba@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9480 qdisc noqueue master cni0 state UP group default 
    link/ether ba:68:cc:a7:7d:a3 brd ff:ff:ff:ff:ff:ff link-netns cni-9ebdf794-a2f7-5460-6d21-78f82e9d4ba3
    inet6 fe80::b868:ccff:fea7:7da3/64 scope link 
       valid_lft forever preferred_lft forever
6: veth658091d4@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9480 qdisc noqueue master cni0 state UP group default 
    link/ether 1a:f4:bc:3c:bb:48 brd ff:ff:ff:ff:ff:ff link-netns cni-369690a1-a496-8cbf-bba6-2bdc1ffd448f
    inet6 fe80::18f4:bcff:fe3c:bb48/64 scope link 
       valid_lft forever preferred_lft forever
27: eth0@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.5/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::5/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:5/64 scope link 
       valid_lft forever preferred_lft forever
31: net0@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default 
    link/ether aa:c1:ab:bf:0d:6b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.1.8.10/24 scope global net0
       valid_lft forever preferred_lft forever
    inet6 fe80::a8c1:abff:febf:d6b/64 scope link 
       valid_lft forever preferred_lft forever

## 路由信息
root@clab-flannel-ipip-directrouting-worker2:/# ip r s
default via 10.1.8.1 dev net0 
10.1.8.0/24 dev net0 proto kernel scope link src 10.1.8.10 
10.244.0.0/24 via 10.1.5.10 dev flannel.ipip onlink 
10.244.1.0/24 via 10.1.5.11 dev flannel.ipip onlink 
10.244.2.0/24 via 10.1.8.11 dev net0 onlink 
10.244.3.0/24 dev cni0 proto kernel scope link src 10.244.3.1 
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.5

Pod 节点进行 ping 包测试，访问 cni-gqht9 Pod 节点

root@kind:~# kubectl exec -it net -- ping 10.244.1.2 -c 1
PING 10.244.1.2 (10.244.1.2): 56 data bytes
64 bytes from 10.244.1.2: seq=0 ttl=62 time=1.562 ms

--- 10.244.1.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.562/1.562/1.562 ms

Pod 节点 eth0 网卡抓包

net~$ tcpdump -pne -i eth0
07:32:41.821687 66:1e:0c:01:b0:60 > 6a:99:0f:73:74:63, ethertype IPv4 (0x0800), length 98: 10.244.3.3 > 10.244.1.2: ICMP echo request, id 60, seq 0, length 64
07:32:41.821796 6a:99:0f:73:74:63 > 66:1e:0c:01:b0:60, ethertype IPv4 (0x0800), length 98: 10.244.1.2 > 10.244.3.3: ICMP echo reply, id 60, seq 0, length 64

数据包源 mac 地址: 66:1e:0c:01:b0:60 为 eth0 网卡 mac 地址，而目的 mac 地址: 6a:99:0f:73:74:63 为 net Pod 节点 cni0 网卡对应的网卡 mac 地址，cni0
网卡 ip 地址为网络网关地址 10.244.2.1 ， flannel 为 2 层网络模式通过路由送往数据到网关地址

net~$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.244.3.1               ether   6a:99:0f:73:74:63   C                     eth0

而通过 veth pair 可以确定 Pod 节点 eth0 网卡对应的 veth pair 为 veth658091d4@if4 网卡

clab-flannel-ipip-directrouting-worker2 节点 veth658091d4 网卡抓包

root@clab-flannel-ipip-directrouting-worker2:/# tcpdump -pne -i veth658091d4
07:32:41.821687 66:1e:0c:01:b0:60 > 6a:99:0f:73:74:63, ethertype IPv4 (0x0800), length 98: 10.244.3.3 > 10.244.1.2: ICMP echo request, id 60, seq 0, length 64
07:32:41.821796 6a:99:0f:73:74:63 > 66:1e:0c:01:b0:60, ethertype IPv4 (0x0800), length 98: 10.244.1.2 > 10.244.3.3: ICMP echo reply, id 60, seq 0, length 64

因为他们互为 veth pair 所以抓包信息相同

clab-flannel-ipip-directrouting-worker2 节点 cni0 网卡抓包

root@clab-flannel-ipip-directrouting-worker2:/# tcpdump -pne -i cni0
07:32:41.821687 66:1e:0c:01:b0:60 > 6a:99:0f:73:74:63, ethertype IPv4 (0x0800), length 98: 10.244.3.3 > 10.244.1.2: ICMP echo request, id 60, seq 0, length 64
07:32:41.821796 6a:99:0f:73:74:63 > 66:1e:0c:01:b0:60, ethertype IPv4 (0x0800), length 98: 10.244.1.2 > 10.244.3.3: ICMP echo reply, id 60, seq 0, length 64

数据包源 mac 地址: 66:1e:0c:01:b0:60 为 net Pod 节点 eth0 网卡 mac 地址，而目的 mac 地址: 6a:99:0f:73:74:63 为 cni0 网卡 mac 地址

查看 clab-flannel-ipip-directrouting-worker2 主机路由信息，发现并在数据包会在通过 10.244.1.0/24 via 10.1.5.11 dev flannel.ipip onlink 路由信息转发

clab-flannel-ipip-directrouting-worker2 节点 flannel.ipip 网卡抓包

root@clab-flannel-ipip-directrouting-worker2:/# tcpdump -pne -i flannel.ipip icmp
listening on flannel.ipip, link-type RAW (Raw IP), snapshot length 262144 bytes
07:34:59.737937 ip: 10.244.3.3 > 10.244.1.2: ICMP echo request, id 86, seq 0, length 64
07:34:59.738009 ip: 10.244.1.2 > 10.244.3.3: ICMP echo reply, id 86, seq 0, length 64

icmp 包中，没有 mac 信息，只有源 ip 目的 ip 信息，这也是 ipip 数据包的特性: IPIP 隧道的工作原理是将源主机的IP数据包封装在一个新的IP数据包中

clab-flannel-ipip-directrouting-worker2 节点 net0 网卡抓包

request 数据包信息信息
- icmp 包中，外部 mac 信息中，源 mac: aa:c1:ab:bf:0d:6b 为 clab-flannel-ipip-directrouting-worker2 的 net0 网卡 mac ，目的 mac: aa:c1:ab:25:67:20 为 gw0 主机的 eth2 网卡 mac
clab-flannel-ipip-directrouting-worker2 节点 ipip 信息

root@clab-flannel-ipip-directrouting-worker2:/# ip -d link show
3: flannel.ipip@NONE: <NOARP,UP,LOWER_UP> mtu 9480 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/ipip 10.1.8.10 brd 0.0.0.0 promiscuity 0 minmtu 0 maxmtu 0 
    ipip any remote any local 10.1.8.10 ttl inherit nopmtudisc addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

数据包流向

数据从 pod 服务发出，通过查看本机路由表，送往 10.244.3.1 网卡。路由: 10.244.0.0/16 via 10.244.3.1 dev eth0
通过 veth pair 网卡 veth658091d4 发送数据到 clab-flannel-ipip-directrouting-worker2 主机上，在转送到 cni0: 10.244.3.1 网卡
clab-flannel-ipip-directrouting-worker2 主机查看自身路由后，会送往 flannel.ipip 接口，因为目的地址为 10.244.1.2。路由: 10.244.1.0/24 via 10.1.5.11 dev flannel.ipip onlink
flannel.ipip 接口为 ipip 模式，会重新封装数据包。
数据封装完成后，会送往 net0 接口，并送往 gw0 主机。
gw0 主机接受到数据包后，发目的地址为 10.1.5.11，会查看路由表，送往 eth1 接口。路由: 10.1.5.0/24 dev eth1 proto kernel scope link src 10.1.5.1
通过 gw0 主机 eth1 网卡重新封装数据包后，最终会把数据包送到 clab-flannel-ipip-directrouting-worker 主机
对端 flannel-ipip-directrouting-worker 主机接受到数据包后，发现这个是一个 ipip 数据包，接收端会将外层 IP 头部去掉，提取内层的 IP 数据包。
内层数据包会被交给 flannel.ipip 接口进行处理，就像是接收到了一个普通的 IP 数据包一样。
解封装后发现内部的数据包，目的地址为 10.244.1.2 ，通过查看本机路由表，送往 cni0 网卡。路由: 10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1
通过 cni0 网卡 brctl showmacs cni0 mac 信息，最终会把数据包送到 cni-gqht9 主机

`Service` 网络通讯

可以查看此文档 Flannel UDP 模式中，Service 网络通讯，数据包转发流程一致

posted @ 2024-08-21 15:47 evescn 阅读(77) 评论(50) 编辑收藏举报

刷新页面返回顶部

登录后才能查看或发表评论，立即登录或者逛逛博客园首页

相关博文：

· Flannel VxLAN DR 模式

· Flannel IPIP 模式

· k8s-flannel

· 网络组件flannel总结

· Flannel网络组件

阅读排行：
· 震惊！C++程序真的从main开始吗？99%的程序员都答错了
· 【硬核科普】Trae如何「偷看」你的代码？零基础破解AI编程运行原理
· 单元测试从入门到精通
· 上周热点回顾（3.3-3.9）
· Vue3状态管理终极指南：Pinia保姆级教程

公告

昵称： evescn
园龄： 8年6个月
粉丝： 28
关注： 3

+加关注

2025年3月

日

一

二

三

四

五

六

evescn

Flannel IPIP DR 模式

Flannel IPIP DR 模式

一、环境信息

二、安装服务

`kind` 配置文件信息

创建 `clab` 容器环境

创建网桥

`clab` 网络拓扑文件

`VyOS` 配置文件

部署服务

检查 k8s 集群信息

安装 `flannel` 服务

`k8s` 集群安装 `Pod` 测试网络

三、测试网络

同节点 `Pod` 网络通讯

跨节点同 `Node` 网段 `Pod` 网络通讯

跨节点不同 `Node` 网段 `Pod` 网络通讯

数据包流向

`Service` 网络通讯

公告

我的标签

合集

随笔分类

随笔档案

evescn

Flannel IPIP DR 模式

Flannel IPIP DR 模式

一、环境信息

二、安装服务

kind 配置文件信息

创建 clab 容器环境

创建网桥

clab 网络拓扑文件

VyOS 配置文件

部署服务

检查 k8s 集群信息

安装 flannel 服务

k8s 集群安装 Pod 测试网络

三、测试网络

同节点 Pod 网络通讯

跨节点同 Node 网段 Pod 网络通讯

跨节点不同 Node 网段 Pod 网络通讯

数据包流向

Service 网络通讯

公告

我的标签

合集

随笔分类

随笔档案

`kind` 配置文件信息

创建 `clab` 容器环境

`clab` 网络拓扑文件

`VyOS` 配置文件

安装 `flannel` 服务

`k8s` 集群安装 `Pod` 测试网络

同节点 `Pod` 网络通讯

跨节点同 `Node` 网段 `Pod` 网络通讯

跨节点不同 `Node` 网段 `Pod` 网络通讯

`Service` 网络通讯