Calico BGP RouteReflector 模式
Calico BGP RouteReflector 模式
Calico BGP RR 模式
一、环境信息
| 主机 | IP |
|---|---|
| ubuntu | 172.16.94.141 |
| 软件 | 版本 |
|---|---|
| docker | 26.1.4 |
| helm | v3.15.0-rc.2 |
| kind | 0.18.0 |
| clab | 0.54.2 |
| kubernetes | 1.23.4 |
| ubuntu os | Ubuntu 20.04.6 LTS |
| kernel | 5.11.5 内核升级文档 |
二、安装服务
网络架构
kind 配置文件信息
$ cat install.sh
#!/bin/bash
date
set -v
# 1.prep noCNI env
cat <<EOF | kind create cluster --image=kindest/node:v1.23.4 --config=-
kind: Cluster
name: calico-bgp-rr
apiVersion: kind.x-k8s.io/v1alpha4
networking:
disableDefaultCNI: true
podSubnet: "10.244.0.0/16"
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.5.10
node-labels: "rack=rack0"
- role: worker
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.5.11
node-labels: "rack=rack0"
- role: worker
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.8.10
node-labels: "rack=rack1"
- role: worker
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.8.11
node-labels: "rack=rack1"
containerdConfigPatches:
- |-
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.evescn.com"]
endpoint = ["https://harbor.evescn.com"]
EOF
# 2.remove taints
kubectl taint nodes $(kubectl get nodes -o name | grep control-plane) node-role.kubernetes.io/master:NoSchedule-
# 3.install necessary tools
# cd /usr/bin/
# curl -o calicoctl -O -L "https://gh.api.99988866.xyz/https://github.com/projectcalico/calico/releases/download/v3.23.2/calicoctl-linux-amd64"
# chmod +x calicoctl
for i in $(docker ps -a --format "table {{.Names}}" | grep calico)
do
echo $i
docker cp /usr/bin/calicoctl $i:/usr/bin/calicoctl
docker cp /usr/bin/ping $i:/usr/bin/ping
docker exec -it $i bash -c "sed -i -e 's/jp.archive.ubuntu.com\|archive.ubuntu.com\|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list"
docker exec -it $i bash -c "apt-get -y update >/dev/null && apt-get -y install net-tools tcpdump lrzsz bridge-utils >/dev/null 2>&1"
done
- 安装
k8s集群
root@kind:~# ./install.sh
Creating cluster "calico-bgp-rr" ...
✓ Ensuring node image (kindest/node:v1.23.4) 🖼
✓ Preparing nodes 📦 📦 📦 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
Set kubectl context to "kind-calico-bgp-rr"
You can now use your cluster with:
kubectl cluster-info --context kind-calico-bgp-rr
Have a nice day! 👋
root@kind:~# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
calico-bgp-rr-control-plane NotReady control-plane,master 2m28s v1.23.4 <none> <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
calico-bgp-rr-worker NotReady <none> 114s v1.23.4 <none> <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
calico-bgp-rr-worker2 NotReady <none> 114s v1.23.4 <none> <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
calico-bgp-rr-worker3 NotReady <none> 114s v1.23.4 <none> <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
集群创建完成后，可以看到节点状态因为没有
cni插件，状态为NotReady，同时也没有ip地址。这是正常现象
创建 clab 容器环境
创建网桥
root@kind:~# brctl addbr br-leaf0
root@kind:~# ifconfig br-leaf0 up
root@kind:~# brctl addbr br-leaf1
root@kind:~# ifconfig br-leaf1 up
root@kind:~# ip a l
19: br-leaf0: <BROADCAST,MULTICAST,UP> mtu 9500 qdisc noqueue state UP group default qlen 1000
link/ether aa:c1:ab:0e:19:6f brd ff:ff:ff:ff:ff:ff
inet6 fe80::2c9c:78ff:fe54:f7b6/64 scope link
valid_lft forever preferred_lft forever
20: br-leaf1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 6a:2b:69:01:aa:9c brd ff:ff:ff:ff:ff:ff
inet6 fe80::682b:69ff:fe01:aa9c/64 scope link
valid_lft forever preferred_lft forever
创建这两个网桥主要是为了让
kind上节点通过虚拟交换机连接到containerLab，为什么不直连接containerLab，如果10.1.5.10/24使用vethPair和containerLab进行连接，10.1.5.11/24就没有额外的端口进行连接
clab 网络拓扑文件
# calico.bgp.clab.yml
name: calico-bgp-rr
topology:
nodes:
spine0:
kind: linux
image: vyos/vyos:1.2.8
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/spine0-boot.cfg:/opt/vyatta/etc/config/config.boot
spine1:
kind: linux
image: vyos/vyos:1.2.8
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/spine1-boot.cfg:/opt/vyatta/etc/config/config.boot
leaf0:
kind: linux
image: vyos/vyos:1.2.8
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/leaf0-boot.cfg:/opt/vyatta/etc/config/config.boot
leaf1:
kind: linux
image: vyos/vyos:1.2.8
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/leaf1-boot.cfg:/opt/vyatta/etc/config/config.boot
br-leaf0:
kind: bridge
br-leaf1:
kind: bridge
server1:
kind: linux
image: harbor.dayuan1997.com/devops/nettool:0.9
# 复用节点网络，共享网络命名空间
network-mode: container:calico-bgp-rr-control-plane
# 配置是为了设置节点上的业务网卡，同时将默认路由的网关进行更改，使用业务网卡为出接口。
exec:
- ip addr add 10.1.5.10/24 dev net0
- ip route replace default via 10.1.5.1
server2:
kind: linux
image: harbor.dayuan1997.com/devops/nettool:0.9
# 复用节点网络，共享网络命名空间
network-mode: container:calico-bgp-rr-worker
# 配置是为了设置节点上的业务网卡，同时将默认路由的网关进行更改，使用业务网卡为出接口。
exec:
- ip addr add 10.1.5.11/24 dev net0
- ip route replace default via 10.1.5.1
server3:
kind: linux
image: harbor.dayuan1997.com/devops/nettool:0.9
# 复用节点网络，共享网络命名空间
network-mode: container:calico-bgp-rr-worker2
# 配置是为了设置节点上的业务网卡，同时将默认路由的网关进行更改，使用业务网卡为出接口。
exec:
- ip addr add 10.1.8.10/24 dev net0
- ip route replace default via 10.1.8.1
server4:
kind: linux
image: harbor.dayuan1997.com/devops/nettool:0.9
# 复用节点网络，共享网络命名空间
network-mode: container:calico-bgp-rr-worker3
# 配置是为了设置节点上的业务网卡，同时将默认路由的网关进行更改，使用业务网卡为出接口。
exec:
- ip addr add 10.1.8.11/24 dev net0
- ip route replace default via 10.1.8.1
links:
- endpoints: ["br-leaf0:br-leaf0-net0", "server1:net0"]
- endpoints: ["br-leaf0:br-leaf0-net1", "server2:net0"]
- endpoints: ["br-leaf1:br-leaf1-net0", "server3:net0"]
- endpoints: ["br-leaf1:br-leaf1-net1", "server4:net0"]
- endpoints: ["leaf0:eth1", "spine0:eth1"]
- endpoints: ["leaf0:eth2", "spine1:eth1"]
- endpoints: ["leaf0:eth3", "br-leaf0:br-leaf0-net2"]
- endpoints: ["leaf1:eth1", "spine0:eth2"]
- endpoints: ["leaf1:eth2", "spine1:eth2"]
- endpoints: ["leaf1:eth3", "br-leaf1:br-leaf1-net2"]
VyOS 配置文件
spine0-boot.cfg
配置文件
# ./startup-conf/spine0-boot.cfg
interfaces {
ethernet eth1 {
address 10.1.10.2/24
duplex auto
smp-affinity auto
speed auto
}
ethernet eth2 {
address 10.1.34.2/24
duplex auto
smp-affinity auto
speed auto
}
loopback lo {
}
}
protocols {
# 配置 bgp 信息，bpg 自治系统编号 500
bgp 500 {
# 配置其他 AS 信息 包括 自治系统编号
neighbor 10.1.10.1 {
remote-as 65005
}
# 配置其他 AS 信息 包括 自治系统编号
neighbor 10.1.34.1 {
remote-as 65008
}
parameters {
# 指定了 BGP 路由器的路由器 ID
router-id 10.1.10.2
}
}
}
system {
config-management {
commit-revisions 100
}
console {
device ttyS0 {
speed 9600
}
}
host-name spine0
login {
user vyos {
authentication {
encrypted-password $6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/
plaintext-password ""
}
level admin
}
}
ntp {
server 0.pool.ntp.org {
}
server 1.pool.ntp.org {
}
server 2.pool.ntp.org {
}
}
syslog {
global {
facility all {
level info
}
facility protocols {
level debug
}
}
}
time-zone UTC
}
/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "qos@1:dhcp-server@5:webgui@1:pppoe-server@2:webproxy@2:firewall@5:pptp@1:dns-forwarding@1:mdns@1:quagga@7:webproxy@1:snmp@1:system@10:conntrack@1:l2tp@1:broadcast-relay@1:dhcp-relay@2:conntrack-sync@1:vrrp@2:ipsec@5:ntp@1:config-management@1:wanloadbalance@3:ssh@1:nat@4:zone-policy@1:cluster@1" === */
/* Release version: 1.2.8 */
spine1-boot.cfg
配置文件
# ./startup-conf/spine1-boot.cfg
interfaces {
ethernet eth1 {
address 10.1.12.2/24
duplex auto
smp-affinity auto
speed auto
}
ethernet eth2 {
address 10.1.11.2/24
duplex auto
smp-affinity auto
speed auto
}
loopback lo {
}
}
protocols {
bgp 800 {
neighbor 10.1.11.1 {
remote-as 65008
}
neighbor 10.1.12.1 {
remote-as 65005
}
parameters {
router-id 10.1.12.2
}
}
}
system {
config-management {
commit-revisions 100
}
console {
device ttyS0 {
speed 9600
}
}
host-name spine1
login {
user vyos {
authentication {
encrypted-password $6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/
plaintext-password ""
}
level admin
}
}
ntp {
server 0.pool.ntp.org {
}
server 1.pool.ntp.org {
}
server 2.pool.ntp.org {
}
}
syslog {
global {
facility all {
level info
}
facility protocols {
level debug
}
}
}
time-zone UTC
}
/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "qos@1:dhcp-server@5:webgui@1:pppoe-server@2:webproxy@2:firewall@5:pptp@1:dns-forwarding@1:mdns@1:quagga@7:webproxy@1:snmp@1:system@10:conntrack@1:l2tp@1:broadcast-relay@1:dhcp-relay@2:conntrack-sync@1:vrrp@2:ipsec@5:ntp@1:config-management@1:wanloadbalance@3:ssh@1:nat@4:zone-policy@1:cluster@1" === */
/* Release version: 1.2.8 */
leaf0-boot.cfg
配置文件
# ./startup-conf/leaf0-boot.cfg
interfaces {
ethernet eth1 {
address 10.1.10.1/24
duplex auto
smp-affinity auto
speed auto
}
ethernet eth2 {
address 10.1.12.1/24
duplex auto
smp-affinity auto
speed auto
}
ethernet eth3 {
address 10.1.5.1/24
duplex auto
smp-affinity auto
speed auto
}
loopback lo {
}
}
# 配置 nat 信息，网络下的服务器可以访问外网
nat {
source {
rule 100 {
outbound-interface eth0
source {
address 10.1.0.0/16
}
translation {
address masquerade
}
}
}
}
protocols {
# 配置 bgp 信息，bpg 自治系统编号 65005
bgp 65005 {
# 本地网络段
address-family {
ipv4-unicast {
network 10.1.5.0/24 {
}
}
}
# 配置 客户机（Client） 以及 Client 自治系统编号
neighbor 10.1.5.10 {
address-family {
ipv4-unicast {
route-reflector-client
}
}
remote-as 65005
}
# 配置 客户机（Client） 以及 Client 自治系统编号
neighbor 10.1.5.11 {
address-family {
ipv4-unicast {
route-reflector-client
}
}
remote-as 65005
}
# 配置其他 AS 信息 包括 自治系统编号
neighbor 10.1.10.2 {
remote-as 500
}
# 配置其他 AS 信息 包括 自治系统编号
neighbor 10.1.12.2 {
remote-as 800
}
parameters {
# BGP 能够在不同的 AS-PATH 下使用多路径路由 ECMP
bestpath {
as-path {
multipath-relax
}
}
# 指定了 BGP 路由器的路由器 ID
router-id 10.1.5.1
}
}
}
system {
config-management {
commit-revisions 100
}
console {
device ttyS0 {
speed 9600
}
}
host-name leaf0
login {
user vyos {
authentication {
encrypted-password $6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/
plaintext-password ""
}
level admin
}
}
ntp {
server 0.pool.ntp.org {
}
server 1.pool.ntp.org {
}
server 2.pool.ntp.org {
}
}
syslog {
global {
facility all {
level info
}
facility protocols {
level debug
}
}
}
time-zone UTC
}
/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "qos@1:dhcp-server@5:webgui@1:pppoe-server@2:webproxy@2:firewall@5:pptp@1:dns-forwarding@1:mdns@1:quagga@7:webproxy@1:snmp@1:system@10:conntrack@1:l2tp@1:broadcast-relay@1:dhcp-relay@2:conntrack-sync@1:vrrp@2:ipsec@5:ntp@1:config-management@1:wanloadbalance@3:ssh@1:nat@4:zone-policy@1:cluster@1" === */
/* Release version: 1.2.8 */
leaf1-boot.cfg
配置文件
# ./startup-conf/leaf1-boot.cfg
interfaces {
ethernet eth1 {
address 10.1.34.1/24
duplex auto
smp-affinity auto
speed auto
}
ethernet eth2 {
address 10.1.11.1/24
duplex auto
smp-affinity auto
speed auto
}
ethernet eth3 {
address 10.1.8.1/24
duplex auto
smp-affinity auto
speed auto
}
loopback lo {
}
}
nat {
source {
rule 100 {
outbound-interface eth0
source {
address 10.1.0.0/16
}
translation {
address masquerade
}
}
}
}
protocols {
bgp 65008 {
address-family {
ipv4-unicast {
network 10.1.8.0/24 {
}
}
}
neighbor 10.1.8.10 {
address-family {
ipv4-unicast {
route-reflector-client
}
}
remote-as 65008
}
neighbor 10.1.8.11 {
address-family {
ipv4-unicast {
route-reflector-client
}
}
remote-as 65008
}
neighbor 10.1.11.2 {
remote-as 800
}
neighbor 10.1.34.2 {
remote-as 500
}
parameters {
bestpath {
as-path {
multipath-relax
}
}
router-id 10.1.8.1
}
}
}
system {
config-management {
commit-revisions 100
}
console {
device ttyS0 {
speed 9600
}
}
host-name leaf1
login {
user vyos {
authentication {
encrypted-password $6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/
plaintext-password ""
}
level admin
}
}
ntp {
server 0.pool.ntp.org {
}
server 1.pool.ntp.org {
}
server 2.pool.ntp.org {
}
}
syslog {
global {
facility all {
level info
}
facility protocols {
level debug
}
}
}
time-zone UTC
}
/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "qos@1:dhcp-server@5:webgui@1:pppoe-server@2:webproxy@2:firewall@5:pptp@1:dns-forwarding@1:mdns@1:quagga@7:webproxy@1:snmp@1:system@10:conntrack@1:l2tp@1:broadcast-relay@1:dhcp-relay@2:conntrack-sync@1:vrrp@2:ipsec@5:ntp@1:config-management@1:wanloadbalance@3:ssh@1:nat@4:zone-policy@1:cluster@1" === */
/* Release version: 1.2.8 */
部署服务
# tree -L 2 ./
./
├── calico.bgp.clab.yml
└── startup-conf
├── spine0-boot.cfg
├── spine1-boot.cfg
├── leaf0-boot.cfg
└── leaf1-boot.cfg
# clab deploy -t calico.bgp.clab.yml
INFO[0000] Containerlab v0.54.2 started
INFO[0000] Parsing & checking topology file: clab.yaml
INFO[0000] Creating docker network: Name="clab", IPv4Subnet="172.20.20.0/24", IPv6Subnet="2001:172:20:20::/64", MTU=1500
INFO[0000] Creating lab directory: /root/wcni-kind/calico/6-kind-calico-bgp-rr/clab-calico-bgp-rr
WARN[0000] node calico-bgp-rr-control-plane referenced in namespace sharing not found in topology definition, considering it an external dependency.
WARN[0000] node calico-bgp-rr-worker3 referenced in namespace sharing not found in topology definition, considering it an external dependency.
WARN[0000] node calico-bgp-rr-worker referenced in namespace sharing not found in topology definition, considering it an external dependency.
WARN[0000] node calico-bgp-rr-worker2 referenced in namespace sharing not found in topology definition, considering it an external dependency.
INFO[0000] Creating container: "spine1"
INFO[0000] Creating container: "leaf1"
INFO[0002] Created link: leaf1:eth2 <--> spine1:eth2
INFO[0002] Creating container: "spine0"
INFO[0002] Creating container: "leaf0"
INFO[0004] Created link: leaf1:eth1 <--> spine0:eth2
INFO[0004] Created link: leaf1:eth3 <--> br-leaf1:br-leaf1-net2
INFO[0004] Creating container: "server4"
INFO[0004] Created link: leaf0:eth1 <--> spine0:eth1
INFO[0004] Created link: leaf0:eth2 <--> spine1:eth1
INFO[0005] Created link: leaf0:eth3 <--> br-leaf0:br-leaf0-net2
INFO[0005] Created link: br-leaf1:br-leaf1-net1 <--> server4:net0
INFO[0005] Creating container: "server1"
INFO[0006] Creating container: "server3"
INFO[0007] Created link: br-leaf0:br-leaf0-net0 <--> server1:net0
INFO[0007] Created link: br-leaf1:br-leaf1-net0 <--> server3:net0
INFO[0008] Creating container: "server2"
INFO[0009] Created link: br-leaf0:br-leaf0-net1 <--> server2:net0
INFO[0009] Executed command "ip addr add 10.1.8.11/24 dev net0" on the node "server4". stdout:
INFO[0009] Executed command "ip route replace default via 10.1.8.1" on the node "server4". stdout:
INFO[0009] Executed command "ip addr add 10.1.5.10/24 dev net0" on the node "server1". stdout:
INFO[0009] Executed command "ip route replace default via 10.1.5.1" on the node "server1". stdout:
INFO[0009] Executed command "ip addr add 10.1.8.10/24 dev net0" on the node "server3". stdout:
INFO[0009] Executed command "ip route replace default via 10.1.8.1" on the node "server3". stdout:
INFO[0009] Executed command "ip addr add 10.1.5.11/24 dev net0" on the node "server2". stdout:
INFO[0009] Executed command "ip route replace default via 10.1.5.1" on the node "server2". stdout:
INFO[0009] Adding containerlab host entries to /etc/hosts file
INFO[0010] Adding ssh config for containerlab nodes
INFO[0010] 🎉 New containerlab version 0.56.0 is available! Release notes: https://containerlab.dev/rn/0.56/
Run 'containerlab version upgrade' to upgrade or go check other installation options at https://containerlab.dev/install/
+---+----------------------------+--------------+------------------------------------------+-------+---------+----------------+----------------------+
| # | Name | Container ID | Image | Kind | State | IPv4 Address | IPv6 Address |
+---+----------------------------+--------------+------------------------------------------+-------+---------+----------------+----------------------+
| 1 | clab-calico-bgp-rr-leaf0 | f0193dfa4fa8 | vyos/vyos:1.2.8 | linux | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 2 | clab-calico-bgp-rr-leaf1 | 79b6dacb155d | vyos/vyos:1.2.8 | linux | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 3 | clab-calico-bgp-rr-server1 | 33f3138d53e5 | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A | N/A |
| 4 | clab-calico-bgp-rr-server2 | 1b9ef1142550 | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A | N/A |
| 5 | clab-calico-bgp-rr-server3 | bc6d9ccd46d0 | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A | N/A |
| 6 | clab-calico-bgp-rr-server4 | 3010cb806d17 | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A | N/A |
| 7 | clab-calico-bgp-rr-spine0 | 9a988b928fbe | vyos/vyos:1.2.8 | linux | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 8 | clab-calico-bgp-rr-spine1 | bf694b65bd2a | vyos/vyos:1.2.8 | linux | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
+---+----------------------------+--------------+------------------------------------------+-------+---------+----------------+----------------------+
检查 k8s 集群信息
root@kind:~# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
calico-bgp-rr-control-plane NotReady control-plane,master 10m v1.23.4 10.1.5.10 <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
calico-bgp-rr-worker NotReady <none> 9m44s v1.23.4 10.1.5.11 <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
calico-bgp-rr-worker2 NotReady <none> 9m44s v1.23.4 10.1.8.10 <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
calico-bgp-rr-worker3 NotReady <none> 9m44s v1.23.4 10.1.8.11 <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
# 查看 node 节点 ip 信息
root@kind:~# docker exec -it calico-bgp-rr-control-plane ip a l
11: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fc00:f853:ccd:e793::2/64 scope global nodad
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe12:2/64 scope link
valid_lft forever preferred_lft forever
34: net0@if35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default
link/ether aa:c1:ab:21:21:b1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.1.5.10/24 scope global net0
valid_lft forever preferred_lft forever
inet6 fe80::a8c1:abff:fe21:21b1/64 scope link
valid_lft forever preferred_lft forever
# 查看 node 节点路由信息
root@kind:~# docker exec -it calico-bgp-rr-control-plane ip r s
default via 10.1.5.1 dev net0
10.1.5.0/24 dev net0 proto kernel scope link src 10.1.5.10
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.2
查看 k8s 集群发现
node节点ip地址分配了，登陆容器查看到了新的ip地址，并且默认路由信息调整为了10.1.5.0/24 dev net0 proto kernel scope link src 10.1.5.10
安装 calico 服务
calico.yaml
配置文件
# ./calico.yaml
---
# Source: calico/templates/calico-config.yaml
# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
name: calico-config
namespace: kube-system
data:
# Typha is disabled.
typha_service_name: "none"
# Configure the backend to use.
calico_backend: "bird"
# Configure the MTU to use for workload interfaces and tunnels.
# By default, MTU is auto-detected, and explicitly setting this field should not be required.
# You can override auto-detection by providing a non-zero value.
veth_mtu: "0"
# The CNI network configuration to install on each node. The special
# values in this config will be automatically populated.
cni_network_config: |-
{
"name": "k8s-pod-network",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "calico",
"log_level": "info",
"log_file_path": "/var/log/calico/cni/cni.log",
"datastore_type": "kubernetes",
"nodename": "__KUBERNETES_NODE_NAME__",
"mtu": __CNI_MTU__,
"ipam": {
"type": "calico-ipam"
},
"policy": {
"type": "k8s"
},
"kubernetes": {
"kubeconfig": "__KUBECONFIG_FILEPATH__"
}
},
{
"type": "portmap",
"snat": true,
"capabilities": {"portMappings": true}
},
{
"type": "bandwidth",
"capabilities": {"bandwidth": true}
}
]
}
---
# Source: calico/templates/kdd-crds.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: bgpconfigurations.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: BGPConfiguration
listKind: BGPConfigurationList
plural: bgpconfigurations
singular: bgpconfiguration
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
description: BGPConfiguration contains the configuration for any BGP routing.
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: BGPConfigurationSpec contains the values of the BGP configuration.
properties:
asNumber:
description: 'ASNumber is the default AS number used by a node. [Default:
64512]'
format: int32
type: integer
bindMode:
description: BindMode indicates whether to listen for BGP connections
on all addresses (None) or only on the node's canonical IP address
Node.Spec.BGP.IPvXAddress (NodeIP). Default behaviour is to listen
for BGP connections on all addresses.
type: string
communities:
description: Communities is a list of BGP community values and their
arbitrary names for tagging routes.
items:
description: Community contains standard or large community value
and its name.
properties:
name:
description: Name given to community value.
type: string
value:
description: Value must be of format `aa:nn` or `aa:nn:mm`.
For standard community use `aa:nn` format, where `aa` and
`nn` are 16 bit number. For large community use `aa:nn:mm`
format, where `aa`, `nn` and `mm` are 32 bit number. Where,
`aa` is an AS Number, `nn` and `mm` are per-AS identifier.
pattern: ^(\d+):(\d+)$|^(\d+):(\d+):(\d+)$
type: string
type: object
type: array
listenPort:
description: ListenPort is the port where BGP protocol should listen.
Defaults to 179
maximum: 65535
minimum: 1
type: integer
logSeverityScreen:
description: 'LogSeverityScreen is the log severity above which logs
are sent to the stdout. [Default: INFO]'
type: string
nodeMeshMaxRestartTime:
description: Time to allow for software restart for node-to-mesh peerings. When
specified, this is configured as the graceful restart timeout. When
not specified, the BIRD default of 120s is used. This field can
only be set on the default BGPConfiguration instance and requires
that NodeMesh is enabled
type: string
nodeMeshPassword:
description: Optional BGP password for full node-to-mesh peerings.
This field can only be set on the default BGPConfiguration instance
and requires that NodeMesh is enabled
properties:
secretKeyRef:
description: Selects a key of a secret in the node pod's namespace.
properties:
key:
description: The key of the secret to select from. Must be
a valid secret key.
type: string
name:
description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Add other useful fields. apiVersion, kind, uid?'
type: string
optional:
description: Specify whether the Secret or its key must be
defined
type: boolean
required:
- key
type: object
type: object
nodeToNodeMeshEnabled:
description: 'NodeToNodeMeshEnabled sets whether full node to node
BGP mesh is enabled. [Default: true]'
type: boolean
prefixAdvertisements:
description: PrefixAdvertisements contains per-prefix advertisement
configuration.
items:
description: PrefixAdvertisement configures advertisement properties
for the specified CIDR.
properties:
cidr:
description: CIDR for which properties should be advertised.
type: string
communities:
description: Communities can be list of either community names
already defined in `Specs.Communities` or community value
of format `aa:nn` or `aa:nn:mm`. For standard community use
`aa:nn` format, where `aa` and `nn` are 16 bit number. For
large community use `aa:nn:mm` format, where `aa`, `nn` and
`mm` are 32 bit number. Where,`aa` is an AS Number, `nn` and
`mm` are per-AS identifier.
items:
type: string
type: array
type: object
type: array
serviceClusterIPs:
description: ServiceClusterIPs are the CIDR blocks from which service
cluster IPs are allocated. If specified, Calico will advertise these
blocks, as well as any cluster IPs within them.
items:
description: ServiceClusterIPBlock represents a single allowed ClusterIP
CIDR block.
properties:
cidr:
type: string
type: object
type: array
serviceExternalIPs:
description: ServiceExternalIPs are the CIDR blocks for Kubernetes
Service External IPs. Kubernetes Service ExternalIPs will only be
advertised if they are within one of these blocks.
items:
description: ServiceExternalIPBlock represents a single allowed
External IP CIDR block.
properties:
cidr:
type: string
type: object
type: array
serviceLoadBalancerIPs:
description: ServiceLoadBalancerIPs are the CIDR blocks for Kubernetes
Service LoadBalancer IPs. Kubernetes Service status.LoadBalancer.Ingress
IPs will only be advertised if they are within one of these blocks.
items:
description: ServiceLoadBalancerIPBlock represents a single allowed
LoadBalancer IP CIDR block.
properties:
cidr:
type: string
type: object
type: array
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: bgppeers.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: BGPPeer
listKind: BGPPeerList
plural: bgppeers
singular: bgppeer
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: BGPPeerSpec contains the specification for a BGPPeer resource.
properties:
asNumber:
description: The AS Number of the peer.
format: int32
type: integer
keepOriginalNextHop:
description: Option to keep the original nexthop field when routes
are sent to a BGP Peer. Setting "true" configures the selected BGP
Peers node to use the "next hop keep;" instead of "next hop self;"(default)
in the specific branch of the Node on "bird.cfg".
type: boolean
maxRestartTime:
description: Time to allow for software restart. When specified,
this is configured as the graceful restart timeout. When not specified,
the BIRD default of 120s is used.
type: string
node:
description: The node name identifying the Calico node instance that
is targeted by this peer. If this is not set, and no nodeSelector
is specified, then this BGP peer selects all nodes in the cluster.
type: string
nodeSelector:
description: Selector for the nodes that should have this peering. When
this is set, the Node field must be empty.
type: string
numAllowedLocalASNumbers:
description: Maximum number of local AS numbers that are allowed in
the AS path for received routes. This removes BGP loop prevention
and should only be used if absolutely necesssary.
format: int32
type: integer
password:
description: Optional BGP password for the peerings generated by this
BGPPeer resource.
properties:
secretKeyRef:
description: Selects a key of a secret in the node pod's namespace.
properties:
key:
description: The key of the secret to select from. Must be
a valid secret key.
type: string
name:
description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Add other useful fields. apiVersion, kind, uid?'
type: string
optional:
description: Specify whether the Secret or its key must be
defined
type: boolean
required:
- key
type: object
type: object
peerIP:
description: The IP address of the peer followed by an optional port
number to peer with. If port number is given, format should be `[<IPv6>]:port`
or `<IPv4>:<port>` for IPv4. If optional port number is not set,
and this peer IP and ASNumber belongs to a calico/node with ListenPort
set in BGPConfiguration, then we use that port to peer.
type: string
peerSelector:
description: Selector for the remote nodes to peer with. When this
is set, the PeerIP and ASNumber fields must be empty. For each
peering between the local node and selected remote nodes, we configure
an IPv4 peering if both ends have NodeBGPSpec.IPv4Address specified,
and an IPv6 peering if both ends have NodeBGPSpec.IPv6Address specified. The
remote AS number comes from the remote node's NodeBGPSpec.ASNumber,
or the global default if that is not set.
type: string
sourceAddress:
description: Specifies whether and how to configure a source address
for the peerings generated by this BGPPeer resource. Default value
"UseNodeIP" means to configure the node IP as the source address. "None"
means not to configure a source address.
type: string
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: blockaffinities.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: BlockAffinity
listKind: BlockAffinityList
plural: blockaffinities
singular: blockaffinity
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: BlockAffinitySpec contains the specification for a BlockAffinity
resource.
properties:
cidr:
type: string
deleted:
description: Deleted indicates that this block affinity is being deleted.
This field is a string for compatibility with older releases that
mistakenly treat this field as a string.
type: string
node:
type: string
state:
type: string
required:
- cidr
- deleted
- node
- state
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: (devel)
creationTimestamp: null
name: caliconodestatuses.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: CalicoNodeStatus
listKind: CalicoNodeStatusList
plural: caliconodestatuses
singular: caliconodestatus
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: CalicoNodeStatusSpec contains the specification for a CalicoNodeStatus
resource.
properties:
classes:
description: Classes declares the types of information to monitor
for this calico/node, and allows for selective status reporting
about certain subsets of information.
items:
type: string
type: array
node:
description: The node name identifies the Calico node instance for
node status.
type: string
updatePeriodSeconds:
description: UpdatePeriodSeconds is the period at which CalicoNodeStatus
should be updated. Set to 0 to disable CalicoNodeStatus refresh.
Maximum update period is one day.
format: int32
type: integer
type: object
status:
description: CalicoNodeStatusStatus defines the observed state of CalicoNodeStatus.
No validation needed for status since it is updated by Calico.
properties:
agent:
description: Agent holds agent status on the node.
properties:
birdV4:
description: BIRDV4 represents the latest observed status of bird4.
properties:
lastBootTime:
description: LastBootTime holds the value of lastBootTime
from bird.ctl output.
type: string
lastReconfigurationTime:
description: LastReconfigurationTime holds the value of lastReconfigTime
from bird.ctl output.
type: string
routerID:
description: Router ID used by bird.
type: string
state:
description: The state of the BGP Daemon.
type: string
version:
description: Version of the BGP daemon
type: string
type: object
birdV6:
description: BIRDV6 represents the latest observed status of bird6.
properties:
lastBootTime:
description: LastBootTime holds the value of lastBootTime
from bird.ctl output.
type: string
lastReconfigurationTime:
description: LastReconfigurationTime holds the value of lastReconfigTime
from bird.ctl output.
type: string
routerID:
description: Router ID used by bird.
type: string
state:
description: The state of the BGP Daemon.
type: string
version:
description: Version of the BGP daemon
type: string
type: object
type: object
bgp:
description: BGP holds node BGP status.
properties:
numberEstablishedV4:
description: The total number of IPv4 established bgp sessions.
type: integer
numberEstablishedV6:
description: The total number of IPv6 established bgp sessions.
type: integer
numberNotEstablishedV4:
description: The total number of IPv4 non-established bgp sessions.
type: integer
numberNotEstablishedV6:
description: The total number of IPv6 non-established bgp sessions.
type: integer
peersV4:
description: PeersV4 represents IPv4 BGP peers status on the node.
items:
description: CalicoNodePeer contains the status of BGP peers
on the node.
properties:
peerIP:
description: IP address of the peer whose condition we are
reporting.
type: string
since:
description: Since the state or reason last changed.
type: string
state:
description: State is the BGP session state.
type: string
type:
description: Type indicates whether this peer is configured
via the node-to-node mesh, or via en explicit global or
per-node BGPPeer object.
type: string
type: object
type: array
peersV6:
description: PeersV6 represents IPv6 BGP peers status on the node.
items:
description: CalicoNodePeer contains the status of BGP peers
on the node.
properties:
peerIP:
description: IP address of the peer whose condition we are
reporting.
type: string
since:
description: Since the state or reason last changed.
type: string
state:
description: State is the BGP session state.
type: string
type:
description: Type indicates whether this peer is configured
via the node-to-node mesh, or via en explicit global or
per-node BGPPeer object.
type: string
type: object
type: array
required:
- numberEstablishedV4
- numberEstablishedV6
- numberNotEstablishedV4
- numberNotEstablishedV6
type: object
lastUpdated:
description: LastUpdated is a timestamp representing the server time
when CalicoNodeStatus object last updated. It is represented in
RFC3339 form and is in UTC.
format: date-time
nullable: true
type: string
routes:
description: Routes reports routes known to the Calico BGP daemon
on the node.
properties:
routesV4:
description: RoutesV4 represents IPv4 routes on the node.
items:
description: CalicoNodeRoute contains the status of BGP routes
on the node.
properties:
destination:
description: Destination of the route.
type: string
gateway:
description: Gateway for the destination.
type: string
interface:
description: Interface for the destination
type: string
learnedFrom:
description: LearnedFrom contains information regarding
where this route originated.
properties:
peerIP:
description: If sourceType is NodeMesh or BGPPeer, IP
address of the router that sent us this route.
type: string
sourceType:
description: Type of the source where a route is learned
from.
type: string
type: object
type:
description: Type indicates if the route is being used for
forwarding or not.
type: string
type: object
type: array
routesV6:
description: RoutesV6 represents IPv6 routes on the node.
items:
description: CalicoNodeRoute contains the status of BGP routes
on the node.
properties:
destination:
description: Destination of the route.
type: string
gateway:
description: Gateway for the destination.
type: string
interface:
description: Interface for the destination
type: string
learnedFrom:
description: LearnedFrom contains information regarding
where this route originated.
properties:
peerIP:
description: If sourceType is NodeMesh or BGPPeer, IP
address of the router that sent us this route.
type: string
sourceType:
description: Type of the source where a route is learned
from.
type: string
type: object
type:
description: Type indicates if the route is being used for
forwarding or not.
type: string
type: object
type: array
type: object
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: clusterinformations.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: ClusterInformation
listKind: ClusterInformationList
plural: clusterinformations
singular: clusterinformation
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
description: ClusterInformation contains the cluster specific information.
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: ClusterInformationSpec contains the values of describing
the cluster.
properties:
calicoVersion:
description: CalicoVersion is the version of Calico that the cluster
is running
type: string
clusterGUID:
description: ClusterGUID is the GUID of the cluster
type: string
clusterType:
description: ClusterType describes the type of the cluster
type: string
datastoreReady:
description: DatastoreReady is used during significant datastore migrations
to signal to components such as Felix that it should wait before
accessing the datastore.
type: boolean
variant:
description: Variant declares which variant of Calico should be active.
type: string
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: felixconfigurations.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: FelixConfiguration
listKind: FelixConfigurationList
plural: felixconfigurations
singular: felixconfiguration
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
description: Felix Configuration contains the configuration for Felix.
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: FelixConfigurationSpec contains the values of the Felix configuration.
properties:
allowIPIPPacketsFromWorkloads:
description: 'AllowIPIPPacketsFromWorkloads controls whether Felix
will add a rule to drop IPIP encapsulated traffic from workloads
[Default: false]'
type: boolean
allowVXLANPacketsFromWorkloads:
description: 'AllowVXLANPacketsFromWorkloads controls whether Felix
will add a rule to drop VXLAN encapsulated traffic from workloads
[Default: false]'
type: boolean
awsSrcDstCheck:
description: 'Set source-destination-check on AWS EC2 instances. Accepted
value must be one of "DoNothing", "Enable" or "Disable". [Default:
DoNothing]'
enum:
- DoNothing
- Enable
- Disable
type: string
bpfConnectTimeLoadBalancingEnabled:
description: 'BPFConnectTimeLoadBalancingEnabled when in BPF mode,
controls whether Felix installs the connection-time load balancer. The
connect-time load balancer is required for the host to be able to
reach Kubernetes services and it improves the performance of pod-to-service
connections. The only reason to disable it is for debugging purposes. [Default:
true]'
type: boolean
bpfDataIfacePattern:
description: BPFDataIfacePattern is a regular expression that controls
which interfaces Felix should attach BPF programs to in order to
catch traffic to/from the network. This needs to match the interfaces
that Calico workload traffic flows over as well as any interfaces
that handle incoming traffic to nodeports and services from outside
the cluster. It should not match the workload interfaces (usually
named cali...).
type: string
bpfDisableUnprivileged:
description: 'BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled
sysctl to disable unprivileged use of BPF. This ensures that unprivileged
users cannot access Calico''s BPF maps and cannot insert their own
BPF programs to interfere with Calico''s. [Default: true]'
type: boolean
bpfEnabled:
description: 'BPFEnabled, if enabled Felix will use the BPF dataplane.
[Default: false]'
type: boolean
bpfEnforceRPF:
description: 'BPFEnforceRPF enforce strict RPF on all interfaces with
BPF programs regardless of what is the per-interfaces or global
setting. Possible values are Disabled or Strict. [Default: Strict]'
type: string
bpfExtToServiceConnmark:
description: 'BPFExtToServiceConnmark in BPF mode, control a 32bit
mark that is set on connections from an external client to a local
service. This mark allows us to control how packets of that connection
are routed within the host and how is routing intepreted by RPF
check. [Default: 0]'
type: integer
bpfExternalServiceMode:
description: 'BPFExternalServiceMode in BPF mode, controls how connections
from outside the cluster to services (node ports and cluster IPs)
are forwarded to remote workloads. If set to "Tunnel" then both
request and response traffic is tunneled to the remote node. If
set to "DSR", the request traffic is tunneled but the response traffic
is sent directly from the remote node. In "DSR" mode, the remote
node appears to use the IP of the ingress node; this requires a
permissive L2 network. [Default: Tunnel]'
type: string
bpfKubeProxyEndpointSlicesEnabled:
description: BPFKubeProxyEndpointSlicesEnabled in BPF mode, controls
whether Felix's embedded kube-proxy accepts EndpointSlices or not.
type: boolean
bpfKubeProxyIptablesCleanupEnabled:
description: 'BPFKubeProxyIptablesCleanupEnabled, if enabled in BPF
mode, Felix will proactively clean up the upstream Kubernetes kube-proxy''s
iptables chains. Should only be enabled if kube-proxy is not running. [Default:
true]'
type: boolean
bpfKubeProxyMinSyncPeriod:
description: 'BPFKubeProxyMinSyncPeriod, in BPF mode, controls the
minimum time between updates to the dataplane for Felix''s embedded
kube-proxy. Lower values give reduced set-up latency. Higher values
reduce Felix CPU usage by batching up more work. [Default: 1s]'
type: string
bpfLogLevel:
description: 'BPFLogLevel controls the log level of the BPF programs
when in BPF dataplane mode. One of "Off", "Info", or "Debug". The
logs are emitted to the BPF trace pipe, accessible with the command
`tc exec bpf debug`. [Default: Off].'
type: string
bpfMapSizeConntrack:
description: 'BPFMapSizeConntrack sets the size for the conntrack
map. This map must be large enough to hold an entry for each active
connection. Warning: changing the size of the conntrack map can
cause disruption.'
type: integer
bpfMapSizeIPSets:
description: BPFMapSizeIPSets sets the size for ipsets map. The IP
sets map must be large enough to hold an entry for each endpoint
matched by every selector in the source/destination matches in network
policy. Selectors such as "all()" can result in large numbers of
entries (one entry per endpoint in that case).
type: integer
bpfMapSizeNATAffinity:
type: integer
bpfMapSizeNATBackend:
description: BPFMapSizeNATBackend sets the size for nat back end map.
This is the total number of endpoints. This is mostly more than
the size of the number of services.
type: integer
bpfMapSizeNATFrontend:
description: BPFMapSizeNATFrontend sets the size for nat front end
map. FrontendMap should be large enough to hold an entry for each
nodeport, external IP and each port in each service.
type: integer
bpfMapSizeRoute:
description: BPFMapSizeRoute sets the size for the routes map. The
routes map should be large enough to hold one entry per workload
and a handful of entries per host (enough to cover its own IPs and
tunnel IPs).
type: integer
bpfPSNATPorts:
anyOf:
- type: integer
- type: string
description: 'BPFPSNATPorts sets the range from which we randomly
pick a port if there is a source port collision. This should be
within the ephemeral range as defined by RFC 6056 (1024–65535) and
preferably outside the ephemeral ranges used by common operating
systems. Linux uses 32768–60999, while others mostly use the IANA
defined range 49152–65535. It is not necessarily a problem if this
range overlaps with the operating systems. Both ends of the range
are inclusive. [Default: 20000:29999]'
pattern: ^.*
x-kubernetes-int-or-string: true
chainInsertMode:
description: 'ChainInsertMode controls whether Felix hooks the kernel''s
top-level iptables chains by inserting a rule at the top of the
chain or by appending a rule at the bottom. insert is the safe default
since it prevents Calico''s rules from being bypassed. If you switch
to append mode, be sure that the other rules in the chains signal
acceptance by falling through to the Calico rules, otherwise the
Calico policy will be bypassed. [Default: insert]'
type: string
dataplaneDriver:
description: DataplaneDriver filename of the external dataplane driver
to use. Only used if UseInternalDataplaneDriver is set to false.
type: string
dataplaneWatchdogTimeout:
description: 'DataplaneWatchdogTimeout is the readiness/liveness timeout
used for Felix''s (internal) dataplane driver. Increase this value
if you experience spurious non-ready or non-live events when Felix
is under heavy load. Decrease the value to get felix to report non-live
or non-ready more quickly. [Default: 90s]'
type: string
debugDisableLogDropping:
type: boolean
debugMemoryProfilePath:
type: string
debugSimulateCalcGraphHangAfter:
type: string
debugSimulateDataplaneHangAfter:
type: string
defaultEndpointToHostAction:
description: 'DefaultEndpointToHostAction controls what happens to
traffic that goes from a workload endpoint to the host itself (after
the traffic hits the endpoint egress policy). By default Calico
blocks traffic from workload endpoints to the host itself with an
iptables "DROP" action. If you want to allow some or all traffic
from endpoint to host, set this parameter to RETURN or ACCEPT. Use
RETURN if you have your own rules in the iptables "INPUT" chain;
Calico will insert its rules at the top of that chain, then "RETURN"
packets to the "INPUT" chain once it has completed processing workload
endpoint egress policy. Use ACCEPT to unconditionally accept packets
from workloads after processing workload endpoint egress policy.
[Default: Drop]'
type: string
deviceRouteProtocol:
description: This defines the route protocol added to programmed device
routes, by default this will be RTPROT_BOOT when left blank.
type: integer
deviceRouteSourceAddress:
description: This is the IPv4 source address to use on programmed
device routes. By default the source address is left blank, leaving
the kernel to choose the source address used.
type: string
deviceRouteSourceAddressIPv6:
description: This is the IPv6 source address to use on programmed
device routes. By default the source address is left blank, leaving
the kernel to choose the source address used.
type: string
disableConntrackInvalidCheck:
type: boolean
endpointReportingDelay:
type: string
endpointReportingEnabled:
type: boolean
externalNodesList:
description: ExternalNodesCIDRList is a list of CIDR's of external-non-calico-nodes
which may source tunnel traffic and have the tunneled traffic be
accepted at calico nodes.
items:
type: string
type: array
failsafeInboundHostPorts:
description: 'FailsafeInboundHostPorts is a list of UDP/TCP ports
and CIDRs that Felix will allow incoming traffic to host endpoints
on irrespective of the security policy. This is useful to avoid
accidentally cutting off a host with incorrect configuration. For
back-compatibility, if the protocol is not specified, it defaults
to "tcp". If a CIDR is not specified, it will allow traffic from
all addresses. To disable all inbound host ports, use the value
none. The default value allows ssh access and DHCP. [Default: tcp:22,
udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]'
items:
description: ProtoPort is combination of protocol, port, and CIDR.
Protocol and port must be specified.
properties:
net:
type: string
port:
type: integer
protocol:
type: string
required:
- port
- protocol
type: object
type: array
failsafeOutboundHostPorts:
description: 'FailsafeOutboundHostPorts is a list of UDP/TCP ports
and CIDRs that Felix will allow outgoing traffic from host endpoints
to irrespective of the security policy. This is useful to avoid
accidentally cutting off a host with incorrect configuration. For
back-compatibility, if the protocol is not specified, it defaults
to "tcp". If a CIDR is not specified, it will allow traffic from
all addresses. To disable all outbound host ports, use the value
none. The default value opens etcd''s standard ports to ensure that
Felix does not get cut off from etcd as well as allowing DHCP and
DNS. [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666,
tcp:6667, udp:53, udp:67]'
items:
description: ProtoPort is combination of protocol, port, and CIDR.
Protocol and port must be specified.
properties:
net:
type: string
port:
type: integer
protocol:
type: string
required:
- port
- protocol
type: object
type: array
featureDetectOverride:
description: FeatureDetectOverride is used to override the feature
detection. Values are specified in a comma separated list with no
spaces, example; "SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=".
"true" or "false" will force the feature, empty or omitted values
are auto-detected.
type: string
floatingIPs:
default: Disabled
description: FloatingIPs configures whether or not Felix will program
floating IP addresses.
enum:
- Enabled
- Disabled
type: string
genericXDPEnabled:
description: 'GenericXDPEnabled enables Generic XDP so network cards
that don''t support XDP offload or driver modes can use XDP. This
is not recommended since it doesn''t provide better performance
than iptables. [Default: false]'
type: boolean
healthEnabled:
type: boolean
healthHost:
type: string
healthPort:
type: integer
interfaceExclude:
description: 'InterfaceExclude is a comma-separated list of interfaces
that Felix should exclude when monitoring for host endpoints. The
default value ensures that Felix ignores Kubernetes'' IPVS dummy
interface, which is used internally by kube-proxy. If you want to
exclude multiple interface names using a single value, the list
supports regular expressions. For regular expressions you must wrap
the value with ''/''. For example having values ''/^kube/,veth1''
will exclude all interfaces that begin with ''kube'' and also the
interface ''veth1''. [Default: kube-ipvs0]'
type: string
interfacePrefix:
description: 'InterfacePrefix is the interface name prefix that identifies
workload endpoints and so distinguishes them from host endpoint
interfaces. Note: in environments other than bare metal, the orchestrators
configure this appropriately. For example our Kubernetes and Docker
integrations set the ''cali'' value, and our OpenStack integration
sets the ''tap'' value. [Default: cali]'
type: string
interfaceRefreshInterval:
description: InterfaceRefreshInterval is the period at which Felix
rescans local interfaces to verify their state. The rescan can be
disabled by setting the interval to 0.
type: string
ipipEnabled:
description: 'IPIPEnabled overrides whether Felix should configure
an IPIP interface on the host. Optional as Felix determines this
based on the existing IP pools. [Default: nil (unset)]'
type: boolean
ipipMTU:
description: 'IPIPMTU is the MTU to set on the tunnel device. See
Configuring MTU [Default: 1440]'
type: integer
ipsetsRefreshInterval:
description: 'IpsetsRefreshInterval is the period at which Felix re-checks
all iptables state to ensure that no other process has accidentally
broken Calico''s rules. Set to 0 to disable iptables refresh. [Default:
90s]'
type: string
iptablesBackend:
description: IptablesBackend specifies which backend of iptables will
be used. The default is legacy.
type: string
iptablesFilterAllowAction:
type: string
iptablesLockFilePath:
description: 'IptablesLockFilePath is the location of the iptables
lock file. You may need to change this if the lock file is not in
its standard location (for example if you have mapped it into Felix''s
container at a different path). [Default: /run/xtables.lock]'
type: string
iptablesLockProbeInterval:
description: 'IptablesLockProbeInterval is the time that Felix will
wait between attempts to acquire the iptables lock if it is not
available. Lower values make Felix more responsive when the lock
is contended, but use more CPU. [Default: 50ms]'
type: string
iptablesLockTimeout:
description: 'IptablesLockTimeout is the time that Felix will wait
for the iptables lock, or 0, to disable. To use this feature, Felix
must share the iptables lock file with all other processes that
also take the lock. When running Felix inside a container, this
requires the /run directory of the host to be mounted into the calico/node
or calico/felix container. [Default: 0s disabled]'
type: string
iptablesMangleAllowAction:
type: string
iptablesMarkMask:
description: 'IptablesMarkMask is the mask that Felix selects its
IPTables Mark bits from. Should be a 32 bit hexadecimal number with
at least 8 bits set, none of which clash with any other mark bits
in use on the system. [Default: 0xff000000]'
format: int32
type: integer
iptablesNATOutgoingInterfaceFilter:
type: string
iptablesPostWriteCheckInterval:
description: 'IptablesPostWriteCheckInterval is the period after Felix
has done a write to the dataplane that it schedules an extra read
back in order to check the write was not clobbered by another process.
This should only occur if another application on the system doesn''t
respect the iptables lock. [Default: 1s]'
type: string
iptablesRefreshInterval:
description: 'IptablesRefreshInterval is the period at which Felix
re-checks the IP sets in the dataplane to ensure that no other process
has accidentally broken Calico''s rules. Set to 0 to disable IP
sets refresh. Note: the default for this value is lower than the
other refresh intervals as a workaround for a Linux kernel bug that
was fixed in kernel version 4.11. If you are using v4.11 or greater
you may want to set this to, a higher value to reduce Felix CPU
usage. [Default: 10s]'
type: string
ipv6Support:
description: IPv6Support controls whether Felix enables support for
IPv6 (if supported by the in-use dataplane).
type: boolean
kubeNodePortRanges:
description: 'KubeNodePortRanges holds list of port ranges used for
service node ports. Only used if felix detects kube-proxy running
in ipvs mode. Felix uses these ranges to separate host and workload
traffic. [Default: 30000:32767].'
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
logDebugFilenameRegex:
description: LogDebugFilenameRegex controls which source code files
have their Debug log output included in the logs. Only logs from
files with names that match the given regular expression are included. The
filter only applies to Debug level logs.
type: string
logFilePath:
description: 'LogFilePath is the full path to the Felix log. Set to
none to disable file logging. [Default: /var/log/calico/felix.log]'
type: string
logPrefix:
description: 'LogPrefix is the log prefix that Felix uses when rendering
LOG rules. [Default: calico-packet]'
type: string
logSeverityFile:
description: 'LogSeverityFile is the log severity above which logs
are sent to the log file. [Default: Info]'
type: string
logSeverityScreen:
description: 'LogSeverityScreen is the log severity above which logs
are sent to the stdout. [Default: Info]'
type: string
logSeveritySys:
description: 'LogSeveritySys is the log severity above which logs
are sent to the syslog. Set to None for no logging to syslog. [Default:
Info]'
type: string
maxIpsetSize:
type: integer
metadataAddr:
description: 'MetadataAddr is the IP address or domain name of the
server that can answer VM queries for cloud-init metadata. In OpenStack,
this corresponds to the machine running nova-api (or in Ubuntu,
nova-api-metadata). A value of none (case insensitive) means that
Felix should not set up any NAT rule for the metadata path. [Default:
127.0.0.1]'
type: string
metadataPort:
description: 'MetadataPort is the port of the metadata server. This,
combined with global.MetadataAddr (if not ''None''), is used to
set up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort.
In most cases this should not need to be changed [Default: 8775].'
type: integer
mtuIfacePattern:
description: MTUIfacePattern is a regular expression that controls
which interfaces Felix should scan in order to calculate the host's
MTU. This should not match workload interfaces (usually named cali...).
type: string
natOutgoingAddress:
description: NATOutgoingAddress specifies an address to use when performing
source NAT for traffic in a natOutgoing pool that is leaving the
network. By default the address used is an address on the interface
the traffic is leaving on (ie it uses the iptables MASQUERADE target)
type: string
natPortRange:
anyOf:
- type: integer
- type: string
description: NATPortRange specifies the range of ports that is used
for port mapping when doing outgoing NAT. When unset the default
behavior of the network stack is used.
pattern: ^.*
x-kubernetes-int-or-string: true
netlinkTimeout:
type: string
openstackRegion:
description: 'OpenstackRegion is the name of the region that a particular
Felix belongs to. In a multi-region Calico/OpenStack deployment,
this must be configured somehow for each Felix (here in the datamodel,
or in felix.cfg or the environment on each compute node), and must
match the [calico] openstack_region value configured in neutron.conf
on each node. [Default: Empty]'
type: string
policySyncPathPrefix:
description: 'PolicySyncPathPrefix is used to by Felix to communicate
policy changes to external services, like Application layer policy.
[Default: Empty]'
type: string
prometheusGoMetricsEnabled:
description: 'PrometheusGoMetricsEnabled disables Go runtime metrics
collection, which the Prometheus client does by default, when set
to false. This reduces the number of metrics reported, reducing
Prometheus load. [Default: true]'
type: boolean
prometheusMetricsEnabled:
description: 'PrometheusMetricsEnabled enables the Prometheus metrics
server in Felix if set to true. [Default: false]'
type: boolean
prometheusMetricsHost:
description: 'PrometheusMetricsHost is the host that the Prometheus
metrics server should bind to. [Default: empty]'
type: string
prometheusMetricsPort:
description: 'PrometheusMetricsPort is the TCP port that the Prometheus
metrics server should bind to. [Default: 9091]'
type: integer
prometheusProcessMetricsEnabled:
description: 'PrometheusProcessMetricsEnabled disables process metrics
collection, which the Prometheus client does by default, when set
to false. This reduces the number of metrics reported, reducing
Prometheus load. [Default: true]'
type: boolean
prometheusWireGuardMetricsEnabled:
description: 'PrometheusWireGuardMetricsEnabled disables wireguard
metrics collection, which the Prometheus client does by default,
when set to false. This reduces the number of metrics reported,
reducing Prometheus load. [Default: true]'
type: boolean
removeExternalRoutes:
description: Whether or not to remove device routes that have not
been programmed by Felix. Disabling this will allow external applications
to also add device routes. This is enabled by default which means
we will remove externally added routes.
type: boolean
reportingInterval:
description: 'ReportingInterval is the interval at which Felix reports
its status into the datastore or 0 to disable. Must be non-zero
in OpenStack deployments. [Default: 30s]'
type: string
reportingTTL:
description: 'ReportingTTL is the time-to-live setting for process-wide
status reports. [Default: 90s]'
type: string
routeRefreshInterval:
description: 'RouteRefreshInterval is the period at which Felix re-checks
the routes in the dataplane to ensure that no other process has
accidentally broken Calico''s rules. Set to 0 to disable route refresh.
[Default: 90s]'
type: string
routeSource:
description: 'RouteSource configures where Felix gets its routing
information. - WorkloadIPs: use workload endpoints to construct
routes. - CalicoIPAM: the default - use IPAM data to construct routes.'
type: string
routeTableRange:
description: Deprecated in favor of RouteTableRanges. Calico programs
additional Linux route tables for various purposes. RouteTableRange
specifies the indices of the route tables that Calico should use.
properties:
max:
type: integer
min:
type: integer
required:
- max
- min
type: object
routeTableRanges:
description: Calico programs additional Linux route tables for various
purposes. RouteTableRanges specifies a set of table index ranges
that Calico should use. Deprecates`RouteTableRange`, overrides `RouteTableRange`.
items:
properties:
max:
type: integer
min:
type: integer
required:
- max
- min
type: object
type: array
serviceLoopPrevention:
description: 'When service IP advertisement is enabled, prevent routing
loops to service IPs that are not in use, by dropping or rejecting
packets that do not get DNAT''d by kube-proxy. Unless set to "Disabled",
in which case such routing loops continue to be allowed. [Default:
Drop]'
type: string
sidecarAccelerationEnabled:
description: 'SidecarAccelerationEnabled enables experimental sidecar
acceleration [Default: false]'
type: boolean
usageReportingEnabled:
description: 'UsageReportingEnabled reports anonymous Calico version
number and cluster size to projectcalico.org. Logs warnings returned
by the usage server. For example, if a significant security vulnerability
has been discovered in the version of Calico being used. [Default:
true]'
type: boolean
usageReportingInitialDelay:
description: 'UsageReportingInitialDelay controls the minimum delay
before Felix makes a report. [Default: 300s]'
type: string
usageReportingInterval:
description: 'UsageReportingInterval controls the interval at which
Felix makes reports. [Default: 86400s]'
type: string
useInternalDataplaneDriver:
description: UseInternalDataplaneDriver, if true, Felix will use its
internal dataplane programming logic. If false, it will launch
an external dataplane driver and communicate with it over protobuf.
type: boolean
vxlanEnabled:
description: 'VXLANEnabled overrides whether Felix should create the
VXLAN tunnel device for VXLAN networking. Optional as Felix determines
this based on the existing IP pools. [Default: nil (unset)]'
type: boolean
vxlanMTU:
description: 'VXLANMTU is the MTU to set on the IPv4 VXLAN tunnel
device. See Configuring MTU [Default: 1410]'
type: integer
vxlanMTUV6:
description: 'VXLANMTUV6 is the MTU to set on the IPv6 VXLAN tunnel
device. See Configuring MTU [Default: 1390]'
type: integer
vxlanPort:
type: integer
vxlanVNI:
type: integer
wireguardEnabled:
description: 'WireguardEnabled controls whether Wireguard is enabled.
[Default: false]'
type: boolean
wireguardHostEncryptionEnabled:
description: 'WireguardHostEncryptionEnabled controls whether Wireguard
host-to-host encryption is enabled. [Default: false]'
type: boolean
wireguardInterfaceName:
description: 'WireguardInterfaceName specifies the name to use for
the Wireguard interface. [Default: wg.calico]'
type: string
wireguardKeepAlive:
description: 'WireguardKeepAlive controls Wireguard PersistentKeepalive
option. Set 0 to disable. [Default: 0]'
type: string
wireguardListeningPort:
description: 'WireguardListeningPort controls the listening port used
by Wireguard. [Default: 51820]'
type: integer
wireguardMTU:
description: 'WireguardMTU controls the MTU on the Wireguard interface.
See Configuring MTU [Default: 1420]'
type: integer
wireguardRoutingRulePriority:
description: 'WireguardRoutingRulePriority controls the priority value
to use for the Wireguard routing rule. [Default: 99]'
type: integer
workloadSourceSpoofing:
description: WorkloadSourceSpoofing controls whether pods can use
the allowedSourcePrefixes annotation to send traffic with a source
IP address that is not theirs. This is disabled by default. When
set to "Any", pods can request any prefix.
type: string
xdpEnabled:
description: 'XDPEnabled enables XDP acceleration for suitable untracked
incoming deny rules. [Default: true]'
type: boolean
xdpRefreshInterval:
description: 'XDPRefreshInterval is the period at which Felix re-checks
all XDP state to ensure that no other process has accidentally broken
Calico''s BPF maps or attached programs. Set to 0 to disable XDP
refresh. [Default: 90s]'
type: string
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: globalnetworkpolicies.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: GlobalNetworkPolicy
listKind: GlobalNetworkPolicyList
plural: globalnetworkpolicies
singular: globalnetworkpolicy
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
properties:
applyOnForward:
description: ApplyOnForward indicates to apply the rules in this policy
on forward traffic.
type: boolean
doNotTrack:
description: DoNotTrack indicates whether packets matched by the rules
in this policy should go through the data plane's connection tracking,
such as Linux conntrack. If True, the rules in this policy are
applied before any data plane connection tracking, and packets allowed
by this policy are marked as not to be tracked.
type: boolean
egress:
description: The ordered set of egress rules. Each rule contains
a set of packet match criteria and a corresponding action to apply.
items:
description: "A Rule encapsulates a set of match criteria and an
action. Both selector-based security Policy and security Profiles
reference rules - separated out as a list of rules for both ingress
and egress packet matching. \n Each positive match criteria has
a negated version, prefixed with \"Not\". All the match criteria
within a rule must be satisfied for a packet to match. A single
rule can contain the positive and negative version of a match
and both must be satisfied for the rule to match."
properties:
action:
type: string
destination:
description: Destination contains the match criteria that apply
to destination entity.
properties:
namespaceSelector:
description: "NamespaceSelector is an optional field that
contains a selector expression. Only traffic that originates
from (or terminates at) endpoints within the selected
namespaces will be matched. When both NamespaceSelector
and another selector are defined on the same rule, then
only workload endpoints that are matched by both selectors
will be selected by the rule. \n For NetworkPolicy, an
empty NamespaceSelector implies that the Selector is limited
to selecting only workload endpoints in the same namespace
as the NetworkPolicy. \n For NetworkPolicy, `global()`
NamespaceSelector implies that the Selector is limited
to selecting only GlobalNetworkSet or HostEndpoint. \n
For GlobalNetworkPolicy, an empty NamespaceSelector implies
the Selector applies to workload endpoints across all
namespaces."
type: string
nets:
description: Nets is an optional field that restricts the
rule to only apply to traffic that originates from (or
terminates at) IP addresses in any of the given subnets.
items:
type: string
type: array
notNets:
description: NotNets is the negated version of the Nets
field.
items:
type: string
type: array
notPorts:
description: NotPorts is the negated version of the Ports
field. Since only some protocols have ports, if any ports
are specified it requires the Protocol match in the Rule
to be set to "TCP" or "UDP".
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
notSelector:
description: NotSelector is the negated version of the Selector
field. See Selector field for subtleties with negated
selectors.
type: string
ports:
description: "Ports is an optional field that restricts
the rule to only apply to traffic that has a source (destination)
port that matches one of these ranges/values. This value
is a list of integers or strings that represent ranges
of ports. \n Since only some protocols have ports, if
any ports are specified it requires the Protocol match
in the Rule to be set to \"TCP\" or \"UDP\"."
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
selector:
description: "Selector is an optional field that contains
a selector expression (see Policy for sample syntax).
\ Only traffic that originates from (terminates at) endpoints
matching the selector will be matched. \n Note that: in
addition to the negated version of the Selector (see NotSelector
below), the selector expression syntax itself supports
negation. The two types of negation are subtly different.
One negates the set of matched endpoints, the other negates
the whole match: \n \tSelector = \"!has(my_label)\" matches
packets that are from other Calico-controlled \tendpoints
that do not have the label \"my_label\". \n \tNotSelector
= \"has(my_label)\" matches packets that are not from
Calico-controlled \tendpoints that do have the label \"my_label\".
\n The effect is that the latter will accept packets from
non-Calico sources whereas the former is limited to packets
from Calico-controlled endpoints."
type: string
serviceAccounts:
description: ServiceAccounts is an optional field that restricts
the rule to only apply to traffic that originates from
(or terminates at) a pod running as a matching service
account.
properties:
names:
description: Names is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account whose name is in the list.
items:
type: string
type: array
selector:
description: Selector is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account that matches the given label selector. If
both Names and Selector are specified then they are
AND'ed.
type: string
type: object
services:
description: "Services is an optional field that contains
options for matching Kubernetes Services. If specified,
only traffic that originates from or terminates at endpoints
within the selected service(s) will be matched, and only
to/from each endpoint's port. \n Services cannot be specified
on the same rule as Selector, NotSelector, NamespaceSelector,
Nets, NotNets or ServiceAccounts. \n Ports and NotPorts
can only be specified with Services on ingress rules."
properties:
name:
description: Name specifies the name of a Kubernetes
Service to match.
type: string
namespace:
description: Namespace specifies the namespace of the
given Service. If left empty, the rule will match
within this policy's namespace.
type: string
type: object
type: object
http:
description: HTTP contains match criteria that apply to HTTP
requests.
properties:
methods:
description: Methods is an optional field that restricts
the rule to apply only to HTTP requests that use one of
the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
methods are OR'd together.
items:
type: string
type: array
paths:
description: 'Paths is an optional field that restricts
the rule to apply to HTTP requests that use one of the
listed HTTP Paths. Multiple paths are OR''d together.
e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
ONLY specify either a `exact` or a `prefix` match. The
validator will check for it.'
items:
description: 'HTTPPath specifies an HTTP path to match.
It may be either of the form: exact: <path>: which matches
the path exactly or prefix: <path-prefix>: which matches
the path prefix'
properties:
exact:
type: string
prefix:
type: string
type: object
type: array
type: object
icmp:
description: ICMP is an optional field that restricts the rule
to apply to a specific type and code of ICMP traffic. This
should only be specified if the Protocol field is set to "ICMP"
or "ICMPv6".
properties:
code:
description: Match on a specific ICMP code. If specified,
the Type value must also be specified. This is a technical
limitation imposed by the kernel's iptables firewall,
which Calico uses to enforce the rule.
type: integer
type:
description: Match on a specific ICMP type. For example
a value of 8 refers to ICMP Echo Request (i.e. pings).
type: integer
type: object
ipVersion:
description: IPVersion is an optional field that restricts the
rule to only match a specific IP version.
type: integer
metadata:
description: Metadata contains additional information for this
rule
properties:
annotations:
additionalProperties:
type: string
description: Annotations is a set of key value pairs that
give extra information about the rule
type: object
type: object
notICMP:
description: NotICMP is the negated version of the ICMP field.
properties:
code:
description: Match on a specific ICMP code. If specified,
the Type value must also be specified. This is a technical
limitation imposed by the kernel's iptables firewall,
which Calico uses to enforce the rule.
type: integer
type:
description: Match on a specific ICMP type. For example
a value of 8 refers to ICMP Echo Request (i.e. pings).
type: integer
type: object
notProtocol:
anyOf:
- type: integer
- type: string
description: NotProtocol is the negated version of the Protocol
field.
pattern: ^.*
x-kubernetes-int-or-string: true
protocol:
anyOf:
- type: integer
- type: string
description: "Protocol is an optional field that restricts the
rule to only apply to traffic of a specific IP protocol. Required
if any of the EntityRules contain Ports (because ports only
apply to certain protocols). \n Must be one of these string
values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
\"UDPLite\" or an integer in the range 1-255."
pattern: ^.*
x-kubernetes-int-or-string: true
source:
description: Source contains the match criteria that apply to
source entity.
properties:
namespaceSelector:
description: "NamespaceSelector is an optional field that
contains a selector expression. Only traffic that originates
from (or terminates at) endpoints within the selected
namespaces will be matched. When both NamespaceSelector
and another selector are defined on the same rule, then
only workload endpoints that are matched by both selectors
will be selected by the rule. \n For NetworkPolicy, an
empty NamespaceSelector implies that the Selector is limited
to selecting only workload endpoints in the same namespace
as the NetworkPolicy. \n For NetworkPolicy, `global()`
NamespaceSelector implies that the Selector is limited
to selecting only GlobalNetworkSet or HostEndpoint. \n
For GlobalNetworkPolicy, an empty NamespaceSelector implies
the Selector applies to workload endpoints across all
namespaces."
type: string
nets:
description: Nets is an optional field that restricts the
rule to only apply to traffic that originates from (or
terminates at) IP addresses in any of the given subnets.
items:
type: string
type: array
notNets:
description: NotNets is the negated version of the Nets
field.
items:
type: string
type: array
notPorts:
description: NotPorts is the negated version of the Ports
field. Since only some protocols have ports, if any ports
are specified it requires the Protocol match in the Rule
to be set to "TCP" or "UDP".
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
notSelector:
description: NotSelector is the negated version of the Selector
field. See Selector field for subtleties with negated
selectors.
type: string
ports:
description: "Ports is an optional field that restricts
the rule to only apply to traffic that has a source (destination)
port that matches one of these ranges/values. This value
is a list of integers or strings that represent ranges
of ports. \n Since only some protocols have ports, if
any ports are specified it requires the Protocol match
in the Rule to be set to \"TCP\" or \"UDP\"."
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
selector:
description: "Selector is an optional field that contains
a selector expression (see Policy for sample syntax).
\ Only traffic that originates from (terminates at) endpoints
matching the selector will be matched. \n Note that: in
addition to the negated version of the Selector (see NotSelector
below), the selector expression syntax itself supports
negation. The two types of negation are subtly different.
One negates the set of matched endpoints, the other negates
the whole match: \n \tSelector = \"!has(my_label)\" matches
packets that are from other Calico-controlled \tendpoints
that do not have the label \"my_label\". \n \tNotSelector
= \"has(my_label)\" matches packets that are not from
Calico-controlled \tendpoints that do have the label \"my_label\".
\n The effect is that the latter will accept packets from
non-Calico sources whereas the former is limited to packets
from Calico-controlled endpoints."
type: string
serviceAccounts:
description: ServiceAccounts is an optional field that restricts
the rule to only apply to traffic that originates from
(or terminates at) a pod running as a matching service
account.
properties:
names:
description: Names is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account whose name is in the list.
items:
type: string
type: array
selector:
description: Selector is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account that matches the given label selector. If
both Names and Selector are specified then they are
AND'ed.
type: string
type: object
services:
description: "Services is an optional field that contains
options for matching Kubernetes Services. If specified,
only traffic that originates from or terminates at endpoints
within the selected service(s) will be matched, and only
to/from each endpoint's port. \n Services cannot be specified
on the same rule as Selector, NotSelector, NamespaceSelector,
Nets, NotNets or ServiceAccounts. \n Ports and NotPorts
can only be specified with Services on ingress rules."
properties:
name:
description: Name specifies the name of a Kubernetes
Service to match.
type: string
namespace:
description: Namespace specifies the namespace of the
given Service. If left empty, the rule will match
within this policy's namespace.
type: string
type: object
type: object
required:
- action
type: object
type: array
ingress:
description: The ordered set of ingress rules. Each rule contains
a set of packet match criteria and a corresponding action to apply.
items:
description: "A Rule encapsulates a set of match criteria and an
action. Both selector-based security Policy and security Profiles
reference rules - separated out as a list of rules for both ingress
and egress packet matching. \n Each positive match criteria has
a negated version, prefixed with \"Not\". All the match criteria
within a rule must be satisfied for a packet to match. A single
rule can contain the positive and negative version of a match
and both must be satisfied for the rule to match."
properties:
action:
type: string
destination:
description: Destination contains the match criteria that apply
to destination entity.
properties:
namespaceSelector:
description: "NamespaceSelector is an optional field that
contains a selector expression. Only traffic that originates
from (or terminates at) endpoints within the selected
namespaces will be matched. When both NamespaceSelector
and another selector are defined on the same rule, then
only workload endpoints that are matched by both selectors
will be selected by the rule. \n For NetworkPolicy, an
empty NamespaceSelector implies that the Selector is limited
to selecting only workload endpoints in the same namespace
as the NetworkPolicy. \n For NetworkPolicy, `global()`
NamespaceSelector implies that the Selector is limited
to selecting only GlobalNetworkSet or HostEndpoint. \n
For GlobalNetworkPolicy, an empty NamespaceSelector implies
the Selector applies to workload endpoints across all
namespaces."
type: string
nets:
description: Nets is an optional field that restricts the
rule to only apply to traffic that originates from (or
terminates at) IP addresses in any of the given subnets.
items:
type: string
type: array
notNets:
description: NotNets is the negated version of the Nets
field.
items:
type: string
type: array
notPorts:
description: NotPorts is the negated version of the Ports
field. Since only some protocols have ports, if any ports
are specified it requires the Protocol match in the Rule
to be set to "TCP" or "UDP".
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
notSelector:
description: NotSelector is the negated version of the Selector
field. See Selector field for subtleties with negated
selectors.
type: string
ports:
description: "Ports is an optional field that restricts
the rule to only apply to traffic that has a source (destination)
port that matches one of these ranges/values. This value
is a list of integers or strings that represent ranges
of ports. \n Since only some protocols have ports, if
any ports are specified it requires the Protocol match
in the Rule to be set to \"TCP\" or \"UDP\"."
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
selector:
description: "Selector is an optional field that contains
a selector expression (see Policy for sample syntax).
\ Only traffic that originates from (terminates at) endpoints
matching the selector will be matched. \n Note that: in
addition to the negated version of the Selector (see NotSelector
below), the selector expression syntax itself supports
negation. The two types of negation are subtly different.
One negates the set of matched endpoints, the other negates
the whole match: \n \tSelector = \"!has(my_label)\" matches
packets that are from other Calico-controlled \tendpoints
that do not have the label \"my_label\". \n \tNotSelector
= \"has(my_label)\" matches packets that are not from
Calico-controlled \tendpoints that do have the label \"my_label\".
\n The effect is that the latter will accept packets from
non-Calico sources whereas the former is limited to packets
from Calico-controlled endpoints."
type: string
serviceAccounts:
description: ServiceAccounts is an optional field that restricts
the rule to only apply to traffic that originates from
(or terminates at) a pod running as a matching service
account.
properties:
names:
description: Names is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account whose name is in the list.
items:
type: string
type: array
selector:
description: Selector is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account that matches the given label selector. If
both Names and Selector are specified then they are
AND'ed.
type: string
type: object
services:
description: "Services is an optional field that contains
options for matching Kubernetes Services. If specified,
only traffic that originates from or terminates at endpoints
within the selected service(s) will be matched, and only
to/from each endpoint's port. \n Services cannot be specified
on the same rule as Selector, NotSelector, NamespaceSelector,
Nets, NotNets or ServiceAccounts. \n Ports and NotPorts
can only be specified with Services on ingress rules."
properties:
name:
description: Name specifies the name of a Kubernetes
Service to match.
type: string
namespace:
description: Namespace specifies the namespace of the
given Service. If left empty, the rule will match
within this policy's namespace.
type: string
type: object
type: object
http:
description: HTTP contains match criteria that apply to HTTP
requests.
properties:
methods:
description: Methods is an optional field that restricts
the rule to apply only to HTTP requests that use one of
the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
methods are OR'd together.
items:
type: string
type: array
paths:
description: 'Paths is an optional field that restricts
the rule to apply to HTTP requests that use one of the
listed HTTP Paths. Multiple paths are OR''d together.
e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
ONLY specify either a `exact` or a `prefix` match. The
validator will check for it.'
items:
description: 'HTTPPath specifies an HTTP path to match.
It may be either of the form: exact: <path>: which matches
the path exactly or prefix: <path-prefix>: which matches
the path prefix'
properties:
exact:
type: string
prefix:
type: string
type: object
type: array
type: object
icmp:
description: ICMP is an optional field that restricts the rule
to apply to a specific type and code of ICMP traffic. This
should only be specified if the Protocol field is set to "ICMP"
or "ICMPv6".
properties:
code:
description: Match on a specific ICMP code. If specified,
the Type value must also be specified. This is a technical
limitation imposed by the kernel's iptables firewall,
which Calico uses to enforce the rule.
type: integer
type:
description: Match on a specific ICMP type. For example
a value of 8 refers to ICMP Echo Request (i.e. pings).
type: integer
type: object
ipVersion:
description: IPVersion is an optional field that restricts the
rule to only match a specific IP version.
type: integer
metadata:
description: Metadata contains additional information for this
rule
properties:
annotations:
additionalProperties:
type: string
description: Annotations is a set of key value pairs that
give extra information about the rule
type: object
type: object
notICMP:
description: NotICMP is the negated version of the ICMP field.
properties:
code:
description: Match on a specific ICMP code. If specified,
the Type value must also be specified. This is a technical
limitation imposed by the kernel's iptables firewall,
which Calico uses to enforce the rule.
type: integer
type:
description: Match on a specific ICMP type. For example
a value of 8 refers to ICMP Echo Request (i.e. pings).
type: integer
type: object
notProtocol:
anyOf:
- type: integer
- type: string
description: NotProtocol is the negated version of the Protocol
field.
pattern: ^.*
x-kubernetes-int-or-string: true
protocol:
anyOf:
- type: integer
- type: string
description: "Protocol is an optional field that restricts the
rule to only apply to traffic of a specific IP protocol. Required
if any of the EntityRules contain Ports (because ports only
apply to certain protocols). \n Must be one of these string
values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
\"UDPLite\" or an integer in the range 1-255."
pattern: ^.*
x-kubernetes-int-or-string: true
source:
description: Source contains the match criteria that apply to
source entity.
properties:
namespaceSelector:
description: "NamespaceSelector is an optional field that
contains a selector expression. Only traffic that originates
from (or terminates at) endpoints within the selected
namespaces will be matched. When both NamespaceSelector
and another selector are defined on the same rule, then
only workload endpoints that are matched by both selectors
will be selected by the rule. \n For NetworkPolicy, an
empty NamespaceSelector implies that the Selector is limited
to selecting only workload endpoints in the same namespace
as the NetworkPolicy. \n For NetworkPolicy, `global()`
NamespaceSelector implies that the Selector is limited
to selecting only GlobalNetworkSet or HostEndpoint. \n
For GlobalNetworkPolicy, an empty NamespaceSelector implies
the Selector applies to workload endpoints across all
namespaces."
type: string
nets:
description: Nets is an optional field that restricts the
rule to only apply to traffic that originates from (or
terminates at) IP addresses in any of the given subnets.
items:
type: string
type: array
notNets:
description: NotNets is the negated version of the Nets
field.
items:
type: string
type: array
notPorts:
description: NotPorts is the negated version of the Ports
field. Since only some protocols have ports, if any ports
are specified it requires the Protocol match in the Rule
to be set to "TCP" or "UDP".
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
notSelector:
description: NotSelector is the negated version of the Selector
field. See Selector field for subtleties with negated
selectors.
type: string
ports:
description: "Ports is an optional field that restricts
the rule to only apply to traffic that has a source (destination)
port that matches one of these ranges/values. This value
is a list of integers or strings that represent ranges
of ports. \n Since only some protocols have ports, if
any ports are specified it requires the Protocol match
in the Rule to be set to \"TCP\" or \"UDP\"."
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
selector:
description: "Selector is an optional field that contains
a selector expression (see Policy for sample syntax).
\ Only traffic that originates from (terminates at) endpoints
matching the selector will be matched. \n Note that: in
addition to the negated version of the Selector (see NotSelector
below), the selector expression syntax itself supports
negation. The two types of negation are subtly different.
One negates the set of matched endpoints, the other negates
the whole match: \n \tSelector = \"!has(my_label)\" matches
packets that are from other Calico-controlled \tendpoints
that do not have the label \"my_label\". \n \tNotSelector
= \"has(my_label)\" matches packets that are not from
Calico-controlled \tendpoints that do have the label \"my_label\".
\n The effect is that the latter will accept packets from
non-Calico sources whereas the former is limited to packets
from Calico-controlled endpoints."
type: string
serviceAccounts:
description: ServiceAccounts is an optional field that restricts
the rule to only apply to traffic that originates from
(or terminates at) a pod running as a matching service
account.
properties:
names:
description: Names is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account whose name is in the list.
items:
type: string
type: array
selector:
description: Selector is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account that matches the given label selector. If
both Names and Selector are specified then they are
AND'ed.
type: string
type: object
services:
description: "Services is an optional field that contains
options for matching Kubernetes Services. If specified,
only traffic that originates from or terminates at endpoints
within the selected service(s) will be matched, and only
to/from each endpoint's port. \n Services cannot be specified
on the same rule as Selector, NotSelector, NamespaceSelector,
Nets, NotNets or ServiceAccounts. \n Ports and NotPorts
can only be specified with Services on ingress rules."
properties:
name:
description: Name specifies the name of a Kubernetes
Service to match.
type: string
namespace:
description: Namespace specifies the namespace of the
given Service. If left empty, the rule will match
within this policy's namespace.
type: string
type: object
type: object
required:
- action
type: object
type: array
namespaceSelector:
description: NamespaceSelector is an optional field for an expression
used to select a pod based on namespaces.
type: string
order:
description: Order is an optional field that specifies the order in
which the policy is applied. Policies with higher "order" are applied
after those with lower order. If the order is omitted, it may be
considered to be "infinite" - i.e. the policy will be applied last. Policies
with identical order will be applied in alphanumerical order based
on the Policy "Name".
type: number
preDNAT:
description: PreDNAT indicates to apply the rules in this policy before
any DNAT.
type: boolean
selector:
description: "The selector is an expression used to pick pick out
the endpoints that the policy should be applied to. \n Selector
expressions follow this syntax: \n \tlabel == \"string_literal\"
\ -> comparison, e.g. my_label == \"foo bar\" \tlabel != \"string_literal\"
\ -> not equal; also matches if label is not present \tlabel in
{ \"a\", \"b\", \"c\", ... } -> true if the value of label X is
one of \"a\", \"b\", \"c\" \tlabel not in { \"a\", \"b\", \"c\",
... } -> true if the value of label X is not one of \"a\", \"b\",
\"c\" \thas(label_name) -> True if that label is present \t! expr
-> negation of expr \texpr && expr -> Short-circuit and \texpr
|| expr -> Short-circuit or \t( expr ) -> parens for grouping \tall()
or the empty selector -> matches all endpoints. \n Label names are
allowed to contain alphanumerics, -, _ and /. String literals are
more permissive but they do not support escape characters. \n Examples
(with made-up labels): \n \ttype == \"webserver\" && deployment
== \"prod\" \ttype in {\"frontend\", \"backend\"} \tdeployment !=
\"dev\" \t! has(label_name)"
type: string
serviceAccountSelector:
description: ServiceAccountSelector is an optional field for an expression
used to select a pod based on service accounts.
type: string
types:
description: "Types indicates whether this policy applies to ingress,
or to egress, or to both. When not explicitly specified (and so
the value on creation is empty or nil), Calico defaults Types according
to what Ingress and Egress rules are present in the policy. The
default is: \n - [ PolicyTypeIngress ], if there are no Egress rules
(including the case where there are also no Ingress rules) \n
- [ PolicyTypeEgress ], if there are Egress rules but no Ingress
rules \n - [ PolicyTypeIngress, PolicyTypeEgress ], if there are
both Ingress and Egress rules. \n When the policy is read back again,
Types will always be one of these values, never empty or nil."
items:
description: PolicyType enumerates the possible values of the PolicySpec
Types field.
type: string
type: array
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: globalnetworksets.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: GlobalNetworkSet
listKind: GlobalNetworkSetList
plural: globalnetworksets
singular: globalnetworkset
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
description: GlobalNetworkSet contains a set of arbitrary IP sub-networks/CIDRs
that share labels to allow rules to refer to them via selectors. The labels
of GlobalNetworkSet are not namespaced.
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: GlobalNetworkSetSpec contains the specification for a NetworkSet
resource.
properties:
nets:
description: The list of IP networks that belong to this set.
items:
type: string
type: array
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: hostendpoints.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: HostEndpoint
listKind: HostEndpointList
plural: hostendpoints
singular: hostendpoint
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: HostEndpointSpec contains the specification for a HostEndpoint
resource.
properties:
expectedIPs:
description: "The expected IP addresses (IPv4 and IPv6) of the endpoint.
If \"InterfaceName\" is not present, Calico will look for an interface
matching any of the IPs in the list and apply policy to that. Note:
\tWhen using the selector match criteria in an ingress or egress
security Policy \tor Profile, Calico converts the selector into
a set of IP addresses. For host \tendpoints, the ExpectedIPs field
is used for that purpose. (If only the interface \tname is specified,
Calico does not learn the IPs of the interface for use in match
\tcriteria.)"
items:
type: string
type: array
interfaceName:
description: "Either \"*\", or the name of a specific Linux interface
to apply policy to; or empty. \"*\" indicates that this HostEndpoint
governs all traffic to, from or through the default network namespace
of the host named by the \"Node\" field; entering and leaving that
namespace via any interface, including those from/to non-host-networked
local workloads. \n If InterfaceName is not \"*\", this HostEndpoint
only governs traffic that enters or leaves the host through the
specific interface named by InterfaceName, or - when InterfaceName
is empty - through the specific interface that has one of the IPs
in ExpectedIPs. Therefore, when InterfaceName is empty, at least
one expected IP must be specified. Only external interfaces (such
as \"eth0\") are supported here; it isn't possible for a HostEndpoint
to protect traffic through a specific local workload interface.
\n Note: Only some kinds of policy are implemented for \"*\" HostEndpoints;
initially just pre-DNAT policy. Please check Calico documentation
for the latest position."
type: string
node:
description: The node name identifying the Calico node instance.
type: string
ports:
description: Ports contains the endpoint's named ports, which may
be referenced in security policy rules.
items:
properties:
name:
type: string
port:
type: integer
protocol:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
required:
- name
- port
- protocol
type: object
type: array
profiles:
description: A list of identifiers of security Profile objects that
apply to this endpoint. Each profile is applied in the order that
they appear in this list. Profile rules are applied after the selector-based
security policy.
items:
type: string
type: array
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: ipamblocks.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: IPAMBlock
listKind: IPAMBlockList
plural: ipamblocks
singular: ipamblock
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: IPAMBlockSpec contains the specification for an IPAMBlock
resource.
properties:
affinity:
description: Affinity of the block, if this block has one. If set,
it will be of the form "host:<hostname>". If not set, this block
is not affine to a host.
type: string
allocations:
description: Array of allocations in-use within this block. nil entries
mean the allocation is free. For non-nil entries at index i, the
index is the ordinal of the allocation within this block and the
value is the index of the associated attributes in the Attributes
array.
items:
type: integer
# TODO: This nullable is manually added in. We should update controller-gen
# to handle []*int properly itself.
nullable: true
type: array
attributes:
description: Attributes is an array of arbitrary metadata associated
with allocations in the block. To find attributes for a given allocation,
use the value of the allocation's entry in the Allocations array
as the index of the element in this array.
items:
properties:
handle_id:
type: string
secondary:
additionalProperties:
type: string
type: object
type: object
type: array
cidr:
description: The block's CIDR.
type: string
deleted:
description: Deleted is an internal boolean used to workaround a limitation
in the Kubernetes API whereby deletion will not return a conflict
error if the block has been updated. It should not be set manually.
type: boolean
sequenceNumber:
default: 0
description: We store a sequence number that is updated each time
the block is written. Each allocation will also store the sequence
number of the block at the time of its creation. When releasing
an IP, passing the sequence number associated with the allocation
allows us to protect against a race condition and ensure the IP
hasn't been released and re-allocated since the release request.
format: int64
type: integer
sequenceNumberForAllocation:
additionalProperties:
format: int64
type: integer
description: Map of allocated ordinal within the block to sequence
number of the block at the time of allocation. Kubernetes does not
allow numerical keys for maps, so the key is cast to a string.
type: object
strictAffinity:
description: StrictAffinity on the IPAMBlock is deprecated and no
longer used by the code. Use IPAMConfig StrictAffinity instead.
type: boolean
unallocated:
description: Unallocated is an ordered list of allocations which are
free in the block.
items:
type: integer
type: array
required:
- allocations
- attributes
- cidr
- strictAffinity
- unallocated
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: ipamconfigs.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: IPAMConfig
listKind: IPAMConfigList
plural: ipamconfigs
singular: ipamconfig
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: IPAMConfigSpec contains the specification for an IPAMConfig
resource.
properties:
autoAllocateBlocks:
type: boolean
maxBlocksPerHost:
description: MaxBlocksPerHost, if non-zero, is the max number of blocks
that can be affine to each host.
type: integer
strictAffinity:
type: boolean
required:
- autoAllocateBlocks
- strictAffinity
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: ipamhandles.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: IPAMHandle
listKind: IPAMHandleList
plural: ipamhandles
singular: ipamhandle
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: IPAMHandleSpec contains the specification for an IPAMHandle
resource.
properties:
block:
additionalProperties:
type: integer
type: object
deleted:
type: boolean
handleID:
type: string
required:
- block
- handleID
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: ippools.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: IPPool
listKind: IPPoolList
plural: ippools
singular: ippool
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: IPPoolSpec contains the specification for an IPPool resource.
properties:
allowedUses:
description: AllowedUse controls what the IP pool will be used for. If
not specified or empty, defaults to ["Tunnel", "Workload"] for back-compatibility
items:
type: string
type: array
blockSize:
description: The block size to use for IP address assignments from
this pool. Defaults to 26 for IPv4 and 122 for IPv6.
type: integer
cidr:
description: The pool CIDR.
type: string
disableBGPExport:
description: 'Disable exporting routes from this IP Pool''s CIDR over
BGP. [Default: false]'
type: boolean
disabled:
description: When disabled is true, Calico IPAM will not assign addresses
from this pool.
type: boolean
ipip:
description: 'Deprecated: this field is only used for APIv1 backwards
compatibility. Setting this field is not allowed, this field is
for internal use only.'
properties:
enabled:
description: When enabled is true, ipip tunneling will be used
to deliver packets to destinations within this pool.
type: boolean
mode:
description: The IPIP mode. This can be one of "always" or "cross-subnet". A
mode of "always" will also use IPIP tunneling for routing to
destination IP addresses within this pool. A mode of "cross-subnet"
will only use IPIP tunneling when the destination node is on
a different subnet to the originating node. The default value
(if not specified) is "always".
type: string
type: object
ipipMode:
description: Contains configuration for IPIP tunneling for this pool.
If not specified, then this is defaulted to "Never" (i.e. IPIP tunneling
is disabled).
type: string
nat-outgoing:
description: 'Deprecated: this field is only used for APIv1 backwards
compatibility. Setting this field is not allowed, this field is
for internal use only.'
type: boolean
natOutgoing:
description: When nat-outgoing is true, packets sent from Calico networked
containers in this pool to destinations outside of this pool will
be masqueraded.
type: boolean
nodeSelector:
description: Allows IPPool to allocate for a specific node by label
selector.
type: string
vxlanMode:
description: Contains configuration for VXLAN tunneling for this pool.
If not specified, then this is defaulted to "Never" (i.e. VXLAN
tunneling is disabled).
type: string
required:
- cidr
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: (devel)
creationTimestamp: null
name: ipreservations.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: IPReservation
listKind: IPReservationList
plural: ipreservations
singular: ipreservation
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: IPReservationSpec contains the specification for an IPReservation
resource.
properties:
reservedCIDRs:
description: ReservedCIDRs is a list of CIDRs and/or IP addresses
that Calico IPAM will exclude from new allocations.
items:
type: string
type: array
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: kubecontrollersconfigurations.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: KubeControllersConfiguration
listKind: KubeControllersConfigurationList
plural: kubecontrollersconfigurations
singular: kubecontrollersconfiguration
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: KubeControllersConfigurationSpec contains the values of the
Kubernetes controllers configuration.
properties:
controllers:
description: Controllers enables and configures individual Kubernetes
controllers
properties:
namespace:
description: Namespace enables and configures the namespace controller.
Enabled by default, set to nil to disable.
properties:
reconcilerPeriod:
description: 'ReconcilerPeriod is the period to perform reconciliation
with the Calico datastore. [Default: 5m]'
type: string
type: object
node:
description: Node enables and configures the node controller.
Enabled by default, set to nil to disable.
properties:
hostEndpoint:
description: HostEndpoint controls syncing nodes to host endpoints.
Disabled by default, set to nil to disable.
properties:
autoCreate:
description: 'AutoCreate enables automatic creation of
host endpoints for every node. [Default: Disabled]'
type: string
type: object
leakGracePeriod:
description: 'LeakGracePeriod is the period used by the controller
to determine if an IP address has been leaked. Set to 0
to disable IP garbage collection. [Default: 15m]'
type: string
reconcilerPeriod:
description: 'ReconcilerPeriod is the period to perform reconciliation
with the Calico datastore. [Default: 5m]'
type: string
syncLabels:
description: 'SyncLabels controls whether to copy Kubernetes
node labels to Calico nodes. [Default: Enabled]'
type: string
type: object
policy:
description: Policy enables and configures the policy controller.
Enabled by default, set to nil to disable.
properties:
reconcilerPeriod:
description: 'ReconcilerPeriod is the period to perform reconciliation
with the Calico datastore. [Default: 5m]'
type: string
type: object
serviceAccount:
description: ServiceAccount enables and configures the service
account controller. Enabled by default, set to nil to disable.
properties:
reconcilerPeriod:
description: 'ReconcilerPeriod is the period to perform reconciliation
with the Calico datastore. [Default: 5m]'
type: string
type: object
workloadEndpoint:
description: WorkloadEndpoint enables and configures the workload
endpoint controller. Enabled by default, set to nil to disable.
properties:
reconcilerPeriod:
description: 'ReconcilerPeriod is the period to perform reconciliation
with the Calico datastore. [Default: 5m]'
type: string
type: object
type: object
debugProfilePort:
description: DebugProfilePort configures the port to serve memory
and cpu profiles on. If not specified, profiling is disabled.
format: int32
type: integer
etcdV3CompactionPeriod:
description: 'EtcdV3CompactionPeriod is the period between etcdv3
compaction requests. Set to 0 to disable. [Default: 10m]'
type: string
healthChecks:
description: 'HealthChecks enables or disables support for health
checks [Default: Enabled]'
type: string
logSeverityScreen:
description: 'LogSeverityScreen is the log severity above which logs
are sent to the stdout. [Default: Info]'
type: string
prometheusMetricsPort:
description: 'PrometheusMetricsPort is the TCP port that the Prometheus
metrics server should bind to. Set to 0 to disable. [Default: 9094]'
type: integer
required:
- controllers
type: object
status:
description: KubeControllersConfigurationStatus represents the status
of the configuration. It's useful for admins to be able to see the actual
config that was applied, which can be modified by environment variables
on the kube-controllers process.
properties:
environmentVars:
additionalProperties:
type: string
description: EnvironmentVars contains the environment variables on
the kube-controllers that influenced the RunningConfig.
type: object
runningConfig:
description: RunningConfig contains the effective config that is running
in the kube-controllers pod, after merging the API resource with
any environment variables.
properties:
controllers:
description: Controllers enables and configures individual Kubernetes
controllers
properties:
namespace:
description: Namespace enables and configures the namespace
controller. Enabled by default, set to nil to disable.
properties:
reconcilerPeriod:
description: 'ReconcilerPeriod is the period to perform
reconciliation with the Calico datastore. [Default:
5m]'
type: string
type: object
node:
description: Node enables and configures the node controller.
Enabled by default, set to nil to disable.
properties:
hostEndpoint:
description: HostEndpoint controls syncing nodes to host
endpoints. Disabled by default, set to nil to disable.
properties:
autoCreate:
description: 'AutoCreate enables automatic creation
of host endpoints for every node. [Default: Disabled]'
type: string
type: object
leakGracePeriod:
description: 'LeakGracePeriod is the period used by the
controller to determine if an IP address has been leaked.
Set to 0 to disable IP garbage collection. [Default:
15m]'
type: string
reconcilerPeriod:
description: 'ReconcilerPeriod is the period to perform
reconciliation with the Calico datastore. [Default:
5m]'
type: string
syncLabels:
description: 'SyncLabels controls whether to copy Kubernetes
node labels to Calico nodes. [Default: Enabled]'
type: string
type: object
policy:
description: Policy enables and configures the policy controller.
Enabled by default, set to nil to disable.
properties:
reconcilerPeriod:
description: 'ReconcilerPeriod is the period to perform
reconciliation with the Calico datastore. [Default:
5m]'
type: string
type: object
serviceAccount:
description: ServiceAccount enables and configures the service
account controller. Enabled by default, set to nil to disable.
properties:
reconcilerPeriod:
description: 'ReconcilerPeriod is the period to perform
reconciliation with the Calico datastore. [Default:
5m]'
type: string
type: object
workloadEndpoint:
description: WorkloadEndpoint enables and configures the workload
endpoint controller. Enabled by default, set to nil to disable.
properties:
reconcilerPeriod:
description: 'ReconcilerPeriod is the period to perform
reconciliation with the Calico datastore. [Default:
5m]'
type: string
type: object
type: object
debugProfilePort:
description: DebugProfilePort configures the port to serve memory
and cpu profiles on. If not specified, profiling is disabled.
format: int32
type: integer
etcdV3CompactionPeriod:
description: 'EtcdV3CompactionPeriod is the period between etcdv3
compaction requests. Set to 0 to disable. [Default: 10m]'
type: string
healthChecks:
description: 'HealthChecks enables or disables support for health
checks [Default: Enabled]'
type: string
logSeverityScreen:
description: 'LogSeverityScreen is the log severity above which
logs are sent to the stdout. [Default: Info]'
type: string
prometheusMetricsPort:
description: 'PrometheusMetricsPort is the TCP port that the Prometheus
metrics server should bind to. Set to 0 to disable. [Default:
9094]'
type: integer
required:
- controllers
type: object
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: networkpolicies.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: NetworkPolicy
listKind: NetworkPolicyList
plural: networkpolicies
singular: networkpolicy
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
properties:
egress:
description: The ordered set of egress rules. Each rule contains
a set of packet match criteria and a corresponding action to apply.
items:
description: "A Rule encapsulates a set of match criteria and an
action. Both selector-based security Policy and security Profiles
reference rules - separated out as a list of rules for both ingress
and egress packet matching. \n Each positive match criteria has
a negated version, prefixed with \"Not\". All the match criteria
within a rule must be satisfied for a packet to match. A single
rule can contain the positive and negative version of a match
and both must be satisfied for the rule to match."
properties:
action:
type: string
destination:
description: Destination contains the match criteria that apply
to destination entity.
properties:
namespaceSelector:
description: "NamespaceSelector is an optional field that
contains a selector expression. Only traffic that originates
from (or terminates at) endpoints within the selected
namespaces will be matched. When both NamespaceSelector
and another selector are defined on the same rule, then
only workload endpoints that are matched by both selectors
will be selected by the rule. \n For NetworkPolicy, an
empty NamespaceSelector implies that the Selector is limited
to selecting only workload endpoints in the same namespace
as the NetworkPolicy. \n For NetworkPolicy, `global()`
NamespaceSelector implies that the Selector is limited
to selecting only GlobalNetworkSet or HostEndpoint. \n
For GlobalNetworkPolicy, an empty NamespaceSelector implies
the Selector applies to workload endpoints across all
namespaces."
type: string
nets:
description: Nets is an optional field that restricts the
rule to only apply to traffic that originates from (or
terminates at) IP addresses in any of the given subnets.
items:
type: string
type: array
notNets:
description: NotNets is the negated version of the Nets
field.
items:
type: string
type: array
notPorts:
description: NotPorts is the negated version of the Ports
field. Since only some protocols have ports, if any ports
are specified it requires the Protocol match in the Rule
to be set to "TCP" or "UDP".
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
notSelector:
description: NotSelector is the negated version of the Selector
field. See Selector field for subtleties with negated
selectors.
type: string
ports:
description: "Ports is an optional field that restricts
the rule to only apply to traffic that has a source (destination)
port that matches one of these ranges/values. This value
is a list of integers or strings that represent ranges
of ports. \n Since only some protocols have ports, if
any ports are specified it requires the Protocol match
in the Rule to be set to \"TCP\" or \"UDP\"."
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
selector:
description: "Selector is an optional field that contains
a selector expression (see Policy for sample syntax).
\ Only traffic that originates from (terminates at) endpoints
matching the selector will be matched. \n Note that: in
addition to the negated version of the Selector (see NotSelector
below), the selector expression syntax itself supports
negation. The two types of negation are subtly different.
One negates the set of matched endpoints, the other negates
the whole match: \n \tSelector = \"!has(my_label)\" matches
packets that are from other Calico-controlled \tendpoints
that do not have the label \"my_label\". \n \tNotSelector
= \"has(my_label)\" matches packets that are not from
Calico-controlled \tendpoints that do have the label \"my_label\".
\n The effect is that the latter will accept packets from
non-Calico sources whereas the former is limited to packets
from Calico-controlled endpoints."
type: string
serviceAccounts:
description: ServiceAccounts is an optional field that restricts
the rule to only apply to traffic that originates from
(or terminates at) a pod running as a matching service
account.
properties:
names:
description: Names is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account whose name is in the list.
items:
type: string
type: array
selector:
description: Selector is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account that matches the given label selector. If
both Names and Selector are specified then they are
AND'ed.
type: string
type: object
services:
description: "Services is an optional field that contains
options for matching Kubernetes Services. If specified,
only traffic that originates from or terminates at endpoints
within the selected service(s) will be matched, and only
to/from each endpoint's port. \n Services cannot be specified
on the same rule as Selector, NotSelector, NamespaceSelector,
Nets, NotNets or ServiceAccounts. \n Ports and NotPorts
can only be specified with Services on ingress rules."
properties:
name:
description: Name specifies the name of a Kubernetes
Service to match.
type: string
namespace:
description: Namespace specifies the namespace of the
given Service. If left empty, the rule will match
within this policy's namespace.
type: string
type: object
type: object
http:
description: HTTP contains match criteria that apply to HTTP
requests.
properties:
methods:
description: Methods is an optional field that restricts
the rule to apply only to HTTP requests that use one of
the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
methods are OR'd together.
items:
type: string
type: array
paths:
description: 'Paths is an optional field that restricts
the rule to apply to HTTP requests that use one of the
listed HTTP Paths. Multiple paths are OR''d together.
e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
ONLY specify either a `exact` or a `prefix` match. The
validator will check for it.'
items:
description: 'HTTPPath specifies an HTTP path to match.
It may be either of the form: exact: <path>: which matches
the path exactly or prefix: <path-prefix>: which matches
the path prefix'
properties:
exact:
type: string
prefix:
type: string
type: object
type: array
type: object
icmp:
description: ICMP is an optional field that restricts the rule
to apply to a specific type and code of ICMP traffic. This
should only be specified if the Protocol field is set to "ICMP"
or "ICMPv6".
properties:
code:
description: Match on a specific ICMP code. If specified,
the Type value must also be specified. This is a technical
limitation imposed by the kernel's iptables firewall,
which Calico uses to enforce the rule.
type: integer
type:
description: Match on a specific ICMP type. For example
a value of 8 refers to ICMP Echo Request (i.e. pings).
type: integer
type: object
ipVersion:
description: IPVersion is an optional field that restricts the
rule to only match a specific IP version.
type: integer
metadata:
description: Metadata contains additional information for this
rule
properties:
annotations:
additionalProperties:
type: string
description: Annotations is a set of key value pairs that
give extra information about the rule
type: object
type: object
notICMP:
description: NotICMP is the negated version of the ICMP field.
properties:
code:
description: Match on a specific ICMP code. If specified,
the Type value must also be specified. This is a technical
limitation imposed by the kernel's iptables firewall,
which Calico uses to enforce the rule.
type: integer
type:
description: Match on a specific ICMP type. For example
a value of 8 refers to ICMP Echo Request (i.e. pings).
type: integer
type: object
notProtocol:
anyOf:
- type: integer
- type: string
description: NotProtocol is the negated version of the Protocol
field.
pattern: ^.*
x-kubernetes-int-or-string: true
protocol:
anyOf:
- type: integer
- type: string
description: "Protocol is an optional field that restricts the
rule to only apply to traffic of a specific IP protocol. Required
if any of the EntityRules contain Ports (because ports only
apply to certain protocols). \n Must be one of these string
values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
\"UDPLite\" or an integer in the range 1-255."
pattern: ^.*
x-kubernetes-int-or-string: true
source:
description: Source contains the match criteria that apply to
source entity.
properties:
namespaceSelector:
description: "NamespaceSelector is an optional field that
contains a selector expression. Only traffic that originates
from (or terminates at) endpoints within the selected
namespaces will be matched. When both NamespaceSelector
and another selector are defined on the same rule, then
only workload endpoints that are matched by both selectors
will be selected by the rule. \n For NetworkPolicy, an
empty NamespaceSelector implies that the Selector is limited
to selecting only workload endpoints in the same namespace
as the NetworkPolicy. \n For NetworkPolicy, `global()`
NamespaceSelector implies that the Selector is limited
to selecting only GlobalNetworkSet or HostEndpoint. \n
For GlobalNetworkPolicy, an empty NamespaceSelector implies
the Selector applies to workload endpoints across all
namespaces."
type: string
nets:
description: Nets is an optional field that restricts the
rule to only apply to traffic that originates from (or
terminates at) IP addresses in any of the given subnets.
items:
type: string
type: array
notNets:
description: NotNets is the negated version of the Nets
field.
items:
type: string
type: array
notPorts:
description: NotPorts is the negated version of the Ports
field. Since only some protocols have ports, if any ports
are specified it requires the Protocol match in the Rule
to be set to "TCP" or "UDP".
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
notSelector:
description: NotSelector is the negated version of the Selector
field. See Selector field for subtleties with negated
selectors.
type: string
ports:
description: "Ports is an optional field that restricts
the rule to only apply to traffic that has a source (destination)
port that matches one of these ranges/values. This value
is a list of integers or strings that represent ranges
of ports. \n Since only some protocols have ports, if
any ports are specified it requires the Protocol match
in the Rule to be set to \"TCP\" or \"UDP\"."
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
selector:
description: "Selector is an optional field that contains
a selector expression (see Policy for sample syntax).
\ Only traffic that originates from (terminates at) endpoints
matching the selector will be matched. \n Note that: in
addition to the negated version of the Selector (see NotSelector
below), the selector expression syntax itself supports
negation. The two types of negation are subtly different.
One negates the set of matched endpoints, the other negates
the whole match: \n \tSelector = \"!has(my_label)\" matches
packets that are from other Calico-controlled \tendpoints
that do not have the label \"my_label\". \n \tNotSelector
= \"has(my_label)\" matches packets that are not from
Calico-controlled \tendpoints that do have the label \"my_label\".
\n The effect is that the latter will accept packets from
non-Calico sources whereas the former is limited to packets
from Calico-controlled endpoints."
type: string
serviceAccounts:
description: ServiceAccounts is an optional field that restricts
the rule to only apply to traffic that originates from
(or terminates at) a pod running as a matching service
account.
properties:
names:
description: Names is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account whose name is in the list.
items:
type: string
type: array
selector:
description: Selector is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account that matches the given label selector. If
both Names and Selector are specified then they are
AND'ed.
type: string
type: object
services:
description: "Services is an optional field that contains
options for matching Kubernetes Services. If specified,
only traffic that originates from or terminates at endpoints
within the selected service(s) will be matched, and only
to/from each endpoint's port. \n Services cannot be specified
on the same rule as Selector, NotSelector, NamespaceSelector,
Nets, NotNets or ServiceAccounts. \n Ports and NotPorts
can only be specified with Services on ingress rules."
properties:
name:
description: Name specifies the name of a Kubernetes
Service to match.
type: string
namespace:
description: Namespace specifies the namespace of the
given Service. If left empty, the rule will match
within this policy's namespace.
type: string
type: object
type: object
required:
- action
type: object
type: array
ingress:
description: The ordered set of ingress rules. Each rule contains
a set of packet match criteria and a corresponding action to apply.
items:
description: "A Rule encapsulates a set of match criteria and an
action. Both selector-based security Policy and security Profiles
reference rules - separated out as a list of rules for both ingress
and egress packet matching. \n Each positive match criteria has
a negated version, prefixed with \"Not\". All the match criteria
within a rule must be satisfied for a packet to match. A single
rule can contain the positive and negative version of a match
and both must be satisfied for the rule to match."
properties:
action:
type: string
destination:
description: Destination contains the match criteria that apply
to destination entity.
properties:
namespaceSelector:
description: "NamespaceSelector is an optional field that
contains a selector expression. Only traffic that originates
from (or terminates at) endpoints within the selected
namespaces will be matched. When both NamespaceSelector
and another selector are defined on the same rule, then
only workload endpoints that are matched by both selectors
will be selected by the rule. \n For NetworkPolicy, an
empty NamespaceSelector implies that the Selector is limited
to selecting only workload endpoints in the same namespace
as the NetworkPolicy. \n For NetworkPolicy, `global()`
NamespaceSelector implies that the Selector is limited
to selecting only GlobalNetworkSet or HostEndpoint. \n
For GlobalNetworkPolicy, an empty NamespaceSelector implies
the Selector applies to workload endpoints across all
namespaces."
type: string
nets:
description: Nets is an optional field that restricts the
rule to only apply to traffic that originates from (or
terminates at) IP addresses in any of the given subnets.
items:
type: string
type: array
notNets:
description: NotNets is the negated version of the Nets
field.
items:
type: string
type: array
notPorts:
description: NotPorts is the negated version of the Ports
field. Since only some protocols have ports, if any ports
are specified it requires the Protocol match in the Rule
to be set to "TCP" or "UDP".
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
notSelector:
description: NotSelector is the negated version of the Selector
field. See Selector field for subtleties with negated
selectors.
type: string
ports:
description: "Ports is an optional field that restricts
the rule to only apply to traffic that has a source (destination)
port that matches one of these ranges/values. This value
is a list of integers or strings that represent ranges
of ports. \n Since only some protocols have ports, if
any ports are specified it requires the Protocol match
in the Rule to be set to \"TCP\" or \"UDP\"."
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
selector:
description: "Selector is an optional field that contains
a selector expression (see Policy for sample syntax).
\ Only traffic that originates from (terminates at) endpoints
matching the selector will be matched. \n Note that: in
addition to the negated version of the Selector (see NotSelector
below), the selector expression syntax itself supports
negation. The two types of negation are subtly different.
One negates the set of matched endpoints, the other negates
the whole match: \n \tSelector = \"!has(my_label)\" matches
packets that are from other Calico-controlled \tendpoints
that do not have the label \"my_label\". \n \tNotSelector
= \"has(my_label)\" matches packets that are not from
Calico-controlled \tendpoints that do have the label \"my_label\".
\n The effect is that the latter will accept packets from
non-Calico sources whereas the former is limited to packets
from Calico-controlled endpoints."
type: string
serviceAccounts:
description: ServiceAccounts is an optional field that restricts
the rule to only apply to traffic that originates from
(or terminates at) a pod running as a matching service
account.
properties:
names:
description: Names is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account whose name is in the list.
items:
type: string
type: array
selector:
description: Selector is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account that matches the given label selector. If
both Names and Selector are specified then they are
AND'ed.
type: string
type: object
services:
description: "Services is an optional field that contains
options for matching Kubernetes Services. If specified,
only traffic that originates from or terminates at endpoints
within the selected service(s) will be matched, and only
to/from each endpoint's port. \n Services cannot be specified
on the same rule as Selector, NotSelector, NamespaceSelector,
Nets, NotNets or ServiceAccounts. \n Ports and NotPorts
can only be specified with Services on ingress rules."
properties:
name:
description: Name specifies the name of a Kubernetes
Service to match.
type: string
namespace:
description: Namespace specifies the namespace of the
given Service. If left empty, the rule will match
within this policy's namespace.
type: string
type: object
type: object
http:
description: HTTP contains match criteria that apply to HTTP
requests.
properties:
methods:
description: Methods is an optional field that restricts
the rule to apply only to HTTP requests that use one of
the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
methods are OR'd together.
items:
type: string
type: array
paths:
description: 'Paths is an optional field that restricts
the rule to apply to HTTP requests that use one of the
listed HTTP Paths. Multiple paths are OR''d together.
e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
ONLY specify either a `exact` or a `prefix` match. The
validator will check for it.'
items:
description: 'HTTPPath specifies an HTTP path to match.
It may be either of the form: exact: <path>: which matches
the path exactly or prefix: <path-prefix>: which matches
the path prefix'
properties:
exact:
type: string
prefix:
type: string
type: object
type: array
type: object
icmp:
description: ICMP is an optional field that restricts the rule
to apply to a specific type and code of ICMP traffic. This
should only be specified if the Protocol field is set to "ICMP"
or "ICMPv6".
properties:
code:
description: Match on a specific ICMP code. If specified,
the Type value must also be specified. This is a technical
limitation imposed by the kernel's iptables firewall,
which Calico uses to enforce the rule.
type: integer
type:
description: Match on a specific ICMP type. For example
a value of 8 refers to ICMP Echo Request (i.e. pings).
type: integer
type: object
ipVersion:
description: IPVersion is an optional field that restricts the
rule to only match a specific IP version.
type: integer
metadata:
description: Metadata contains additional information for this
rule
properties:
annotations:
additionalProperties:
type: string
description: Annotations is a set of key value pairs that
give extra information about the rule
type: object
type: object
notICMP:
description: NotICMP is the negated version of the ICMP field.
properties:
code:
description: Match on a specific ICMP code. If specified,
the Type value must also be specified. This is a technical
limitation imposed by the kernel's iptables firewall,
which Calico uses to enforce the rule.
type: integer
type:
description: Match on a specific ICMP type. For example
a value of 8 refers to ICMP Echo Request (i.e. pings).
type: integer
type: object
notProtocol:
anyOf:
- type: integer
- type: string
description: NotProtocol is the negated version of the Protocol
field.
pattern: ^.*
x-kubernetes-int-or-string: true
protocol:
anyOf:
- type: integer
- type: string
description: "Protocol is an optional field that restricts the
rule to only apply to traffic of a specific IP protocol. Required
if any of the EntityRules contain Ports (because ports only
apply to certain protocols). \n Must be one of these string
values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
\"UDPLite\" or an integer in the range 1-255."
pattern: ^.*
x-kubernetes-int-or-string: true
source:
description: Source contains the match criteria that apply to
source entity.
properties:
namespaceSelector:
description: "NamespaceSelector is an optional field that
contains a selector expression. Only traffic that originates
from (or terminates at) endpoints within the selected
namespaces will be matched. When both NamespaceSelector
and another selector are defined on the same rule, then
only workload endpoints that are matched by both selectors
will be selected by the rule. \n For NetworkPolicy, an
empty NamespaceSelector implies that the Selector is limited
to selecting only workload endpoints in the same namespace
as the NetworkPolicy. \n For NetworkPolicy, `global()`
NamespaceSelector implies that the Selector is limited
to selecting only GlobalNetworkSet or HostEndpoint. \n
For GlobalNetworkPolicy, an empty NamespaceSelector implies
the Selector applies to workload endpoints across all
namespaces."
type: string
nets:
description: Nets is an optional field that restricts the
rule to only apply to traffic that originates from (or
terminates at) IP addresses in any of the given subnets.
items:
type: string
type: array
notNets:
description: NotNets is the negated version of the Nets
field.
items:
type: string
type: array
notPorts:
description: NotPorts is the negated version of the Ports
field. Since only some protocols have ports, if any ports
are specified it requires the Protocol match in the Rule
to be set to "TCP" or "UDP".
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
notSelector:
description: NotSelector is the negated version of the Selector
field. See Selector field for subtleties with negated
selectors.
type: string
ports:
description: "Ports is an optional field that restricts
the rule to only apply to traffic that has a source (destination)
port that matches one of these ranges/values. This value
is a list of integers or strings that represent ranges
of ports. \n Since only some protocols have ports, if
any ports are specified it requires the Protocol match
in the Rule to be set to \"TCP\" or \"UDP\"."
items:
anyOf:
- type: integer
- type: string
pattern: ^.*
x-kubernetes-int-or-string: true
type: array
selector:
description: "Selector is an optional field that contains
a selector expression (see Policy for sample syntax).
\ Only traffic that originates from (terminates at) endpoints
matching the selector will be matched. \n Note that: in
addition to the negated version of the Selector (see NotSelector
below), the selector expression syntax itself supports
negation. The two types of negation are subtly different.
One negates the set of matched endpoints, the other negates
the whole match: \n \tSelector = \"!has(my_label)\" matches
packets that are from other Calico-controlled \tendpoints
that do not have the label \"my_label\". \n \tNotSelector
= \"has(my_label)\" matches packets that are not from
Calico-controlled \tendpoints that do have the label \"my_label\".
\n The effect is that the latter will accept packets from
non-Calico sources whereas the former is limited to packets
from Calico-controlled endpoints."
type: string
serviceAccounts:
description: ServiceAccounts is an optional field that restricts
the rule to only apply to traffic that originates from
(or terminates at) a pod running as a matching service
account.
properties:
names:
description: Names is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account whose name is in the list.
items:
type: string
type: array
selector:
description: Selector is an optional field that restricts
the rule to only apply to traffic that originates
from (or terminates at) a pod running as a service
account that matches the given label selector. If
both Names and Selector are specified then they are
AND'ed.
type: string
type: object
services:
description: "Services is an optional field that contains
options for matching Kubernetes Services. If specified,
only traffic that originates from or terminates at endpoints
within the selected service(s) will be matched, and only
to/from each endpoint's port. \n Services cannot be specified
on the same rule as Selector, NotSelector, NamespaceSelector,
Nets, NotNets or ServiceAccounts. \n Ports and NotPorts
can only be specified with Services on ingress rules."
properties:
name:
description: Name specifies the name of a Kubernetes
Service to match.
type: string
namespace:
description: Namespace specifies the namespace of the
given Service. If left empty, the rule will match
within this policy's namespace.
type: string
type: object
type: object
required:
- action
type: object
type: array
order:
description: Order is an optional field that specifies the order in
which the policy is applied. Policies with higher "order" are applied
after those with lower order. If the order is omitted, it may be
considered to be "infinite" - i.e. the policy will be applied last. Policies
with identical order will be applied in alphanumerical order based
on the Policy "Name".
type: number
selector:
description: "The selector is an expression used to pick pick out
the endpoints that the policy should be applied to. \n Selector
expressions follow this syntax: \n \tlabel == \"string_literal\"
\ -> comparison, e.g. my_label == \"foo bar\" \tlabel != \"string_literal\"
\ -> not equal; also matches if label is not present \tlabel in
{ \"a\", \"b\", \"c\", ... } -> true if the value of label X is
one of \"a\", \"b\", \"c\" \tlabel not in { \"a\", \"b\", \"c\",
... } -> true if the value of label X is not one of \"a\", \"b\",
\"c\" \thas(label_name) -> True if that label is present \t! expr
-> negation of expr \texpr && expr -> Short-circuit and \texpr
|| expr -> Short-circuit or \t( expr ) -> parens for grouping \tall()
or the empty selector -> matches all endpoints. \n Label names are
allowed to contain alphanumerics, -, _ and /. String literals are
more permissive but they do not support escape characters. \n Examples
(with made-up labels): \n \ttype == \"webserver\" && deployment
== \"prod\" \ttype in {\"frontend\", \"backend\"} \tdeployment !=
\"dev\" \t! has(label_name)"
type: string
serviceAccountSelector:
description: ServiceAccountSelector is an optional field for an expression
used to select a pod based on service accounts.
type: string
types:
description: "Types indicates whether this policy applies to ingress,
or to egress, or to both. When not explicitly specified (and so
the value on creation is empty or nil), Calico defaults Types according
to what Ingress and Egress are present in the policy. The default
is: \n - [ PolicyTypeIngress ], if there are no Egress rules (including
the case where there are also no Ingress rules) \n - [ PolicyTypeEgress
], if there are Egress rules but no Ingress rules \n - [ PolicyTypeIngress,
PolicyTypeEgress ], if there are both Ingress and Egress rules.
\n When the policy is read back again, Types will always be one
of these values, never empty or nil."
items:
description: PolicyType enumerates the possible values of the PolicySpec
Types field.
type: string
type: array
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: networksets.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: NetworkSet
listKind: NetworkSetList
plural: networksets
singular: networkset
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
description: NetworkSet is the Namespaced-equivalent of the GlobalNetworkSet.
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: NetworkSetSpec contains the specification for a NetworkSet
resource.
properties:
nets:
description: The list of IP networks that belong to this set.
items:
type: string
type: array
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
---
# Source: calico/templates/calico-kube-controllers-rbac.yaml
# Include a clusterrole for the kube-controllers component,
# and bind it to the calico-kube-controllers serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-kube-controllers
rules:
# Nodes are watched to monitor for deletions.
- apiGroups: [""]
resources:
- nodes
verbs:
- watch
- list
- get
# Pods are watched to check for existence as part of IPAM controller.
- apiGroups: [""]
resources:
- pods
verbs:
- get
- list
- watch
# IPAM resources are manipulated in response to node and block updates, as well as periodic triggers.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ipreservations
verbs:
- list
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
- ipamblocks
- ipamhandles
verbs:
- get
- list
- create
- update
- delete
- watch
# Pools are watched to maintain a mapping of blocks to IP pools.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
verbs:
- list
- watch
# kube-controllers manages hostendpoints.
- apiGroups: ["crd.projectcalico.org"]
resources:
- hostendpoints
verbs:
- get
- list
- create
- update
- delete
# Needs access to update clusterinformations.
- apiGroups: ["crd.projectcalico.org"]
resources:
- clusterinformations
verbs:
- get
- list
- create
- update
- watch
# KubeControllersConfiguration is where it gets its config
- apiGroups: ["crd.projectcalico.org"]
resources:
- kubecontrollersconfigurations
verbs:
# read its own config
- get
# create a default if none exists
- create
# update status
- update
# watch for changes
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-kube-controllers
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: calico-kube-controllers
subjects:
- kind: ServiceAccount
name: calico-kube-controllers
namespace: kube-system
---
---
# Source: calico/templates/calico-node-rbac.yaml
# Include a clusterrole for the calico-node DaemonSet,
# and bind it to the calico-node serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-node
rules:
# The CNI plugin needs to get pods, nodes, and namespaces.
- apiGroups: [""]
resources:
- pods
- nodes
- namespaces
verbs:
- get
# EndpointSlices are used for Service-based network policy rule
# enforcement.
- apiGroups: ["discovery.k8s.io"]
resources:
- endpointslices
verbs:
- watch
- list
- apiGroups: [""]
resources:
- endpoints
- services
verbs:
# Used to discover service IPs for advertisement.
- watch
- list
# Used to discover Typhas.
- get
# Pod CIDR auto-detection on kubeadm needs access to config maps.
- apiGroups: [""]
resources:
- configmaps
verbs:
- get
- apiGroups: [""]
resources:
- nodes/status
verbs:
# Needed for clearing NodeNetworkUnavailable flag.
- patch
# Calico stores some configuration information in node annotations.
- update
# Watch for changes to Kubernetes NetworkPolicies.
- apiGroups: ["networking.k8s.io"]
resources:
- networkpolicies
verbs:
- watch
- list
# Used by Calico for policy information.
- apiGroups: [""]
resources:
- pods
- namespaces
- serviceaccounts
verbs:
- list
- watch
# The CNI plugin patches pods/status.
- apiGroups: [""]
resources:
- pods/status
verbs:
- patch
# Used for creating service account tokens to be used by the CNI plugin
- apiGroups: [""]
resources:
- serviceaccounts/token
resourceNames:
- calico-node
verbs:
- create
# Calico monitors various CRDs for config.
- apiGroups: ["crd.projectcalico.org"]
resources:
- globalfelixconfigs
- felixconfigurations
- bgppeers
- globalbgpconfigs
- bgpconfigurations
- ippools
- ipreservations
- ipamblocks
- globalnetworkpolicies
- globalnetworksets
- networkpolicies
- networksets
- clusterinformations
- hostendpoints
- blockaffinities
- caliconodestatuses
verbs:
- get
- list
- watch
# Calico must create and update some CRDs on startup.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
- felixconfigurations
- clusterinformations
verbs:
- create
- update
# Calico must update some CRDs.
- apiGroups: [ "crd.projectcalico.org" ]
resources:
- caliconodestatuses
verbs:
- update
# Calico stores some configuration information on the node.
- apiGroups: [""]
resources:
- nodes
verbs:
- get
- list
- watch
# These permissions are only required for upgrade from v2.6, and can
# be removed after upgrade or on fresh installations.
- apiGroups: ["crd.projectcalico.org"]
resources:
- bgpconfigurations
- bgppeers
verbs:
- create
- update
# These permissions are required for Calico CNI to perform IPAM allocations.
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
- ipamblocks
- ipamhandles
verbs:
- get
- list
- create
- update
- delete
- apiGroups: ["crd.projectcalico.org"]
resources:
- ipamconfigs
verbs:
- get
# Block affinities must also be watchable by confd for route aggregation.
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
verbs:
- watch
# The Calico IPAM migration needs to get daemonsets. These permissions can be
# removed if not upgrading from an installation using host-local IPAM.
- apiGroups: ["apps"]
resources:
- daemonsets
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: calico-node
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: calico-node
subjects:
- kind: ServiceAccount
name: calico-node
namespace: kube-system
---
# Source: calico/templates/calico-node.yaml
# This manifest installs the calico-node container, as well
# as the CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: calico-node
namespace: kube-system
labels:
k8s-app: calico-node
spec:
selector:
matchLabels:
k8s-app: calico-node
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
k8s-app: calico-node
spec:
nodeSelector:
kubernetes.io/os: linux
hostNetwork: true
tolerations:
# Make sure calico-node gets scheduled on all nodes.
- effect: NoSchedule
operator: Exists
# Mark the pod as a critical add-on for rescheduling.
- key: CriticalAddonsOnly
operator: Exists
- effect: NoExecute
operator: Exists
serviceAccountName: calico-node
# Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
# deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
terminationGracePeriodSeconds: 0
priorityClassName: system-node-critical
initContainers:
# This container performs upgrade from host-local IPAM to calico-ipam.
# It can be deleted if this is a fresh installation, or if you have already
# upgraded to use calico-ipam.
- name: upgrade-ipam
image: harbor.dayuan1997.com/devops/calico/cni:v3.23.2
command: ["/opt/cni/bin/calico-ipam", "-upgrade"]
envFrom:
- configMapRef:
# Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
name: kubernetes-services-endpoint
optional: true
env:
- name: KUBERNETES_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CALICO_NETWORKING_BACKEND
valueFrom:
configMapKeyRef:
name: calico-config
key: calico_backend
volumeMounts:
- mountPath: /var/lib/cni/networks
name: host-local-net-dir
- mountPath: /host/opt/cni/bin
name: cni-bin-dir
securityContext:
privileged: true
# This container installs the CNI binaries
# and CNI network config file on each node.
- name: install-cni
image: harbor.dayuan1997.com/devops/calico/cni:v3.23.2
command: ["/opt/cni/bin/install"]
envFrom:
- configMapRef:
# Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
name: kubernetes-services-endpoint
optional: true
env:
# Name of the CNI config file to create.
- name: CNI_CONF_NAME
value: "10-calico.conflist"
# The CNI network config to install on each node.
- name: CNI_NETWORK_CONFIG
valueFrom:
configMapKeyRef:
name: calico-config
key: cni_network_config
# Set the hostname based on the k8s node name.
- name: KUBERNETES_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
# CNI MTU Config variable
- name: CNI_MTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# Prevents the container from sleeping forever.
- name: SLEEP
value: "false"
volumeMounts:
- mountPath: /host/opt/cni/bin
name: cni-bin-dir
- mountPath: /host/etc/cni/net.d
name: cni-net-dir
securityContext:
privileged: true
containers:
# Runs calico-node container on each Kubernetes node. This
# container programs network policy and routes on each
# host.
- name: calico-node
image: harbor.dayuan1997.com/devops/calico/node:v3.23.2
envFrom:
- configMapRef:
# Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
name: kubernetes-services-endpoint
optional: true
env:
# Use Kubernetes API as the backing datastore.
- name: DATASTORE_TYPE
value: "kubernetes"
# Wait for the datastore.
- name: WAIT_FOR_DATASTORE
value: "true"
# Set based on the k8s node name.
- name: NODENAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
# Choose the backend to use.
- name: CALICO_NETWORKING_BACKEND
valueFrom:
configMapKeyRef:
name: calico-config
key: calico_backend
# Cluster type to identify the deployment type
- name: CLUSTER_TYPE
value: "k8s,bgp"
# Auto-detect the BGP IP address.
- name: IP
value: "autodetect"
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
value: "Never"
# Enable or Disable VXLAN on the default IP pool.
- name: CALICO_IPV4POOL_VXLAN
value: "Never"
# Enable or Disable VXLAN on the default IPv6 IP pool.
- name: CALICO_IPV6POOL_VXLAN
value: "Never"
# Set MTU for tunnel device used if ipip is enabled
- name: FELIX_IPINIPMTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# Set MTU for the VXLAN tunnel device.
- name: FELIX_VXLANMTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# Set MTU for the Wireguard tunnel device.
- name: FELIX_WIREGUARDMTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
# - name: CALICO_IPV4POOL_CIDR
# value: "192.168.0.0/16"
# Disable file logging so `kubectl logs` works.
- name: CALICO_DISABLE_FILE_LOGGING
value: "true"
# Set Felix endpoint to host default action to ACCEPT.
- name: FELIX_DEFAULTENDPOINTTOHOSTACTION
value: "ACCEPT"
# Disable IPv6 on Kubernetes.
- name: FELIX_IPV6SUPPORT
value: "false"
- name: FELIX_HEALTHENABLED
value: "true"
securityContext:
privileged: true
resources:
requests:
cpu: 250m
lifecycle:
preStop:
exec:
command:
- /bin/calico-node
- -shutdown
livenessProbe:
exec:
command:
- /bin/calico-node
- -felix-live
- -bird-live
periodSeconds: 10
initialDelaySeconds: 10
failureThreshold: 6
timeoutSeconds: 10
readinessProbe:
exec:
command:
- /bin/calico-node
- -felix-ready
- -bird-ready
periodSeconds: 10
timeoutSeconds: 10
volumeMounts:
# For maintaining CNI plugin API credentials.
- mountPath: /host/etc/cni/net.d
name: cni-net-dir
readOnly: false
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /run/xtables.lock
name: xtables-lock
readOnly: false
- mountPath: /var/run/calico
name: var-run-calico
readOnly: false
- mountPath: /var/lib/calico
name: var-lib-calico
readOnly: false
- name: policysync
mountPath: /var/run/nodeagent
# For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
# parent directory.
- name: sysfs
mountPath: /sys/fs/
# Bidirectional means that, if we mount the BPF filesystem at /sys/fs/bpf it will propagate to the host.
# If the host is known to mount that filesystem already then Bidirectional can be omitted.
mountPropagation: Bidirectional
- name: cni-log-dir
mountPath: /var/log/calico/cni
readOnly: true
volumes:
# Used by calico-node.
- name: lib-modules
hostPath:
path: /lib/modules
- name: var-run-calico
hostPath:
path: /var/run/calico
- name: var-lib-calico
hostPath:
path: /var/lib/calico
- name: xtables-lock
hostPath:
path: /run/xtables.lock
type: FileOrCreate
- name: sysfs
hostPath:
path: /sys/fs/
type: DirectoryOrCreate
# Used to install CNI.
- name: cni-bin-dir
hostPath:
path: /opt/cni/bin
- name: cni-net-dir
hostPath:
path: /etc/cni/net.d
# Used to access CNI logs.
- name: cni-log-dir
hostPath:
path: /var/log/calico/cni
# Mount in the directory for host-local IPAM allocations. This is
# used when upgrading from host-local to calico-ipam, and can be removed
# if not using the upgrade-ipam init container.
- name: host-local-net-dir
hostPath:
path: /var/lib/cni/networks
# Used to create per-pod Unix Domain Sockets
- name: policysync
hostPath:
type: DirectoryOrCreate
path: /var/run/nodeagent
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: calico-node
namespace: kube-system
---
# Source: calico/templates/calico-kube-controllers.yaml
# See https://github.com/projectcalico/kube-controllers
apiVersion: apps/v1
kind: Deployment
metadata:
name: calico-kube-controllers
namespace: kube-system
labels:
k8s-app: calico-kube-controllers
spec:
# The controllers can only have a single active instance.
replicas: 1
selector:
matchLabels:
k8s-app: calico-kube-controllers
strategy:
type: Recreate
template:
metadata:
name: calico-kube-controllers
namespace: kube-system
labels:
k8s-app: calico-kube-controllers
spec:
nodeSelector:
kubernetes.io/os: linux
tolerations:
# Mark the pod as a critical add-on for rescheduling.
- key: CriticalAddonsOnly
operator: Exists
- key: node-role.kubernetes.io/master
effect: NoSchedule
serviceAccountName: calico-kube-controllers
priorityClassName: system-cluster-critical
containers:
- name: calico-kube-controllers
image: harbor.dayuan1997.com/devops/calico/kube-controllers:v3.23.2
env:
# Choose which controllers to run.
- name: ENABLED_CONTROLLERS
value: node
- name: DATASTORE_TYPE
value: kubernetes
livenessProbe:
exec:
command:
- /usr/bin/check-status
- -l
periodSeconds: 10
initialDelaySeconds: 10
failureThreshold: 6
timeoutSeconds: 10
readinessProbe:
exec:
command:
- /usr/bin/check-status
- -r
periodSeconds: 10
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: calico-kube-controllers
namespace: kube-system
---
# This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: calico-kube-controllers
namespace: kube-system
labels:
k8s-app: calico-kube-controllers
spec:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: calico-kube-controllers
---
# Source: calico/templates/calico-etcd-secrets.yaml
---
# Source: calico/templates/calico-typha.yaml
---
# Source: calico/templates/configure-canal.yaml
calico.yaml 参数解释
- name: CALICO_IPV4POOL_IPIP- 含义: 用于启用或禁用
IPIP（IP in IP）隧道。 value: "Never": 禁用IPIP（IP in IP）隧道
- 含义: 用于启用或禁用
- name: CALICO_IPV4POOL_VXLAN- 含义: 用于启用或禁用
VXLAN隧道。 value: "Never": 禁用VXLAN隧道
- 含义: 用于启用或禁用
跨节点之间的通信不经过
overlay[ ipip 、 vxlan ]的封装。自然而然就是通过路由的方式实现跨节点pod通信了
root@kind:~# kubectl apply -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created
- 查看
k8s集群和安装的服务
root@kind:~# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
calico-bgp-rr-control-plane Ready control-plane,master 17m v1.23.4 10.1.5.10 <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
calico-bgp-rr-worker Ready <none> 16m v1.23.4 10.1.5.11 <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
calico-bgp-rr-worker2 Ready <none> 16m v1.23.4 10.1.8.10 <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
calico-bgp-rr-worker3 Ready <none> 16m v1.23.4 10.1.8.11 <none> Ubuntu 21.10 5.11.5-051105-generic containerd://1.5.10
root@kind:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-8948fd975-j82kp 1/1 Running 0 3m51s
kube-system calico-node-9h9kf 1/1 Running 0 3m51s
kube-system calico-node-h5fjf 1/1 Running 0 3m51s
kube-system calico-node-jzn9l 1/1 Running 0 3m51s
kube-system calico-node-pt575 1/1 Running 0 3m51s
kube-system coredns-64897985d-6xjf9 1/1 Running 0 16m
kube-system coredns-64897985d-qf9t6 1/1 Running 0 16m
kube-system etcd-calico-bgp-rr-control-plane 1/1 Running 0 16m
kube-system kube-apiserver-calico-bgp-rr-control-plane 1/1 Running 0 16m
kube-system kube-controller-manager-calico-bgp-rr-control-plane 1/1 Running 0 16m
kube-system kube-proxy-28k5l 1/1 Running 0 16m
kube-system kube-proxy-dsm7h 1/1 Running 0 16m
kube-system kube-proxy-j8gv6 1/1 Running 0 16m
kube-system kube-proxy-xbrjj 1/1 Running 0 16m
kube-system kube-scheduler-calico-bgp-rr-control-plane 1/1 Running 0 16m
local-path-storage local-path-provisioner-5ddd94ff66-2bhdv 1/1 Running 0 16m
创建 Calico BGPPeeringPolicy 规则
bgp_peering_policy 规则配置文件
root@kind:~# cat bgp_peering_policy.sh
## 关闭bgp fullmesh的模式，采取bgp route reflector
cat <<EOF | calicoctl apply -f -
apiVersion: projectcalico.org/v3
items:
- apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Info
nodeToNodeMeshEnabled: false
kind: BGPConfigurationList
metadata:
EOF
## Calico node 节点配置BGP RR规则
cat <<EOF | calicoctl apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
annotations:
labels:
rack: rack0
name: calico-bgp-rr-control-plane
spec:
addresses:
- address: 10.1.5.10
type: InternalIP
bgp:
asNumber: 65005
ipv4Address: 10.1.5.10/24
orchRefs:
- nodeName: calico-bgp-rr-control-plane
orchestrator: k8s
EOF
## Calico node 节点配置BGP RR规则
cat <<EOF | calicoctl apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
labels:
rack: rack0
name: calico-bgp-rr-worker
spec:
addresses:
- address: 10.1.5.11
type: InternalIP
bgp:
asNumber: 65005
ipv4Address: 10.1.5.11/24
orchRefs:
- nodeName: calico-bgp-rr-worker
orchestrator: k8s
EOF
## Calico node 节点配置BGP RR规则
cat <<EOF | calicoctl apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
labels:
rack: rack1
name: calico-bgp-rr-worker2
spec:
addresses:
- address: 10.1.8.10
type: InternalIP
bgp:
asNumber: 65008
ipv4Address: 10.1.8.10/24
orchRefs:
- nodeName: calico-bgp-rr-worker2
orchestrator: k8s
EOF
## Calico node 节点配置BGP RR规则
cat <<EOF | calicoctl apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
labels:
rack: rack1
name: calico-bgp-rr-worker3
spec:
addresses:
- address: 10.1.8.11
type: InternalIP
bgp:
asNumber: 65008
ipv4Address: 10.1.8.11/24
orchRefs:
- nodeName: calico-bgp-rr-worker3
orchestrator: k8s
EOF
# peer to leaf0 switch
cat <<EOF | calicoctl apply -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
name: rack0-to-leaf0
spec:
peerIP: 10.1.5.1
asNumber: 65005
nodeSelector: rack == 'rack0'
EOF
# peer to leaf1 switch
cat <<EOF | calicoctl apply -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
name: rack1-to-leaf1
spec:
peerIP: 10.1.8.1
asNumber: 65008
nodeSelector: rack == 'rack1'
EOF
- 创建
CalicoBGPPeeringPolicy规则
root@kind:~# bash bgp_peering_policy.sh
Successfully applied 1 'BGPConfiguration' resource(s)
Successfully applied 1 'Node' resource(s)
Successfully applied 1 'Node' resource(s)
Successfully applied 1 'Node' resource(s)
Successfully applied 1 'Node' resource(s)
Successfully applied 1 'BGPPeer' resource(s)
Successfully applied 1 'BGPPeer' resource(s)
root@kind:~/wcni-kind/calico/6-kind-calico-bgp-rr# kubectl get BGPPeer
NAME AGE
rack0-to-leaf0 9s
rack1-to-leaf1 9s
calico 配置信息
root@kind:~# docker exec -it calico-bgp-rr-control-plane calicoctl node status
Calico process is running.
IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+---------------+-------+----------+-------------+
| 10.1.5.1 | node specific | up | 10:23:11 | Established |
+--------------+---------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
rr 详细信息
## 进入到 calico-node 节点查看 rr 信息
root@kind:~/wcni-kind/calico/5-kind-calico-fullmesh# kubectl -n kube-system exec -it ds/calico-node bash
## 进入 brid 控制台
[root@calico-bgp-rr-worker2 /]# birdcl
BIRD v0.3.3+birdv1.6.8 ready.
bird> show ?
show bfd ... Show information about BFD protocol
show interfaces Show network interfaces
show memory Show memory usage
show protocols [<protocol> | "<pattern>"] Show routing protocols
show roa ... Show ROA table
show route ... Show routing table
show static [<name>] Show details of static protocol
show status Show router status
show symbols ... Show all known symbolic names
## 接口信息
bird> show interfaces
lo up (index=1)
MultiAccess AdminUp LinkUp Loopback Ignored MTU=65536
127.0.0.1/8 (Primary, scope host)
eth0 up (index=11)
MultiAccess Broadcast Multicast AdminUp LinkUp MTU=1500
172.18.0.2/16 (Primary, scope site)
net0 up (index=34)
MultiAccess Broadcast Multicast AdminUp LinkUp MTU=9500
10.1.5.10/24 (Primary, scope site)
## BGP 路由信息
bird> show route all
0.0.0.0/0 via 10.1.5.1 on net0 [kernel1 10:14:26] * (10)
Type: inherit unicast univ
Kernel.source: 3
Kernel.metric: 0
10.1.8.0/24 via 10.1.5.1 on net0 [Node_10_1_5_1 10:23:12] * (100/?) [AS65008i]
Type: BGP unicast univ
BGP.origin: IGP
BGP.as_path: 500 65008
BGP.next_hop: 10.1.10.2
BGP.local_pref: 100
10.1.5.0/24 dev net0 [direct1 10:14:25] * (240)
Type: device unicast univ
via 10.1.5.1 on net0 [Node_10_1_5_1 10:23:12] (100/0) [i]
Type: BGP unicast univ
BGP.origin: IGP
BGP.as_path:
BGP.next_hop: 10.1.5.1
BGP.med: 0
BGP.local_pref: 100
10.244.192.0/26 via 10.1.5.1 on net0 [Node_10_1_5_1 10:23:12] * (100/?) [AS65008i]
Type: BGP unicast univ
BGP.origin: IGP
BGP.as_path: 500 65008
BGP.next_hop: 10.1.10.2
BGP.local_pref: 100
## 详细的 BGP 信息
bird> show protocols all
Node_10_1_5_1 BGP master up 10:23:11 Established
Description: Connection to BGP peer
Preference: 100
Input filter: ACCEPT
Output filter: calico_export_to_bgp_peers
Routes: 5 imported, 1 exported, 4 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 5 0 0 0 5
Import withdraws: 1 0 --- 1 0
Export updates: 21 10 9 --- 2
Export withdraws: 2 --- --- --- 8
BGP state: Established
Neighbor address: 10.1.5.1
Neighbor AS: 65005
Neighbor ID: 10.1.5.1
Neighbor caps: refresh restart-aware AS4 add-path-rx
Session: internal multihop AS4 add-path-tx
Source address: 10.1.5.10
Hold timer: 107/180
Keepalive timer: 38/60
k8s 集群安装 Pod 测试网络
root@kind:~# cat cni.yaml
apiVersion: apps/v1
kind: DaemonSet
#kind: Deployment
metadata:
labels:
app: cni
name: cni
spec:
#replicas: 1
selector:
matchLabels:
app: cni
template:
metadata:
labels:
app: cni
spec:
containers:
- image: harbor.dayuan1997.com/devops/nettool:0.9
name: nettoolbox
securityContext:
privileged: true
---
apiVersion: v1
kind: Service
metadata:
name: serversvc
spec:
type: NodePort
selector:
app: cni
ports:
- name: cni
port: 80
targetPort: 80
nodePort: 32000
root@kind:~# kubectl apply -f cni.yaml
daemonset.apps/cni created
service/serversvc created
root@kind:~# kubectl run net --image=harbor.dayuan1997.com/devops/nettool:0.9
pod/net created
- 查看安装服务信息
root@kind:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cni-4qs9x 1/1 Running 0 2m9s 10.244.81.66 calico-bgp-rr-worker <none> <none>
cni-8tljw 1/1 Running 0 2m9s 10.244.205.66 calico-bgp-rr-control-plane <none> <none>
cni-9ghj2 1/1 Running 0 2m9s 10.244.192.6 calico-bgp-rr-worker2 <none> <none>
cni-9tlvp 1/1 Running 0 2m9s 10.244.210.66 calico-bgp-rr-worker3 <none> <none>
net 1/1 Running 0 113s 10.244.81.67 calico-bgp-rr-worker <none> <none>
root@kind:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 52m
serversvc NodePort 10.96.40.243 <none> 80:32000/TCP 2m13s
serversvc NodePort 10.96.24.93 <none> 80:32000/TCP 2m13s
leaf路由器BGP信息查看
# lo clab-calico-bgp-rr-leaf0 bash
root@leaf0:/# show ip bgp
BGP table version is 6, local router ID is 10.1.5.1, vrf id 0
Default local pref 100, local AS 65005
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 10.1.5.0/24 0.0.0.0 0 32768 i
*> 10.1.8.0/24 10.1.10.2 0 500 65008 i
*= 10.1.12.2 0 800 65008 i
*>i10.244.81.64/26 10.1.5.11 100 0 i
*> 10.244.192.0/26 10.1.10.2 0 500 65008 i
*= 10.1.12.2 0 800 65008 i
*>i10.244.205.64/26 10.1.5.10 100 0 i
*= 10.244.210.64/26 10.1.12.2 0 800 65008 i
*> 10.1.10.2 0 500 65008 i
root@leaf0:/# show ip bgp neighbors | grep neighbor
BGP neighbor is 10.1.5.10, remote AS 65005, local AS 65005, internal link
Community attribute sent to this neighbor(all)
BGP neighbor is 10.1.5.11, remote AS 65005, local AS 65005, internal link
Community attribute sent to this neighbor(all)
BGP neighbor is 10.1.10.2, remote AS 500, local AS 65005, external link
Community attribute sent to this neighbor(all)
BGP neighbor is 10.1.12.2, remote AS 800, local AS 65005, external link
Community attribute sent to this neighbor(all)
root@leaf0:/# show ip bgp summary
IPv4 Unicast Summary:
BGP router identifier 10.1.5.1, local AS number 65005 vrf-id 0
BGP table version 6
RIB entries 11, using 2024 bytes of memory
Peers 4, using 82 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.5.10 4 65005 42 43 0 0 0 00:31:04 1
10.1.5.11 4 65005 41 44 0 0 0 00:31:04 1
10.1.10.2 4 500 55 55 0 0 0 00:46:04 3
10.1.12.2 4 800 55 55 0 0 0 00:46:04 3
Total number of neighbors 4
三、测试网络
同节点 Pod 网络通讯
可以查看此文档 Calico IPIP 模式 中，同节点网络通讯，数据包转发流程一致
跨节点同 Node 网段 Pod 网络通讯
Pod节点信息
## ip 信息
root@kind:~# kubectl exec -it net -- ip a l
3: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 86:93:18:5d:6d:4f brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.81.67/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::8493:18ff:fe5d:6d4f/64 scope link
valid_lft forever preferred_lft forever
## 路由信息
root@kind:~# kubectl exec -it net -- ip r s
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
查看 Pod 信息发现在 calico 中主机的 IP 地址为 32 位掩码，意味着该 IP 地址是单个主机的唯一标识，而不是一个子网。这个主机访问其他 IP 均会走路由到达
Pod节点所在Node节点信息
root@kind:~# docker exec -it calico-bgp-rr-worker bash
## ip 信息
root@calico-bgp-rr-worker:/# ip a l
5: cali7b54c5a7378@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-30f0d3f7-763a-0e2e-cce4-fa4063f0615a
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
6: cali8eecb1f59c6@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-8178c65d-7955-22f8-2bb0-a22b2c7997c3
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
17: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:12:00:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.18.0.5/16 brd 172.18.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fc00:f853:ccd:e793::5/64 scope global nodad
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe12:5/64 scope link
valid_lft forever preferred_lft forever
36: net0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default
link/ether aa:c1:ab:f6:82:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.1.5.11/24 scope global net0
valid_lft forever preferred_lft forever
inet6 fe80::a8c1:abff:fef6:8204/64 scope link
valid_lft forever preferred_lft forever
## 路由信息
root@calico-bgp-rr-worker:/# ip r s
default via 10.1.5.1 dev net0
10.1.5.0/24 dev net0 proto kernel scope link src 10.1.5.11
10.1.8.0/24 via 10.1.5.1 dev net0 proto bird
blackhole 10.244.81.64/26 proto bird
10.244.81.66 dev cali7b54c5a7378 scope link
10.244.81.67 dev cali8eecb1f59c6 scope link
10.244.192.0/26 via 10.1.5.1 dev net0 proto bird
10.244.205.64/26 via 10.1.5.10 dev net0 proto bird
10.244.210.64/26 via 10.1.5.1 dev net0 proto bird
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.5
Pod节点进行ping包测试，访问cni-8tljwPod节点
root@kind:~# kubectl exec -it net -- ping 10.244.205.66 -c 1
PING 10.244.205.66 (10.244.205.66): 56 data bytes
64 bytes from 10.244.205.66: seq=0 ttl=62 time=0.694 ms
--- 10.244.205.66 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.694/0.694/0.694 ms
查看 Node 节点 calico-bgp-rr-worker 网卡信息可以确定 6: cali8eecb1f59c6@if3 网卡和 Pod 节点 eth0 网卡互为 veth pair 网卡
Node节点calico-bgp-rr-workercali8eecb1f59c6网卡抓包
root@calico-bgp-rr-worker:/# tcpdump -pne -i cali8eecb1f59c6
08:11:20.164564 86:93:18:5d:6d:4f > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 98: 10.244.81.67 > 10.244.205.66: ICMP echo request, id 68, seq 26, length 64
08:11:20.164622 ee:ee:ee:ee:ee:ee > 86:93:18:5d:6d:4f, ethertype IPv4 (0x0800), length 98: 10.244.205.66 > 10.244.81.67: ICMP echo reply, id 68, seq 26, length 64
Pod 节点数据出来后，先到达了 veth pair 网卡，数据包源 mac 地址: 86:93:18:5d:6d:4f 为 Pod 节点 eth0 网卡 mac 地址，而目的 mac 地址: ee:ee:ee:ee:ee:ee 为cali8eecb1f59c6 网卡 mac 地址，查看宿主机路由信息，发现并在数据包会在通过 10.244.205.64/26 via 10.1.5.10 dev net0 proto bird 路由信息转发
Node节点calico-bgp-rr-workernet0网卡抓包
root@calico-bgp-rr-worker:/# tcpdump -pne -i net0 icmp
08:12:42.256121 aa:c1:ab:f6:82:04 > aa:c1:ab:21:21:b1, ethertype IPv4 (0x0800), length 98: 10.244.81.67 > 10.244.205.66: ICMP echo request, id 68, seq 108, length 64
08:12:42.256158 aa:c1:ab:21:21:b1 > aa:c1:ab:f6:82:04, ethertype IPv4 (0x0800), length 98: 10.244.205.66 > 10.244.81.67: ICMP echo reply, id 68, seq 108, length 64
数据包源 mac 地址: aa:c1:ab:f6:82:04 为 calico-bgp-rr-worker 节点 net0 网卡 mac 地址，而目的 mac 地址: aa:c1:ab:21:21:b1 为 calico-bgp-rr-control-plane 节点 10.1.5.10 网卡 mac 地址
root@calico-bgp-rr-worker:/# ip n s
10.1.5.10 dev net0 lladdr aa:c1:ab:21:21:b1 REACHABLE
数据包流向
- 数据从
pod服务发出，通过查看本机路由表，送往cali8eecb1f59c6网关。路由:default via 169.254.1.1 dev eth0 cali8eecb1f59c6网关在calico-bgp-rr-worker主机上，calico-bgp-rr-worker主机查看自身路由后，会送往net0接口，因为目的地址为10.244.205.66。路由:10.244.205.64/26 via 10.1.5.10 dev net0 proto bird- 通过
net0 接口，并送往对端calico-bgp-rr-control-plane主机。 - 对端
calico-bgp-rr-control-plane主机接受到数据包后，对数据包进行接封装 - 解封装后发现内部的数据包，目的地址为
10.244.205.66，通过查看本机路由表，送往calif87242eb33c网卡。路由:10.244.205.66 dev calif87242eb33c scope link - 通过
cali4565calif87242eb33cd0d993f网卡，最终会把数据包送到cni-8tljw主机
跨节点不同 Node 网段 Pod 网络通讯
Pod节点信息
## ip 信息
root@kind:~# kubectl exec -it net -- ip a l
3: eth0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 56:4d:84:69:19:7b brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.83.130/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::544d:84ff:fe69:197b/64 scope link
valid_lft forever preferred_lft forever
## 路由信息
root@kind:~# kubectl exec -it net -- ip r s
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
查看 Pod 信息发现在 calico 中主机的 IP 地址为 32 位掩码，意味着该 IP 地址是单个主机的唯一标识，而不是一个子网。这个主机访问其他 IP 均会走路由到达
-
Pod节点所在Node节点信息 -
Pod节点信息，查看前面: 同节点Pod网络通讯 -
Pod节点所在Node节点信息，查看前面: 跨节点同Node网段Pod网络通讯 -
Pod节点进行ping包测试，访问cni-9ghj2Pod节点
root@kind:~# kubectl exec -it net -- ping 10.244.192.6
PING 10.244.192.6 (10.244.192.6): 56 data bytes
64 bytes from 10.244.192.6: seq=0 ttl=59 time=0.145 ms
64 bytes from 10.244.192.6: seq=1 ttl=59 time=0.120 ms
64 bytes from 10.244.192.6: seq=2 ttl=59 time=0.131 ms
^C
--- 10.244.192.6 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.120/0.132/0.145 ms
查看 Node 节点 calico-bgp-rr-worker 网卡信息可以确定 4: cali8eecb1f59c6@if3 网卡和 Pod 节点 eth0 网卡互为 veth pair 网卡
Node节点calico-bgp-rr-workercali8eecb1f59c6网卡抓包
root@calico-bgp-rr-worker:/# tcpdump -pne -i cali8eecb1f59c6
08:19:16.785307 86:93:18:5d:6d:4f > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 98: 10.244.81.67 > 10.244.192.6: ICMP echo request, id 74, seq 114, length 64
08:19:16.785415 ee:ee:ee:ee:ee:ee > 86:93:18:5d:6d:4f, ethertype IPv4 (0x0800), length 98: 10.244.192.6 > 10.244.81.67: ICMP echo reply, id 74, seq 114, length 64
Pod 节点数据出来后，先到达了 veth pair 网卡，数据包源 mac 地址: 86:93:18:5d:6d:4f 为 Pod 节点 eth0 网卡 mac 地址，而目的 mac 地址: ee:ee:ee:ee:ee:ee 为cali8eecb1f59c6 网卡 mac 地址，查看宿主机路由信息，发现并在数据包会在通过 10.244.192.0/26 via 10.1.5.1 dev net0 proto bird 路由信息转发
Node节点calico-bgp-rr-workernet0网卡抓包
root@calico-bgp-rr-worker:/# tcpdump -pne -i net0 icmp
08:18:12.754289 aa:c1:ab:f6:82:04 > aa:c1:ab:ac:b8:8a, ethertype IPv4 (0x0800), length 98: 10.244.81.67 > 10.244.192.6: ICMP echo request, id 74, seq 50, length 64
08:18:12.754403 aa:c1:ab:ac:b8:8a > aa:c1:ab:f6:82:04, ethertype IPv4 (0x0800), length 98: 10.244.192.6 > 10.244.81.67: ICMP echo reply, id 74, seq 50, length 64
数据包源 mac 地址: aa:c1:ab:f6:82:04 为 calico-bgp-rr-worker 节点 net0 网卡 mac 地址，而目的 mac 地址: aa:c1:ab:ac:b8:8a 为 leaf0 节点 10.1.5.1 网卡 mac 地址
root@calico-bgp-rr-worker:/# ip n s
10.1.5.1 dev net0 lladdr aa:c1:ab:ac:b8:8a REACHABLE
leaf0节点10.1.5.1网卡抓包
root@kind:~# docker exec -it clab-calico-bgp-rr-leaf0 bash
root@leaf0:/# tcpdump -pne -i eth3
08:21:21.851056 aa:c1:ab:f6:82:04 > aa:c1:ab:ac:b8:8a, ethertype IPv4 (0x0800), length 98: 10.244.81.67 > 10.244.192.6: ICMP echo request, id 74, seq 239, length 64
08:21:21.851120 aa:c1:ab:ac:b8:8a > aa:c1:ab:f6:82:04, ethertype IPv4 (0x0800), length 98: 10.244.192.6 > 10.244.81.67: ICMP echo reply, id 74, seq 239, length 64
leaf0节点10.1.5.1网卡抓包信息同calico-bgp-rr-worker节点net0网卡抓包
leaf0 节点 IP 信息，路由信息
## ip 地址信息
root@leaf0:/# ip a l
32: eth2@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default
link/ether aa:c1:ab:8d:4d:8f brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet 10.1.12.1/24 brd 10.1.12.255 scope global eth2
valid_lft forever preferred_lft forever
inet6 fe80::a8c1:abff:fe8d:4d8f/64 scope link
valid_lft forever preferred_lft forever
38: eth3@if39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default
link/ether aa:c1:ab:ac:b8:8a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.1.5.1/24 brd 10.1.5.255 scope global eth3
valid_lft forever preferred_lft forever
inet6 fe80::a8c1:abff:feac:b88a/64 scope link
valid_lft forever preferred_lft forever
42: eth0@if43: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:14:14:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.20.20.5/24 brd 172.20.20.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 2001:172:20:20::5/64 scope global nodad
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe14:1405/64 scope link
valid_lft forever preferred_lft forever
44: eth1@if45: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default
link/ether aa:c1:ab:39:a1:aa brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet 10.1.10.1/24 brd 10.1.10.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::a8c1:abff:fe39:a1aa/64 scope link
valid_lft forever preferred_lft forever
## 路由信息
root@leaf0:/# ip r s
default via 172.20.20.1 dev eth0
10.1.5.0/24 dev eth3 proto kernel scope link src 10.1.5.1
10.1.8.0/24 proto bgp metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
10.1.10.0/24 dev eth1 proto kernel scope link src 10.1.10.1
10.1.12.0/24 dev eth2 proto kernel scope link src 10.1.12.1
10.244.81.64/26 via 10.1.5.11 dev eth3 proto bgp metric 20
10.244.192.0/26 proto bgp metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
10.244.205.64/26 via 10.1.5.10 dev eth3 proto bgp metric 20
10.244.210.64/26 proto bgp metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
172.20.20.0/24 dev eth0 proto kernel scope link src 172.20.20.5
查看路由信息后， 10.244.192.6 下一跳地址为 10.1.10.2 或者 10.1.12.2 ，这个就是 BGP 的 ECMP 特性，网卡流量分摊。
- 分别在
leaf0节点eth1: 10.1.10.1和eth2: 10.1.12.1这2个网卡抓包，查看路由是否符合ECMP特性
root@leaf0:/# tcpdump -pne -i eth1
08:26:01.053484 aa:c1:ab:75:59:16 > aa:c1:ab:39:a1:aa, ethertype IPv4 (0x0800), length 98: 10.244.192.6 > 10.244.81.67: ICMP echo reply, id 74, seq 518, length 64
08:26:02.053669 aa:c1:ab:75:59:16 > aa:c1:ab:39:a1:aa, ethertype IPv4 (0x0800), length 98: 10.244.192.6 > 10.244.81.67: ICMP echo reply, id 74, seq 519, length 64
root@leaf0:/# tcpdump -pne -i eth2
08:27:30.975368 aa:c1:ab:8d:4d:8f > aa:c1:ab:8e:35:d3, ethertype IPv4 (0x0800), length 98: 10.244.81.67 > 10.244.192.6: ICMP echo request, id 87, seq 0, length 64
08:27:31.975560 aa:c1:ab:8d:4d:8f > aa:c1:ab:8e:35:d3, ethertype IPv4 (0x0800), length 98: 10.244.81.67 > 10.244.192.6: ICMP echo request, id 87, seq 1, length 64
通过抓包我们发现了 ICMP 的 reply 包和 request 包，分别通过 eth1 和 eth2 2个网卡进行数据传输，符合 ECMP 特性，同理的我们在上层 spine0 和 spine1 节点，也只能分别抓到一条对应的 ICMP 数据包信息
- 查看
leaf0节点 bgp 信息
root@leaf0:/# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route
K>* 0.0.0.0/0 [0/0] via 172.20.20.1, eth0, 00:48:16
C>* 10.1.5.0/24 is directly connected, eth3, 00:48:16
B>* 10.1.8.0/24 [20/0] via 10.1.10.2, eth1, 00:48:16
* via 10.1.12.2, eth2, 00:48:16
C>* 10.1.10.0/24 is directly connected, eth1, 00:48:16
C>* 10.1.12.0/24 is directly connected, eth2, 00:48:16
B>* 10.244.81.64/26 [200/0] via 10.1.5.11, eth3, 00:44:29
B>* 10.244.192.0/26 [20/0] via 10.1.10.2, eth1, 00:44:30
* via 10.1.12.2, eth2, 00:44:30
B>* 10.244.205.64/26 [200/0] via 10.1.5.10, eth3, 00:44:29
B>* 10.244.210.64/26 [20/0] via 10.1.10.2, eth1, 00:44:29
* via 10.1.12.2, eth2, 00:44:29
C>* 172.20.20.0/24 is directly connected, eth0, 00:48:16
root@leaf0:/# show ip bgp
BGP table version is 6, local router ID is 10.1.5.1, vrf id 0
Default local pref 100, local AS 65005
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 10.1.5.0/24 0.0.0.0 0 32768 i
*> 10.1.8.0/24 10.1.10.2 0 500 65008 i
*= 10.1.12.2 0 800 65008 i
*>i10.244.81.64/26 10.1.5.11 100 0 i
*> 10.244.192.0/26 10.1.10.2 0 500 65008 i
*= 10.1.12.2 0 800 65008 i
*>i10.244.205.64/26 10.1.5.10 100 0 i
*= 10.244.210.64/26 10.1.12.2 0 800 65008 i
*> 10.1.10.2 0 500 65008 i
- 查看
spine0节点 bgp 信息
root@spine0:/# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route
K>* 0.0.0.0/0 [0/0] via 172.20.20.1, eth0, 00:48:51
B>* 10.1.5.0/24 [20/0] via 10.1.10.1, eth1, 00:48:28
B>* 10.1.8.0/24 [20/0] via 10.1.34.1, eth2, 00:48:28
C>* 10.1.10.0/24 is directly connected, eth1, 00:48:28
C>* 10.1.34.0/24 is directly connected, eth2, 00:48:28
B>* 10.244.81.64/26 [20/0] via 10.1.10.1, eth1, 00:48:21
B>* 10.244.192.0/26 [20/0] via 10.1.34.1, eth2, 00:48:21
B>* 10.244.205.64/26 [20/0] via 10.1.10.1, eth1, 00:48:21
B>* 10.244.210.64/26 [20/0] via 10.1.34.1, eth2, 00:48:21
C>* 172.20.20.0/24 is directly connected, eth0, 00:48:51
root@spine0:/# show ip bgp
BGP table version is 6, local router ID is 10.1.10.2, vrf id 0
Default local pref 100, local AS 500
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 10.1.5.0/24 10.1.10.1 0 0 65005 i
*> 10.1.8.0/24 10.1.34.1 0 0 65008 i
*> 10.244.81.64/26 10.1.10.1 0 65005 i
*> 10.244.192.0/26 10.1.34.1 0 65008 i
*> 10.244.205.64/26 10.1.10.1 0 65005 i
*> 10.244.210.64/26 10.1.34.1 0 65008 i
spine0节点eth1网卡抓包
root@spine0:/# tcpdump -pne -i eth1
08:32:22.070871 aa:c1:ab:75:59:16 > aa:c1:ab:39:a1:aa, ethertype IPv4 (0x0800), length 98: 10.244.192.6 > 10.244.81.67: ICMP echo reply, id 87, seq 291, length 64
08:32:23.071067 aa:c1:ab:75:59:16 > aa:c1:ab:39:a1:aa, ethertype IPv4 (0x0800), length 98: 10.244.192.6 > 10.244.81.67: ICMP echo reply, id 87, seq 292, length 64
数据包流向
- 数据从
pod服务发出，通过查看本机路由表，送往cali8eecb1f59c6网关。路由:default via 169.254.1.1 dev eth0 cali8eecb1f59c6网关在calico-bgp-rr-worker主机上，calico-bgp-rr-worker主机查看自身路由后，会送往net0接口，因为目的地址为10.244.192.6。路由:10.244.192.0/26 via 10.1.5.1 dev net0 proto bird- 通过
net0 接口，并送往leaf0主机。 leaf0主机查看自身路由后，会送往eth1或者eth2接口，因为目的地址为10.244.192.6。- 路由:
B>* 10.244.192.0/26 [20/0] via 10.1.10.2, eth1, 00:44:30via 10.1.12.2, eth2, 00:44:30
- 路由:
- 通过
eth1或者eth2接口，并送往spine0或者spine1主机。 spine0或者spine1主机查看自身路由后，会送往eth2接口，因为目的地址为10.244.192.6。路由:B>* 10.244.192.0/26 [20/0] via xxxxxxx, eth2, 00:48:21- 对端
calico-bgp-rr-worker2主机接受到数据包后，对数据包进行接封装 - 解封装后发现内部的数据包，目的地址为
10.244.192.6，通过查看本机路由表，送往cali9b947619378网卡。路由:10.244.192.6 dev cali9b947619378 scope link - 通过
cali9b947619378网卡，最终会把数据包送到cni-9ghj2主机
Service 网络通讯
可以查看此文档 Calico IPIP 模式 中，Service 网络通讯，数据包转发流程一致
Service IP 宣告
- 宣告
ServiceIP
root@kind:~# calicoctl patch BGPConfig default --patch '{"spec": {"serviceClusterIPs": [{"cidr": "10.96.0.0/16"}]}}'
Successfully patched 1 'BGPConfiguration' resource
- 查看
leaf0节点路由并测试
root@kind:~# lo clab-calico-bgp-rr-leaf0 bash
root@leaf0:/# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route
K>* 0.0.0.0/0 [0/0] via 172.20.20.1, eth0, 00:48:16
C>* 10.1.5.0/24 is directly connected, eth3, 00:48:16
B>* 10.1.8.0/24 [20/0] via 10.1.10.2, eth1, 00:48:16
* via 10.1.12.2, eth2, 00:48:16
C>* 10.1.10.0/24 is directly connected, eth1, 00:48:16
C>* 10.1.12.0/24 is directly connected, eth2, 00:48:16
B>* 10.96.0.0/16 [200/0] via 10.1.5.10, eth3, 00:00:53
* via 10.1.5.11, eth3, 00:00:53
B>* 10.244.81.64/26 [200/0] via 10.1.5.11, eth3, 00:44:29
B>* 10.244.192.0/26 [20/0] via 10.1.10.2, eth1, 00:44:30
* via 10.1.12.2, eth2, 00:44:30
B>* 10.244.205.64/26 [200/0] via 10.1.5.10, eth3, 00:44:29
B>* 10.244.210.64/26 [20/0] via 10.1.10.2, eth1, 00:44:29
* via 10.1.12.2, eth2, 00:44:29
C>* 172.20.20.0/24 is directly connected, eth0, 00:48:16
root@leaf0:/# curl http://10.96.24.93
PodName: cni-9ghj2 | PodIP: eth0 10.244.192.6/32
root@leaf0:/# curl http://10.96.24.93
PodName: cni-8tljw | PodIP: eth0 10.244.205.66/32
root@leaf0:/# curl http://10.96.24.93
PodName: cni-4qs9x | PodIP: eth0 10.244.81.66/32
root@leaf0:/# curl http://10.96.24.93
PodName: cni-9tlvp | PodIP: eth0 10.244.210.66/32
root@leaf0:/# curl http://10.96.24.93
PodName: cni-8tljw | PodIP: eth0 10.244.205.66/32
- 查看
leaf1节点路由并测试
root@kind:~# lo clab-calico-bgp-rr-leaf1 bash
sroot@leaf1:/# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route
K>* 0.0.0.0/0 [0/0] via 172.20.20.1, eth0, 00:31:50
B>* 10.1.5.0/24 [20/0] via 10.1.11.2, eth2, 00:31:23
* via 10.1.34.2, eth1, 00:31:23
C>* 10.1.8.0/24 is directly connected, eth3, 00:31:36
C>* 10.1.11.0/24 is directly connected, eth2, 00:31:35
C>* 10.1.34.0/24 is directly connected, eth1, 00:31:33
B>* 10.96.0.0/16 [200/0] via 10.1.8.10, eth3, 00:06:41
* via 10.1.8.11, eth3, 00:06:41
B>* 10.244.81.64/26 [20/0] via 10.1.11.2, eth2, 00:29:00
* via 10.1.34.2, eth1, 00:29:00
B>* 10.244.192.0/26 [200/0] via 10.1.8.10, eth3, 00:28:41
B>* 10.244.205.64/26 [20/0] via 10.1.11.2, eth2, 00:28:42
* via 10.1.34.2, eth1, 00:28:42
B>* 10.244.210.64/26 [200/0] via 10.1.8.11, eth3, 00:28:42
C>* 172.20.20.0/24 is directly connected, eth0, 00:31:50
root@leaf1:/# curl http://10.96.24.93
PodName: cni-4qs9x | PodIP: eth0 10.244.81.66/32
- 查看
spine0节点路由并测试
root@kind:~# lo clab-calico-bgp-rr-spine0 bash
root@spine0:/# shwo ip route
bash: shwo: command not found
root@spine0:/# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route
K>* 0.0.0.0/0 [0/0] via 172.20.20.1, eth0, 00:27:33
B>* 10.1.5.0/24 [20/0] via 10.1.10.1, eth1, 00:27:04
B>* 10.1.8.0/24 [20/0] via 10.1.34.1, eth2, 00:27:03
C>* 10.1.10.0/24 is directly connected, eth1, 00:27:19
C>* 10.1.34.0/24 is directly connected, eth2, 00:27:17
B>* 10.96.0.0/16 [20/0] via 10.1.34.1, eth2, 00:02:22
B>* 10.244.81.64/26 [20/0] via 10.1.10.1, eth1, 00:24:41
B>* 10.244.192.0/26 [20/0] via 10.1.34.1, eth2, 00:24:22
B>* 10.244.205.64/26 [20/0] via 10.1.10.1, eth1, 00:24:23
B>* 10.244.210.64/26 [20/0] via 10.1.34.1, eth2, 00:24:23
C>* 172.20.20.0/24 is directly connected, eth0, 00:27:33
root@spine0:/# curl http://10.96.24.93
PodName: cni-9ghj2 | PodIP: eth0 10.244.192.6/32
