Calico在K8S中配置BGP反射模式

kunernetes中路由器反射模式下与外部路由器配置BGP,减少外部物理设备与K8S集群节点之间建立的BGP对等连接。如下为本次的拓扑图

1.配置K8S集群为路由器反射模式

1.1.关闭全互联模式

root@master:~/bgp# cat disable-node-mesh.yml 
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  # 关闭calico默认的node-to-node mesh
  nodeToNodeMeshEnabled: false
  # 默认BGP集群的as号64512
  asNumber: 64512
root@master:~/bgp# kubectl apply -f disable-node-mesh.yml
root@master:~/bgp# 

1.2.配置反射路由器

root@master:~/bgp# kubectl label node master.sec.lab route-reflector=true
root@master:~/bgp# kubectl annotate node master.sec.lab projectcalico.org/RouteReflectorClusterID=244.0.0.1
root@master:~/bgp# cat peer-int-bgp.yml 
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: peer-with-route-reflectors
spec:
  nodeSelector: all()
  peerSelector: route-reflector == 'true'
root@master:~/bgp# kubectl apply -f peer-int-bgp.yml
root@master:~/bgp# 

1.3.查看反射路由器与各节点建立连接

# Master节点中
root@master:~/bgp# calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.22.4.12   | node specific | up    | 05:20:48 | Established |
| 10.22.4.13   | node specific | up    | 05:20:46 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

root@master:~/bgp# 

# Node节点中
root@node01:~# calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.22.4.11   | node specific | up    | 05:20:48 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

root@node01:~# 

2.配置K8S反射路由器与外部路由建立BGP

2.1. 配置反射路由器

root@master:~/bgp# cat peer-ext-bgp.yml 
apiVersion: projectcalico.org/v3  
kind: BGPPeer  
metadata:  
  name: bgppeer-to-external-device  
spec:  
  nodeSelector: route-reflector == 'true'
  peerIP: 10.22.4.1 
  keepOriginalNextHop: true
  asNumber: 65002
root@master:~/bgp# kubectl apply -f peer-ext-bgp.yml
root@master:~/bgp# 

2.2. 配置物理路由器

• 此处物理路由器为华为三层

<AR200>system-view 
Enter system view, return user view with Ctrl+Z.
[AR200]
[AR200]bgp 65002
[AR200-bgp]display this 
#
bgp 65002
 router-id 10.22.1.1
 peer 10.22.4.11 as-number 64512
 peer 10.22.4.11 connect-interface Vlanif14
 #
 ipv4-family unicast
  undo synchronization
  default-route imported
  import-route static
  peer 10.22.4.11 enable
#
return
[AR200-bgp]

3.查看K8S BGP路由

3.1.查看邻居关系

• 在Master节点和Node节点

# 在Master节点上
root@master:~/bgp# calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.22.4.12   | node specific | up    | 05:20:48 | Established |
| 10.22.4.13   | node specific | up    | 05:20:46 | Established |
| 10.22.4.1    | node specific | up    | 05:31:20 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

root@master:~/bgp#
# 在Node节点上
root@node01:~# calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.22.4.11   | node specific | up    | 05:20:48 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

• 在华为路由器中查看

<AR200>display bgp peer

 BGP local router ID : 10.22.1.1
 Local AS number : 65002
 Total number of peers : 1                Peers in established state : 1

  Peer            V          AS  MsgRcvd  MsgSent  OutQ  Up/Down       State PrefRcv

  10.22.4.11      4       64512       27       25     0 00:19:05 Established       3
<AR200>

3.2.查看路由

• 在华为路由器中查看BGP路由

<AR200>display bgp routing-table 

 BGP Local router ID is 10.22.1.1 
 Status codes: * - valid, > - best, d - damped,
               h - history,  i - internal, s - suppressed, S - Stale
               Origin : i - IGP, e - EGP, ? - incomplete


 Total Number of Routes: 4
      Network            NextHop        MED        LocPrf    PrefVal Path/Ogn

 *>   0.0.0.0            0.0.0.0         0                     0      ?
 *>   10.244.25.0/24     10.22.4.13                            0      64512i
 *>   10.244.182.0/24    10.22.4.12                            0      64512i
 *>   10.244.187.0/24    10.22.4.11                            0      64512i
<AR200>