CentOS高可用运维案例之---配置bond0

案例说明：
在CentOS 7系统下，两个Server之间网卡部署bond0连接测试。

系统版本：

[root@node201 ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

网络架构：

bond绑定模式介绍：
mode=1(active-backup)(主-备份策略)
这个是主备模式，只有一块网卡是active，另一块是备用的standby，所有流量都在active链路上处理，交换机配置的是捆绑的话将不能工作，因为交换机往两块网卡发包，有一半包是丢弃的。
特点：只有一个设备处于活动状态，当一个宕掉另一个马上由备份转换为主设备。mac地址是外部可见得，从外面看来，bond的MAC地址是唯一的，以避免switch(交换机)发生混乱。
此模式只提供了容错能力；由此可见此算法的优点是可以提供高网络连接的可用性，但是它的资源利用率较低，只有一个接口处于工作状态，在有 N 个网络接口的情况下，资源利用率为1/N。

一、查看系统内核是否支持bond
如下所示，如果可以获取到信息，则系统内核支持bond：

[root@node201 ~]# modinfo bonding |more
filename:       /lib/modules/3.10.0-1160.118.1.el7.x86_64/kernel/drivers/net/bonding/bonding.ko.xz
author:         Thomas Davis, tadavis@lbl.gov and many others
description:    Ethernet Channel Bonding Driver, v3.7.1
version:        3.7.1
license:        GPL
alias:          rtnl-link-bond
retpoline:      Y
rhelversion:    7.9
srcversion:     B395E7507BE97AC98A6E886
depends:
intree:         Y
vermagic:       3.10.0-1160.118.1.el7.x86_64 SMP mod_unload modversions
signer:         CentOS Linux kernel signing key
sig_key:        7C:18:B6:12:D5:11:92:49:73:9A:2C:83:4F:26:1F:AC:0B:15:18:19
sig_hashalgo:   sha256
......

二、查看主机网卡信息

[root@node201 ~]# ip add sh
.......
4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:34:0a:8f brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.115/24 brd 192.168.56.255 scope global noprefixroute dynamic enp0s9
  
5: enp0s10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:58:bd:ac brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.114/24 brd 192.168.56.255 scope global noprefixroute dynamic enp0s10


[root@node201 network-scripts]# ethtool enp0s9
Settings for enp0s9:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off (auto)
        Supports Wake-on: umbg
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes


Settings for enp0s10:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off (auto)
        Supports Wake-on: umbg
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

三、配置网卡
1、配置物理网卡enp0s9

[root@node201 network-scripts]# cat ifcfg-enp0s9
TYPE=Ethernet
BOOTPROTO=none
NAME=enp0s9
DEVICE=enp0s9
ONBOOT=yes
MASTER=bond0
SLAVE=yes

2、配置物理网卡enp0s10

[root@node201 network-scripts]# cat ifcfg-enp0s10
TYPE=Ethernet
BOOTPROTO=none
NAME=enp0s10
DEVICE=enp0s10
ONBOOT=yes
MASTER=bond0
SLAVE=yes

3、配置bond网卡

[root@node201 network-scripts]# cat ifcfg-bond0
DEVICE=bond0
NAME='System bond0'
TYPE=Ethernet
NM_CONTROLLED=no
USERCTL=no
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.10.100
NETMASK=255.255.255.0
BONDING_OPTS='mode=1 miimon=100'
IPV6INIT=no

# miimon=100
# 每100毫秒 (即0.1秒) 监测一次路连接状态，如果有一条线路不通就转入另一条线路； 
Linux的多网卡绑定功能使用的是内核中的"bonding"模块，如果修改为其它模式，
只需要在BONDING_OPTS中指定mode=Number即可。

4、配置内核加载

[root@node201 network-scripts]# echo 'alias bond0 bonding' >> /etc/modprobe.d/dist.conf
[root@node201 network-scripts]# echo 'options bonding mode=1 miimon=100 fail_over_mac=1' >> /etc/modprobe.d/dist.conf
[root@node201 network-scripts]# echo 'ifenslave bond0 enp0s9 enp0s10' >>/etc/rc.local

[root@node201 network-scripts]# cat /etc/modprobe.d/dist.conf
alias bond0 bonding
options bonding mode=1 miimon=100 fail_over_mac=1

# 内核文档中有说明：bond0获取mac地址有两种方式,一种是从第一个活跃网卡中获取mac地址，
然后其余的SLAVE网卡的mac地址都使用该mac地址；另一种是使用fail_over_mac参数，
是bond0使用当前活跃网卡的mac地址，mac地址或者活跃网卡的转换而变。

四、激活bond0网卡

1、重启网络服务network
如下所示，network服务启动异常：

[root@node201 network-scripts]# systemctl restart network.service
[root@node201 network-scripts]# systemctl status network.service
● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2024-10-15 14:00:02 CST; 1min 11s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 24746 ExecStop=/etc/rc.d/init.d/network stop (code=exited, status=0/SUCCESS)
  Process: 25044 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=1/FAILURE)
    Tasks: 0

Oct 15 14:00:01 node201 network[25044]: Bringing up interface enp0s3:  Connection successfully activate...on/6)
Oct 15 14:00:01 node201 network[25044]: [  OK  ]
Oct 15 14:00:02 node201 network[25044]: Bringing up interface enp0s8:  Connection successfully activate...on/7)
Oct 15 14:00:02 node201 network[25044]: [  OK  ]
Oct 15 14:00:02 node201 network[25044]: Bringing up interface enp0s9:  Error: Connection activation fai...ation
Oct 15 14:00:02 node201 network[25044]: [FAILED]
Oct 15 14:00:02 node201 systemd[1]: network.service: control process exited, code=exited status=1
Oct 15 14:00:02 node201 systemd[1]: Failed to start LSB: Bring up/down networking.
Oct 15 14:00:02 node201 systemd[1]: Unit network.service entered failed state.
Oct 15 14:00:02 node201 systemd[1]: network.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

2、手工激活bond0网卡
如下所示，激活bond0网卡异常：

[root@node201 network-scripts]# ifdown bond0;ifup bond0
Error: Connection activation failed: Master device 'enp0s10' can't be activated: Device unmanaged or not available for activation
WARN      : [/etc/sysconfig/network-scripts/ifup-eth] Unable to start slave device ifcfg-enp0s10 for master bond0.
Error: Connection activation failed: Master device 'enp0s9' can't be activated: Device unmanaged or not available for activation
WARN      : [/etc/sysconfig/network-scripts/ifup-eth] Unable to start slave device ifcfg-enp0s9 for master bond0.

# 激活物理网卡失败
[root@node201 network-scripts]# ifup enp0s9
Error: Connection activation failed: Master device 'enp0s9' can't be activated: Device unmanaged or not available for activation

3、查看bond0绑定信息
如下所示，未显示bond0的物理网卡（slave）的信息：

[root@node201 network-scripts]#  modprobe bonding
[root@node201 network-scripts]#  cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: None
MII Status: down
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

4、停止networkmanager服务

[root@node201 network-scripts]# service NetworkManager stop
Redirecting to /bin/systemctl stop NetworkManager.service

5、激活bond0网卡
如下所示，在停止networkmanager服务后，可以正常激活bond0及物理网卡，network网络服务启动正常。

[root@node201 network-scripts]# ifup enp0s9
[root@node201 network-scripts]# ifup enp0s10
[root@node201 network-scripts]# ifup bond0

# 启动network服务
[root@node201 network-scripts]# systemctl restart network
[root@node201 network-scripts]# systemctl status network
● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
   Active: active (running) since Tue 2024-10-15 14:22:07 CST; 6s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 38194 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=0/SUCCESS)
    Tasks: 1
   CGroup: /system.slice/network.service
           └─38604 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--enp0s8.lease -pf /var/run/dhclient-e...

Oct 15 14:22:07 node201 network[38194]: RTNETLINK answers: File exists
Oct 15 14:22:07 node201 network[38194]: RTNETLINK answers: File exists
Oct 15 14:22:07 node201 network[38194]: RTNETLINK answers: File exists
Oct 15 14:22:07 node201 network[38194]: RTNETLINK answers: File exists
Oct 15 14:22:07 node201 network[38194]: RTNETLINK answers: File exists
Oct 15 14:22:07 node201 network[38194]: RTNETLINK answers: File exists
Oct 15 14:22:07 node201 network[38194]: RTNETLINK answers: File exists
Oct 15 14:22:07 node201 network[38194]: RTNETLINK answers: File exists
Oct 15 14:22:07 node201 network[38194]: RTNETLINK answers: File exists
Oct 15 14:22:07 node201 systemd[1]: Started LSB: Bring up/down networking.

# 查看bond0网卡绑定信息
[root@node201 network-scripts]#  cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: enp0s9
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: enp0s9
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:34:0a:8f
Slave queue ID: 0

Slave Interface: enp0s10
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:58:bd:ac
Slave queue ID: 0

6、禁止networkmanager服务

[root@node201 network-scripts]# systemctl stop NetworkManager
[root@node201 network-scripts]# systemctl disable NetworkManager

7、查看网络信息
如下所示，bond0网卡上获取到ip，并且物理网卡和bond0网卡具有相同的mac地址：

[root@node201 network-scripts]# ip add sh

4: enp0s9: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000
    link/ether 08:00:27:34:0a:8f brd ff:ff:ff:ff:ff:ff
5: enp0s10: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000
    link/ether 08:00:27:34:0a:8f brd ff:ff:ff:ff:ff:ff
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 08:00:27:34:0a:8f brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.100/24 brd 192.168.10.255 scope global bond0

如下所示，物理网卡和bond网卡显示相同的mac地址：

五、测试网络连通

1、重启系统，应用bond生效。
2、ping对方主机bond0网卡ip

[root@node201 network-scripts]# ping 192.168.10.101
PING 192.168.10.101 (192.168.10.101) 56(84) bytes of data.
64 bytes from 192.168.10.101: icmp_seq=1 ttl=64 time=222 ms
64 bytes from 192.168.10.101: icmp_seq=2 ttl=64 time=0.402 ms
64 bytes from 192.168.10.101: icmp_seq=3 ttl=64 time=0.378 ms
64 bytes from 192.168.10.101: icmp_seq=4 ttl=64 time=0.420 ms
64 bytes from 192.168.10.101: icmp_seq=5 ttl=64 time=0.474 ms

3、模拟物理网卡down （本地或对方节点）
无论是本地或对方节点，bond中的任意一个物理网卡down，都不影响bond的正常通讯。
[root@node201 ~]# ifdown enp0s9

八、总结
在CentOS下配置bond网卡绑定，注意关闭NetworkManager服务，否则无法激活物理网卡和bond。