KingbaseES V8R6 集群运维系列 -- trusted_server

案例说明：
在KingbaseES V8R3及V8R6早期的版本，对于读写分离的集群如果网关地址无法连通，将会导致整个集群关闭，数据库服务无法访问。在后期版本的改进中，降低了对网关的依赖性，当网关地址不通时，会影响集群的部分高可用功能比如failover切换，但集群可以正常对外提供数据库访问服务。如下图所示：

适用版本：
KingbaseES V8R6

集群网关配置：

[kingbase@node101 bin]$ cat ../etc/repmgr.conf |grep trust
trusted_servers='192.168.1.1'
running_under_failure_trusted_servers='on'

一、查看集群节点状态

[kingbase@node101 bin]$ ./repmgr cluster show

 ID | Name  | Role    | Status    | Upstream | Location | Priority | Timeline | LSN_Lag | Connection string               
----+-------+---------+-----------+----------+----------+----------+----------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------
 1  | node1 | standby |   running | node2    | default  | 100      | 4        | 0 bytes | host=192.168.1.102 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node2 | primary | * running |          | default  | 100      | 4        |         | host=192.168.1.101 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3

二、模拟网关故障

[kingbase@node101 ~]$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
From 192.168.1.101 icmp_seq=10 Destination Host Unreachable
From 192.168.1.101 icmp_seq=11 Destination Host Unreachable
From 192.168.1.101 icmp_seq=12 Destination Host Unreachable
.....

---如上所示，所有集群节点已经无法ping通网关地址。

三、查看网关失败后集群状态

1、集群节点状态

[kingbase@node101 bin]$ ./repmgr cluster show

 ID | Name  | Role    | Status    | Upstream | Location | Priority | Timeline | LSN_Lag | Connection string               
----+-------+---------+-----------+----------+----------+----------+----------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------
 1  | node1 | standby |   running | node2    | default  | 100      | 4        | 0 bytes | host=192.168.1.102 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node2 | primary | * running |          | default  | 100      | 4        |         | host=192.168.1.101 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3

2、数据库连接测试

[kingbase@node102 bin]$ ./ksql -U system test
ksql (V8.0)
Type "help" for help.
                                                       version
----------------------------------------------------------------------------------------------------------------------
 KingbaseES V008R006C007B0012 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-46), 64-bit
(1 row)

---如上所示，网关无法连通后，集群节点状态及数据库服务仍都正常。

3、查看kbha.log日志

Tips：
KingbaseES V8R6集群通过kbha进程每过三秒执行一次网关连通性的测试。

[2023-04-10 15:57:30] [WARNING] ping host"192.168.1.1" failed
[2023-04-10 15:57:31] [NOTICE] PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.

--- 192.168.1.1 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 999ms
pipe 2

[2023-04-10 15:57:31] [WARNING] ping host"192.168.1.1" failed
[2023-04-10 15:57:31] [DETAIL] average RTT value is not greater than zero
[2023-04-10 15:57:31] [DEBUG] ping process end early. usleep(994400)

----如上所示，kbha.log日志记录了网关地址连接失败的日志。

四、集群failover切换测试
1、关闭主库数据库服务
[kingbase@node101 bin]$ ./sys_ctl stop -D ../../data

2、查看备库hamgr.log日志

[2023-04-10 16:13:41] [DEBUG] monitoring node in degraded state for 640 seconds
[2023-04-10 16:13:43] [DEBUG] connecting to: "user=esrep connect_timeout=10 dbname=esrep host=192.168.1.101 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr options=-csearch_path="
[2023-04-10 16:13:43] [DEBUG] monitoring node in degraded state for 642 seconds
[2023-04-10 16:13:45] [DEBUG] connecting to: "user=esrep connect_timeout=10 dbname=esrep host=192.168.1.101 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr options=-csearch_path="
[2023-04-10 16:13:45] [DEBUG] monitoring node in degraded state for 644 seconds
[2023-04-10 16:13:47] [DEBUG] connecting to: "user=esrep connect_timeout=10 dbname=esrep host=192.168.1.101 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr options=-csearch_path="
[2023-04-10 16:13:47] [DEBUG] monitoring node in degraded state for 646 seconds
[2023-04-10 16:13:49] [DEBUG] connecting to: "user=esrep connect_timeout=10 dbname=esrep host=192.168.1.101 port=54321 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3 fallback_application_name=repmgr options=-csearch_path="
[2023-04-10 16:13:49] [DEBUG] monitoring node in degraded state for 648 seconds

---如以上所示，备库检测到主库连接失败，但是并没有触发主备切换。

3、查看集群节点状态

[kingbase@node102 bin]$ ./repmgr cluster show
 ID | Name  | Role    | Status        | Upstream | Location | Priority | Timeline | LSN_Lag | Connection string           
----+-------+---------+---------------+----------+----------+----------+----------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------
 1  | node1 | standby |   running     | ? node2  | default  | 100      | 4        | ?       | host=192.168.1.102 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
 2  | node2 | primary | ? unreachable | ?        | default  | 100      |          |         | host=192.168.1.101 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3

[WARNING] following issues were detected
  - unable to connect to node "node1" (ID: 1)'s upstream node "node2" (ID: 2)
  - unable to determine if node "node1" (ID: 1) is attached to its upstream node "node2" (ID: 2)
  - unable to connect to node "node2" (ID: 2)
  - node "node2" (ID: 2) is registered as an active primary but is unreachable

[HINT] execute with --verbose option to see connection error messages

如下图所示，primary处于不可连接状态，未产生failover切换：

五、总结
KingbaseES集群节点通过ping网关地址，测试集群节点之间的网络的互通，如果网关失败，会影响到集群的正常运行，可以在集群中配置多个网关，保证网关地址的高可用。

[kingbase@node101 bin]$ cat ../etc/repmgr.conf |grep trust
trusted_servers='192.168.1.1,192.168.1.254'
running_under_failure_trusted_servers='on'