KingbaseES V8R3集群维护案例之---pcp_node_refresh应用
案例说明：
在一次KingbaseES V8R3集群切换分析中，运维人员执行了pcp_node_refresh，导致集群发生了failover的切换。此文档对pcp_node_refresh工具做了应用说明及测试，不建议在生产环境使用此工具。
适用版本：
KingbaseES V8R3
一、pcp_node_refresh应用说明
1. pcp_node_refresh 必须要指定节点ID（就是show pool_nodes中查出来的id），刷新这个节点的状态。动作是将数据库踢出集群，然后再加入进来。
2. pcp_node_refresh 指向主库，将主库踢出集群，这个时候会认为主库故障触发会切换 ----------- 集群failover切换。
3. pcp_node_refresh 指向备库，将备库踢出集群再加入，集群无影响。
二、测试案例
1、集群节点状态
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
0 | 192.168.1.101 | 54321 | up | 0.500000 | primary | 0 | true | 0
1 | 192.168.1.102 | 54321 | down | 0.500000 | standby | 0 | false | 0
(2 rows)
---如上所示，备库节点状态是down，执行pcp_node_refresh刷新节点。
2、执行pcp_node_refresh刷新备库节点
1）执行pcp_node_refresh
# 执行pcp_node_refresh
[kingbase@node101 bin]$ ./pcp_node_refresh -U kingbase -h 192.168.1.101 1
Password:
pcp_node_refresh -- Command Successful
# 查看节点状态
[kingbase@node101 bin]$ ./ksql -U SYSTEM -W 123456 TEST -p 9999
ksql (V008R003C002B0290)
Type "help" for help.
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
0 | 192.168.1.101 | 54321 | up | 0.500000 | primary | 0 | false | 0
1 | 192.168.1.102 | 54321 | up | 0.500000 | standby | 0 | true | 0
(2 rows)
TEST=# select * from sys_stat_replication;
PID | USESYSID | USENAME | APPLICATION_NAME | CLIENT_ADDR | CLIENT_HOSTNAME | CLIENT_PORT | BACKEND_START | B
ACKEND_XMIN | STATE | SENT_LOCATION | WRITE_LOCATION | FLUSH_LOCATION | REPLAY_LOCATION | SYNC_PRIORITY | SYNC_STATE
-------+----------+---------+------------------+---------------+-----------------+-------------+-------------------------------+--
------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
28587 | 10 | SYSTEM | node101 | 192.168.1.102 | | 47629 | 2022-09-05 13:42:43.870560+08 |
| streaming | 1/E92607D0 | 1/E92607D0 | 1/E92607D0 | 1/E9260780 | 0 | async
(1 row)
---如上所示，备库节点状态恢复正常。
2）查看日志信息
如下cluster.log所示：执行pcp_node_refresh后，对备库节点作为故障节点进行了recovery的恢复。
如下recovery.log所示，执行pcp_node_refresh后，对备库节点调用sys_rewind恢复备库节点。
3、执行pcp_node_refresh刷新主库节点
1）执行pcp_node_refresh
[kingbase@node101 bin]$ ./pcp_node_refresh -U kingbase -h 192.168.1.101 0
Password:
pcp_node_refresh -- Command Successful
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
0 | 192.168.1.101 | 54321 | up | 0.500000 | standby | 0 | true | 0
1 | 192.168.1.102 | 54321 | up | 0.500000 | primary | 0 | false | 0
(2 rows)
---如上所示，主备节点发生了切换。
2）查看日志信息
如下cluster.log所示：执行pcp_node_refresh后，对主库节点作为故障调用failover_stream.sh执行了failover切换。
如下failover.log所示：执行pcp_node_refresh后，failover切换过程。
-----------------2022-09-05 13:45:42 failover beging---------------------------------------
----failover-stats is %H = hostname of the new master node [192.168.1.102], %P = old primary node id [1], %d = node id[0], %h = host name [192.168.1.101], %O = old primary host[192.168.1.102] %m = new master node id [1], %M = old master node id [1], %D = database cluster path [/home/kingbase/cluster/R3HA/db/data].
----ping trust ip
ping trust ip 192.168.1.1 success
----determine whether the faulty db is master or standby
standby down, master still 192.168.1.102
The sys_stat_replication view result is : []
2022-09-05 13:45:44 sync to async
ALTER SYSTEM
SYS_RELOAD_CONF
-----------------
t
(1 row)
-----------------2022-09-05 13:45:44 failover end---------------------------------------
三、总结
pcp_node_refresh不建议在生产环境使用，对于备库节点在数据库服务和流复制状态都正常的情况下，如果用show pool_nodes显示的是‘down’状态，可以使用pcp_attach_node重新注册备库节点，使节点状态恢复正常。
KINGBASE研究院
