KingbaseES V8R3集群维护案例之---在线添加备库管理节点
案例说明：
在KingbaseES V8R3主备流复制的集群中 ，一般有两个节点是集群的管理节点，分为master和standby；如对于一主二备的架构，其中有两个节点是管理节点，三个数据节点；管理节点运行kingbasecluster服务 ，负责集群节点状态的监控及集群主备切换等操作。
本案例详细介绍在一主一备的架构下，其中一个管理节点宕机的情况下，如何在线添加新的管理节点，如果宕机的节点是主备复制中的primary主库，将自动发生主备切换，所以在线添加的节点都是管理节点的备节点。
管理节点的添加和普通数据节点的在线添加不同，相对较复杂，有关数据节点的添加可以参考以下文档：https://www.cnblogs.com/tiany1224/p/15749993.html KingbaseES V8R3集群维护案例之--- 在线添加数据节点
适用版本：
KingbaseES V8R3通用机环境（专业机可参考）
本次案例数据库版本：
TEST=# select version();
VERSION
-------------------------------------------------------------------------------------------------------------------------
Kingbase V008R003C002B0290 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-46), 64-bit
(1 row)
一、集群原节点信息
[kingbase@node101 bin]$ ./ksql -U SYSTEM -W 123456 TEST -p 9999
ksql (V008R003C002B0290)
Type "help" for help.
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
0 | 192.168.1.101 | 54321 | up | 0.500000 | primary | 0 | true | 0
1 | 192.168.1.102 | 54321 | up | 0.500000 | standby | 0 | false | 0
(2 rows)
TEST=# select * from sys_stat_replication;
PID | USESYSID | USENAME | APPLICATION_NAME | CLIENT_ADDR | CLIENT_HOSTNAME | CLIENT_PORT | BACKEND_START | BACKEND_XMIN | STATE | SENT_LOCATION | WRITE_LOCATION | FLUSH_LOCATION | REPLAY_LOCATION | SYNC_PRIORITY | SYNC_STATE
-------+----------+---------+------------------+---------------+-----------------+-------------+-------------------------------+--------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
25143 | 10 | SYSTEM | node102 | 192.168.1.102 | | 61622 | 2022-06-22 10:22:48.771995+08 | | streaming | 0/1A0000D0 | 0/1A0000D0 | 0/1A0000D0 | 0/1A0000D0 | 2 | sync
(1 row)
二、备库数据库主机宕机
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
0 | 192.168.1.101 | 54321 | up | 0.500000 | primary | 3 | true | 0
1 | 192.168.1.102 | 54321 | down | 0.500000 | standby | 0 | false | 0
(2 rows)
三、准备新的主机环境
Tips：
1）系统环境准备参考《KingbaseES 官方文档》
https://help.kingbase.com.cn/stage-api/profile/document/kes/v8r3/html/highly/clusterware/cluster-manage.html
2）通用机不需要安装数据库软件，专用机需要安装和主库相同版本软件。
3）创建和主库相同的集群目录结构。
1、主库集群目录结构：
[kingbase@node101 R3HA]$ pwd
/home/kingbase/cluster/R3HA
[kingbase@node101 R3HA]$ ls -lh
total 33M
drwxrwxr-x 2 kingbase kingbase 4.0K Jun 21 19:04 archivedir
drwxrwxr-x 8 kingbase kingbase 95 Jun 13 19:28 db
-rw-r--r-- 1 kingbase root 29M Mar 29 15:10 db.zip
drwxrwxr-x 6 kingbase kingbase 48 Apr 1 2021 kingbasecluster
-rwxr-xr-x 1 kingbase root 4.3M Mar 29 15:10 kingbasecluster.zip
drwxr-xr-x 3 kingbase root 4.0K Apr 12 15:08 log
drwxr-xr-x 3 kingbase root 28 Mar 29 15:11 run
-rw------- 1 kingbase kingbase 8.0K Jun 22 10:42 template.bk
2、备库创建和主库相同的目录结构
[kingbase@node102 cluster]$ mkdir -p /home/kingbase/cluster/R3HA/
[kingbase@node102 cluster]$ cd R3HA
[kingbase@node102 R3HA]$ mkdir archivedir db kingbasecluster log run
[kingbase@node102 R3HA]$ ls -lh
total 0
drwxrwxr-x 2 kingbase kingbase 6 Jun 22 10:30 archivedir
drwxrwxr-x 2 kingbase kingbase 6 Jun 22 10:30 db
drwxrwxr-x 2 kingbase kingbase 6 Jun 22 10:30 kingbasecluster
drwxrwxr-x 2 kingbase kingbase 6 Jun 22 10:30 log
drwxrwxr-x 2 kingbase kingbase 6 Jun 22 10:30 run
3、从主库拷贝db目录下数据到备库（主库运行排除data目录）
[kingbase@node101 db]$ scp -r bin node102:/home/kingbase/cluster/R3HA/db
[kingbase@node101 db]$ scp -r etc node102:/home/kingbase/cluster/R3HA/db
[kingbase@node101 db]$ scp -r kb_scripts node102:/home/kingbase/cluster/R3HA/db
[kingbase@node101 db]$ scp -r lib node102:/home/kingbase/cluster/R3HA/db
[kingbase@node101 db]$ scp -r share node102:/home/kingbase/cluster/R3HA/db
[kingbase@node101 db]$ scp -r kingbase.log node102:/home/kingbase/cluster/R3HA/db
4、从主库拷贝kingbasecluster目录下所有数据到备库
[kingbase@node101 kingbasecluster]$ scp -r * node102:/home/kingbase/cluster/R3HA/kingbasecluster/
5、从主库拷贝log目录下所有数据到备库：（kingbasecluster.pid文件无需拷贝）
[kingbase@node101 log]$ scp -r * node102:/home/kingbase/cluster/R3HA/log/
6、从主库拷贝run目录下所有数据到备库：
[kingbase@node101 run]$ scp -r * node102:/home/kingbase/cluster/R3HA/run
7、从主库拷贝文件到备库
[kingbase@node101 R3HA]$ scp template.bk node102:/home/kingbase/cluster/R3HA
四、创建主备流复制
1、执行sys_basebackup克隆备库
[kingbase@node102 bin]$ ./sys_basebackup -h 192.168.1.101 -U SYSTEM -W 123456 -p 54321 -F p -x -v -P -D /home/kingbase/cluster/R3HA/db/data
transaction log start point: 0/1B000028 on timeline 7
108616/108616 kB (100%), 1/1 tablespace
transaction log end point: 0/1B0000F8
sys_basebackup: base backup completed
2、配置备库data目录权限
[kingbase@node102 db]$ chmod 700 data
3、配置recovery.conf
[kingbase@node102 data]$ cp ../etc/recovery.done ./recovery.conf
# 编辑recovery.conf:
[kingbase@node102 data]$ cat recovery.conf
standby_mode='on'
primary_conninfo='port=54321 host=192.168.1.101 user=SYSTEM password=MTIzNDU2 application_name=node101'
recovery_target_timeline='latest'
primary_slot_name ='slot_node101'
4、执行sys_ctl启动备库数据库服务
[kingbase@node102 bin]$ ./sys_ctl start -D /home/kingbase/cluster/R3HA/db/data
server starting
[kingbase@node102 bin]$ LOG: redirecting log output to logging collector process
HINT: Future log output will appear in directory "/home/kingbase/cluster/R3HA/db/data/sys_log".
# 查看备库数据库服务
[kingbase@node102 bin]$ ps -ef|grep kingbase
kingbase 17885 1 0 11:10 pts/0 00:00:00 /home/kingbase/cluster/R3HA/db/bin/kingbase -D /home/kingbase/cluster/R3HA/db/data
kingbase 17886 17885 0 11:10 ? 00:00:00 kingbase: logger process
kingbase 17887 17885 0 11:10 ? 00:00:00 kingbase: startup process recovering 00000007000000000000001C
kingbase 17891 17885 0 11:10 ? 00:00:00 kingbase: checkpointer process
kingbase 17892 17885 0 11:10 ? 00:00:00 kingbase: writer process
kingbase 17893 17885 0 11:10 ? 00:00:00 kingbase: stats collector process
kingbase 17894 17885 0 11:10 ? 00:00:00 kingbase: wal receiver process streaming 0/1C000060
5、查看流复制状态
# 查看节点状态（此时备库状态为down）
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
0 | 192.168.1.101 | 54321 | up | 0.500000 | primary | 4 | true | 0
1 | 192.168.1.102 | 54321 | down | 0.500000 | standby | 0 | false | 0
(2 rows)
# 查看流复制状态
TEST=# select * from sys_stat_replication;
PID | USESYSID | USENAME | APPLICATION_NAME | CLIENT_ADDR | CLIENT_HOSTNAME | CLIENT_PORT | BACKEND_START | BACKEND_XMIN | STATE | SENT_LOCATION | WRITE_LOCATION | FLUSH_LOCATION | REPLAY_LOCATION | SYNC_PRIORITY | SYNC_STATE
-------+----------+---------+------------------+---------------+-----------------+-------------+-------------------------------+--------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
25242 | 10 | SYSTEM | node101 | 192.168.1.102 | | 33495 | 2022-06-22 11:11:02.085186+08 | | streaming | 0/1C000060 | 0/1C000060 | 0/1C000060 | 0/1C000060 | 0 | async
(1 row)
# 查看复制槽信息
TEST=# select * from sys_replication_slots;
SLOT_NAME | PLUGIN | SLOT_TYPE | DATOID | DATABASE | ACTIVE | ACTIVE_PID | XMIN | CATALOG_XMIN | RESTART_LSN | CONFIRMED_FLUSH_LSN
--------------+--------+-----------+--------+----------+--------+------------+------+--------------+-------------+---------------------
slot_node101 | | physical | | | t | 25242 | 2112 | | 0/1C000060 |
slot_node102 | | physical | | | f | | 2112 | | 0/1A0000D0 |
(2 rows)
# 从以上可知，流复制状态正常。
五、备库配置kingbasecluster管理
Tips：
1）由于kingbasecluster的配置文件从主库复制而来，在备库需要做修改。
2）修改文件为kingbasecluster.conf 和 HAmodule.conf文件。
1、配置kingbasecluster.conf 文件
2、修改HAmodule.conf文件（包括kingbasecluster和db目录下）
[kingbase@node102 etc]$ vi HAmodule.conf
#the current node ip.example:KB_LOCALHOST_IP="192.168.28.128"
KB_LOCALHOST_IP="192.168.1.102"
#recoord the names of local node.example:NODE_NAME="node1"
NODE_NAME="node102"
3、root用户手工启动kingbasecluster
[root@node102 bin]# ./kingbasecluster -n >/home/kingbase/cluster/R3HA/log/cluster.log 2>&1 &
[1] 23929
4、查看kingbasecluster服务启动状态
[root@node102 bin]# netstat -an |grep 9999
tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTEN
tcp6 0 0 :::9999 :::* LISTEN
# 服务端口处于监听，kingbasecluster服务启动成功。
5、将备库节点注册到集群
[kingbase@node102 bin]$ ./pcp_attach_node -h 192.168.1.101 -U kingbase 1
Password:
pcp_attach_node -- Command Successful
# 备库注册后，节点状态为”up“。
[kingbase@node101 bin]$ ./ksql -U SYSTEM -W 123456 TEST -p 9999
ksql (V008R003C002B0290)
Type "help" for help.
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
0 | 192.168.1.101 | 54321 | up | 0.500000 | primary | 6 | true | 0
1 | 192.168.1.102 | 54321 | up | 0.500000 | standby | 0 | false | 0
(2 rows)
六、重启集群测试（生产环境在无业务时执行）
1、关闭备库的kingbasecluster服务
[root@node102 bin]# cd /home/kingbase/cluster/R3HA/kingbasecluster/bin
[root@node102 bin]# ./kingbasecluster -m fast stop
2022-06-22 11:41:01: pid 28321: LOG: stop request sent to kingbasecluster. waiting for termination...
.done.
[1]+ Done ./kingbasecluster -n > /home/kingbase/cluster/R3HA/log/cluster.log 2>&1
2、重启集群
[kingbase@node101 bin]$ ./kingbase_monitor.sh restart
-----------------------------------------------------------------------
2022-06-22 11:42:50 KingbaseES automation beging...
......
......................
all started..
...
now we check again
=======================================================================
| ip | program| [status]
[ 192.168.1.101]| [kingbasecluster]| [active]
[ 192.168.1.102]| [kingbasecluster]| [active]
[ 192.168.1.101]| [kingbase]| [active]
[ 192.168.1.102]| [kingbase]| [active]
=======================================================================
# 如上所示，集群启动成功。
3、备库配置cron计划任务
[root@node101 ~]# cat /etc/cron.d/KINGBASECRON
*/1 * * * * kingbase /home/kingbase/cluster/R3HA/db/bin/network_rewind.sh
*/1 * * * * root /home/kingbase/cluster/R3HA/kingbasecluster/bin/restartcluster.sh
七、集群failover切换测试（生产环境业务低峰测试）
1、配置主备库arping文件属主及权限（专用机不需要此操作）
[root@node102 bin]# cd /home/kingbase/cluster/R3HA/db/bin
[root@node102 bin]# chown root.root arping
[root@node102 bin]# chmod u+s arping
[root@node102 bin]# ls -lh arping
-rwsr-xr-x 1 root root 33K Apr 1 2021 arping
2、failover主备切换测试
1）关闭主库数据库服务
[kingbase@node101 bin]$ ./sys_ctl stop -D ../data
waiting for server to shut down.... done
server stopped
2）查看切换后的节点状态
[kingbase@node102 bin]$ ./ksql -U SYSTEM -W 123456 TEST -p 9999
ksql (V008R003C002B0290)
Type "help" for help.
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
0 | 192.168.1.101 | 54321 | down | 0.500000 | standby | 0 | false | 0
1 | 192.168.1.102 | 54321 | up | 0.500000 | primary | 0 | true | 0
(2 rows)
# 如上所示，node102已经切换为primary。
3、recovery原主库作为备库加入集群
1）配置recovery.conf
[kingbase@node101 data]$ cp ../etc/recovery.done ./recovery.conf
[kingbase@node101 data]$ cat recovery.conf
standby_mode='on'
primary_conninfo='port=54321 host=192.168.1.102 user=SYSTEM password=MTIzNDU2 application_name=node101'
recovery_target_timeline='latest'
primary_slot_name ='slot_node101'
2）在新主库创建复制槽
TEST=# select * from sys_replication_slots;
SLOT_NAME | PLUGIN | SLOT_TYPE | DATOID | DATABASE | ACTIVE | ACTIVE_PID | XMIN | CATALOG_XMIN | RESTART_LSN | CONFIRMED_FLUSH_LSN
-----------+--------+-----------+--------+----------+--------+------------+------+--------------+-------------+---------------------
(0 rows)
TEST=# select sys_create_physical_replication_slot('slot_node101');
SYS_CREATE_PHYSICAL_REPLICATION_SLOT
--------------------------------------
(slot_node101,)
(1 row)
TEST=# select sys_create_physical_replication_slot('slot_node102');
SYS_CREATE_PHYSICAL_REPLICATION_SLOT
--------------------------------------
(slot_node102,)
(1 row)
TEST=# select * from sys_replication_slots;
SLOT_NAME | PLUGIN | SLOT_TYPE | DATOID | DATABASE | ACTIVE | ACTIVE_PID | XMIN | CATALOG_XMIN | RESTART_LSN | CONFIRMED_FLUSH_LSN
--------------+--------+-----------+--------+----------+--------+------------+------+--------------+-------------+---------------------
slot_node101 | | physical | | | f | | | | |
slot_node102 | | physical | | | f | | | | |
(2 rows)
3、启动原主库（新备库）数据库服务
[kingbase@node101 bin]$ ./sys_ctl start -D ../data
server starting
[kingbase@node101 bin]$ LOG: redirecting log output to logging collector process
HINT: Future log output will appear in directory "/home/kingbase/cluster/R3HA/db/data/sys_log".
4、在新主库查看流复制和节点状态
# 查看流复制状态
TEST=# select * from sys_stat_replication;
PID | USESYSID | USENAME | APPLICATION_NAME | CLIENT_ADDR | CLIENT_HOSTNAME | CLIENT_PORT | BACKEND_START | BACKEND_XMIN | STATE | SENT_LOCATION | WRITE_LOCATION | FLUSH_LOCATION | REPLAY_LOCATION | SYNC_PRIORITY | SYNC_STATE
-------+----------+---------+------------------+---------------+-----------------+-------------+-------------------------------+--------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
17271 | 10 | SYSTEM | node101 | 192.168.1.101 | | 17540 | 2022-06-22 14:09:44.848740+08 | | streaming | 0/22050C18 | 0/22050C18 | 0/22050C18 | 0/2204FFD8 | 0 | async
(1 row)
# 查看集群节点状态
[kingbase@node102 bin]$ ./ksql -U SYSTEM -W 123456 TEST -p 9999
ksql (V008R003C002B0290)
Type "help" for help.
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+-------------------
0 | 192.168.1.101 | 54321 | up | 0.500000 | standby | 0 | false | 0
1 | 192.168.1.102 | 54321 | up | 0.500000 | primary | 0 | true | 0
(2 rows)
# 由以上可知，集群failover切换完成！！！
八、总结
对于KingbaseES V8R3集群在线添加新的管理节点，操作较复杂，尤其是对kingbasecluster配置文件的修改，注意细节部分，否则集群的启动和切换都会受到影响。
KINGBASE研究院
