KingbaseES V8R6集群安装部署案例---脚本在线一键扩容
案例说明：
KingbaseES V8R6支持图形化方式在线扩容，但是在一些生产环境，在服务器不支持图形化界面的情况下 ，只能通过脚本命令行的方式执行集群的部署或在线扩容。
Tips：
KingbaseES V8R6C5默认情况下部署脚本（V8R6_cluster_install.sh）和配置文件（install.conf）不支持在线扩容，需要从KingbaseES V8R6C6版本下拷贝脚本和配置文件到 KingbaseES V8R6C5环境下使用。
适用版本：
KingbaseES V8R6
一、集群节点状态信息
1、主机节点信息
2、扩容前集群节点状态
[kingbase@node101 bin]$ ./repmgr cluster show
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+-----------+----------+----------+----------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------
1 | node101 | primary | * running | | default | 100 | 3 | host=192.168.1.101 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
2 | node102 | standby | running | node101 | default | 100 | 3 | host=192.168.1.102 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
[kingbase@node101 bin]$ ./repmgr service status
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------+---------+-----------+----------+---------+------+---------+--------------------
1 | node101 | primary | * running | | running | 2997 | no | n/a
2 | node102 | standby | running | node101 | running | 2777 | no | 0 second(s) ago
二、新增节点系统配置
扩容说明参考官方文档：
https://help.kingbase.com.cn/stage-api/profile/document/kes/v8r6/html/highly/availability/cluster-use/cluster-use-6.html#sshd
1、配置新增节点系统环境
1）扩容节点需要和原集群节点具有相同的操作系统环境，并且版本一致；
2）配置新增节点系统内核参数、防火墙、selinux、进程资源限制等，具体配置可见《KingbaseES官方文档》。
3）需在新增节点建立数据库用户及root用户到集群其他节点的ssh互信。
2、在新增节点创建目录和相关文件
=以下文件，可以从主库的数据库软件安装目录下获取。=
Tips：
V8R6_cluster_install.sh和install.conf文件从KingbaseES V8R6C6版本下拷贝，KingbaseES V8R6C5版本目前不支持。
[kingbase@node103 r6_install]$ ls -lh
total 264M
-rw-rw-r-- 1 kingbase kingbase 261M Apr 7 10:33 db.zip
-rw-rw-r-- 1 kingbase kingbase 8.2K Apr 7 10:54 install.conf
-rwxr-x--- 1 kingbase kingbase 3.2K Apr 7 10:42 license.dat
-rw-rw-r-- 1 kingbase kingbase 2.1M Apr 7 10:33 securecmdd.zip
-rwxrwxr-x 1 kingbase kingbase 3.3K Apr 7 10:33 trust_cluster.sh
-rwxrwxr-x 1 kingbase kingbase 87K Apr 7 10:33 V8R6_cluster_install.sh
三、在线脚本扩容（新增节点执行）
1、编辑install.conf
[kingbase@node103 r6_install]$ cat install.conf |grep -v ^$|grep -v ^#
[install]
on_bmj=0
all_ip=(192.168.1.101 192.168.1.102)
witness_ip=""
production_ip=()
local_disaster_recovery_ip=()
remote_disaster_recovery_ip=()
install_dir="/home/kingbase/cluster/HA_R6/kha/kingbase"
zip_package="/home/kingbase/r6_install/db.zip"
license_file=(license.dat)
db_user="system" # the user name of database
db_port="54321" # the port of database, defaults is 54321
db_mode="oracle" # database mode: pg, oracle
db_auth="scram-sha-256" # database authority: scram-sha-256, md5, default is scram-sha-256
db_case_sensitive="yes" # database case sensitive settings: yes, no. default is yes - case sensitive; no - case insensitive (NOTE. cannot set to 'no' when db_mode="pg").
trusted_servers="192.168.1.1"
data_directory="/home/kingbase/cluster/HA_R6/kha/kingbase/data"
virtual_ip=""
net_device=(enp0s3 enp0s3)
net_device_ip=(192.168.1.101 192.168.1.102)
ipaddr_path="/usr/sbin"
arping_path="/opt/Kingbase/ES/V8R6_041/Server/bin/"
ping_path="/bin"
super_user="root"
execute_user="kingbase"
deploy_by_sshd=1 # choose whether to use sshd when deploy, 0 means not to use (deploy by sys_securecmdd), 1 means to use (deploy by sshd), default value is 1; when on_bmj=1, it will auto set to no(deploy_by_sshd=0)
use_scmd=1 # Is the cluster running on sys_securecmdd or sshd? 1 means yes (on sys_securecmdd), 0 means no (on sshd), default value is 1; when on_bmj=1, it will auto set to yes(use_scmd=1)
reconnect_attempts="10" # the number of retries in the event of an error
reconnect_interval="6" # retry interval
recovery="standby" # the way of cluster recovery: standby/automatic/manual
ssh_port="22" # the port of ssh, default is 22
scmd_port="8890" # the port of sys_securecmdd, default is 8890
auto_cluster_recovery_level='1'
use_check_disk='off'
synchronous='quorum'
###### 以下为expand配置信息#######
[expand]
expand_type="0" # The node type of standby/witness node, which would be add to cluster. 0:standby 1:witness
primary_ip="192.168.101" # The ip addr of cluster primary node, which need to expand a standby/witness node.
expand_ip="192.168.1.103" # The ip addr of standby/witness node, which would be add to cluster.
node_id="3" # The node_id of standby/witness node, which would be add to cluster. It does not the same with any one in cluster node
# for example: node_id="3"
install_dir="/home/kingbase/cluster/HA_R6/kha"
zip_package="/home/kingbase/r6_install/db.zip"
net_device=(enp0s3) # if virtual_ip set,it must be set
net_device_ip=(192.168.1.103) # if virtual_ip set,it must be set
license_file=(license.dat)
deploy_by_sshd="1"
ssh_port="22"
scmd_port="8890"
############################################
[shrink]
shrink_type="" # The node type of standby/witness node, which would be delete from cluster. 0:standby 1:witness
primary_ip="" # The ip addr of cluster primary node, which need to shrink a standby/witness node.
shrink_ip="" # The ip addr of standby/witness node, which would be delete from cluster.
node_id="" # The node_id of standby/witness node, which would be delete from cluster. It does not the same with any one in cluster node
# for example: node_id="3"
install_dir=""
ssh_port="22" # the port of ssh, default is 22
scmd_port="8890" # the port of sys_securecmd, default is 8890
=以下为扩容（expand）节点配置信息，详细说明参考官方文档。=
2、执行扩容（expand）
[kingbase@node103 r6_install]$ sh V8R6_cluster_install.sh expand
[CONFIG_CHECK] will deploy the cluster of
[RUNNING] success connect to the target "192.168.1.103" ..... OK
.......
INFO: connecting to local node "node3" (ID: 3)
INFO: connecting to primary database
WARNING: --upstream-node-id not supplied, assuming upstream node is primary (node ID 1)
INFO: standby registration complete
NOTICE: standby node "node3" (ID: 3) successfully registered
Usage: /home/kingbase/cluster/R6HA/kha//kingbase/bin/sys_monitor.sh {start|stop|restart|stoplocal|set [--restart]|change_password user password}
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+---------------+----------+----------+----------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------
1 | node101 | primary | * running | | default | 100 | 3 | host=192.168.1.101 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
2 | node102 | standby | running | node101 | default | 100 | 3 | host=192.168.1.102 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
3 | node3 | standby | ? unreachable | node101 | default | 100 | ? | host=192.168.1.103 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
WARNING: following issues were detected
- unable to connect to node "node3" (ID: 3)
- node "node3" (ID: 3) is registered as an active standby but is unreachable
=如上所示，扩容成功，但是新增节点node3的状态为“unreachable”。=
三、扩容故障问题解决
1、查看集群节点状态信息
=如下所示，在主库上查询，node3状态信息为“unreachable”。=
[kingbase@node101 bin]$ ./repmgr cluster show
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+---------------+----------+----------+----------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------
1 | node101 | primary | * running | | default | 100 | 3 | host=192.168.1.101 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
2 | node102 | standby | running | node101 | default | 100 | 3 | host=192.168.1.102 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
3 | node3 | standby | ? unreachable | node101 | default | 100 | ? | host=192.168.1.103 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
WARNING: following issues were detected
- unable to connect to node "node3" (ID: 3)
2、查看新增节点防火墙信息
=如下所示，新增节点防火墙未关闭，将防火墙关闭。=
[root@node103 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-07-20 15:09:55 CST; 47min ago
Main PID: 815 (firewalld)
CGroup: /system.slice/firewalld.service
└─815 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
Jul 20 15:09:52 node103 systemd[1]: Starting firewalld - dynamic firewall daemon...
Jul 20 15:09:55 node103 systemd[1]: Started firewalld - dynamic firewall daemon.
[root@node103 ~]# systemctl stop firewalld
[root@node103 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
[root@node103 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Jul 20 15:09:52 node103 systemd[1]: Starting firewalld - dynamic firewall daemon...
Jul 20 15:09:55 node103 systemd[1]: Started firewalld - dynamic firewall daemon.
Jul 20 15:57:05 node103 systemd[1]: Stopping firewalld - dynamic firewall daemon...
Jul 20 15:57:08 node103 systemd[1]: Stopped firewalld - dynamic firewall daemon.
[root@node103 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
3、重新注册新增节点
[kingbase@node103 bin]$ ./repmgr standby register --force
INFO: connecting to local node "node3" (ID: 3)
INFO: connecting to primary database
INFO: standby registration complete
NOTICE: standby node "node3" (ID: 3) successfully registered
4、查看集群节点状态信息
[kingbase@node101 bin]$ ./repmgr cluster show
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
----+---------+---------+-----------+----------+----------+----------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------
1 | node101 | primary | * running | | default | 100 | 3 | host=192.168.1.101 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
2 | node102 | standby | running | node101 | default | 100 | 3 | host=192.168.1.102 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
3 | node3 | standby | running | node101 | default | 100 | 3 | host=192.168.1.103 user=system dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
=如上所示，集群所有节点状态正常，集群扩容完成。=
四、总结
1、注意配置新增节点的系统环境，如防火墙、网络、内核参数、Selinux等。
2、需要建立新增节点数据库用户和root用户与集群原节点之间的ssh互信。
3、install.conf中【install】部分为原脚本部署集群时的配置，扩容需要配置【expand】部分。
