KingbaseES V8R6集群运维系列 -- 修改ssh通信为 sys_securecmdd 通信
一、适用于：
本文档使用于KingbaseES V008R006版本。
二、关于SYS_SECURECMDD：
sys_securecmdd是KingbaseES集群自带的工具，集群监控、管理集群时通过sys_securecmdd安全执行命令而不使用ssh服务。
sys_securecmdd主要包含以下文件：
| 服务端 | sys_securecmdd | 默认监听8890端口，接受客户端连接。 |
|---|---|---|
| sys_secureftp | 服务端调用，用于接收文件。 | |
| sys_HAscmdd.sh | 脚本，管理服务端。 | |
| 客户端 | sys_securecmd | 客户端，用于连接服务端。 |
| 密钥文件 | accept_hosts | 免密文件 |
| key_file | 私钥文件 | |
| 其他文件 | securecmdd_config | 服务端配置文件 |
| securecmd_config | 客户端配置文件 | |
| sys_HAscmdd.conf | 脚本配置文件 | |
| securecmdd.service | 服务模板文件，服务端可以使用此文件 注册为服务。 | |
| 依赖库文件 | libcrypto.so.10 | 依赖openssl编译，为了不同环境能够 使用，需要带上编译使用的库文件。 |
sys_HAscmdd.conf是sys_securecmdd的配置文件，其中参数说明如下：
| 参数名称 | 描述 | 取值约束 |
|---|---|---|
| start_method | 启动sys_securecmdd进程并保证进程 高可用的方式。systemd，通用机集群 默认值，通过service服务启动 sys_securecmdd；crontab,专用机 集群默认值，通过crond服务定时启动 sys_securecmdd。 | crontab, systemd 默认为crontab。 |
| scmd_port | 进程sys_securecmdd的监听端口，修 改后，需要使用 sys_HAscmdd.sh脚 本初始化。 | INT，默认8890。 |
三、安装部署SYS_SECURECMDD服务：
进行安装部署SYS_SECURECMDD服务期间，不要停止数据库。
1. 部署SYS_SECURECMDD服务：
1.1 查询服务器防火墙是否开启：
所有的节点执行此操作：
systemctl status firewalld.service
# 如果active状态是running，表示防火墙是开启的。
Active: active (running)
# 防火墙开启的话需要添加对应策略
# 其中10046是为sys_securecmdd服务预留的端口
firewall-cmd --permanent --add-port=10046/tcp
firewall-cmd --permanent --add-port=10046/udp
firewall-cmd --reload
# 添加完策略后可以使用以下命令查看是否生效
# 如果有添加的端口输出说明是生效的
firewall-cmd --list-port
54321/tcp 54321/udp 10046/tcp 10046/udp
1.2 上传securecmdd.zip到集群所有节点:
# zip包默认路径
../V008R006C007B0012/ClientTools/guitools/DeployTools/zip/Lin64/
$ ls -l
total 2260
drwxrwxr-x. 3 kes_v8r6c7b12 kes_v8r6c7b12 83 Feb 27 16:04 cluster
-rw-r--r--. 1 kes_v8r6c7b12 kes_v8r6c7b12 2115099 Mar 1 14:22 securecmdd.zip
# scp securecmdd.zip到node2节点
$ scp securecmdd.zip kes_v8r6c7b12@node2:~
The authenticity of host 'node2 (192.168.10.43)' can't be established.
securecmdd.zip 100% 2066KB 14.8MB/s 00:00
1.3 解压securecmdd.zip并安装securecmdd：
集群所有节点都执行以下操作：
# 解压securecmdd.zip包
$ unzip securecmdd.zip
Archive: securecmdd.zip
creating: securecmdd/
creating: securecmdd/lib/
inflating: securecmdd/lib/libcrypto.so.10
inflating: securecmdd/lib/libssl.so.10
creating: securecmdd/bin/
inflating: securecmdd/bin/sys_securecmd
inflating: securecmdd/bin/sys_secureftp
inflating: securecmdd/bin/sys_HAscmdd.sh
inflating: securecmdd/bin/sys_securecmdd
creating: securecmdd/share/
inflating: securecmdd/share/sys_HAscmdd.conf
inflating: securecmdd/share/key_file
inflating: securecmdd/share/securecmdd_config
inflating: securecmdd/share/securecmdd.service
inflating: securecmdd/share/securecmd_config
inflating: securecmdd/share/accept_hosts
# 修改sys_HAscmdd.conf更改默认端口8890为10046
# sys_HAscmdd.conf配置文件在securecmdd/share目录下
vi sys_HAscmdd.conf
scmd_port=10046
# 执行sys_HAscmdd.sh init进行初始化
# 如果出现以下错误，需要切换为root用户执行
$ sys_HAscmdd.sh init
Only execute by root, current user is kes_v8r6c7b12
# init成功
./sys_HAscmdd.sh init
successfully initialized the sys_securecmdd, please use "./sys_HAscmdd.sh start" to start the sys_securecmdd
# 使用./sys_HAscmdd.sh start启动
./sys_HAscmdd.sh start
Created symlink /etc/systemd/system/multi-user.target.wants/securecmdd.service → /etc/systemd/system/securecmdd.service.
# 查看是否正常启动
systemctl status securecmdd
● securecmdd.service - KingbaseES - sys_securecmdd daemon
Loaded: loaded (/etc/systemd/system/securecmdd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2023-03-01 14:55:36 CST; 12s ago
Main PID: 39535 (sys_securecmdd)
Tasks: 1 (limit: 12498)
Memory: 668.0K
CGroup: /system.slice/securecmdd.service
└─39535 sys_securecmdd: /home/kes_v8r6c7b12/securecmdd/bin/sys_securecmdd -f /etc/.kes/securecmdd_config [listener] 0 of 128-256 startups
Mar 01 14:55:36 node1 systemd[1]: Started KingbaseES - sys_securecmdd daemon.
# 测试连接是否正常
./sys_securecmd kes_v8r6c7b12@192.168.10.43 date
Wed Mar 1 15:02:29 CST 2023
./sys_securecmd kes_v8r6c7b12@192.168.10.40 date
Wed Mar 1 15:02:38 CST 2023
测试通过后，至此完成securecmdd的安装。
sys_securecmdd安装完成后，会在root、kingbase用户目录生成.es目录，包含以下文件（.es目录的文档建议不要修改）：
key_file是sys_securecmdd服务私钥文件。
accept_hosts是sys_securecmdd服务密钥文件（集群节点互信使用）。
# root用户目录
[root@node2 ~]# ls -l .es/
total 8
-rw------- 1 root root 381 Mar 3 14:07 accept_hosts
-rw------- 1 root root 1675 Mar 3 14:07 key_file
# 数据库用户目录
[root@node2 ~]# ls -l /home/kes_v8r6c7b12/.es/
total 8
-rw------- 1 kes_v8r6c7b12 kes_v8r6c7b12 381 Mar 3 14:07 accept_hosts
-rw------- 1 kes_v8r6c7b12 kes_v8r6c7b12 1675 Mar 3 14:07 key_file
[root@node2 ~]#
修改accept_hosts文件会导致集群节点互信失效，失效后的处理：
在所有的节点执行以下操作：
# 停止sys_securecmdd服务
./sys_HAscmdd.sh stop
# 重新初始化sys_securecmdd服务
./sys_HAscmdd.sh init
# 启动sys_securecmdd服务
./sys_HAscmdd.sh start
# 测试节点连通性
[root@node2 bin]# ./sys_securecmd root@node1 date
Fri Mar 3 14:07:33 CST 2023
[root@node2 bin]# ./sys_securecmd root@node2 date
Fri Mar 3 14:07:36 CST 2023
[root@node2 bin]# ./sys_securecmd root@192.168.10.40 date
Fri Mar 3 14:07:45 CST 2023
[root@node2 bin]# ./sys_securecmd root@192.168.10.43 date
Fri Mar 3 14:07:48 CST 2023
四、修改数据库集群使用SYS_SECURECMDD通信
1. 修改repmgr.conf配置文件使用SYS_SECURECMDD通信：
在集群所有节点进行以下操作：
# 修改repmgr.con文件里面use_scmd=off为on
# use_scmd=off 是不使用SYS_SECURECMDD通信，使用系统SSH进行通信。use_scmd=on 使用SYS_SECURECMDD通信
use_scmd=on
# 修改scmd_options 选项里面端口为sys_HAscmdd.conf文件里面scmd_port=10046
scmd_options='-q -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ServerAliveInterval=2 -o ServerAliveCountMax=5 -p 10046'
2. 修改完成后，使用sys_monitor.sh重启数据库集群。
[kes_v8r6c7b12@node1 ~]$ sys_monitor.sh restart
2023-03-03 14:19:38 Ready to stop all DB ...
Service process "node_export" was killed at process 9578
Service process "postgres_ex" was killed at process 9579
Service process "node_export" was killed at process 1344
Service process "postgres_ex" was killed at process 1345
2023-03-03 14:19:42 begin to stop repmgrd on "[192.168.10.40]".
2023-03-03 14:19:42 repmgrd on "[192.168.10.40]" stop success.
2023-03-03 14:19:42 begin to stop repmgrd on "[192.168.10.43]".
2023-03-03 14:19:43 repmgrd on "[192.168.10.43]" stop success.
2023-03-03 14:19:43 begin to stop DB on "[192.168.10.43]".
waiting for server to shut down.... done
server stopped
2023-03-03 14:19:43 DB on "[192.168.10.43]" stop success.
2023-03-03 14:19:43 begin to stop DB on "[192.168.10.40]".
waiting for server to shut down.... done
server stopped
2023-03-03 14:19:43 DB on "[192.168.10.40]" stop success.
2023-03-03 14:19:44 Done.
2023-03-03 14:19:44 Ready to start all DB ...
2023-03-03 14:19:44 begin to start DB on "[192.168.10.40]".
waiting for server to start.... done
server started
2023-03-03 14:19:44 execute to start DB on "[192.168.10.40]" success, connect to check it.
2023-03-03 14:19:45 DB on "[192.168.10.40]" start success.
2023-03-03 14:19:45 Try to ping trusted_servers on host 192.168.10.40 ...
2023-03-03 14:19:48 Try to ping trusted_servers on host 192.168.10.43 ...
2023-03-03 14:19:50 begin to start DB on "[192.168.10.43]".
waiting for server to start.... done
server started
2023-03-03 14:19:51 execute to start DB on "[192.168.10.43]" success, connect to check it.
2023-03-03 14:19:52 DB on "[192.168.10.43]" start success.
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | LSN_Lag | Connection string
----+-------+---------+-----------+----------+----------+----------+----------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------
1 | node1 | primary | * running | | default | 100 | 1 | | host=192.168.10.40 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
2 | node2 | standby | running | node1 | default | 100 | 1 | 0 bytes | host=192.168.10.43 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=3
2023-03-03 14:19:52 The primary DB is started.
2023-03-03 14:19:52 begin to start repmgrd on "[192.168.10.40]".
[2023-03-03 14:19:53] [NOTICE] using provided configuration file "/home/kes_v8r6c7b12/cluster/kingbase/etc/repmgr.conf"
[2023-03-03 14:19:53] [NOTICE] redirecting logging output to "/home/kes_v8r6c7b12/cluster/kingbase/log/hamgr.log"
2023-03-03 14:19:54 repmgrd on "[192.168.10.40]" start success.
2023-03-03 14:19:54 begin to start repmgrd on "[192.168.10.43]".
[2023-03-03 14:19:55] [NOTICE] using provided configuration file "/home/kes_v8r6c7b12/cluster/kingbase/etc/repmgr.conf"
[2023-03-03 14:19:55] [NOTICE] redirecting logging output to "/home/kes_v8r6c7b12/cluster/kingbase/log/hamgr.log"
2023-03-03 14:19:56 repmgrd on "[192.168.10.43]" start success.
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+-------+---------+--------------------
1 | node1 | primary | * running | | running | 10436 | no | n/a
2 | node2 | standby | running | node1 | running | 5932 | no | 1 second(s) ago
[2023-03-03 14:19:58] [NOTICE] redirecting logging output to "/home/kes_v8r6c7b12/cluster/kingbase/log/kbha.log"
[2023-03-03 14:20:00] [NOTICE] redirecting logging output to "/home/kes_v8r6c7b12/cluster/kingbase/log/kbha.log"
2023-03-03 14:20:01 Done.
[kes_v8r6c7b12@node1 ~]$
至此，集群通信切换完成。
五、验证是否切换成功：
使用KingbaseES备份进行验证，集群使用的通信服务：
执行以下命令，观察备份执行输出：
sh -x sys_backup.sh init
# 如果有以下内容输出说明集群通信服务切换成功
sys_securecmd -q -n -o ConnectTimeout=30 -o StrictHostKeyChecking=no -o PreferredAuthentications=publickey
KINGBASE研究院
