KingbaseES V8R3 集群运维案例 -- cluster.log无日志输出问题诊断
案例说明：
KingbaseES V8R3集群正常运行期间，现场发现cluster.log日志无任何信息输出，针对这一问题做了复现及提出解决方案。后现场检查发现，cluster.log文件曾被删除：
适用版本：
KingbaseES V8R3
一、查看集群的服务状态
1、集群节点状态
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | repli
cation_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+------
-------------
0 | 192.168.1.101 | 54321 | up | 0.500000 | primary | 0 | false | 0
1 | 192.168.1.102 | 54321 | up | 0.500000 | standby | 0 | true | 0
(2 rows)
2、流复制状态
TEST=# select * from sys_stat_replication;
PID | USESYSID | USENAME | APPLICATION_NAME | CLIENT_ADDR | CLIENT_HOSTNAME | CLIENT_PORT |
BACKEND_START | BACKEND_XMIN | STATE | SENT_LOCATION | WRITE_LOCATION | FLUSH_LOCATION | REP
LAY_LOCATION | SYNC_PRIORITY | SYNC_STATE
------+----------+---------+------------------+---------------+-----------------+-------------+---------
----------------------+--------------+-----------+---------------+----------------+----------------+----
-------------+---------------+------------
1366 | 10 | SYSTEM | node2 | 192.168.1.102 | | 38968 | 2023-04-
12 14:29:09.881587+08 | | streaming | 1/2E0001B0 | 1/2E0001B0 | 1/2E0001B0 | 1/2
E0001B0 | 0 | async
(1 row)
二、通过lsof查看进程访问的文件
1、查看kingbasecluster进程访问的日志文件
[root@node101 ~]# cd /home/kingbase/cluster/HAR3/db/bin
[root@node101 bin]# lsof -c kingbasecluster |grep cluster.log
kingbasec 1689 root 1w REG 253,2 29453500 34403567 /home/kingbase/cluster/HAR3/log/cluster.log
kingbasec 1689 root 2w REG 253,2 29453500 34403567 /home/kingbase/cluster/HAR3/log/cluster.log
kingbasec 1724 root 1w REG 253,2 29453500 34403567 /home/kingbase/cluster/HAR3/log/cluster.log
---如上所示，kingbasecluster服务启动后会访问cluster.log日志文件。
2、检查cluster.log对应的进程
[root@node101 bin]# lsof /home/kingbase/cluster/HAR3/log/cluster.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
kingbasec 1689 root 1w REG 253,2 29455673 34403567 /home/kingbase/cluster/HAR3/log/cluster.log
kingbasec 1689 root 2w REG 253,2 29455673 34403567 /home/kingbase/cluster/HAR3/log/cluster.log
kingbasec 1724 root 1w REG 253,2 29455673 34403567 /home/kingbase/cluster/HAR3/log/cluster.log
# 查看对应的进程
[root@node101 bin]# ps -ef |grep 1689
root 1689 1 0 14:29 ? 00:00:00 ./kingbasecluster -n
root 1724 1689 0 14:29 ? 00:00:00 kingbasecluster: watchdog
root 1788 1689 0 14:29 ? 00:00:00 kingbasecluster: lifecheck
root 1794 1689 0 14:29 ? 00:00:00 kingbasecluster: wait for connection request
root 1795 1689 0 14:29 ? 00:00:00 kingbasecluster: wait for connection request
root 1796 1689 0 14:29 ? 00:00:00 kingbasecluster: wait for connection request
---如上所示，cluster.log日志文件会被kingbasecluster访问。
三、模拟cluster.log文件被删除
1、将cluster.log文件改名
[root@node101 log]# ls -lh cluster*
-rw-r--r-- 1 root root 2.4M Apr 12 14:33 cluster_restart.log
-rw-r--r-- 1 root root 110 Apr 12 14:09 clusterstop
[root@node101 log]# mv cluster.log cluster.log.bk
[root@node101 log]# ls -lh cluster*
-rw-r--r-- 1 root root 29M Apr 12 14:33 cluster.log.bk
-rw-r--r-- 1 root root 2.4M Apr 12 14:33 cluster_restart.log
-rw-r--r-- 1 root root 110 Apr 12 14:09 clusterstop
2、通过lsof查看kingbasecluster进程访问的日志文件
# 如下所示，kingbasecluster输出日志文件为cluster.log.bk
[root@node101 log]# lsof -c kingbasecluster |grep cluster.log
kingbasec 1689 root 1w REG 253,2 29464573 34403567 /home/kingbase/cluster/HAR3/log/cluster.log.bk
kingbasec 1689 root 2w REG 253,2 29464573 34403567 /home/kingbase/cluster/HAR3/log/cluster.log.bk
kingbasec 1724 root 1w REG 253,2 29464573 34403567 /home/kingbase/cluster/HAR3/log/cluster.log.bk
# 删除cluster.log.bk文件
[root@node101 log]# rm cluster.log.bk
[root@node101 log]# lsof -c kingbasecluster |grep cluster.log
kingbasec 1689 root 1w REG 253,2 29467057 34403567 /home/kingbase/cluster/HAR3/log/cluster.log.bk (deleted)
kingbasec 1689 root 2w REG 253,2 29467057 34403567 /home/kingbase/cluster/HAR3/log/cluster.log.bk (deleted)
kingbasec 1724 root 1w REG 253,2 29467057 34403567 /home/kingbase/cluster/HAR3/log/cluster.log.bk (deleted)
# 手工创建cluster.log文件
[root@node101 log]# touch cluster.log
[root@node101 log]# lsof /home/kingbase/cluster/HAR3/log/cluster.log
---如上所示，手工创建cluster.log文件后，也没有被kingbasecluster进程访问。
如下图所示，删除cluster.log后，lsof标识文件被‘delete’：
四、测试failover切换
1、切换前集群节点状态
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | repli
cation_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+------
-------------
0 | 192.168.1.101 | 54321 | up | 0.500000 | primary | 0 | false | 0
1 | 192.168.1.102 | 54321 | up | 0.500000 | standby | 0 | true | 0
(2 rows)
2、模拟主库数据库服务down
[kingbase@node101 bin]$ ./sys_ctl stop -D ../data
waiting for server to shut down.... done
server stopped
3、查看切换后的新主库
TEST=# select sys_is_in_recovery();
SYS_IS_IN_RECOVERY
--------------------
f
(1 row)
4、切换后集群节点状态
TEST=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | repli
cation_delay
---------+---------------+-------+--------+-----------+---------+------------+-------------------+------
-------------
0 | 192.168.1.101 | 54321 | up | 0.500000 | standby | 0 | false | 0
1 | 192.168.1.102 | 54321 | up | 0.500000 | primary | 0 | true | 0
(2 rows)
---如上所示，对于cluster.log文件被误删除，不会影响到集群的切换。
五、恢复cluster.log日志输出
Tips：
如果需要kingbasecluster重新输出日志到cluster.log文件，需要重启kingbasecluster服务，可以通过root用户执行restartcluster.sh；但是必须先kill原来的kingbasecluster进程，然后再执行此脚本。这样可以在不影响数据库服务的情况下，重启集群服务。
1、查看kingbasecluster进程
[root@node101 ~]# ps -ef |grep kingbasecluster
root 1689 1 0 14:29 ? 00:00:00 ./kingbasecluster -n
.......
2、kill进程（注意：不用kill -9）
[root@node101 ~]# kill -2 1689
[root@node101 ~]# ps -ef |grep kingbasecluster
3、执行restartcluster.sh
[root@node101 ~]# /home/kingbase/cluster/HAR3/kingbasecluster/bin/restartcluster.sh
# 进程重启pid发生变化
[root@node101 ~]# ps -ef |grep kingbasecluster
root 14943 1 0 14:50 pts/0 00:00:00 ./kingbasecluster -n
.......
4、查看cluster.log日志输出
[root@node101 ~]# tail -f /home/kingbase/cluster/HAR3/log/cluster.log
2023-04-12 14:50:46: pid 14943: LOG: Setting up socket for :::9999
2023-04-12 14:50:46: pid 14943: LOG: kingbasecluster successfully started. version 3.6.7 (release)
2023-04-12 14:50:47: pid 14966: LOG: creating socket for sending heartbeat
2023-04-12 14:50:47: pid 14966: DETAIL: bind send socket to device: enp0s3
.......
六、总结
对于集群及数据库服务的日志文件，不要轻易通过手工方式删除文件，如果文件日益增大占用磁盘空间，可以通过Linux系统的logrotate工具切割日志文件并自动删除陈旧日志文件。
通过restartcluster.sh重启集群kingbasecluster服务，不影响集群的正常访问，但是会导致cluster vip漂移，对于生产环境，应该在业务访问低峰期间执行。
KINGBASE研究院
