ClickHouse集群+ZooKeeper集群架构下的ZooKeeper集群迁移
ClickHouse集群+ZooKeeper集群架构下的ZooKeeper集群迁移
目前测试环境的ck集群+zk集群架构如下:
ClickHouse | ZooKeeper |
192.168.1.171 192.168.1.172 192.168.1.173 192.168.1.174 |
192.168.1.171 192.168.1.172 192.168.1.173 |
由于ck和zk混搭在一块,存在较为严重的性能问题,计划将zk从171,172,173迁移到34,35,36上。
通过向现有的zk集群中添加成员34,35,36变为6个节点的集群后,在剔除171,172,173这三个成员达到迁移目的,即扩容然后缩容。
最终结构如下:
ClickHouse | ZooKeeper |
192.168.1.171 192.168.1.172 192.168.1.173 192.168.1.174 |
192.168.1.34 192.168.1.35 192.168.1.36 |
启用动态配置功能
由于目前zk(171,172,173)并没有启动动态配置功能,需要停机修改配置。
先停止ck全部节点,在停止zk全部节点。(略)
修改171,172,173的zk配置:
vi /usr/local/zookeeper/conf/zoo.cfg 删除server列表配置: server.1=192.168.1.171:2888:3888 server.2=192.168.1.172:2888:3888 server.3=192.168.1.173:2888:3888 追加: skipACL=yes reconfigEnabled=true dynamicConfigFile=/usr/local/zookeeper/conf/zoo.cfg.dynamic
添加文件zoo.cfg.dynamic,指定原来的server列表节点:
vi /usr/local/zookeeper/conf/zoo.cfg.dynamic server.1=192.168.1.171:2888:3888:participant;2181 server.2=192.168.1.172:2888:3888:participant;2181 server.3=192.168.1.173:2888:3888:participant;2181
启动171,172,173的zk,然后启动ck集群。(略)
安装配置zk服务器
按照企业规范对新的34,35,36安装配置zk软件,但先不启动zk服务(略)。
特别需要注意:ck集群(171-174),旧zk集群(171-173),新zk集群(34-36)的/etc/hosts要对应有全部节点的解析。
cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.171 dev-app76 192.168.1.172 dev-app77 192.168.1.173 dev-app78 192.168.1.174 dev-app79 192.168.1.34 zk01 192.168.1.35 zk02 192.168.1.36 zk03
zk集群扩容
添加新节点zk01(34)
vi /usr/local/zookeeper/conf/zoo.cfg.dynamic server.1=192.168.1.171:2888:3888:participant;2181 server.2=192.168.1.172:2888:3888:participant;2181 server.3=192.168.1.173:2888:3888:participant;2181 server.4=192.168.1.34:2888:3888:participant;2181 启动zk01: zkServer.sh start 或者:systemctl start zookeeper
旧的zk服务器171,172,173随便挑选一台执行。
[zk: localhost:2181(CONNECTED) 0] reconfig -add 4=192.168.1.34:2888:3888;2181 Committed new configuration: server.1=192.168.1.171:2888:3888:participant;0.0.0.0:2181 server.2=192.168.1.172:2888:3888:participant;0.0.0.0:2181 server.3=192.168.1.173:2888:3888:participant;0.0.0.0:2181 server.4=192.168.1.34:2888:3888:participant;0.0.0.0:2181 version=630000e4be
检查各个节点zk状态。
echo conf | nc zk01 2181 | grep server. echo conf | nc 192.168.1.171 2181 | grep server. echo conf | nc 192.168.1.172 2181 | grep server. echo conf | nc 192.168.1.173 2181 | grep server.
样例:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
[root@zk01 conf]# echo conf | nc zk01 2181 | grep server. serverId=4 server.4=192.168.1.34:2888:3888:participant;0.0.0.0:2181 server.5=192.168.1.35:2888:3888:participant;0.0.0.0:2181 server.6=192.168.1.36:2888:3888:participant;0.0.0.0:2181
检查ck读写是否异常。
参考语句:(关于gv$parts,参考clickhouse集群跨节点查询其他节点系统表信息)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
CREATE TABLE default.zkm on cluster ceb_cluster ( `id` String DEFAULT 'NULL' COMMENT '用户编号', `repo` String DEFAULT 'NULL' COMMENT '仓库编号' ) ENGINE = Distributed('ceb_cluster', 'default', 'zkm_local', rand()); CREATE TABLE default.zkm_local on cluster ceb_cluster ( `id` String DEFAULT 'NULL' COMMENT '用户编号', `repo` String DEFAULT 'NULL' COMMENT '仓库编号' ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/ceb_cluster-{shard}/default/zkm_local', '{replica}') ORDER BY id SETTINGS index_granularity = 8192; insert into zkm values('a','b'); insert into zkm select * from zkm settings insert_deduplicate=0; --这条多次执行 SELECT host AS `主机名`, sum(rows) AS `总行数`, formatReadableSize(sum(data_uncompressed_bytes)) AS `原始大小`, formatReadableSize(sum(data_compressed_bytes)) AS `压缩大小`, round((sum(data_compressed_bytes) / sum(data_uncompressed_bytes)) * 100, 0) AS `压缩率` FROM `gv$parts` WHERE (database = 'default') AND (table = 'zkm_local') AND (active = 1) AND (engine = 'ReplicatedMergeTree') GROUP BY host ORDER BY host ASC;
添加新节点zk02(35)
同上诉添加zk01。略。
添加新节点zk03(36)
同上诉添加zk01。略。
修改ck服务器zk配置
将ck作为客户端连接的原zk(171,172,173)替换为新的zk节点(34,35,36)。
所有的ck(171,172,173,174)节点:
vi /etc/clickhouse-server/config.d/zks.xml <?xml version="1.0"?> <clickhouse> <zookeeper> <node index="1"> <host>zk01</host> <port>2181</port> </node> <node index="2"> <host>zk02</host> <port>2181</port> </node> <node index="3"> <host>zk03</host> <port>2181</port> </node> </zookeeper> </clickhouse>
服务器跟踪配置文件中的更改,以及执行替换和复盖时使用的文件和ZooKeeper节点,并动态重新加载用户和集群的设置。 这意味着您可以在不重新启动服务器的情况下修改群集、用户及其设置。
确认新配置信息是否实时生效(在ck的数据文件所在根目录):
cat /clickhouse/preprocessed_configs/config.xml
zk集群缩容
从zk服务器34,35,36随便挑选一台执行。
zkCli.sh进去交互式: reconfig -remove 1 reconfig -remove 2 reconfig -remove 3
检查各个节点zk状态。
echo conf | nc zk01 2181 | grep server. echo conf | nc zk02 2181 | grep server. echo conf | nc zk03 2181 | grep server. echo conf | nc 192.168.1.171 2181 | grep server. echo conf | nc 192.168.1.172 2181 | grep server. echo conf | nc 192.168.1.173 2181 | grep server.
关闭旧的zk节点(171,172,173):
zkServer.sh stop
检查各个节点zk状态。
echo conf | nc zk01 2181 | grep server. echo conf | nc zk02 2181 | grep server. echo conf | nc zk03 2181 | grep server.
检查ck是否有会话连接到zk:
echo cons | nc zk01 2181 echo cons | nc zk02 2181 echo cons | nc zk03 2181 或者 echo srvr | nc zk01 2181 echo srvr | nc zk02 2181 echo srvr | nc zk03 2181 最终如下: [root@zk03 clickhouse]# echo conf | nc zk01 2181 | grep server. serverId=4 server.4=192.168.1.34:2888:3888:participant;0.0.0.0:2181 server.5=192.168.1.35:2888:3888:participant;0.0.0.0:2181 server.6=192.168.1.36:2888:3888:participant;0.0.0.0:2181
卸载旧的集群zk节点171,172,173:
绿色安装的zk,按照安装文档反向操作进行卸载即可。
略。
参考文档:
https://www.cnblogs.com/taiyang-li/p/14594208.html
https://zookeeper.apache.org/doc/r3.7.0/zookeeperReconfig.html
https://clickhouse.com/docs/zh/operations/configuration-files#implementation-details