使用codis-admin搭建codis集群
目的
在Redis Codis 部署安装的文章中,介绍了通过fe在web上搭建codis的基本步骤和方法,也介绍了codis-admin的相关说明,为了更好的熟悉codis-admin的使用,本文将使用codis-admin直接搭建codis集群(和fe进行相关的对比)。这样做的另一个目的是为实现自动化脚本部署的时候做相关的准备。
环境
和Redis Codis 部署安装中的环境一样,包括各个服务的IP和端口,以及安装方法。
机器 |
服务 |
端口 |
端口说明 |
依赖 |
192.168.163.131/132/133(Ubuntu 16.04) |
Codis |
7021/7022 |
server端口:主/从(三台) |
GO |
11080 |
proxy管理端口(三台) |
|||
18080 |
dashboard管理端口(一台) |
|||
10890 |
fe管理端口(一台) |
|||
10086 |
sentinel(三台) |
|||
192.168.163.131/132/133(Ubuntu 16.04) |
zookeeper |
2181 |
zk客户端监听端口(三台) |
JDK |
2888 |
zk内部通讯端口(三台) |
|||
3888 |
zk选举端口(三台) |
开始
首先要保证ZooKeeper、Dashboard、Proxy、Server、Sentine、fe等都已经开启。根据Redis Codis 部署安装文章的安装和配置,进行codis相关组件的开启、关闭(ZK通过上篇文文章说明):
开启关闭进程
1)codis-server
开启(三台)
codis-server /etc/codis/codis-server/redis7021.conf codis-server /etc/codis/codis-server/redis7022.conf
关闭(三台)
进入到redis里面进行shutdown或则kill。
2)dashboard
开启(一台)
codis-dashboard --ncpu=1 --config=/etc/codis/codis-dashboard/dashboard.toml --log=/usr/local/codis/logs/dashboard.log --log-level=warn &
关闭(一台)
codis-admin --dashboard=192.168.163.131:18080 --shutdown
3)codis-fe
开启(一台)
cd /etc/codis/codis-fe/ codis-admin --dashboard-list --zookeeper=192.168.163.131:2181 | tee codis.json codis-fe --ncpu=1 --dashboard-list=/etc/codis/codis-fe/codis.json --listen=192.168.163.131:18090 --log=/usr/local/codis/logs/fe.log --log-level=warn --assets-dir=/usr/local/codis/bin/assets/ &
关闭(一台)
ps -ef|grep codis-fe|grep -v grep|awk '{print $2}'|xargs kill
4)Proxy
开启(三台)
codis-proxy --ncpu=1 --config=/etc/codis/codis-proxy/proxy.toml --log=/usr/local/codis/logs/proxy.log --log-level=warn &
关闭(三台)
codis-admin --proxy=192.168.163.131:11080 --shutdown codis-admin --proxy=192.168.163.132:11080 --shutdown codis-admin --proxy=192.168.163.133:11080 --shutdown
codis-admin应用
1)Proxy
① 添加Proxy到Dashboard,并且设置成Online(3个)。
Proxy进程起来之后一直处于proxy waiting online ...状态,不会接受请求(该状态不会注册到zk的codis3和jodis里)。先加入到Dashboard、注册到zk(jodis、codis3)并设置成online:
codis-admin --dashboard=192.168.163.131:18080 --create-proxy --addr=192.168.163.131:11080 codis-admin --dashboard=192.168.163.132:18080 --create-proxy --addr=192.168.163.131:11080 codis-admin --dashboard=192.168.163.133:18080 --create-proxy --addr=192.168.163.131:11080
等于codis-fe的操作:
执行完毕之后,api会调用:
[WARN] [0xc4202bf440] API call /api/topom/proxy/create/4c0ca749efb5aad2b20b8d84b1bb6905/192.168.163.133:11080 from 192.168.163.131:44504 [192.168.163.1] [WARN] create proxy-[cfa02d4002da74e6b1f3b51f1416aa1d]
可以在日志里看到 proxy is working ...
②:从Dashboard上删除掉Proxy。
codis-admin --dashboard=192.168.163.131:18080 --remove-proxy --addr=192.168.163.133:11080
等于codis-fe的操作:
执行完毕之后,api会调用:
[WARN] [0xc4202bf440] API call /api/topom/proxy/remove/4c0ca749efb5aad2b20b8d84b1bb6905/cfa02d4002da74e6b1f3b51f1416aa1d/0 from 192.168.163.132:42134 [] [WARN] remove proxy-[cfa02d4002da74e6b1f3b51f1416aa1d]
需要注意的是,执行该操作之后,不仅从DashBoard退出,而且也会把进程退出掉,后续要开启才能加入。
③:重新同步所有的slots,当Proxy的slots出现异常的时候执行。
codis-admin --dashboard=192.168.163.131:18080 --reinit-proxy --addr=192.168.163.131:11080
等于codis-fe的操作:
执行完毕之后,api会调用:
[WARN] [0xc4202bf440] API call /api/topom/proxy/reinit/4c0ca749efb5aad2b20b8d84b1bb6905/4f07158ae347d67b1af825fd8e84b2f9 from 192.168.163.131:44612 [192.168.163.1] [WARN] proxy-[4f07158ae347d67b1af825fd8e84b2f9] reinit:
2)Group
① 添加Group到Dashboard(3个)
codis-admin --dashboard=192.168.163.131:18080 --create-group --gid=1 codis-admin --dashboard=192.168.163.131:18080 --create-group --gid=2 codis-admin --dashboard=192.168.163.131:18080 --create-group --gid=3
等于codis-fe的操作:
执行完毕之后,api会调用:
[WARN] [0xc4202bf440] API call /api/topom/group/create/4c0ca749efb5aad2b20b8d84b1bb6905/1 from 192.168.163.132:42138 [] [WARN] create group-[1]: { "id": 1, "servers": [], "promoting": {}, "out_of_sync": false } ...
② 删除Group
codis-admin --dashboard=192.168.163.131:18080 --remove-group --gid=4
等于codis-fe的操作:
执行完毕之后,api会调用:
[WARN] [0xc4202bf440] API call /api/topom/group/remove/4c0ca749efb5aad2b20b8d84b1bb6905/4 from 192.168.163.132:42146 [] [WARN] remove group-[4]: { "id": 4, "servers": [], "promoting": {}, "out_of_sync": false }
③ 添加Server到Group
#group1 codis-admin --dashboard=192.168.163.131:18080 --group-add --gid=1 --addr=192.168.163.131:7021 codis-admin --dashboard=192.168.163.131:18080 --group-add --gid=1 --addr=192.168.163.132:7022 #group2 ... #group3 ...
等于codis-fe的操作:
执行完之后,api会调用:
[WARN] [0xc4202bf440] API call /api/topom/group/add/4c0ca749efb5aad2b20b8d84b1bb6905/1/192.168.163.132:7022 from 192.168.163.132:42156 [] [WARN] update group-[1]: { "id": 1, "servers": [ { "server": "192.168.163.131:7021", "datacenter": "", "action": {}, "replica_group": false }, { "server": "192.168.163.132:7022", "datacenter": "", "action": {}, "replica_group": false } ], "promoting": {}, "out_of_sync": false }
...
③ 移除Group中的Server
codis-admin --dashboard=192.168.163.131:18080 --group-del --gid=1 --addr=192.168.163.132:7022
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/group/del/4c0ca749efb5aad2b20b8d84b1bb6905/1/192.168.163.132:7022 from 192.168.163.132:42158 [] [WARN] update group-[1]: { "id": 1, "servers": [ { "server": "192.168.163.131:7021", "datacenter": "", "action": {}, "replica_group": false } ], "promoting": {}, "out_of_sync": false }
④ 给Group添加Server
需要注意,做高可用的话每个Group里需要至少2个Server,一主一从。默认第2个Server是从。
codis-admin --dashboard=192.168.163.131:18080 --sync-action --create --addr=192.168.163.132:7022 --第2个server
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/group/action/create/4c0ca749efb5aad2b20b8d84b1bb6905/192.168.163.132:7022 from 192.168.163.132:42172 [] [WARN] update group-[1] ... [WARN] server-[192.168.163.132:7022] action prepare [WARN] update group-[1] ... [WARN] sync-[192.168.163.132:7022] process action [WARN] server-[192.168.163.132:7022] action failed = false [WARN] update group-[1]: { "id": 1, "servers": [ { "server": "192.168.163.131:7021", "datacenter": "", "action": {}, "replica_group": false }, { "server": "192.168.163.132:7022", "datacenter": "", "action": { "state": "synced" }, "replica_group": false } ], "promoting": {}, "out_of_sync": false }
⑤ 提升Group的一个Slave为Master
codis-admin --dashboard=192.168.163.131:18080 --promote-server --gid=3 --addr=192.168.163.133:7021
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/group/promote/4c0ca749efb5aad2b20b8d84b1bb6905/3/192.168.163.133:7021 from 192.168.163.132:42190 [] [WARN] group-[3] will promote index = 1 [WARN] update group-[3] "state": "preparing" ... [WARN] group-[3] resync to prepared [WARN] update group-[3]: "state": "prepared" ... [WARN] update group-[3]: "state": "finished" ... [WARN] group-[3] resync to finished [WARN] update group-[3]: { "id": 3, "servers": [ { "server": "192.168.163.133:7021", "datacenter": "", "action": {}, "replica_group": false }, { "server": "192.168.163.131:7022", "datacenter": "", "action": {}, "replica_group": false } ], "promoting": {}, "out_of_sync": false }
这里还要注意,Slave提升成Master之后,老Master还需要执行slaveof才能进行新的复制。
codis-admin --dashboard=192.168.163.131:18080 --sync-action --create --addr=192.168.163.131:7022
3)Slots
注意:此时,所有的slots都处在offline状态,需要进行分配slots。否则客户端会报错:
ERR handle request, slot is not ready, may be offline
① 分配slots,离开offline状态。迁移某一段范围的slots到指定的group,分配所有的slots到Group中。
codis-admin --dashboard=192.168.163.131:18080 --slot-action --create-range --beg=1 --end=100 --gid=1 codis-admin --dashboard=192.168.163.131:18080 --slot-action --create-range --beg=101 --end=1000 --gid=2 codis-admin --dashboard=192.168.163.131:18080 --slot-action --create-range --beg=1001 --end=1023 --gid=3
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/slots/action/create-range/4c0ca749efb5aad2b20b8d84b1bb6905/0/0/1 from 192.168.163.131:45936 [192.168.163.1] [WARN] update slot-[0]: "state": "pending", ... [WARN] slot-[0] action prepare: "state": "pending", [WARN] update slot-[0]: "state": "preparing", ... [WARN] slot-[0] resync to prepared "state": "prepared", ... [WARN] slot-[0] resync to migrating "state": "migrating", ... [WARN] slot-[0] process action [WARN] slot-[0] action complete "state": "migrating", ... [WARN] slot-[0] resync to finished "state": "finished", ...
注意:正在迁移的slots中的key被访问,该key会被立马迁移到新的group,才能被访问。迁移好之后的slots通过--slots-status查看,发生了变化:
{ "id": 0, "backend_addr": "192.168.163.131:7021", --迁移后新增 "backend_addr_group_id": 1, --迁移后新增 "forward_method": 1 },
② 迁移slots,迁移指定数量的slots从一个Group到另一个Group。
从Group2中迁移100个slots到Group3中:
codis-admin --dashboard=192.168.163.131:18080 --slot-action --create-some --gid-from=2 --gid-to=3 --num-slots=100
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/slots/action/create-some/4c0ca749efb5aad2b20b8d84b1bb6905/2/3/100 from 192.168.163.132:42498 [] "state": "pending", ... [WARN] slot-[104] action prepare: "state": "pending", ... "state": "preparing", [WARN] slot-[104] resync to prepared "state": "prepared", [WARN] slot-[104] resync to migrating "state": "migrating", [WARN] slot-[104] resync to finished "state": "finished", ...
注意,勾选Action Status可以查看迁移的进度。
③ 停止、开启Slots迁移。
停止:
codis-admin --dashboard=192.168.163.131:18080 --slot-action --disabled=1
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/slots/action/disabled/4c0ca749efb5aad2b20b8d84b1bb6905/1 from 192.168.163.132:42524 [] [WARN] set action disabled = true
开启:
codis-admin --dashboard=192.168.163.131:18080 --slot-action --disabled=0
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/slots/action/disabled/4c0ca749efb5aad2b20b8d84b1bb6905/0 from 192.168.163.132:42526 [] [WARN] set action disabled = false
④ 平均分配slots,各个Group平均分配1024个slots。
codis-admin --dashboard=192.168.163.131:18080 --rebalance --confirm
--confirm 表示执行rebalance,不加表示查看(不执行)
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/slots/rebalance/4c0ca749efb5aad2b20b8d84b1bb6905/1 from 192.168.163.132:42534 [] ... [WARN] slot-[882] resync to finished ...
⑤ slots迁移的时间间隔
codis-admin --dashboard=192.168.163.131:18080 --slot-action --interval=1000
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/slots/action/interval/4c0ca749efb5aad2b20b8d84b1bb6905/1000 from 192.168.163.132:42540 [] [WARN] set action interval = 1000
4)Sentinel 高可用
① 添加Sentinel(3个)
codis-admin --dashboard=192.168.163.131:18080 --sentinel-add --addr=192.168.163.131:10086 codis-admin --dashboard=192.168.163.131:18080 --sentinel-add --addr=192.168.163.132:10086 codis-admin --dashboard=192.168.163.131:18080 --sentinel-add --addr=192.168.163.133:10086
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/sentinels/add/4c0ca749efb5aad2b20b8d84b1bb6905/192.168.163.131:10086 from 192.168.163.132:42544 [] [WARN] update sentinel: { "servers": [ "192.168.163.131:10086" ], "out_of_sync": true } ...
为了保证Sentinel服务的正常,需要再resync下sentinel:
codis-admin --dashboard=192.168.163.131:18080 --sentinel-resync
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/sentinels/resync-all/4c0ca749efb5aad2b20b8d84b1bb6905 from 192.168.163.132:42550 [] [WARN] rewatch sentinels = [192.168.163.131:10086 192.168.163.132:10086 192.168.163.133:10086]
② 移除Sentinel
codis-admin --dashboard=192.168.163.131:18080 --sentinel-del --addr=192.168.163.133:10086
等于codis-fe的操作:
执行时,调用的api:
[WARN] [0xc4202bf440] API call /api/topom/sentinels/del/4c0ca749efb5aad2b20b8d84b1bb6905/192.168.163.133:10086/0 from 192.168.163.132:42598 []
到此,通过codis-admin已经完成了Codis的搭建。
总结
通过本文的说明,进一步理顺了codis-admin的各个命令。可以不依赖fe的web界面进行Codis集群的搭建,可以为自动化的一些脚本做好更好的支持工作。
posted on 2019-08-14 14:13 ExplorerMan 阅读(418) 评论(0) 编辑 收藏 举报