redis数据库迁移工具类 redis-shake 使用速查

　　由于各种原因，我们都可能不会一直使用同一个redis实例，从而需要在不定的时候进行切换。

　　切换数据本身应该只是一个更改链接的过程而已，但进行数据源切换的唯一要点就是，原有数据如何平滑迁移到新实例，从而可以实现无缝迁移！

网上进行redis数据库迁移的方式罗列:

　　　　1. redis-dump,
　　　　2. 基于 slave 方式的 rdb 备份恢复;
　　　　3. 自己写的python/shell脚本进行全量数据操作;
　　　　4. 其他...

　　总体来说，都可行，但是自己可能遇到各种问题。略去不说。

　　今天，我们就来看另一个redis数据库迁移工具: redis-shake.

　　redis-shake是阿里云自研的开源工具,可以顺利用于进行redis数据迁移！

　　官方文档藏得比较深: https://tech.antfin.com/docs/2/117311

　　github项目地址: https://github.com/alibaba/RedisShake

　　应该说，看了官方说明文档，基本就是依葫芦画瓢了。

本也没啥好说！就检需要的说吧！

步骤罗列:

1. 下载安装包,这个可以github的release页面找到:

wget https://github.com/alibaba/RedisShake/releases/download/release-v1.6.6-20190609/redis-shake.tar.gz

2. 解压文件:

tar -xvf redis-shake.tar.gz

3. 编辑 redis-shake.conf 配置文件, 主要是更改 source, target 选项:

vim redis-shake.conf    # 注意按照普通的redis连接来更改即可;

4. 运行 redis-shake 同步脚本:

./redis-shake.linux64 -type=rump -conf=redis-shake.conf

 

其他说明:

　　1. 同步过程可能出现异常，有可能是因为数据正在被写入，在同步时出了问题。解决办法是：没关系，再次同步就好了！
　　2. 针对大数据量的同步，还得自己注意下！

     

redis-shake.conf 文件如下：

# this is the configuration of redis-shake.
# if you have any problem, please visit https://github.com/alibaba/RedisShake/wiki/FAQ

# id
id = redis-shake

# log file，日志文件，不配置将打印到stdout (e.g. /var/log/redis-shake.log )
log.file =
# log level: "none", "error", "warn", "info", "all". default is "info".
log.level = info
# pid path，进程文件存储地址（e.g. /var/run/)，不配置将默认输出到执行下面,
# 注意这个是目录，真正的pid是`{pid_path}/{id}.pid`
pid_path = 

# pprof port
system_profile = 9310
# restful port，查看metric端口
http_profile = 9320

# runtime.GOMAXPROCS, 0 means use cpu core number: runtime.NumCPU()
ncpu = 0

# parallel routines number used in RDB file syncing. default is 64.
parallel = 32

# source redis configuration.
# used in `dump`, `sync` and `rump`.
# source redis type, e.g. "standalone" (default), "sentinel" or "cluster".
#   1. "standalone": standalone db mode.
#   2. "sentinel": the redis address is read from sentinel.
#   3. "cluster": the source redis has several db.
#   4. "proxy": the proxy address, currently, only used in "rump" mode.
# 源端redis的类型，支持standalone，sentinel，cluster和proxy四种模式，注意：目前proxy只用于rump模式。
source.type = standalone
# ip:port
# the source address can be the following:
#   1. single db address. for "standalone" type.
#   2. ${sentinel_master_name}:${master or slave}@sentinel single/cluster address, e.g., mymaster:master@127.0.0.1:26379;127.0.0.1:26380. for "sentinel" type.
#   3. cluster that has several db nodes split by semicolon(;). for "cluster" type. e.g., 10.1.1.1:20331;10.1.1.2:20441.
#   4. proxy address(used in "rump" mode only). for "proxy" type.
# 源redis地址。对于sentinel模式，输入格式为"master名字:拉取角色为master或者slave@sentinel的地址"
source.address = r-a.redis.rds.aliyuncs.com:6379
# password.
source.password_raw = 123456
# auth type, don't modify it
source.auth_type = auth
# tls enable, true or false. Currently, only support standalone.
# open source redis does NOT support tls so far, but some cloud versions do.
source.tls_enable = false

# target redis configuration. used in `restore`, `sync` and `rump`.
# the type of target redis can be "standalone", "proxy" or "cluster".
#   1. "standalone": standalone db mode.
#   2. "sentinel": the redis address is read from sentinel.
#   3. "cluster": open source cluster (not supported currently).
#   4. "proxy": proxy layer ahead redis. Data will be inserted in a round-robin way if more than 1 proxy given.
# 目的redis的类型，支持standalone，sentinel，cluster和proxy四种模式。
target.type = standalone
# ip:port
# the target address can be the following:
#   1. single db address. for "standalone" type.
#   2. sentinel_master_name@sentinel single/cluster address, e.g., mymaster@127.0.0.1:26379;127.0.0.1:26380. for "sentinel" type.
#   3. cluster that has several db nodes split by semicolon(;). for "cluster" type.
#   4. proxy address(used in "rump" mode only). for "proxy" type.
target.address = r-b.redis.rds.aliyuncs.com:6379
# password.
target.password_raw = 123456
# auth type, don't modify it
target.auth_type = auth
# all the data will be written into this db. < 0 means disable.
target.db = -1
# tls enable, true or false. Currently, only support standalone.
# open source redis does NOT support tls so far, but some cloud versions do.
target.tls_enable = false

# input RDB file.
# used in `decode` and `restore`.
# if the input is list split by semicolon(;), redis-shake will restore the list one by one.
# 如果是decode或者restore，这个参数表示读取的rdb文件。支持输入列表，例如：rdb.0;rdb.1;rdb.2
# redis-shake将会挨个进行恢复。
rdb.input = local
# output RDB file prefix.
# used in `decode` and `dump`.
# 如果是decode或者dump，这个参数表示输出的rdb前缀，比如输入有3个db，那么dump分别是:
# ${output_rdb}.0, ${output_rdb}.1, ${output_rdb}.2
rdb.output = local_dump
# the concurrence of fetching data, default is len(source.address) or len(rdb.input).
# used in `dump`, `sync` and `restore`. 0 means default.
# 拉取的并发度，如果是`dump`或者`sync`，默认是source.address中db的个数，`restore`模式默认len(rdb.input)。
# 假如db节点/输入的rdb有5个，但rdb.parallel=3，那么一次只会
# 并发拉取3个db的全量数据，直到某个db的rdb拉取完毕，才会拉取第4个db节点的rdb，以此类推。
rdb.parallel = 0
# for special cloud vendor: ucloud
# ucloud集群版的rdb文件添加了slot前缀，进行特判剥离: ucloud_cluster。
rdb.special_cloud = 

# use for expire key, set the time gap when source and target timestamp are not the same.
# 用于处理过期的键值，当迁移两端不一致的时候，目的端需要加上这个值
fake_time =

# force rewrite when destination restore has the key
# used in `restore`, `sync` and `rump`.
# 当源目的有重复key，是否进行覆写
rewrite = true

# filter db or key or slot
# choose these db, e.g., 5, only choose db5. defalut is all.
# used in `restore` and `sync`.
# 支持过滤db，只让指定的db通过
filter.db =
# filter key with prefix string. multiple keys are separated by ';'.
# e.g., a;b;c
# default is all.
# used in `restore` and `sync`.
# 支持过滤key，只让指定的key通过，分号分隔
filter.key =
# filter given slot, multiple slots are separated by ';'.
# e.g., 1;2;3
# used in `sync`.
# 指定过滤slot，只让指定的slot通过
filter.slot =

# big key threshold, the default is 500 * 1024 * 1024 bytes. If the value is bigger than
# this given value, all the field will be spilt and write into the target in order.
# 正常key如果不大，那么都是直接调用restore写入到目的端，如果key对应的value字节超过了给定
# 的值，那么会分批依次一个一个写入。
big_key_threshold = 524288000

# use psync command.
# used in `sync`.
# 默认使用sync命令，启用将会使用psync命令
psync = false

# enable metric
# used in `sync`.
# 是否启用metric
metric = true
# print in log
# 是否将metric打印到log中
metric.print_log = false

# heartbeat
# send heartbeat to this url
# used in `sync`.
# 心跳的url地址，redis-shake将会发送到这个地址
#heartbeat.url = http://127.0.0.1:8000
heartbeat.url =
# interval by seconds
# 心跳保活周期
heartbeat.interval = 3
# external info which will be included in heartbeat data.
# 在心跳报文中添加额外的信息
heartbeat.external = test external
# local network card to get ip address, e.g., "lo", "eth0", "en0"
# 获取ip的网卡
heartbeat.network_interface =

# sender information.
# sender flush buffer size of byte.
# used in `sync`.
# 发送缓存的字节长度，超过这个阈值将会强行刷缓存发送
sender.size = 104857600
# sender flush buffer size of oplog number.
# used in `sync`.
# 发送缓存的报文个数，超过这个阈值将会强行刷缓存发送
sender.count = 5000
# delay channel size. once one oplog is sent to target redis, the oplog id and timestamp will also
# stored in this delay queue. this timestamp will be used to calculate the time delay when receiving
# ack from target redis.
# used in `sync`.
# 用于metric统计时延的队列
sender.delay_channel_size = 65535

# enable keep_alive option in TCP when connecting redis.
# the unit is second.
# 0 means disable.
# TCP keep-alive保活参数，单位秒，0表示不启用。
keep_alive = 0

# used in `rump`.
# number of keys captured each time. default is 100.
# 每次scan的个数，不配置则默认100.
scan.key_number = 50

# used in `rump`.
# we support some special redis types that don't use default `scan` command like alibaba cloud and tencent cloud.
# 有些版本具有特殊的格式，与普通的scan命令有所不同，我们进行了特殊的适配。目前支持腾讯云的集群版"tencent_cluster"
# 和阿里云的集群版"aliyun_cluster"。
scan.special_cloud =

# used in `rump`.
# we support to fetching data from given file which marks the key list.
# 有些云版本，既不支持sync/psync，也不支持scan，我们支持从文件中进行读取所有key列表并进行抓取：一行一个key。
scan.key_file =

# ----------------splitter----------------
# below variables are useless for current open source version so don't set.

# replace hash tag.
# used in `sync`.
replace_hash_tag = false

# used in `restore` and `dump`.
extra = false