rdb解析rdb文件

 

环境:
Os:Centos 7
Python:3.6.5

IP:192.168.1.134

 

1.安装rdbtools

[root@localhost ~]# pip3 install rdbtools
Collecting rdbtools
  Downloading https://files.pythonhosted.org/packages/dd/e5/eebb748863d74b7f9d433e359b874d97e830e9d1b3e3d876b91b5cba6b9d/rdbtools-0.1.15.tar.gz
Collecting redis (from rdbtools)
  Downloading https://files.pythonhosted.org/packages/91/f1/cbae04b7bdf1ae5f952dcf6babfe413647663e73fef8061a5d3c08b0b6f4/redis-4.0.0-py3-none-any.whl (118kB)
    100% |████████████████████████████████| 122kB 486kB/s 
Collecting deprecated (from redis->rdbtools)
  Downloading https://files.pythonhosted.org/packages/51/6a/c3a0408646408f7283b7bc550c30a32cc791181ec4618592eec13e066ce3/Deprecated-1.2.13-py2.py3-none-any.whl
Collecting wrapt<2,>=1.10 (from deprecated->redis->rdbtools)
  Downloading https://files.pythonhosted.org/packages/e2/0f/89c9c2d8ba06709a3d471507a78be443e2c2d9f1321d3e1154c76f44150c/wrapt-1.13.3-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (78kB)
    100% |████████████████████████████████| 81kB 1.6MB/s 
Installing collected packages: wrapt, deprecated, redis, rdbtools
  Running setup.py install for rdbtools ... done
Successfully installed deprecated-1.2.13 rdbtools-0.1.15 redis-4.0.0 wrapt-1.13.3
You are using pip version 9.0.3, however version 21.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

 

2.安装python-lzf
pip3 install python-lzf

3.使用

默认安装的路径

 

[root@localhost /]# find ./ -name rdb
./usr/local/services/Python-3.7.8/bin/rdb

 

可以将如上命令加入到环境变量

 

[root@localhost ~]# rdb -help

 

 

转换成json格式
rdb -c json /tmp/7001-dump.rdb>/tmp/7001dump.txt

内存中的表现形式

rdb -c memory /tmp/dump.rdb>/tmp/mem_dump.txt  

 

4.rdb文件比较差异
[root@localhost ~]# rdb --command diff /tmp/dump.rdb | sort > /tmp/dump1.txt
[root@localhost ~]# rdb --command diff /tmp/7001-dump.rdb| sort > /tmp/dump2.txt

运行差异程序
安装kdiff3(图形化工具)
[root@localhost tmp]#yum install kdiff3
[root@localhost tmp]# kdiff3 dump1.txt dump2.txt

 

命令行工具命令

[root@localhost tmp]# diff /tmp/dump1.txt /tmp/dump2.txt

 

5.导出内存字节排名前3的keys：
rdb -c memory --largest 3 /tmp/dump.rdb

6.导出字节大于1024000的key：
rdb --command memory --bytes 1024000 /tmp/dump.rdb

 

 

-------------------------------------rdb文件内容导入到表中分析------------------------------------------

数据库服务器:192.168.1.14(mysql 5.7)

 

1.创建表
CREATE TABLE `tb_redis_rdb_mem0831` (
`redis_database` int(128) DEFAULT NULL COMMENT 'key在redis的db',
`type` varchar(128) DEFAULT NULL COMMENT 'key类型',
`redis_key` varchar(1024) DEFAULT NULL COMMENT 'key值',
`size_in_bytes` bigint(20) DEFAULT NULL COMMENT 'key的内存大小(byte)',
`encoding` varchar(128) DEFAULT NULL COMMENT 'value的存储编码形式',
`num_elements` bigint(20) DEFAULT NULL COMMENT 'key中的value的个数',
`len_largest_element` varchar(128) DEFAULT NULL COMMENT 'key中的value的长度',
`expiry` varchar(32) DEFAULT NULL COMMENT 'key过期时间'
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='RDB数据分析临时表';

 

2.将rdb导出成csv文件
rdb -c memory /tmp/dump.rdb > /tmp/redis_memory_report.csv
需要将文件拷贝到mysql服务器并放在导出的目录下，注意修改权限(mysql)

cp /tmp/redis_memory_report.csv /opt/mysql57/secure_file/

 

说明：这里导出的数据是以逗号分隔的，若加载到数据库中若key带有逗号的话，解析会不对

 

3.加载到mysql表中(无密码)

/opt/mysql57/bin/mysql -h localhost -uroot -pmysql -P13306 -S /opt/mysql57/data/mysql.sock
use db_test;

 

LOAD DATA INFILE '/opt/mysql57/secure_file/7003_memory_report.csv'
ignore INTO TABLE tb_redis_rdb_mem0831
character set gbk
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
IGNORE 1 ROWS
(`redis_database`,`type`,`redis_key`,`size_in_bytes`,`encoding`,`num_elements`,`len_largest_element`,`expiry`);