【redis】关于查询和分析redis中的bigkeys问题

一、场景

    今年的实际业务中，出现了一次redis bigkeys导致的生产事故，导致业务1个小时收不到指令。

 

二、关于bigkeys查询方法

1、使用redis-cli加--bigkeys参数

$ redis-cli -h 192.168.3.74 -p 6379 --bigkeys

 

$ redis-cli -h 192.168.3.74 -p 6379 --bigkeys

# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).

[00.00%] Biggest string found so far '"command-work-device-info:IGV807"' with 37 bytes
[00.00%] Biggest string found so far '"factor:vms_crane:CR40"' with 124 bytes
[00.00%] Biggest string found so far '"factor:crane:CR16"' with 375 bytes
[00.00%] Biggest string found so far '"factor:vehicle_instr:IGV820"' with 470 bytes
[05.18%] Biggest hash   found so far '"remote-gantry-info"' with 173 fields
[05.18%] Biggest string found so far '"factor:vehicle_instr_queue:IGV819"' with 6795 bytes
[06.91%] Biggest set    found so far '"online-truck-no-set"' with 1 members
[12.09%] Biggest string found so far '"pscp_route_info_T811"' with 89885 bytes
[42.31%] Biggest set    found so far '"truck-white-list:Node1-1"' with 197 members
[44.21%] Biggest list   found so far '"factor:command_parse_result:IGV841"' with 2 items
[45.94%] Biggest hash   found so far '"truck-info"' with 187 fields
[45.94%] Biggest list   found so far '"run-event-alarms"' with 12 items

-------- summary -------

Sampled 579 keys in the keyspace!
Total key length in bytes is 13082 (avg len 22.59)

Biggest   list found '"run-event-alarms"' has 12 items
Biggest   hash found '"truck-info"' has 187 fields
Biggest string found '"pscp_route_info_T811"' has 89885 bytes
Biggest    set found '"truck-white-list:Node1-1"' has 197 members

2 lists with 14 items (00.35% of keys, avg size 7.00)
8 hashs with 784 fields (01.38% of keys, avg size 98.00)
565 strings with 388320 bytes (97.58% of keys, avg size 687.29)
0 streams with 0 entries (00.00% of keys, avg size 0.00)
4 sets with 200 members (00.69% of keys, avg size 50.00)
0 zsets with 0 members (00.00% of keys, avg size 0.00)
$ redis-cli -h 192.168.3.74 -p 6379 --bigkeys

# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).

[00.00%] Biggest string found so far '"command-work-device-info:IGV807"' with 37 bytes
[00.00%] Biggest string found so far '"factor:vms_crane:CR40"' with 124 bytes
[00.00%] Biggest string found so far '"factor:crane:CR16"' with 375 bytes
[00.00%] Biggest string found so far '"factor:vehicle_instr:IGV820"' with 470 bytes
[05.18%] Biggest hash   found so far '"remote-gantry-info"' with 173 fields
[05.18%] Biggest string found so far '"factor:vehicle_instr_queue:IGV819"' with 6795 bytes
[06.91%] Biggest set    found so far '"online-truck-no-set"' with 1 members
[12.09%] Biggest string found so far '"pscp_route_info_T811"' with 89885 bytes
[42.31%] Biggest set    found so far '"truck-white-list:Node1-1"' with 197 members
[44.21%] Biggest list   found so far '"factor:command_parse_result:IGV841"' with 2 items
[45.94%] Biggest hash   found so far '"truck-info"' with 187 fields
[45.94%] Biggest list   found so far '"run-event-alarms"' with 12 items

-------- summary -------

Sampled 579 keys in the keyspace!
Total key length in bytes is 13082 (avg len 22.59)

Biggest   list found '"run-event-alarms"' has 12 items
Biggest   hash found '"truck-info"' has 187 fields
Biggest string found '"pscp_route_info_T811"' has 89885 bytes
Biggest    set found '"truck-white-list:Node1-1"' has 197 members

2 lists with 14 items (00.35% of keys, avg size 7.00)
8 hashs with 784 fields (01.38% of keys, avg size 98.00)
565 strings with 388320 bytes (97.58% of keys, avg size 687.29)
0 streams with 0 entries (00.00% of keys, avg size 0.00)
4 sets with 200 members (00.69% of keys, avg size 50.00)
0 zsets with 0 members (00.00% of keys, avg size 0.00)

 

 

2、使用redis-cli中的scan和memory usage

https://openatomworkshop.csdn.net/67441ed53a01316874d7ab20.html

 

3、使用开源工具redis-rdb-tools

可以看官方文档

https://github.com/sripathikrishnan/redis-rdb-tools

 

4、使用开源工具rdb_bigkeys

看官方文档

https://github.com/weiyanwei412/rdb_bigkeys.git

 

 

5、使用云服务提供商配套的工具

此项为可选项

 

三、关于预警的方案

采用promethus+grafana+alert-manager+exporter-node实现redis的监控和告警

 

 

 

参考链接：

每个方法有所差异，可以根据需求进行使用，比如key的长度，大小等的统计。

https://blog.csdn.net/jc0803kevin/article/details/141031065

https://openatomworkshop.csdn.net/674403a23a01316874d787c4.html