记一次 habse replication的Lag越来越大
误操作导致habse replication的Lag越来越大解决（没有继续同步，堆积越来越多）：
hdfs fsck /data/logs/ -openforwrite 检查哪些文件处于打开写的过程一直未关闭
/data/logs/2019/11/27/16/data-log-20191127_16.1574841601929.log 861 bytes, replicated: replication=2, 1 block(s), OPENFORWRITE: /data/logs/2019/11/28/14/data-log-20191128_14.1574920800058.log.tmp 351 bytes, replicated: replication=2, 1 block(s), OPENFORWRITE: /data/logs/2019/11/28/14/data-log-20191128_14.1574920812437.log.tmp 703 bytes, replicated: replication=2, 1 block(s), OPENFORWRITE: /data/logs/2019/11/28/14/data-log-20191128_14.1574920817055.log.tmp 766 bytes, replicated: replication=2, 1 block(s), OPENFORWRITE:
Status: HEALTHY
Number of data-nodes: 3
Number of racks: 1
Total dirs: 152
Total symlinks: 0
Replicated Blocks:
Total size: 964521712 B
Total files: 429
Total blocks (validated): 429 (avg. block size 2248302 B)
Minimally replicated blocks: 425 (99.0676 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 1.981352
Missing blocks: 0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Erasure Coded Block Groups:
Total size: 0 B
Total files: 0
Total block groups (validated): 0
Minimally erasure-coded block groups: 0
Over-erasure-coded block groups: 0
Under-erasure-coded block groups: 0
Unsatisfactory placement block groups: 0
Average block group size: 0.0
Missing block groups: 0
Corrupt block groups: 0
Missing internal blocks: 0
FSCK ended at Thu Nov 28 14:16:14 CST 2019 in 10 milliseconds
The filesystem under path '/data/logs' is HEALTHY
通过命令hdfs debug recoverLease -path /data/logs/2019/11/27/16/data-log-20191127_16.1574841601929.log -retries 3
修复该文件即可。
参考 https://blog.csdn.net/mnasd/article/details/84560221
https://codeleading.com/article/88092566172/
https://blog.csdn.net/q495673918/article/details/128817054
HDFS常用命令
HDFS常用命令的学习
HDFS Balance调优参数设置
HDFS数据平衡
hdfs fsck / -list-corruptfileblocks -openforwrite -files -blocks -locations
