记一次 habse replication的Lag越来越大
误操作导致habse replication的Lag越来越大解决(没有继续同步,堆积越来越多):
hdfs fsck /data/logs/ -openforwrite 检查哪些文件处于打开写的过程一直未关闭
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | /data/logs/2019/11/27/16/data-log-20191127_16.1574841601929.log 861 bytes, replicated: replication=2, 1 block(s), OPENFORWRITE: /data/logs/2019/11/28/14/data-log-20191128_14.1574920800058.log.tmp 351 bytes, replicated: replication=2, 1 block(s), OPENFORWRITE: /data/logs/2019/11/28/14/data-log-20191128_14.1574920812437.log.tmp 703 bytes, replicated: replication=2, 1 block(s), OPENFORWRITE: /data/logs/2019/11/28/14/data-log-20191128_14.1574920817055.log.tmp 766 bytes, replicated: replication=2, 1 block(s), OPENFORWRITE: Status: HEALTHY Number of data-nodes: 3 Number of racks: 1 Total dirs: 152 Total symlinks: 0 Replicated Blocks: Total size: 964521712 B Total files: 429 Total blocks (validated): 429 (avg. block size 2248302 B) Minimally replicated blocks: 425 (99.0676 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 2 Average block replication: 1.981352 Missing blocks: 0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Erasure Coded Block Groups: Total size: 0 B Total files: 0 Total block groups (validated): 0 Minimally erasure-coded block groups: 0 Over-erasure-coded block groups: 0 Under-erasure-coded block groups: 0 Unsatisfactory placement block groups: 0 Average block group size: 0.0 Missing block groups: 0 Corrupt block groups: 0 Missing internal blocks: 0 FSCK ended at Thu Nov 28 14:16:14 CST 2019 in 10 milliseconds The filesystem under path '/data/logs' is HEALTHY |
通过命令hdfs debug recoverLease -path /data/logs/2019/11/27/16/data-log-20191127_16.1574841601929.log -retries 3
修复该文件即可。
参考 https://blog.csdn.net/mnasd/article/details/84560221
https://codeleading.com/article/88092566172/
https://blog.csdn.net/q495673918/article/details/128817054
HDFS常用命令
HDFS常用命令的学习
HDFS Balance调优参数设置
HDFS数据平衡
hdfs fsck / -list-corruptfileblocks -openforwrite -files -blocks -locations
【推荐】国内首个AI IDE,深度理解中文开发场景,立即下载体验Trae
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· TypeScript + Deepseek 打造卜卦网站:技术与玄学的结合
· Manus的开源复刻OpenManus初探
· AI 智能体引爆开源社区「GitHub 热点速览」
· C#/.NET/.NET Core技术前沿周刊 | 第 29 期(2025年3.1-3.9)
· 从HTTP原因短语缺失研究HTTP/2和HTTP/3的设计差异
2014-12-17 grub