报错背景
服务器集群扩容,有一台节点忘记扩容并且硬盘可用空间为0的情况下启动了HBase集群,导致HBase处于不健康的运行状态。
报错现象
HBase无法正常关闭
查看日志 hbase-root-regionserver-bigdata2.log
2020-09-28 10:27:53,657 WARN [regionserver/bigdata2:16020] regionserver.HRegionServer: reportForDuty failed; sleeping and then retrying. 2020-09-28 10:27:56,658 INFO [regionserver/bigdata2:16020] regionserver.HRegionServer: reportForDuty to master=bigdata1,16000,1601259923019 with port=16020, startcode=1601259928763 2020-09-28 10:27:56,667 WARN [regionserver/bigdata2:16020] regionserver.HRegionServer: reportForDuty failed; sleeping and then retrying.
报错原因
(1)HBase的不健康状态导致hadoop进入安全模式,因此导致HBase不能正常运行。
(2)HDFS中/hbase/MasterProcWALs下日志文件太多,造成HBase重启失败。
报错解决
hadoop查看安全模式
# hadoop dfsadmin -safemode get
hadoop退出安全模式 # hadoop dfsadmin -safemode leave 重启HBase # stop-hbase.sh # start-hbase.sh
查看HDFS中的数据,我的此目录下数据达到100GB # hadoop dfs -ls /hbase/MasterProcWALs
删除日志文件 # hadoop dfs -rm -r /hbase/MasterProcWALs/pv*
重启HBase
# stop-hbase.sh # start-hbase.sh