There is an overlap in the region chain修复
ERROR: (region day_hotstatic,860010-2355010000_20140417_12_entry_00000000321,1400060700465.fda3b0aca340570aeb64410c97e3cb73.) Multiple regions have the same startkey: 860010-2355010000_20140417_12_entry_00000000321
ERROR: (region day_hotstatic,860010-2355010000_20140417_12_entry_00000000321,1398674475358.0dc205736ec1e890bd2d37a2e3220acc.) Multiple regions have the same startkey: 860010-2355010000_20140417_12_entry_00000000321
ERROR: (regions day_hotstatic,860010-2355010000_20140417_12_entry_00000000321,1398674475358.0dc205736ec1e890bd2d37a2e3220acc. and day_hotstatic,860010-2368000000_20140413_14_visit_00000001964,1400060700465.a590268ef714ef76779486a62fe837a3.) There is an overlap in the region chain.
14/05/15 15:35:16 WARN util.HBaseFsck: reached end of problem group: 860010-2368010000_20140417_14_exit_00000000390
ERROR: Found inconsistency in table day_hotstatic
14/05/15 15:35:16 WARN util.HBaseFsck: Naming new problem group: 860010-2155000000_201404_4_entry_00000001763
ERROR: (region month_hotstatic,860010-2155000000_201404_4_entry_00000001763,1399568279705.1edc38d93e59257da8f1b3dadf68ac0b.) Multiple regions have the same startkey: 860010-2155000000_201404_4_entry_00000001763
ERROR: (region month_hotstatic,860010-2155000000_201404_4_entry_00000001763,1399958842442.ffdf1bbbbf06c0a4ecfb3a1f67568128.) Multiple regions have the same startkey: 860010-2155000000_201404_4_entry_00000001763
ERROR: (region month_hotstatic,860010-2288000000_201405_5_exit_00000047486,1399568279705.b323293466c60bcda712421657c43d5d.) Multiple regions have the same startkey: 860010-2288000000_201405_5_exit_00000047486
ERROR: (region month_hotstatic,860010-2288000000_201405_5_exit_00000047486,1399958848239.fb5eb32a3d25471b61dded04012de31f.) Multiple regions have the same startkey: 860010-2288000000_201405_5_exit_00000047486
14/05/15 15:35:16 WARN util.HBaseFsck: reached end of problem group: null
ERROR: Found inconsistency in table month_hotstatic
ERROR: (region day_hotstatic,860010-2355010000_20140417_12_entry_00000000321,1398674475358.0dc205736ec1e890bd2d37a2e3220acc.) Multiple regions have the same startkey: 860010-2355010000_20140417_12_entry_00000000321
ERROR: (regions day_hotstatic,860010-2355010000_20140417_12_entry_00000000321,1398674475358.0dc205736ec1e890bd2d37a2e3220acc. and day_hotstatic,860010-2368000000_20140413_14_visit_00000001964,1400060700465.a590268ef714ef76779486a62fe837a3.) There is an overlap in the region chain.
14/05/15 15:35:16 WARN util.HBaseFsck: reached end of problem group: 860010-2368010000_20140417_14_exit_00000000390
ERROR: Found inconsistency in table day_hotstatic
14/05/15 15:35:16 WARN util.HBaseFsck: Naming new problem group: 860010-2155000000_201404_4_entry_00000001763
ERROR: (region month_hotstatic,860010-2155000000_201404_4_entry_00000001763,1399568279705.1edc38d93e59257da8f1b3dadf68ac0b.) Multiple regions have the same startkey: 860010-2155000000_201404_4_entry_00000001763
ERROR: (region month_hotstatic,860010-2155000000_201404_4_entry_00000001763,1399958842442.ffdf1bbbbf06c0a4ecfb3a1f67568128.) Multiple regions have the same startkey: 860010-2155000000_201404_4_entry_00000001763
ERROR: (region month_hotstatic,860010-2288000000_201405_5_exit_00000047486,1399568279705.b323293466c60bcda712421657c43d5d.) Multiple regions have the same startkey: 860010-2288000000_201405_5_exit_00000047486
ERROR: (region month_hotstatic,860010-2288000000_201405_5_exit_00000047486,1399958848239.fb5eb32a3d25471b61dded04012de31f.) Multiple regions have the same startkey: 860010-2288000000_201405_5_exit_00000047486
14/05/15 15:35:16 WARN util.HBaseFsck: reached end of problem group: null
ERROR: Found inconsistency in table month_hotstatic
修复方法:找到start_key和end_key相同的几个region,把它们的从hdfs上删除掉。然后用add_table重建meta表(会导致丢失数据)
这个过程也是一个hbase的bug产生的,这个bug来自于重启过程。复现问题也很容易,进行以下几步即可复现:
1 找到一台正在split的region所在的rs
2 kill掉该台rs
3 重启整个集群或master进行切换
原因分析:
当hbase的master在主从切换或者重启的时候,有一个步骤是切换之后的master需要对原来所有的挂掉的regionserver上的region进行processDeadRegion,即重新上线。
该过程在0.90.4之前存在一个bug,即会把meta表中所有处在split期间的region也进行处理,虽然region在meta表中处于split状态并不能证明它己经split结束还是正在split(要对split状态进行标记还是很复杂的,因此目前的代码还没有对split状态进行记录,只能通过一些辅助手段,比如检查子region的状态来说明region是否处于split状态),但是万一它己经split结束的话是绝对不应该上线的。因此有可能一个region己经split结束,但它在这个处理过程中又被新起的master上线了,这就导致父子region同时服务了。而父region上线后又有可能继续split,导致状况更加糟糕,同一段数据被两个region服务,等等。
正确的处理办法是在重启时检查这些region的子region状态,具体检查方案在hbase-0.90.4中己经给出,可参见HBASE-3946。注意:打上3946的patch以后,还必须要打上3995的patch,否则单元测试无法通过。
这个过程也是一个hbase的bug产生的,这个bug来自于重启过程。复现问题也很容易,进行以下几步即可复现:
1 找到一台正在split的region所在的rs
2 kill掉该台rs
3 重启整个集群或master进行切换
原因分析:
当hbase的master在主从切换或者重启的时候,有一个步骤是切换之后的master需要对原来所有的挂掉的regionserver上的region进行processDeadRegion,即重新上线。
该过程在0.90.4之前存在一个bug,即会把meta表中所有处在split期间的region也进行处理,虽然region在meta表中处于split状态并不能证明它己经split结束还是正在split(要对split状态进行标记还是很复杂的,因此目前的代码还没有对split状态进行记录,只能通过一些辅助手段,比如检查子region的状态来说明region是否处于split状态),但是万一它己经split结束的话是绝对不应该上线的。因此有可能一个region己经split结束,但它在这个处理过程中又被新起的master上线了,这就导致父子region同时服务了。而父region上线后又有可能继续split,导致状况更加糟糕,同一段数据被两个region服务,等等。
正确的处理办法是在重启时检查这些region的子region状态,具体检查方案在hbase-0.90.4中己经给出,可参见HBASE-3946。注意:打上3946的patch以后,还必须要打上3995的patch,否则单元测试无法通过。
2.另一种修复方法:
So those are the same two regions (region1 in the first error is region1 in the second error and so on)? If so, it's probably a parent and a daughter region and somehow (due to a bug, can't tell for sure what exactly) they were both re-deployed. To fix, I think you could merge the two regions manually by first disabling the table, moving the files from one region to the other,deleting the folder / entry in .META. for the former region, and finally re-enable the table.