hbase meta中分区信息错误的记录

bulk write hbase 时遇到下面的错误：

19/03/20 02:16:02 ERROR LoadIncrementalHFiles: IOException during splitting
java.util.concurrent.ExecutionException: java.io.IOException: The endkey of one region for table ap:users_v2 is not equal to the startkey of the next region in hbase:meta.Please use hbck tool to fix it first.
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:558)
	at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:414)
	at com.hotstar.ap.ingest.batch.tool.bulkload.BulkloadHelper$.bulkloadWrite(BulkloadHelper.scala:55)
	at com.hotstar.ap.ingest.batch.tool.HBaseWriterHelper$.writeToHBase(HBaseWriterHelper.scala:57)
	at com.hotstar.ap.ingest.batch.ingester.v2.VirtualBatchIngester.runOffline(VirtualBatchIngester.scala:20)
	at com.hotstar.ap.ingest.batch.ingester.VirtualIngester.run(VirtualIngester.scala:12)
	at com.hotstar.ap.ingest.batch.Main$$anonfun$1.apply(Main.scala:52)
	at com.hotstar.ap.ingest.batch.Main$$anonfun$1.apply(Main.scala:52)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at com.hotstar.ap.ingest.batch.Main$.delayedEndpoint$com$hotstar$ap$ingest$batch$Main$1(Main.scala:52)
	at com.hotstar.ap.ingest.batch.Main$delayedInit$body.apply(Main.scala:14)
	at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
	at scala.App$class.main(App.scala:76)
	at com.hotstar.ap.ingest.batch.Main$.main(Main.scala:14)
	at com.hotstar.ap.ingest.batch.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
Caused by: java.io.IOException: The endkey of one region for table ap:users_v2 is not equal to the startkey of the next region in hbase:meta.Please use hbck tool to fix it first.
	at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:682)
	at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:548)
	at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:545)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

 

sudo -u hbase hbase hbck 检查，存在 inconsistence，其中表 ap:users_v2 有错误，但显示为

Table ap:users_v2 is okay.
    Number of regions: 201
    Deployed on:  ip-10-10-240-127.ec2.internal,16020,1552903865915 ip-10-10-240-145.ec2.internal,16020,1552903928202 ip-10-10-240-48.ec2.internal,16020,1552903982881 ip-10-10-241-65.ec2.internal,16020,1552904038213 ip-10-10-243-103.ec2.internal,16020,1552904093359 ip-10-10-243-116.ec2.internal,16020,1552904166150

 

sudo -u hbase hbase hbck -repair 修复，解决了 inconsistence，结果显示为

Table ap:users_v2 is okay.
    Number of regions: 202
    Deployed on:  ip-10-10-240-127.ec2.internal,16020,1552903865915 ip-10-10-240-145.ec2.internal,16020,1552903928202 ip-10-10-240-48.ec2.internal,16020,1552903982881 ip-10-10-241-65.ec2.internal,16020,1552904038213 ip-10-10-243-103.ec2.internal,16020,1552904093359 ip-10-10-243-116.ec2.internal,16020,1552904166150