应用程序连接hbase报错：java.net.SocketTimeoutException: callTimeout=60000

背景说明：

　　今天对生产环境hbase增加了节点，下午的时候一个同事反馈，应用程序后台报错，如下：

Tue Feb 26 17:35:35 CST 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68451: row 'SYSTEM.CATALOG,TARGETCUST_DATA,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host-10-191-36-24,16020,1551146724629, seqNum=0

        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
        at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
        at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1275)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)
        at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
        at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:506)
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:722)
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:692)
        at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1769)
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1724)
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1704)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1301)
        ... 47 more
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68451: row 'SYSTEM.CATALOG,TARGETCUST_DATA,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host-10-191-36-24,16020,1551146724629, seqNum=0
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:169)
        at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
        ... 3 more
Caused by: java.net.UnknownHostException: host-10-191-36-24
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.<init>(AbstractRpcClient.java:315)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.createBlockingRpcChannel(AbstractRpcClient.java:267)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getClient(ConnectionManager.java:1639)
        at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:162)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.prepare(ScannerCallableWithReplicas.java:376)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
        ... 4 more
2019-02-26 17:35:35 [com.asiainfo.cb2.consumer.ReceiveSMSID]-[ERROR]:33 - sms consumer begin 
2019-02-26 17:35:35 [com.asiainfo.cb2.consumer.ReceiveSMSID]-[ERROR]:38 - sms consumer pushType3

 

分析：

　　开始的时候，思路还是纠结在，超时的错误java.net.SocketTimeoutException: callTimeout=60000,想着有没有办法能够增加客户端的超时时间，然后还看了datanode的日志，认为是datanode在写数据导磁盘的时候性能问题，新增加hadoop节点，节点间在进行平衡，导致影响了性能，但是，对于以上的分析都没有更好的方法来解决问题。

结果：

　　等到后续，在仔细看错误，发现，Caused by: java.net.UnknownHostException: host-10-191-36-24，找不到主机的异常，才突然明白，应用程序首先连接到zk，然后zk告知region在哪个regionserver上，然后，应用程序再连接到hbase的regionserver上读写数据。

解决：

　　所以，在应用程序的/etc/hosts上配置新增的hbase regionserver节点的hosts解析，再次观察应用程序，该问题解决

 

文档创建时间：2019年2月27日10:34:38