ERROR: Can't get master address from ZooKeeper; znode data == null

启动hbase-shell  > lIst 报错：ERROR: Can't get master address from ZooKeeper; znode data == null；

zookeeper:

　　

 

 hbase-master:

2018-03-15 10:39:21,945 INFO  [main-SendThread(hdp-datanode1:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3621edf9db10004, likely server has closed socket, closing socket connection and attempting reconnect
2018-03-15 10:39:21,945 INFO  [master/hdp-namenode/192.168.152.134:16000-SendThread(hdp-datanode1:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2621edf9e0f000a, likely server has closed socket, closing socket connection and attempting reconnect
2018-03-15 10:39:22,287 INFO  [master/hdp-namenode/192.168.152.134:16000-SendThread(hdp-datanode2:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp-datanode2/192.168.152.136:2181. Will not attempt to authenticate using SASL (unknown error)
2018-03-15 10:39:22,429 INFO  [master/hdp-namenode/192.168.152.134:16000-SendThread(hdp-datanode2:2181)] zookeeper.ClientCnxn: Socket connection established to hdp-datanode2/192.168.152.136:2181, initiating session
2018-03-15 10:39:22,450 INFO  [master/hdp-namenode/192.168.152.134:16000-SendThread(hdp-datanode2:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2621edf9e0f000a has expired, closing socket connection
2018-03-15 10:39:22,451 WARN  [master/hdp-namenode/192.168.152.134:16000-EventThread] client.ConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:702)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:613)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2018-03-15 10:39:22,463 INFO  [master/hdp-namenode/192.168.152.134:16000-EventThread] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2621edf9e0f000a
2018-03-15 10:39:22,464 INFO  [master/hdp-namenode/192.168.152.134:16000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2018-03-15 10:39:22,473 INFO  [main-SendThread(hdp-datanode2:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp-datanode2/192.168.152.136:2181. Will not attempt to authenticate using SASL (unknown error)
2018-03-15 10:39:22,506 INFO  [main-SendThread(hdp-datanode2:2181)] zookeeper.ClientCnxn: Socket connection established to hdp-datanode2/192.168.152.136:2181, initiating session
2018-03-15 10:39:22,537 INFO  [main-SendThread(hdp-datanode2:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3621edf9db10004 has expired, closing socket connection
2018-03-15 10:39:22,553 FATAL [main-EventThread] master.HMaster: Master server abort: loaded coprocessors are: []
2018-03-15 10:39:22,592 FATAL [main-EventThread] master.HMaster: master:16000-0x3621edf9db10004, quorum=hdp-namenode:2181,hdp-datanode1:2181,hdp-datanode2:2181, baseZNode=/hbase master:16000-0x3621edf9db10004 received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:702)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:613)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2018-03-15 10:39:22,593 INFO  [main-EventThread] regionserver.HRegionServer: STOPPED: master:16000-0x3621edf9db10004, quorum=hdp-namenode:2181,hdp-datanode1:2181,hdp-datanode2:2181, baseZNode=/hbase master:16000-0x3621edf9db10004 received expired from ZooKeeper, aborting
2018-03-15 10:39:22,640 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2018-03-15 10:39:22,769 INFO  [hdp-namenode,16000,1520994687974_splitLogManager__ChoreService_1] hbase.ScheduledChore: Chore: SplitLogManager Timeout Monitor was stopped
2018-03-15 10:39:23,426 INFO  [hdp-namenode,16000,1520994687974_ChoreService_2] hbase.ScheduledChore: Chore: hdp-namenode,16000,1520994687974-DoMetricsChore was stopped
2018-03-15 10:39:23,799 INFO  [master/hdp-namenode/192.168.152.134:16000] regionserver.HRegionServer: Stopping infoServer
2018-03-15 10:39:26,020 INFO  [master/hdp-namenode/192.168.152.134:16000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010
2018-03-15 10:39:27,075 INFO  [master/hdp-namenode/192.168.152.134:16000] procedure2.ProcedureExecutor: Stopping the procedure executor
2018-03-15 10:39:27,075 INFO  [master/hdp-namenode/192.168.152.134:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store
2018-03-15 10:39:30,178 INFO  [master/hdp-namenode/192.168.152.134:16000] regionserver.HRegionServer: stopping server hdp-namenode,16000,1520994687974
2018-03-15 10:39:30,221 INFO  [master/hdp-namenode/192.168.152.134:16000] regionserver.HRegionServer: stopping server hdp-namenode,16000,1520994687974; all regions closed.
2018-03-15 10:39:30,222 INFO  [master/hdp-namenode/192.168.152.134:16000] hbase.ChoreService: Chore service for: hdp-namenode,16000,1520994687974 had [[ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: hdp-namenode,16000,1520994687974-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: hdp-namenode,16000,1520994687974-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: hdp-namenode,16000,1520994687974-RegionNormalizerChore Period: 1800000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-hdp-namenode:16000 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown
2018-03-15 10:39:30,414 WARN  [master/hdp-namenode/192.168.152.134:16000] zookeeper.ZKUtil: master:16000-0x3621edf9db10004, quorum=hdp-namenode:2181,hdp-datanode1:2181,hdp-datanode2:2181, baseZNode=/hbase Unable to get data of znode /hbase/master
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:623)
        at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148)
        at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267)
        at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1199)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1083)
        at java.lang.Thread.run(Thread.java:748)
2018-03-15 10:39:30,416 ERROR [master/hdp-namenode/192.168.152.134:16000] zookeeper.ZooKeeperWatcher: master:16000-0x3621edf9db10004, quorum=hdp-namenode:2181,hdp-datanode1:2181,hdp-datanode2:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:623)
        at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148)
        at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267)
        at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1199)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1083)
        at java.lang.Thread.run(Thread.java:748)
2018-03-15 10:39:30,417 ERROR [master/hdp-namenode/192.168.152.134:16000] master.ActiveMasterManager: master:16000-0x3621edf9db10004, quorum=hdp-namenode:2181,hdp-datanode1:2181,hdp-datanode2:2181, baseZNode=/hbase Error deleting our own master address node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:623)
        at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148)
        at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267)
        at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1199)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1083)
        at java.lang.Thread.run(Thread.java:748)
2018-03-15 10:39:30,492 INFO  [master/hdp-namenode/192.168.152.134:16000] hbase.ChoreService: Chore service for: hdp-namenode,16000,1520994687974_splitLogManager_ had [] on shutdown
2018-03-15 10:39:30,549 INFO  [master/hdp-namenode/192.168.152.134:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down.
2018-03-15 10:39:30,674 INFO  [master/hdp-namenode/192.168.152.134:16000] ipc.RpcServer: Stopping server on 16000
2018-03-15 10:39:30,676 INFO  [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: stopping
2018-03-15 10:39:31,829 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2018-03-15 10:39:41,692 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2018-03-15 10:39:41,683 WARN  [master/hdp-namenode/192.168.152.134:16000] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/hdp-namenode,16000,1520994687974
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1236)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1225)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1436)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1091)
        at java.lang.Thread.run(Thread.java:748)
2018-03-15 10:39:41,773 INFO  [master/hdp-namenode/192.168.152.134:16000] regionserver.HRegionServer: stopping server hdp-namenode,16000,1520994687974; zookeeper connection closed.
2018-03-15 10:39:41,773 INFO  [master/hdp-namenode/192.168.152.134:16000] regionserver.HRegionServer: master/hdp-namenode/192.168.152.134:16000 exiting
View Code

 

可能原因：

　　1）、如果是虚拟机，可能zookeeper不稳定导致集群间通信出现问题，重启hbase集群，正常；

 　　2）、重启hbase集群，仍然无效，检查各节点状态正常，发现zookeeper其中一个节点拒绝连接；

 　　3）、检查发现hbase-maser无法启动，提示：

　　FATAL [hdp-namenode:16000.activeMasterManager] master.HMaster: Failed to become active master

　　org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby（master所在的主机namenode处于备份状态）

　　　　　　切换namenode状态后再启动hbase，检查是否有gc情况；

 

posted on 2018-03-15 12:55  RUI.Z  阅读(6447)  评论(0)  编辑  收藏  举报

导航