YARN HA Failover 导致 RM状态异常问题
2021-11-17 11:41 DataBases 阅读(1682) 评论(0) 编辑 收藏 举报2021-11-15 18:52:15,361 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2021-11-15 18:52:15,372 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8032: starting
2021-11-15 18:52:15,421 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8030: starting
2021-11-15 18:52:19,190 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to active state
2021-11-15 18:52:19,193 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning to standby state
2021-11-15 18:52:19,222 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop OPERATION=refreshQueues TARGET=AdminService RESULT=FAILURE DESCRIPTION=ResourceManager is not active. Can not refresh queues. PERMISSIONS=
2021-11-15 18:52:19,222 ERROR org.apache.hadoop.yarn.server.resourcemanager.AdminService: RefreshAll failed so firing fatal event
org.apache.hadoop.ha.ServiceFailedException: ResourceManager rm1 is not Active!
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:609)
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:313)
at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:813)
at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:418)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2021-11-15 18:52:19,223 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:128)
at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:813)
at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:418)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll during transistion to Active
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:321)
at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
... 4 more
Caused by: org.apache.hadoop.ha.ServiceFailedException: ResourceManager rm1 is not Active!
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:609)
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:313)
... 5 more
2021-11-15 18:52:19,223 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session
2021-11-15 18:52:19,232 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received a org.apache.hadoop.yarn.server.resourcemanager.RMFatalEvent of type TRANSITION_TO_ACTIVE_FAILED. Cause:
org.apache.hadoop.ha.ServiceFailedException: ResourceManager rm1 is not Active!
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:609)
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:313)
at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:813)
at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:418)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2021-11-15 18:52:19,234 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
获取rm的状态
命令:
yarn rmadmin -getServiceState rm1
yarn rmadmin -getServiceState rm2
不能获取rm1,rm2的状态。
https://issues.apache.org/jira/browse/YARN-2588
https://issues.apache.org/jira/browse/YARN-2019
https://issues.apache.org/jira/browse/YARN-2010
Recovery失败导致导致RM无法启动:非常严重的bug。
根源在于RM HA启动时会去zk读取之前的状态(recovery过程),如果zk中的数据有问题,recovery过程会抛出异常,
这个异常没处理好,会直接抛到上层,导致RM进入STOPPED状态。
进入STOPPED状态后,就不能再变成其他状态了。
解决方法:
不改代码的话,只能手动清除zk中的数据。可以手动删除zk中的/rmstore路径:
setAcl /rmstore/ZKRMStateRoot world:anyone:cdrwa
rmr /rmstore
或者在yarn-site.xml中设置yarn.resourcemanager.zk-state-store.parent-path属性,比如/rmstore2,将数据存到另外一个路径。
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步