Hadoop学习问题记录之基础篇
目的
记录学习hadoop过程中遇到的基础问题,无关大小、无关困扰时间长短。
问题一 全分布式环境中运行mapred程序,报异常:java.net.NoRouteToHostException: 没有到主机的路由
在全分布式环境中运行mapred程序,报异常:java.net.NoRouteToHostException: 没有到主机的路由,但同样的配置、同样的程序,在伪分布式环境中是没有问题的。具体异常信息如下:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
2019-09-14 15:37:44,018 INFO mapreduce.Job: Running job: job_1568442070466_0003 2019-09-14 15:43:46,231 INFO mapreduce.Job: Job job_1568442070466_0003 running in uber mode : false 2019-09-14 15:43:46,233 INFO mapreduce.Job: map 0% reduce 0% 2019-09-14 15:43:46,273 INFO mapreduce.Job: Job job_1568442070466_0003 failed with state FAILED due to: Application application_1568442070466_0003 failed 2 times due to Error launching appattempt_1568442070466_0003_000002. Got exception: java.net.NoRouteToHostException: No Route to Host from master/192.168.212.132 to slave1:45816 failed on socket timeout exception: java.net.NoRouteToHostException: 没有到主机的路由; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost at sun.reflect.GeneratedConstructorAccessor56.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1515) at org.apache.hadoop.ipc.Client.call(Client.java:1457) at org.apache.hadoop.ipc.Client.call(Client.java:1367) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy83.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:128) at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy84.startContainers(Unknown Source) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:123) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:308) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.NoRouteToHostException: 没有到主机的路由 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:690) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794) at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:411) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1572) at org.apache.hadoop.ipc.Client.call(Client.java:1403) ... 19 more . Failing the application. 2019-09-14 15:43:46,332 INFO mapreduce.Job: Counters: 0
问题定位
1、虽然异常信息中只提到了一个slave节点,但已知所有节点配置均一模一样,所以这应该是一个共性问题;
2、根据异常的字面意思,即当前提交工作的master节点找不到slave节点,而找不到无非是以下几种情况:
- 当前主机的hosts配置错误
经ping slave1命令测试成功,这个情况可以排除。 - 目标主机对应端口(45816)未打开
打开所有slave节点的45816端口后,重新提交一次作业观察运行情况。
再次提交作业,发现仍然报错,但此时的端口变成了43910,这个端口我们并未处理,自然也是关闭的。由端口的变化,可推测,每次提交作业调度运行时,用到的IPC端口是框架随机使用的,所以再用打开某个端口的方式来解决这个问题已经明显不可取了。
解决思路有两个:1、为slave节点添加对master节点的IP级别允许通过;2、使框架用到的ipc端口固定,然后用允许端口的方式开放。(暂未找到这种配置方法)
在使用解决思路1,使用以下命令开放给master节点访问:firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.212.132" accept' firewall-cmd --reload
再次提交作业,发现作业已经可以提交成功,并有成功的task运行,但仍存在NoRouteToHostException报错,错误信息如下:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
2019-09-14 17:31:55,733 INFO mapreduce.Job: Task Id : attempt_1568452715996_0001_m_000013_2, Status : FAILED Container launch failed for container_1568452715996_0001_01_000056 : java.net.NoRouteToHostException: No Route to Host from slave3/192.168.212.135 to slave1:42387 failed on socket timeout exception: java.net.NoRouteToHostException: 没有到主机的路由; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1515) at org.apache.hadoop.ipc.Client.call(Client.java:1457) at org.apache.hadoop.ipc.Client.call(Client.java:1367) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy84.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:128) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy85.startContainers(Unknown Source) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:160) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:394) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.NoRouteToHostException: 没有到主机的路由 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:690) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794) at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:411) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1572) at org.apache.hadoop.ipc.Client.call(Client.java:1403) ... 19 more
问题定位
看异常信息,虽然同为NoRouteToHostException,也都是在startContainer(启动容器)的时候报错,但与上一次的报错不同,这一次的异常并不是由master节点ipc调用slave1节点报错,而是slave1节点ipc调用slave3节点startContainer报错,可知由master到各个slave节点的通道已经打通,但slave间的通道还是堵塞的,即需要在各个salve节点的防火墙配置中,加入每一个salve节点的“通行许可”,如下
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.212.133" accept' firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.212.134" accept' firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.212.135" accept' firewall-cmd --reload
其中,133、134、135分别是slave1、2、3节点的IP地址。
用以下“在Yarn上运行MapReduce程序的过程”图来说明的话,第一个NoRouteToHostException出在步骤5a,第二个NoRouteToHostException出在步骤9a。
重新提交作业,问题解决。
问题二、MR程序实现HDFS和Mysql数据库间倒数据时报错:java.io.IOException: com.mysql.jdbc.Driver
问题定位
该问题是由于程序找不到mysql连接驱动,网上找到三种方法:
1、将mysql连接驱动jar放入每个节点的{HADOOP_HOME}/lib/下;(在hadoop3.2.0环境下验证无效)
2、将mysql连接驱动jar放入hdfs中,在代码中将jar引入;(在hadoop3.2.0环境下验证可用)
3、将mysql连接驱动jar直接打入mr程序中。(过于粗暴未实验,理论上应该也可行)
问题三、MR程序实现HDFS和Mysql数据库间倒数据时报错:Error: java.io.IOException: null, message from server: "Host 'slave1' is not allowed to connect to this MySQL server"
详细错误栈信息如下:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
Error: java.io.IOException: null, message from server: "Host 'slave1' is not allowed to connect to this MySQL server" at org.apache.hadoop.mapreduce.lib.db.DBOutputFormat.getRecordWriter(DBOutputFormat.java:197) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:659) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
问题定位
从错误信息来看,大意是当前host允许连接到mysql服务,于是更改mysql的用户表中host列为%,意为允许任意host发起连接:
mysql> use mysql; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> update user set host='%' where user='root'; Query OK, 0 rows affected (0.03 sec) Rows matched: 1 Changed: 0 Warnings: 0
再次允许程序,仍然报该错误。检查程序,发现是mysql登录密码写错了(尴尬),把密码改正后,问题解决。
这里记录一下,登录账户密码不对也会报以上错误。