解决Hue/hiveserver2报错：java.io.IOException: Job status not available

sql是：select count(distinct col) from db.table;

排查过程中遇到过几个不同的报错：

1. beeline -u jdbc:hive2://0.0.0.0:10000 -e "select count(distinct col) from db.table;"

INFO  : Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1494385775332_0822
ERROR : Ended Job = job_1494385775332_0822 with exception 'java.io.IOException(Job status not available )'
java.io.IOException: Job status not available 
        at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:333)
        at org.apache.hadoop.mapreduce.Job.getJobState(Job.java:358)
        at org.apache.hadoop.mapred.JobClient$NetworkedJob.getJobState(JobClient.java:295)
        at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:244)
        at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437)
        at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
....
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=1)

Diagnostics:	Staging dir does not exist /mr-history/anonymous/.staging

2. beeline -u jdbc:hive2://0.0.0.0:10000 -n userxxx -e "select count(distinct col) from db.table;"

ERROR : Ended Job = job_1494385775332_0823 with exception 'java.io.IOException(java.io.IOException: Unknown Job job_1494385775332_0823
        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:218)
        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:281)
        at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173)
        at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
)'
java.io.IOException: java.io.IOException: Unknown Job job_1494385775332_0823

Diagnostics:	Job init failed : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://xxx/mr-history/userxxx/.staging/job_1494385775332_0823/job.splitmetainfo

使用不同的用户执行还是报一样的错误。

3. hive -e "select count(distinct col) from db.table;" 却能执行成功

 

解决：

根据第二步Diagnostics的报错信息搜索到一篇文章：https://community.hortonworks.com/questions/17489/job-init-fail-job-splitmetainfo-file-does-not-exis.html

对比了hive和hs2中此属性的值，果然不一样.hive上的值和配置文件中的一样，hs2不一样，在hs2中执行hql前设置为正确值后问题解决。询问了运维同事，这个属性近期刚改，改完后hiveserver2没有重启，导致使用beeline客户端或Hue时报此错。

解决方法：

1. 临时性：hql中显示纠正此属性值：set yarn.app.mapreduce.am.staging-dir=xxx;

2. 重启HS2