用户提交任务到yarn报错

转载 https://www.cnblogs.com/barneywill/p/10628108.html

 

用户提交任务到yarn时有可能遇到下面的错误：

1）

Requested user anything is not whitelisted and has id 980,which is below the minimum allowed 1000

这是因为yarn中配置min.user.id=1000，yarn认为id小于1000的是超级用户，yarn禁止超级用户提交任务；

Each account must have a user ID that is greater than or equal to 1000. In the /etc/hadoop/conf/taskcontroller.cfg file, the default setting for the banned.users property is mapred, hdfs, and bin to prevent jobs from being submitted via those user accounts. The default setting for the min.user.id property is 1000 to prevent jobs from being submitted with a user ID less than 1000, which are conventionally Unix super users.

改为更小即可

然后重启yarn

 

2）

Requested user hdfs is banned

这是因为yarn中配置banned.users=hdfs,yarn,mapred,bin，会阻止这几个用户提交任务

the default setting for the banned.users property is hdfs, yarn, mapred, and bin to prevent jobs from being submitted via those user accounts.

 

3）其他组件比如oozie无法获得提交到yarn上的application的状态

2019-04-01 11:22:23,528 WARN org.apache.oozie.command.wf.ActionCheckXCommand: SERVER[cdp-test-server-04.bj] USER[impala] GROUP[-] TOKEN[] APP[test_sh_wf] JOB[0000000-190330171215585-oozie-oozi-W] ACTION[0000000-190330171215585-oozie-oozi-W@test_sh_action] Exception while executing check(). Error Code [JA017], Message[JA017: Could not lookup launched hadoop Job ID [job_1553937093273_0007] which was associated with  action [0000000-190330171215585-oozie-oozi-W@test_sh_action].  Failing this action!]
org.apache.oozie.action.ActionExecutorException: JA017: Could not lookup launched hadoop Job ID [job_1553937093273_0007] which was associated with  action [0000000-190330171215585-oozie-oozi-W@test_sh_action].  Failing this action!
        at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1507)
        at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:182)
        at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
        at org.apache.oozie.command.XCommand.call(XCommand.java:286)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

此时请检查jobhistory server，如果jobhistory server页面中没有application，也可以通过curl查看：

# curl http:/localhost:19888/jobhistory -v
...
            <script type="text/javascript">
              var jobsTableData=[
]
            </script>
...

正常来说这个数组应该存放很多application，这个问题需要检查hdfs上/user/history及子目录是否存在以及权限

$ hdfs dfs -mkdir /user/history
$ hdfs dfs -chmod 777 /user/history

正常情况下jobhistory会从hdfs以下目录加载应用信息

/user/history/done/$year/$month/$day/
/user/history/done_intermediate/$user/

 

4）从jobhistory server查看application时报错

2019-04-01 16:04:16,115 WARN org.apache.oozie.command.wf.ActionCheckXCommand: SERVER[cdp-test-server-04.bj] USER[impala] GROUP[-] TOKEN[] APP[test_sh_wf] JOB[0000008-190330171215585-oozie-oozi-W] ACTION[0000008-190330171215585-oozie-oozi-W@test_sh_action] Exception while executing check(). Error Code [JA009], Message[JA009: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not load history file hdfs://hdfs:8020/user/history/done_intermediate/impala/job_1553937093273_0018-1554105820398-impala-oozie%3Alauncher%3AT%3Dshell%3AW%3Dtest_sh_wf%3AA%3Dtest_sh_acti-1554105833651-1-0-SUCCEEDED-root.users.impala-1554105825680.jhist
...
Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=READ, inode="/user/history/done_intermediate/impala/job_1553937093273_0018-1554105820398-impala-oozie%3Alauncher%3AT%3Dshell%3AW%3Dtest_sh_wf%3AA%3Dtest_sh_acti-1554105833651-1-0-SUCCEEDED-root.users.impala-1554105825680.jhist":impala:supergroup:-rwxrwx---

hdfs中supergroup配置如下：

dfs.permissions.supergroup=supergroup
dfs.permissions.superusergroup=supergroup

解决方法：

# groupadd supergroup
# usermod -a -G supergroup mapred
# usermod -a -G supergroup hdfs
# usermod -a -G supergroup yarn