Diagnostics: Container [pid=11942**] is running beyond virtual memory limits. Current usage: 97.0 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.

异常信息:
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1550592067516_0001 failed 2 times due to AM Container for appattempt_1550592067516_0001_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://hadoop103:8088/cluster/app/application_1550592067516_0001Then, click on links to logs of each attempt.
Diagnostics: Container [pid=11942,containerID=container_1550592067516_0001_02_000001] is running beyond virtual memory limits. Current usage: 97.0 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1550592067516_0001_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
.....................................................................................
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:911)
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:880)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:205)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:532)
... 8 more

集群虚拟机运行的Container过多,分配给集群使用的内存不足,集群NodeManager杀掉了部分容器,可以关闭虚拟内存检查。
解决办法:
修改hadoop集群yarn配置文件yarn-site.xml,添加属性:

yarn.nodemanager.vmem-check-enabled
false

posted @ 2020-08-02 12:09  Leo-Wong  阅读(364)  评论(0编辑  收藏  举报