hive虚拟内溢出

hive虚拟内存溢出

Task with the most failures(4): 
-----
Task ID:
  task_1652184477422_0005_r_000001
​
URL:
  http://slave1:8088/taskdetails.jsp?jobid=job_1652184477422_0005&tipid=task_1652184477422_0005_r_000001
-----
Diagnostic Messages for this Task:
[2022-05-10 20:22:00.799]Container [pid=3957,containerID=container_1652184477422_0005_01_000009] is running 317848064B beyond the 'VIRTUAL' memory limit. Current usage: 163.4 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1652184477422_0005_01_000009 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 3957 3956 3957 3957 (bash) 0 0 9797632 289 /bin/bash -c /opt/module/jdk1.8.0_301/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN   -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/root/appcache/application_1652184477422_0005/container_1652184477422_0005_01_000009/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1652184477422_0005/container_1652184477422_0005_01_000009 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dyarn.app.mapreduce.shuffle.logger=INFO,shuffleCLA -Dyarn.app.mapreduce.shuffle.logfile=syslog.shuffle -Dyarn.app.mapreduce.shuffle.log.filesize=0 -Dyarn.app.mapreduce.shuffle.log.backups=0 org.apache.hadoop.mapred.YarnChild 192.168.150.129 46270 attempt_1652184477422_0005_r_000001_3 9 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1652184477422_0005/container_1652184477422_0005_01_000009/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1652184477422_0005/container_1652184477422_0005_01_000009/stderr  
    |- 3970 3957 3957 3957 (java) 550 34 2562908160 41553 /opt/module/jdk1.8.0_301/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/root/appcache/application_1652184477422_0005/container_1652184477422_0005_01_000009/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1652184477422_0005/container_1652184477422_0005_01_000009 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dyarn.app.mapreduce.shuffle.logger=INFO,shuffleCLA -Dyarn.app.mapreduce.shuffle.logfile=syslog.shuffle -Dyarn.app.mapreduce.shuffle.log.filesize=0 -Dyarn.app.mapreduce.shuffle.log.backups=0 org.apache.hadoop.mapred.YarnChild 192.168.150.129 46270 attempt_1652184477422_0005_r_000001_3 9 
​
[2022-05-10 20:22:00.851]Container killed on request. Exit code is 143
[2022-05-10 20:22:00.876]Container exited with a non-zero exit code 143. 
​
​
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1  Reduce: 3   Cumulative CPU: 4.75 sec   HDFS Read: 19958 HDFS Write: 654 FAIL
Total MapReduce CPU Time Spent: 4 seconds 750 msec

解决方法(亲测)：

再yarn-site.xml加入配置：

<property>
                <name>yarn.nodemanager.vmem-check-enabled</name>
                <value>false</value>
        </property>