EMR 配置纪录(不断更新)


日志路径 -> /var/log(软连接到 /mnt/var/log)

在 spark master 机器中,配置的路径为

/usr/lib/spark/conf
/usr/lib/hadoop-yarn/etc/hadoop/

  

其中关键的配置

<property><name>yarn.nodemanager.local-dirs</name><value>/mnt/yarn</value></property>
<property><name>yarn.nodemanager.log-dirs</name><value>/var/log/hadoop-yarn/containers</value></property>

local-dirs: 应用的本地文件,官方解释为:

List of directories to store localized files in. An application's localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers' work directories, called container_${contid}, will be subdirectories of this.


log-dirs: 应用的日志路径,官方解释为:
Where to store container logs. An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container.

 

posted @ 2019-02-16 09:45  徐软件  阅读(285)  评论(0编辑  收藏  举报