EMR 配置纪录(不断更新)
日志路径 -> /var/log(软连接到 /mnt/var/log)
在 spark master 机器中,配置的路径为
/usr/lib/spark/conf /usr/lib/hadoop-yarn/etc/hadoop/
其中关键的配置
<property><name>yarn.nodemanager.local-dirs</name><value>/mnt/yarn</value></property> <property><name>yarn.nodemanager.log-dirs</name><value>/var/log/hadoop-yarn/containers</value></property>
local-dirs: 应用的本地文件,官方解释为:
List of directories to store localized files in. An application's localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers' work directories, called container_${contid}, will be subdirectories of this.
log-dirs: 应用的日志路径,官方解释为:
Where to store container logs. An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container.