Hadoop——集群参数配置详解
一、配置文件
在hadoop集群中,需要配置的文件主要包括四个,分别是core-site.xml、hdfs-site.xml、mapred-site.xml和yarn-site.xml,
这四个文件分别是对不同组件的配置参数,主要内容如下表所示:
二、配置时重要参数
1、core-site.xml
2、hdfs-site.xml
3、mapred-site.xml
4、yarn-site.xml
三、搭建集群配置样例
1、core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://cloud01:9000</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file:/home/hduser/tmp</value> </property> </configuration>
2、hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>cloud01:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file://${hadoop.tmp.dir}/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file://${hadoop.tmp.dir}/dfs/data</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file://${hadoop.tmp.dir}/dfs/namesecondary</value>
</property>
<property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> </configuration>
3、mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>cloud01:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>cloud01:19888</value> </property> </configuration>
4、yarn-site.xml
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>cloud01:8132</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>cloud01:8130</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value> cloud01:8131</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value> cloud01:8133</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>cloud01:8188</value> </property> </configuration>
四、更多配置参数信息
1、官方网址查询
- http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xml
- http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
- http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
- http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
通过这些网址,可以了解最新的全部的hadoop 配置信息,而且包括一些过时的定义标识,从而更好地维护您的集群。
2、集群工具查询
集群配置好后,可以通过http://192.168.75.101:8188/conf,查看集群参数状况,其中192.168.75.101为yarn.resourcemanager.webapp.address配置项的地址,8188为该配置项设置的端口。
引用: