Hadoop伪分布环境搭建
核心配置文件
hadoop-env.sh: hadoop依赖环境配置;
core-site.xml : hadoop 公共运行配置项配置文件:
hdfs-site.xml : hdfs相关配置项配置文件;
mapred-env.sh : mapreduce运行环境配置
mapred-site.xml : mapreduce配置项配置文件;
yarn-env.sh : yarn环境变量配置文件
yarn-site.xml : yarn配置项配置文件;
hadoop-env.sh
修改配置:
export JAVA_HOME=/opt/app/jdk1.8.0_321
export HADOOP_CONF_DIR=/opt/app/hadoop-2.8.5/etc/hadoop
core-site.xml
<configuration>
<!--在configuration标签中增加如下配置-->
<!-- 指定HDFS中NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://node1:9000</value>
</property>
<!-- 指定hadoop运行时产生文件的存储目录 HDFS相关文件存放地址-->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/app/hadoop-2.8.5/metaData</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<!-- 指定HDFS副本的数量 -->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
mapred-env.sh
修改配置:
export JAVA_HOME=/opt/app/jdk1.8.0_321
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
yarn-env.sh
export JAVA_HOME=/opt/app/jdk1.8.0_321
yarn-site.xml
<configuration>
<!-- reducer获取数据的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 指定YARN的ResourceManager的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>node1</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>
/opt/app/hadoop-2.8.5/etc/hadoop,
/opt/app/hadoop-2.8.5/share/hadoop/common/*,
/opt/app/hadoop-2.8.5/share/hadoop/common/lib/*,
/opt/app/hadoop-2.8.5/share/hadoop/hdfs/*,
/opt/app/hadoop-2.8.5/share/hadoop/hdfs/lib/*,
/opt/app/hadoop-2.8.5/share/hadoop/mapreduce/*,
/opt/app/hadoop-2.8.5/share/hadoop/mapreduce/lib/*,
/opt/app/hadoop-2.8.5/share/hadoop/yarn/*,
/opt/app/hadoop-2.8.5/share/hadoop/yarn/lib/*
</value>
</property>
</configuration>
slaves:DataNode NodeManager地址
node1
vim /etc/hosts
添加
192.168.200.225 node1
配置主机名
vi /etc/hostname
修改完成后需要重启
格式化namenode
hdfs namenode -format
启动hdfs和yarn
启动hdfs:
start-dfs.sh
启动yarn:
start-yarn.sh
在页面访问
查看namenode
http://192.168.200.225:50070/
查看datanode
http://192.168.200.225:50075/
查看secondarynode
http://192.168.200.225:50090/
查看yarn
http://192.168.200.225:8088/
关闭
关闭dfs
stop-dfs.sh
关闭yarn
stop-yarn.sh
本文来自博客园,作者:jsqup,转载请注明原文链接:https://www.cnblogs.com/jsqup/p/16497147.html