hadoop集群搭建

软件安装：Linux（阿里云）安装Hadoop(详细教程+避坑)_linux 下载并配置hadoop环境-CSDN博客

网络配置、软件配置：Hadoop集群安装和搭建（全面超详细的过程）_hadoop集群搭建-CSDN博客

yarn配置：Hadoop3.1.3Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster_hadoop 3.1.3 exception from container launch-CSDN博客

yum换源：https://www.8a.hk/news/content/8645.html

 

Hadoop异常--There are 3 datanode(s) running and 3 node(s) are excluded in this operation.-CSDN博客

 

注意每一次修改配置文件之后一定要注意用户或者用户组有没用系统更新成root，被更新了需要重新chown，否则hadoop可能读取不到配置

 

/etc/sysconfig/network-scripts/ifcfg-ens33：

IPADDR=192.168.56.150
NETMASK=255.255.255.0
GATEWAY=192.168.56.2
DNS1=8.8.8.8
DNS2=114.114.114.114

 

etc/profile：

export JAVA_HOME=/usr/java8
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HDFS_NAMENODE_USER=yangyuqi
export HDFS_DATANODE_USER=yangyuqi
export HDFS_SECONDARYNAMENODE_USER=yangyuqi
export YARN_RESOURCEMANAGER_USER=yangyuqi
export YARN_NODEMANAGER_USER=yangyuqi

 

/etc/hosts

192.168.56.149 hadoop01

192.168.56.150 hadoop02

192.168.56.151 hadoop03

 

jdk：/usr/java8

 

hadoop：/opt/hadoop

hive：/opt/hive

 

 

用户名：yangyuqi

密码：20040417

启动命令

hadoop01：

 hdfs namenode -format

start-dfs.sh

start-yarn.sh

 

网址：

hadoop:hadoop01:9870

yarn:hadoop01:8088

log:hadoop02:50090

 

hive 启动命令（hive建库一点要让hive自己建立，不要手动建立）

hadoop1:

nohup /opt/hive/bin/hive --service metastore >> /opt/hive/logs/metastore.log 2>&1 &

nohup /opt/hive/bin/hive --service hiveserver2 >> /opt/hive/logs/hiveserver2.log 2>&1 &

 

hdfs dfsadmin -safemode leave

 

 

如果你是复制的想重新部署：

sudo chown -R yangyuqi:yangyuqi /opt

sudo chmod +x /opt/hadoop/bin/hdfs

sudo chmod +x /opt/hadoop/bin/yarn

 sudo chmod +x /opt/hadoop/bin/hadoop

sudo chmod +x /opt/hadoop/sbin/start-dfs.sh

sudo chmod +x /opt/hadoop/sbin/stop-dfs.sh

sudo chmod +x /opt/hadoop/sbin/start-yarn.sh

sudo chmod +x /opt/hadoop/sbin/stop-yarn.sh

sudo chmod +x /opt/hive/bin/hive

 

/opt/hive/bin/hive --service metastore --hiveconf hive.root.logger=TRACE,console