Hadoop3.3.4单机版安装教程及集成kerberos

 

配置java环境变量

wget https://repo.huaweicloud.com/java/jdk/8u202-b08/jdk-8u202-linux-x64.tar.gz
mkdir -p /usr/local/java
cd /usr/local/java
tar -zxvf jdk-8u202-linux-x64.tar.gz 

vi /etc/profile

#添加以下内容
export JAVA_HOME=/usr/local/java/jdk1.8.0_202
export PATH=$JAVA_HOME/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

#使之生效
source /etc/profile

 

安装Hadoop

下载地址：

https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable/

 

mkdir -p /opt/apps
cd /opt/apps
wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable/hadoop-3.3.4.tar.gz --no-check-certificate
tar -zxvf hadoop-3.3.4.tar.gz
mv hadoop-3.3.4 hadoop

 

配置Hadoop环境变量

vi /etc/profile
#添加以下内容
export HADOOP_HOME=/opt/apps/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

#使之生效
source /etc/profile

#验证一下
[root@cenots7 java]# hadoop version
Hadoop 3.3.4
Source code repository https://github.com/apache/hadoop.git -r a585a73c3e02ac62350c136643a5e7f6095a3dbb
Compiled by stevel on 2022-07-29T12:32Z
Compiled with protoc 3.7.1
From source with checksum fb9dd8918a7b8a5b430d61af858f6ec
This command was run using /opt/apps/hadoop/share/hadoop/common/hadoop-common-3.3.4.jar
[root@cenots7 java]#

 

新建用户和组

useradd hdfs
groupadd hadoop
usermod -g hadoop hdfs

 

 设置hostname和hosts文件

#设置 hostname
vi /etc/hostname
设置为 master （重启机器 reboot）
#设置 hosts
vi /etc/hosts
添加以下内容
192.168.30.206 master

 

修改配置文件

修改core-site.xml

vi ${HADOOP_HOME}/etc/hadoop/core-site.xml

添加以下内容

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
<description>配置 hdfs 端口,默认 9000</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/apps/hadoop/tmp</value>
<description>hadoop 临时工作目录</description>
</property>
<property>
<name>hadoop.native.lib</name>
<value>false</value>
<description>Should native hadoop libraries, if present, be used.</description>
</property>
</configuration>

 

修改hadoop-env.sh，修改JAVA_HOME

vi ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh

 

 

vi /opt/apps/hadoop/etc/hadoop/hdfs-site.xml

添加以下内容

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<!--这里是你自己的 ip，端口默认-->
<value>master:50090</value>
</property>
</configuration>

 

 

vi mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

 

vi yarn-site.xml

<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<!-- 自己的 ip 端口默认 -->
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>