3.Hadoop Single Node Cluster安装

3.1安装JDK

java --version 查看java版本，出现下面信息代表未安装。

输入命令 sudo apt-get update 更新软件包信息。

sudo apt-get install defult-jdk 下载jdk，出现提示按y。

再次查看jdk版本。

输入update-alternatives --display java 查看java安装路径。

3.2设置SSH无密码登录

输入命令 sudo apt-get install ssh 下载ssh

输入 sudo apt-get install rsync 安装rsync

输入ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa 产生密钥

查看产生的ssh

输入 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys 把产生的Key放到许可证文件中

3.3下载安装Hadoop

wget https://archive.apache.org/dist/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gz

下载慢换镜像https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gz

输入sudo tar -zxvf hadoop-3.2.4.tar.gz 解压

输入 sudo mv hadoop-3.2.4 /usr/local/hadoop 把hadoop移动到/usr/local下

3.4设置Hadoop环境变量

sudo gedit ~/.bashrc 设置环境变量

设置JDK安装路径

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop

设置PATH
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

设置HADOOP其他环境变量

export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME

链接库相关设置

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH

source ~/.bashrc 让设置生效

3.5修改Hadoop配置文件

编辑Hadoop-env.sh

sudo gedit /usr/local/hadoop/etc/hadoop/hadoop-env.sh

把原来注释掉的JAVA_HOME修改为：export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

设置core-site.xml

sudo gedit /usr/local/hadoop/etc/hadoop/core-site.xml

<configuration>
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>
</configuration>

编辑YARN-site.xml

sudo gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml

<property>
    <name>yarn.nodemanager.aux.services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.nodemanager.aux.services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

设置mapred-site.xml

sudo cp /usr/local/hadoop/etc/hadoop/mapred-site.xml /usr/local/hadoop/mapred-site.xml

sudo gedit /usr/local/hadoop/etc/hadoop/mapred-site.xml

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
</configuration>

编辑hdfs-site.xml

sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
<property>
    <name>dfs.replication</name>
    <value>3</value>
</property>
<property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/usr/local/hadoop/hadoop_data/hdfs/namenode</value>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/usr/local/hadoop/hadoop_data/hdfs/datanode</value>
</property>
</configuration>