hadoop 2.5.1 、Hadoop 2.7 Hadoop 2.6

1  rpm 安装

    yum install rpm

2  JDK安装

   《《 一定要先删除JDK！！！！！！》》

 

    rpm -qa | grep java

 

    rpm -e --nodeps 

 

    rpm -qa | grep java

 

 

    下载 JDK

 

    tar -zxvf  or  xvf java

 

 环境变量更改

 

         /usr/local/opt/jdk1.7.0_67

 

 jdk环境变量

        8.1) vi /etc/profile

        8.2)

export JAVA_HOME=/usr/local/opt/jdk1.7.0_67

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar 

export PATH=$PATH:$JAVA_HOME/bin

   

 

1 下载Hadoop

 

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.1/hadoop-2.5.1.tar.gz

wget http://mirror.bit.edu.cn/apache/hbase/hbase-1.0.2/hbase-1.0.2-bin.tar.gz

 

wget http://mirror.bit.edu.cn/apache/hbase/1.1.2/hbase-1.1.2-bin.tar.gz

 

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz

 

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

 

如果环境变量崩了 就子机whereis vim         应该是这样  /usr/bin/vim /etc/profile

 

 

hadoop 环境变量

vi /etc/profile

/usr/local/opt/hadoop-2.5.1

/home/hadoop/hadoop/hadoop-2.6.0

export HADOOP_HOME=/usr/local/opt/hadoop-2.7.2

export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$HADOOP_HOME/lib:$HIVE_HOME/lib

export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/bin

 

hive 环境变量

export HIVE_HOME=/usr/hive-0.11.0

 

4 hdfs 初始化 yarn 初始化

进入hadoop下面的etc/hadoop/不是系统的etc，是hadoop下面的），ls，看到很多配置文件。

 

 

修改core-site.xml

 

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://t1:9000/</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
 <name>hadoop.tmp.dir</name>
<value>/opt/hadoop-data/tmp/hadoop-${user.name}</value>
<description>A base for other temporary directories.</description>
</property>
</configuration>

 

修改hdfs-site.xml

<configuration>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value></value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>/hdfs/data</value>
        </property>
        <property>
                <name>dfs.permissions</name>
                <value>false</value>
        </property>
        <property>
                <name>dfs.federation.nameservices</name>
                <value>t1</value>
        </property>
        <property>
                <name>dfs.namenode.rpc-address.master</name>
                <value>t1:9000</value>
        </property>
        <property>
                 <name>dfs.namenode.http-address.master</name>
                <value>t1:23001</value>
        </property>
        <property>
                <name>dfs.namenode.secondary.http-address.master</name>
                <value>t1:23002</value>
        </property>

</configuration>

修改mapred-site.xml

<configuration>

        <property>

                <name>mapreduce.framework.name</name>

                <value>yarn</value>

        </property>

        <property>

                <name>mapreduce.jobhistory.address</name>

                <value>t1:10020</value>

        </property>

        <property>

                 <name>mapreduce.jobhistory.webapp.address</name>

                <value>t1:19888</value>

        </property>

</configuration>

修改yarn-site.xml

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

 

<property>

<description>

host is the hostname of the resource manager and port is the port on which the NodeManagers contact the Resource Manager

</description>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>t1:8031</value>

</property>

 

<property>

<description>

the hostsis the hostname of the ResourceManager and the port is the port on which the clients can talk to the Resource Manager.

</description>

<name>yarn.resourcemanager.address</name>

<value>t1:8032</value>

</property>

 

<property>

<description>

host is the hostname of the resourcemanager and port is the port on which the Applications in the cluster talk to the Resource Manager.

</description>

<name>yarn.resourcemanager.scheduler.address</name>

<value>t1:8030</value>

</property>

 

<property>

<description>The address of the RM admin interface.</description>

<name>yarn.resourcemanager.admin.address</name>

<value>t1:8033</value>

</property>

 

<property>

<description>Resource Manager的Web访问地址</description>

<name>yarn.resourcemanager.webapp.address</name>

<value>t1:50030</value>

</property>

 

<property>

<name>yarn.nodemanager.local-dirs</name>

<value>${hadoop.tmp.dir}/nodemanager/local</value>

<description>the local directories used by the nodemanager to store the temp file</description>

</property>

 

<property>

<name>yarn.nodemanager.log-dirs</name>

<value>${hadoop.tmp.dir}/nodemanager/logs</value>

<description>the directories used by Nodemanagers as log directories</description>

</property>

</configuration>
 

修改 hadoop-env.sh、yarn-env.sh

export JAVA_HOME=/usr/local/jdk

 

 

 

 

 

5 配置 slave

vi etc/hadoop/slaves

加入

 

t1

t2

 

t3

 

6 HDFS 初始化

./bin/hdfs namenode -format

 

如果 format 出异常 JAVA环境变量监察一下。

 

 

7 将 hadoop 目录分发至其余节点

整文件夹拷贝

scp -r file root@slave1:/path

 

 

8 启动所有。

./sbin/start-all.sh

 

http://master:50070/       hdfs

 

http://master:50030/       mapReduce

 

成功后可以运行个例子

hadoop-2.7.2/share/hadoop/mapreduce   这里有一坨。

 

****************************  HDFS  ***************************

yum install gcc

查看系统的libc版本 
# ll /lib64/libc.so.6
lrwxrwxrwx 1 root root 11 Apr 24 16:49 /lib64/libc.so.6 -> libc-2.5.so
系统中的版本为2.5
将系统中的glibc升级为2.9
下载glibc
wget  http://ftp.gnu.org/gnu/glibc/glibc-2.9.tar.bz2
下载glibc-linuxthreads
wget http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.5.tar.bz2
解压
$tar -jxvf glibc-2.9.tar.bz2
$cd glibc-2.9
$tar -jxvf ../glibc-linuxthreads-2.5.tar.bz2
$cd ..
$export CFLAGS="-g -O2"
$./glibc-2.9/configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin
$make
#make install
安装编译过程中需要注意三点：
1、要将glibc-linuxthreads解压到glibc目录下。
2、不能在glibc当前目录下运行configure。
3、加上优化开关，export CFLAGS="-g -O2"，否则会出现错误

 

*****************************  HIVE  *****************************

 

http://apache.fayea.com/hive/