hadoop安装全过程
1.下载hadoop-0.20.2,下载地址:http://www.apache.org/dist/hadoop/common/hadoop-0.20.2/
2.下载jdk-6u20-linux-i586.bin,下载地址:http://autosetup1.googlecode.com/files/jdk-6u20-linux-i586.bin
3.安装jdk:./jdk-6u20-linux-i586.bin
4.设置环境变量:在/etc/profile中加入如下语句:
JAVA_HOME=java安装路径(如/usr/java)
CLASSPATH=.:$JAVA_HOME/lib
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH
CLASSPATH=.:$JAVA_HOME/lib
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH
然后重新加载环境变量 source /etc/profile
5.解压缩hadoop-0.20.2,修改hadoop配置,配置文件如下:
1.core-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoopdata</value>(这个是一个文件夹,相当于分布式文件系统的硬盘主目录)
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.1.217:9000</value>(这是namenode的主机名和端口,连接hdfs时需要用到)
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/root/jz/hadoop-0.20.2/excludes</value>(这是一个文件,现在还不知道是干啥用的)
</property>
</configuration>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoopdata</value>(这个是一个文件夹,相当于分布式文件系统的硬盘主目录)
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.1.217:9000</value>(这是namenode的主机名和端口,连接hdfs时需要用到)
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/root/jz/hadoop-0.20.2/excludes</value>(这是一个文件,现在还不知道是干啥用的)
</property>
</configuration>
2.hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/root/jz/hadoop-0.20.2/hdfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/root/jz/hadoop-0.20.2/hdfs/data</value>
</property>
</configuration>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/root/jz/hadoop-0.20.2/hdfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/root/jz/hadoop-0.20.2/hdfs/data</value>
</property>
</configuration>
3.mapred-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>m:9001</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
</configuration>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>m:9001</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
</configuration>
master中填写master主机IP地址
slave中填写slave主机IP地址
遇到问题:hadoop环境变量配置,除了配置系统的环境变量/etc/profile外,还要配置hadoop/conf/hadoop-env.sh,因为hadoop启动时会先通过这个脚本重新加载环境变量,具体调用在hadoop-daemon.sh中
hadoop启动过程如下:先加载配置文件(hadoop-config.sh),启动hdfs(start-dfs.sh),启动mapreduce(start-mapred.sh)
使用jps查看正在运行的服务,namenode,jobtracker,SecondaryNameNode一定要起来
如果出现哪一个没起,就去日志里面查看原因,这次出现的两个错误有:namenode没有format,用命令hadoop namenode -format即可解决;excludes文件不存在,建立一下就好
注意:conf中的每一个配置对应的路径和文件都必须存在,否则就会出错。
第二个问题:如果没把集群配起,hdfs不能使用,配集群要设置SSH,使master不输入口令就可以连接到slave,配置方法如下:
|