hadoop hdfs notes

vi /etc/profile

export JAVA_HOME=/usr/lib/jvm/java

export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib 

export PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/bin:$PATH:$HOME/bin 

export HADOOP_HOME=/opt/hadoop/hadoop-0.20.2

export PATH=$PATH:$HADOOP_HOME/bin

 

 

修改4个文件hadoop-site.xml,hadoop-env.sh,masters,slaves 

masters: 

server1

 

slaves: 

server2

server3

 

hadoop-env.sh 

export JAVA_HOME=/usr/lib/jvm/java

 

conf/core-site.xml

<configuration>

<property>

  <name>fs.default.name</name>

  <value>hdfs://server1:54310</value> //namenode的配置，机器名加端口

</property>

   <property>

      <name>dfs.name.dir</name>

      <value>/data/hdfs/hdfs1</value> //NameNode持久存储名字空间及事务日志的本地文件系统路径。 

   </property>

   <property>

     <name>dfs.data.dir</name> //DataNode存放块数据的本地文件系统路径，逗号分割的列表。

     <value>/data/hdfs/hdfsdata1</value>

   </property>

<property>

  <name>hadoop.tmp.dir</name>>//Hadoop的默认数据路径，这个最好配置，然后在新增节 点或者其他情况下莫名其妙的DataNode启动不了，就删除此文件中的tmp目录即可。不过如果删除了NameNode机器的此目录，那么就需要重新执 行NameNode格式化的命令了。

  <value>/data/hdfs/hdfstmp</value>

  <description>A base for other temporary directories.</description>

</property>

<property>

   <name>dfs.replication</name> //数据需要备份的数量，默认是三

   <value>1</value>

</property>

 

</configuration>

 

conf/mapred-site.xml

<property>

    <name>mapred.job.tracker</name> //JobTracker的主机（或者IP）和端口。

    <value>server1:54311</value>

   </property>

 

./bin/hadoop namenode -format

netstat -tnl//50070,54310,50090

http://master:50070

 

cd /home/hadoop 

tar -cvf hadoop-0.19.0.tar.gz hadoop-0.19.0/ 

scp -r hadoop-0.19.0.tar.gz hadoop@slave1:/home/hadoop 

scp -r hadoop-0.19.0.tar.gz hadoop@slave2:/home/hadoop

 

cd /opt/hadoop/hadoop-0.20.2

bin/hadoop dfs -ls

bin/hadoop dfs -mkdir testdir

bin/hadoop  dfs  -put  /opt/hadoop/chukwa-0.4.0.tar.gz  chukwa-0.4.0.tar.gz

and may need to change the /etc/hosts:

192.168.1.28    yun-phy1.scalethink.com yun-phy1

127.0.0.1 yun-phy1.scalethink.com yun-phy1 localhost.localdomain localhost

::1     localhost6.localdomain6 localhost6

192.168.1.27yun-phy2

192.168.1.25    yun-phy3

 

 

2.  hadoop 报Name node is in safe mode错解决办法

运行hadoop程序时， 中途我把它终止了，然后再向hdfs加文件或删除文件时，出现Name node is in safe mode错误： 
rmr: org.apache.hadoop.dfs.SafeModeException: Cannot delete /user/hadoop/input. Name node is in safe mode 

 

解决的命令： 

bin/hadoop dfsadmin -safemode leave  关闭safe mode