hadoop相关Exception

Problem 1: 程序 'java' 已包含在下列软件包中：
 * default-jre
 * gcj-4.6-jre-headless
 * gcj-4.7-jre-headless
 * openjdk-7-jre-headless
 * openjdk-6-jre-headless
请尝试：sudo apt-get install <选定的软件包>

解决方法：(手动设置系统默认JDK)

在终端中依次输入命令：

sudo update-alternatives --install /usr/bin/java java /home/landen/UntarFile/jdk1.7.0_17/bin/java 300

sudo update-alternatives --install /usr/bin/javac javac /home/landen/UntarFile/jdk1.7.0_17/bin/javac 300

sudo update-alternatives --config java

接下来输入java -version就可以看到所安装JDK的版本信息了。

若仍有异常：

kelvin@ubuntu:~/Downloads$ java -version
bash: /usr/bin/java: Permission denied

此时只需要给/usr/bin/java , javac , jps添加相应的执行权限即可

kelvin@ubuntu:~/UntarFile$ sudo chmod +x /usr/bin/java
kelvin@ubuntu:~/UntarFile$ sudo chmod +x /usr/bin/javac
kelvin@ubuntu:~/UntarFile$ java -version
java version "1.7.0_09"
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) Client VM (build 23.5-b02, mixed mode)

kelvin@ubuntu:~/UntarFile$ sudo chmod +x /usr/bin/jps
kelvin@ubuntu:~/UntarFile$ jps
27132 Jps

 

Problem 2: 程序 'jps' 已包含在下列软件包中：

* openjdk-6-jdk
* openjdk-7-jdk
 Try: sudo apt-get install <selected package>

It is actually not a Hadoop problem. Hadoop does not use JPS.This problem is caused since you have installed JDK from Oracle (may be). 
You can fix this problem by using update-alternatives program to link jps to standard path directory. Use this command to fix this in a terminal

Solution: sudo update-alternatives --install /usr/bin/jps jps /home/landen/UntarFile/jdk1.7.0_17/bin/jps 1
Use the actual jps program path in the appropriate jdk (your version of jdk) instead of jdk1.7 which is specific to me.

extra Problem: Error using SCP: “not a regular file”
scp cannot copy a directory unless you ask for a recursive copy with the -r option (and it would refuse
 to overwrite an existing file with a directory even with -r, but it would quietly overwrite a regular file if the source was a regular file).

Problem 3: 安装使用Hive的时候发现文件无法从本地拷贝到hadoop集群系统，老是报错，错误内容为：

java.io.IOException: File  … could only be replicated to 0 nodes, instead of 1 
归根到底，就是自己DataNode节点没启动。
Exception 1: Sometime ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /home/hadoop/hadoop/hadtmp: namenode namespaceID = 1234649143; datanode namespaceID =10235035473
datanode与namenode之间的id不一致。
此时最好办法就是将hdfs.tmp.dir路径下的相关目录（name）全部删除，重新格式化hdfs，重启集群,另外可以更改其相关目录下的Current/version
landen@landen-Lenovo:~/UntarFile/hadoop-1.0.4$：bin/hadoop namenode -format
landen@landen-Lenovo:~/UntarFile/hadoop-1.0.4$: bin/start-all.sh
landen@landen-Lenovo:~/UntarFile/hadoop-1.0.4$: jps
Exception 2：查看hadoop相关log文件
2013-03-27 16:02:02,075 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: Incorrect permission for /home/landen/UntarFile/hadoop-1.0.4/datas/hdfs/data, expected: rwxr-xr-x, while actual: rwxrwxr-x
2013-03-27 16:02:02,075 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in dfs.data.dir are invalid.
2013-03-27 16:02:02,075 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
原来时权限问题，果断该权限：
landen@landen-Lenovo:~/UntarFile/hadoop-1.0.4/datas/hdfs$ chmod g-w data
landen@landen-Lenovo:~/UntarFile/hadoop-1.0.4$ jps
19844 JobTracker
20133 Jps
19210 NameNode
19445 DataNode
20081 TaskTracker
19692 SecondaryNameNode

DataNode启动Succeed

Exception 3：
2011-10-26 17:57:05,231 INFO org.apache.hadoop.ipc.RPC: Server at /192.168.0.189:9000 not available yet, Zzzzz...

2011-10-26 17:57:07,235 INFO org.apache.hadoop.ipc.Client: Retrying 
connect to server: /192.168.0.189:9000. Already tried 0 time(s).

2011-10-26 17:57:08,236 INFO org.apache.hadoop.ipc.Client: Retrying 
connect to server: /192.168.0.189:9000. Already tried 1 time(s).

2011-10-26 17:57:09,237 INFO org.apache.hadoop.ipc.Client: Retrying 
connect to server: /192.168.0.189:9000. Already tried 2 time(s).

2011-10-26 17:57:10,239 INFO org.apache.hadoop.ipc.Client: Retrying 
connect to server: /192.168.0.189:9000. Already tried 3 time(s).

2011-10-26 17:57:11,240 INFO org.apache.hadoop.ipc.Client: Retrying 
connect to server: /192.168.0.189:9000. Already tried 4 time(s).

2011-10-26 17:57:12,241 INFO org.apache.hadoop.ipc.Client: Retrying 
connect to server: /192.168.0.189:9000. Already tried 5 time(s).

也就是datanode连接不上namenode。而在namenode上面显示：

2011-10-26 14:18:49,686 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9000, call addBlock(/root/hadoop/tmp/mapred/system/jobtracker.info, DFSClient_-1928560478, null, null) from 127.0.0.1:32817: error: java.io.IOException: File /root/hadoop/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1

java.io.IOException: File /root/hadoop/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1448)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:690)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1350)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1346)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:396)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1344)

Reason 1: namenode也在试图将jobtracker.info存入hdfs文件系统中，而又存不进去。然后查了一下网上的说法，之后发现原来是/etc/hosts中的ip映射的问题。由于在master中/etc/hosts的配置为：

#127.0.0.1      master

#127.0.1.1      ubuntu.ubuntu-domain    ubuntu

192.168.0.189 master

192.168.0.104  slave1

192.168.0.135  slave2

192.168.0.149  slave3


因此可能存在一个优先匹配第一个碰见的问题，之后是将前两行注释掉(后来又将第一行改为了127.0.0.1 localhost)。然后在进行正常的hadoop format和启动，就可以连接上了。

Reason 2:  Remember将core-site.xml，mapred-site.xml中的localhost全部改为Master对应的主机名(/etc/hosts(永久修改), hostname 修改的主机名(临时修改))

Problem 4: HDFS暂时处于Safemode模式不能写入故障处理

2013-11-17 16:23:07,392 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9000, call delete(/home/landen/UntarFile/hadoop-1.0.4/datas/tmp/mapred/system, true) from 10.21.244.79:39841: error: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /home/landen/UntarFile/hadoop-1.0.4/datas/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically.
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /home/landen/UntarFile/hadoop-1.0.4/datas/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically.

不过这个safemode会锁定一会而后自动dfs写入(bin/hadoop dfsadmin -safemode leave)

Problem 5: hadoop伪分布式下每次启动时需要重新格式化否则namenode无法启动

      原因是hadoop默认的临时目录在/tmp目录下，这个目录在电脑重启后就会被删除，故需要在hadoop的conf的core-site.xml的配置文件中更改hadoop.tmp.dir在自己指定的目录下

Problem 6: 运行hive (dataprocess)> select * from tweets limit 10时出现如下异常：FAILED: RuntimeException org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /home/landen/UntarFile/hive-0.10.0/tmpHive_ScratchDir/hive-landen/hive_2013-09-25_21-11-14_259_3698608019622175648. Name node is in safe mode.

Solution:

$ bin/hadoop dfsadmin -safemode leave
来关闭safe mode.

Problem 7:

2012-12-14 12:17:50,587 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: can not create directory: /data/hdfs/data
2012-12-14 12:17:50,587 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in dfs.data.dir are invalid.

Solution:

data的相关权限问题：

$ chmod -R 755 hdfs/data

然后再重新启动 整个新装的集群