hadoop : java.lang.OutOfMemoryError: unable to create new native thread
最近在运行hadoop程序，遇到两个问题：
1、OutOfMemoryError in Hadoop
Error: unable to create new native thread Error initializing attempt_201111090003_0013_r_000000_0: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:614) at java.lang.UNIXProcess$1.run(UNIXProcess.java:157) at java.security.AccessController.doPrivileged(Native Method) at java.lang.UNIXProcess.(UNIXProcess.java:119) at java.lang.ProcessImpl.start(ProcessImpl.java:81) at java.lang.ProcessBuilder.start(ProcessBuilder.java:468) at org.apache.hadoop.util.Shell.runCommand(Shell.java:149) at org.apache.hadoop.util.Shell.run(Shell.java:134) at org.apache.hadoop.fs.DF.getAvailable(DF.java:73) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664) at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97) at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
When you have this kind of erros when runnning hadoop jobs, there might be a numer of possible reasons thanks to the feeble implementation of Hadoop. One possible reason is because in your MapReduce programs you open too much processes exceeding the default setting of your OS, for example, the default number is 1024 (you can check this number by executing 'ulimit -u'). A perfect example of using many processes would be such a case, in which you want control the output file name based on key-value pair in the reduce stage. To solve this problem, you need to modify some configuration files to raise up the maximum process number you can use, which can be done by editing /etc/security/limits.conf. Simply adding the following two lines to the llimits.conf to set the 100000 as the maximum number of processs in your system for user hadoop.
hadoop soft nproc 100000
hadoop hard nproc 100000
Other useful resources about OOM in hadoop can be found in the following links:
The dark side of hadoop; NoSQL; Dealing with outofmemoryerror-in-hadoop;
2. Task Id : attempt_200912131946_0001_m_000000_0, Status : FAILED Too many fetch-failures
Reduce task启动后第一个阶段是shuffle，即向map端fetch数据。每次fetch都可能因为connect超时，read超时，checksum错误等原因而失败。Reduce task为每个map设置了一个计数器，用以记录fetch该map输出时失败的次数。当失败次数达到一定阈值时，会通知JobTracker fetch该map输出操作失败次数太多了，并打印如下log：
Failed to fetch map-output from attempt_201105261254_102769_m_001802_0 even after MAX_FETCH_RETRIES_PER_MAP retries... reporting to the JobTracker
来自于博客：http://blog.csdn.net/liangliyin/article/details/6455713
我感觉是机器节点之间的通信问题。
下面还有一些比较好的帖子：
http://www.360doc.com/content/11/0323/23/23378_104035203.shtml
