mapreduce 之 shuffle错误

错误信息

reduce容器报的错误信息如下：

2020-07-01 14:06:19,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#78
	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:395)
	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:310)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:291)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:330)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198)

其他日志信息

2020-07-01 14:06:19,404 INFO [fetcher#60] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#60 - MergeManager returned status WAIT ...
2020-07-01 14:06:19,404 INFO [fetcher#8] org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: xxx118:13562 freed by fetcher#8 in 0ms
2020-07-01 14:06:19,404 INFO [fetcher#60] org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: xxx162:13562 freed by fetcher#60 in 0ms
2020-07-01 14:06:19,404 INFO [fetcher#6] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#6 - MergeManager returned status WAIT ...
2020-07-01 14:06:19,404 INFO [fetcher#6] org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: xxx144:13562 freed by fetcher#6 in 2ms

从信息可以看出来，错误的原因是由于reduce从map拷贝数据的过程当中失败的，并且还是在merge阶段.

解决办法：

修改mapred的参数

mapreduce.reduce.shuffle.memory.limit.percent=0.1 # 默认是0.25 可以在代码里面设置，也可以在mapred-site.xml文件配置,这个参数的意思是说reduce每次fetceh数据的时候，达到reduce jvm内存的百分之多少的时候，就把数据写入到磁盘