Eclipse下使用Hadoop单机模式调试MapReduce程序
在单机模式下Hadoop不会使用HDFS，也不会开启任何Hadoop守护进程，所有程序将在一个JVM上运行并且最多只允许拥有一个reducer
在Eclipse中新创建一个hadoop-test的Java工程（特别要注意的是Hadoop需要1.6或1.6以上版本的JDK）
在Hadoop的官网http://www.apache.org/dyn/closer.cgi/hadoop/common/上选择合适的地址下载hadoop-1.2.1.tar.gz
解压hadoop-1.2.1.tar.gz得到hadoop-1.2.1目录
将hadoop-1.2.1目录下和hadoop-1.2.1\lib目录下的jar包导入到hadoop-test工程中
接下来编写MapReduce程序（该程序用来统计每月收支结余）
Map：
import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
public class MapBus extends MapReduceBase
implements Mapper<LongWritable, Text, Text, LongWritable> {
@Override
public void map(LongWritable key, Text date,
OutputCollector<Text, LongWritable> output,
Reporter reporter) throws IOException {
//2013-01-11,-200
String line = date.toString();
if(line.contains(",")){
String[] tmp = line.split(",");
String month = tmp[0].substring(5, 7);
int money = Integer.valueOf(tmp[1]).intValue();
output.collect(new Text(month), new LongWritable(money));
}
}
}
Reduce:
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;
public class ReduceBus extends MapReduceBase
implements Reducer<Text, LongWritable, Text, LongWritable> {
@Override
public void reduce(Text month, Iterator<LongWritable> money,
OutputCollector<Text, LongWritable> output, Reporter reporter)
throws IOException {
int total_money = 0;
while(money.hasNext()){
total_money += money.next().get();
}
output.collect(month, new LongWritable(total_money));
}
}
Main：
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
public class Wallet {
public static void main(String[] args){
if(args.length != 2){
System.err.println("param error!");
System.exit(-1);
}
JobConf jobConf = new JobConf(Wallet.class);
jobConf.setJobName("My Wallet");
FileInputFormat.addInputPath(jobConf, new Path(args[0]));
FileOutputFormat.setOutputPath(jobConf, new Path(args[1]));
jobConf.setMapperClass(MapBus.class);
jobConf.setReducerClass(ReduceBus.class);
jobConf.setOutputKeyClass(Text.class);
jobConf.setOutputValueClass(LongWritable.class);
try{
JobClient.runJob(jobConf);
}catch(Exception e){
e.printStackTrace();
}
}
}
还需准备待分析的文件，在E:\cygwin_root\home\input路径下创建2个文件，一个文件名为：2013-01.txt，另一个文件名为：2013-02.txt
2013-01.txt：
2013-01-01,100 2013-01-02,-100 2013-01-07,100 2013-01-10,-100 2013-01-11,100 2013-01-21,-100 2013-01-22,100 2013-01-25,-100 2013-01-27,100 2013-01-18,-100 2013-01-09,500
2013-02.txt：
2013-02-01,100
设置好运行参数后，就可以通过Run As -> Java Application运行MapReduce程序了
java.io.IOException: Failed to set permissions of path: \tmp\hadoop-linkage\mapred\staging\linkage1150562408\.staging to 0700
报这个错误的主要原因是后期的hadoop版本增加了对文件路径的校验，我的修改方式比较简单，将hadoop-core-1.2.1.jar替换为hadoop-0.20.2-core.jar即可正常运行
下面是MapReduce程序运行时打印的日志
14/02/11 10:54:16 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 14/02/11 10:54:16 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 14/02/11 10:54:16 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 14/02/11 10:54:16 INFO mapred.FileInputFormat: Total input paths to process : 2 14/02/11 10:54:17 INFO mapred.JobClient: Running job: job_local_0001 14/02/11 10:54:17 INFO mapred.FileInputFormat: Total input paths to process : 2 14/02/11 10:54:17 INFO mapred.MapTask: numReduceTasks: 1 14/02/11 10:54:17 INFO mapred.MapTask: io.sort.mb = 100 14/02/11 10:54:17 INFO mapred.MapTask: data buffer = 79691776/99614720 14/02/11 10:54:17 INFO mapred.MapTask: record buffer = 262144/327680 14/02/11 10:54:17 INFO mapred.MapTask: Starting flush of map output 14/02/11 10:54:18 INFO mapred.MapTask: Finished spill 0 14/02/11 10:54:18 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting 14/02/11 10:54:18 INFO mapred.LocalJobRunner: file:/E:/cygwin_root/home/input/2013-01.txt:0+179 14/02/11 10:54:18 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done. 14/02/11 10:54:18 INFO mapred.MapTask: numReduceTasks: 1 14/02/11 10:54:18 INFO mapred.MapTask: io.sort.mb = 100 14/02/11 10:54:18 INFO mapred.MapTask: data buffer = 79691776/99614720 14/02/11 10:54:18 INFO mapred.MapTask: record buffer = 262144/327680 14/02/11 10:54:18 INFO mapred.MapTask: Starting flush of map output 14/02/11 10:54:18 INFO mapred.MapTask: Finished spill 0 14/02/11 10:54:18 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting 14/02/11 10:54:18 INFO mapred.LocalJobRunner: file:/E:/cygwin_root/home/input/2013-02.txt:0+16 14/02/11 10:54:18 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000001_0' done. 14/02/11 10:54:18 INFO mapred.LocalJobRunner: 14/02/11 10:54:18 INFO mapred.Merger: Merging 2 sorted segments 14/02/11 10:54:18 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 160 bytes 14/02/11 10:54:18 INFO mapred.LocalJobRunner: 14/02/11 10:54:18 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting 14/02/11 10:54:18 INFO mapred.LocalJobRunner: 14/02/11 10:54:18 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now 14/02/11 10:54:18 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/E:/cygwin_root/home/output 14/02/11 10:54:18 INFO mapred.LocalJobRunner: reduce > reduce 14/02/11 10:54:18 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done. 14/02/11 10:54:18 INFO mapred.JobClient: map 100% reduce 100% 14/02/11 10:54:18 INFO mapred.JobClient: Job complete: job_local_0001 14/02/11 10:54:18 INFO mapred.JobClient: Counters: 13 14/02/11 10:54:18 INFO mapred.JobClient: FileSystemCounters 14/02/11 10:54:18 INFO mapred.JobClient: FILE_BYTES_READ=39797 14/02/11 10:54:18 INFO mapred.JobClient: FILE_BYTES_WRITTEN=80473 14/02/11 10:54:18 INFO mapred.JobClient: Map-Reduce Framework 14/02/11 10:54:18 INFO mapred.JobClient: Reduce input groups=2 14/02/11 10:54:18 INFO mapred.JobClient: Combine output records=0 14/02/11 10:54:18 INFO mapred.JobClient: Map input records=12 14/02/11 10:54:18 INFO mapred.JobClient: Reduce shuffle bytes=0 14/02/11 10:54:18 INFO mapred.JobClient: Reduce output records=2 14/02/11 10:54:18 INFO mapred.JobClient: Spilled Records=24 14/02/11 10:54:18 INFO mapred.JobClient: Map output bytes=132 14/02/11 10:54:18 INFO mapred.JobClient: Map input bytes=195 14/02/11 10:54:18 INFO mapred.JobClient: Combine input records=0 14/02/11 10:54:18 INFO mapred.JobClient: Map output records=12 14/02/11 10:54:18 INFO mapred.JobClient: Reduce input records=12
运行完成后将在E:\cygwin_root\home\output路径下生成2个文件：.part-00000.crc和part-00000。.part-00000.crc为一二进制文件，是一个保存了part-00000文件校验和的内部文件；part-00000文件中保存了最终的统计结果
01 500 02 100
特别要注意的是每次运行前都需要先将输出路径删掉，否则会报
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/E:/cygwin_root/home/output already exists
Hadoop做这个校验的目的是为了避免上一次MapReduce程序没有完成时，再次执行MapReduce程序产生的中间文件会覆盖掉上一次的中间文件
