在Windows上用Eclipse开发WordCount jar包, 并运行在hadoop上

目前最新的Eclipse版本号叫 Neon.3 Release (4.6.3), 支持java8

读者阅读这篇文章的时候可能有了新的java版本和对应的eclipse版本出来了, 所以建议先去维基百科或者有关网站了解下应该下载什么版本的Eclipse.

使用Eclipse开发MapReduce程序, 先要给Eclipse添加好引用的类库, 右键项目, 可以找到一个选项"Build Path", 除了JRE库外还要添加hadoop的两个类库 hadoop-common-x.x.x.jar 和 hadoop-mapreduce-client-core-x.x.x.jar(它们在hadoop目录的子目录share/hadoop/commmon和share/hadoop/mapreduce目录里)

Java的类库之上还有一层叫package, 现在假设我们项目叫 HadoopTest, Package叫mapreduce, 而实现WordCount的类库文件叫 WordCount.java. 我们可以对.java文件右键export成jar文件, 也可以对package右键export成jar文件,也可以对项目右键export成jar文件.

按照下面的顺序来调用(以下英文部分转自: http://www.srccodes.com/p/article/45/run-hadoop-wordcount-mapreduce-example-windows):

Create a text file with some content. We'll pass this file as input to the wordcount MapReduce job for counting words.
#C:\file1.txt
#Install Hadoop 
#Run Hadoop Wordcount Mapreduce Example
#Create a directory (say 'input') in HDFS to keep all the text files (say 'file1.txt') to be used for counting words. Hadoop is install in C:\hadoop.

C:\Users> cd c:\hadoop
C:\hadoop> bin\hdfs dfs -mkdir /input

#Copy the text file(say 'file1.txt') from local disk to the newly created 'input' directory in HDFS.

C:\hadoop> bin\hdfs dfs -copyFromLocal c:/file1.txt /input

#Check content of the copied file.

C:\hadoop> hdfs dfs -ls /input

#Found 1 items
#-rw-r--r--   1 ABHIJITG supergroup         55 2014-02-03 13:19 input/file1.txt

C:\hadoop> bin\hdfs dfs -cat /input/file1.txt

#Install Hadoop

#Run Hadoop Wordcount Mapreduce Example

#Run the wordcount MapReduce job provided in %HADOOP_HOME%\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.2.0.jar

#切记是yarn jar, 不是yarn -jar

C:\hadoop\bin\yarn jar pathof.jar package.classname /input /output

C:\hadoop>bin\hdfs dfs -cat /output/*

#Check output.

Example 1
Hadoop  2
Install 1
Mapreduce       1
Run     1
Wordcount       1

这里说两点:

1. 根据我的经验, 所有命令行里提到的文件夹要前缀一个 "/", 比如"/output", 这可能是我在windows系统上跑hadoop造成的.

2.使用yarn 命令的时候要注明package.class的名称(除非你export时设置好了Main class, 那就不要package.classname这个参数了,否则报错),比如

yarn -jar path-of-your.jar package.classname /input /output  #未设置main class
yarn -jar path-of-your.jar /input /output  #设置了main class

最后附上 Hadoop2.8中 WordCount 的源代码(代码来自hadoop官网):

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

  public static class TokenizerMapper
       extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);
      }
    }
  }

  public static class IntSumReducer
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,
                       Context context
                       ) throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

你可能会感到很奇怪, 为什么Mapper和Reducer后面都跟了4个泛型, 其实这4个泛型分别对应:

KEYIN,VALUEIN,KEYOUT,VALUEOUT

具体可以参考下面两个链接:

http://www.aboutyun.com/thread-14243-1-1.html

https://hadoop.apache.org/docs/r2.7.1/api/src-html/org/apache/hadoop/mapreduce/Mapper.html

map后数据的结构: <key,{1,1,1,1,1}>,<key,{1,1,1}>

combine后数据的结构: <key,{5}>  <key,{3}>

shuffle后数据的结构:  <key,{5,3}>
最后reduce会再做一次合并得到: <key,{8}>
因为combine和reduce这两步其实都是对元组里第二项的各个元素求和, 所以代码里看到它们共用一个class