关于MapReduce中自定义分组类（三）

Job类

1.  /**
2.    * Define the comparator that controls which keys are grouped together
3.    * for a single call to
4.    * {@link Reducer#reduce(Object, Iterable,
5.    *                       org.apache.hadoop.mapreduce.Reducer.Context)}
6.    * @param cls the raw comparator to use
7.    * @throws IllegalStateException if the job is submitted
8.    * @see #setCombinerKeyGroupingComparatorClass(Class)
9.    */
10.   publicvoid setGroupingComparatorClass(Class<? extends RawComparator> cls
11.                                          ) throws IllegalStateException{
12.     ensureState(JobState.DEFINE);
13.     conf.setOutputValueGroupingComparator(cls);
14.   }

 

JobConf类

在JobConf类中的setOutputValueGroupingComparator方法：

1.  /**
2.    * Set the user defined {@link RawComparator} comparator for
3.    * grouping keys in the input to the reduce.
4.    *
5.    * <p>This comparator should be provided if the equivalence rules for keys
6.    * for sorting the intermediates are different from those for grouping keys
7.    * before each call to
8.    * {@link Reducer#reduce(Object, java.util.Iterator, OutputCollector, Reporter)}.</p>
9.    * 
10.    * <p>For key-value pairs (K1,V1) and (K2,V2), the values (V1, V2) are passed
11.    * in a single call to the reduce function if K1 and K2 compare as equal.</p>
12.    *
13.    * <p>Since {@link #setOutputKeyComparatorClass(Class)} can be used to control
14.    * how keys are sorted, this can be used in conjunction to simulate
15.    * <i>secondary sort on values</i>.</p>
16.    * 
17.    * <p><i>Note</i>: This is not a guarantee of the reduce sort being
18.    * <i>stable</i> in any sense. (In any case, with the order of available
19.    * map-outputs to the reduce being non-deterministic, it wouldn't make
20.    * that much sense.)</p>
21.    *
22.    * @param theClass the comparator class to be used for grouping keys.
23.    *                 It should implement <code>RawComparator</code>.
24.    * @see #setOutputKeyComparatorClass(Class)
25.    * @see #setCombinerKeyGroupingComparator(Class)
26.    */
27.   publicvoid setOutputValueGroupingComparator(
28.       Class<? extends RawComparator> theClass){
29.     setClass(JobContext.GROUP_COMPARATOR_CLASS,
30.              theClass,RawComparator.class);
31.   }

 

ctrl+O

找到getOutputValueGroupingComparator

1. /**
2.    * Get the user defined {@link WritableComparable} comparator for
3.    * grouping keys of inputs to the reduce.
4.    *
5.    * @return comparator set by the user for grouping values.
6.    * @see #setOutputValueGroupingComparator(Class) for details.
7.    */
8.   publicRawComparator getOutputValueGroupingComparator(){
9.     Class<? extends RawComparator> theClass = getClass(
10.       JobContext.GROUP_COMPARATOR_CLASS, null,RawComparator.class);
11.     if(theClass == null){
12.       return getOutputKeyComparator();
13.     }
14.     returnReflectionUtils.newInstance(theClass,this);
15.   }

 

那么谁调用了getOutputValueGroupingComparator方法

ReduceTask类

在ReduceTask类中：

（这里没有定义属性comparator，因为直接作为返回值接受接好了啊）

1.  RawComparator comparator = job.getOutputValueGroupingComparator();

这里get到的comparator其实就是我们自定义的xxxG

于是查找，哪里用到了comparator

1. if(useNewApi){
2.       runNewReducer(job, umbilical, reporter, rIter, comparator,
3.                     keyClass, valueClass);
4.     }else{
5.       runOldReducer(job, umbilical, reporter, rIter, comparator,
6.                     keyClass, valueClass);
7.     }

因为有新旧API之分啊

 

所以找到该runNewReducer方法：

1. private<INKEY,INVALUE,OUTKEY,OUTVALUE>void runNewReducer(JobConf job,
2.                      final TaskUmbilicalProtocol umbilical,
3.                      final TaskReporter reporter,
4.                      RawKeyValueIterator rIter,
5.                      RawComparator<INKEY> comparator,
6.                      Class<INKEY> keyClass,
7.                      Class<INVALUE> valueClass
8.                      ) throws IOException,InterruptedException,
9.                               ClassNotFoundException{
10.     // wrap value iterator to report progress.
11.     final RawKeyValueIterator rawIter = rIter;
12.     rIter =newRawKeyValueIterator(){
13.       publicvoid close() throws IOException{
14.         rawIter.close();
15.       }
16.       publicDataInputBuffer getKey() throws IOException{
17.         return rawIter.getKey();
18.       }
19.       publicProgress getProgress(){
20.         return rawIter.getProgress();
21.       }
22.       publicDataInputBuffer getValue() throws IOException{
23.         return rawIter.getValue();
24.       }
25.       public boolean next() throws IOException{
26.         boolean ret = rawIter.next();
27.         reporter.setProgress(rawIter.getProgress().getProgress());
28.         return ret;
29.       }
30.     };
31.     // make a task context so we can get the classes
32.     org.apache.hadoop.mapreduce.TaskAttemptContext taskContext =
33.       new org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(job,
34.           getTaskID(), reporter);
35.     // make a reducer
36.     org.apache.hadoop.mapreduce.Reducer<INKEY,INVALUE,OUTKEY,OUTVALUE> reducer =
37.       (org.apache.hadoop.mapreduce.Reducer<INKEY,INVALUE,OUTKEY,OUTVALUE>)
38.         ReflectionUtils.newInstance(taskContext.getReducerClass(), job);
39.     org.apache.hadoop.mapreduce.RecordWriter<OUTKEY,OUTVALUE> trackedRW =
40.       newNewTrackingRecordWriter<OUTKEY, OUTVALUE>(this, taskContext);
41.     job.setBoolean("mapred.skip.on", isSkipping());
42.     job.setBoolean(JobContext.SKIP_RECORDS, isSkipping());
43.     org.apache.hadoop.mapreduce.Reducer.Context
44.          reducerContext = createReduceContext(reducer, job, getTaskID(),
45.                                                rIter, reduceInputKeyCounter,
46.                                                reduceInputValueCounter,
47.                                                trackedRW,
48.                                                committer,
49.                                                reporter, comparator, keyClass,
50.                                                valueClass);
51.     try{
52.       reducer.run(reducerContext);
53.     } finally {
54.       trackedRW.close(reducerContext);
55.     }
56.   }

runNewReducer方法接收该comparator参数后传递给了createReduceContext方法

 

Task类

在Task里面的createReduceContext方法：

1.  @SuppressWarnings("unchecked")
2.   protectedstatic<INKEY,INVALUE,OUTKEY,OUTVALUE>
3.   org.apache.hadoop.mapreduce.Reducer<INKEY,INVALUE,OUTKEY,OUTVALUE>.Context
4.   createReduceContext(org.apache.hadoop.mapreduce.Reducer
5.                         <INKEY,INVALUE,OUTKEY,OUTVALUE> reducer,
6.                       Configuration job,
7.                       org.apache.hadoop.mapreduce.TaskAttemptID taskId,
8.                       RawKeyValueIterator rIter,
9.                       org.apache.hadoop.mapreduce.Counter inputKeyCounter,
10.                       org.apache.hadoop.mapreduce.Counter inputValueCounter,
11.                       org.apache.hadoop.mapreduce.RecordWriter<OUTKEY,OUTVALUE> output,
12.                       org.apache.hadoop.mapreduce.OutputCommitter committer,
13.                       org.apache.hadoop.mapreduce.StatusReporter reporter,
14.                       RawComparator<INKEY> comparator,
15.                       Class<INKEY> keyClass,Class<INVALUE> valueClass
16.   ) throws IOException,InterruptedException{
17.     org.apache.hadoop.mapreduce.ReduceContext<INKEY, INVALUE, OUTKEY, OUTVALUE>
18.     reduceContext =
19.       newReduceContextImpl<INKEY, INVALUE, OUTKEY, OUTVALUE>(job, taskId,
20.                                                               rIter,
21.                                                               inputKeyCounter,
22.                                                               inputValueCounter,
23.                                                               output,
24.                                                               committer,
25.                                                               reporter,
26.                                                               comparator,
27.                                                               keyClass,
28.                                                               valueClass);

 

ReduceContextImpl类

找到ReduceContextImpl中找到：

1.  publicReduceContextImpl(Configuration conf,TaskAttemptID taskid,
2.                            RawKeyValueIterator input,
3.                            Counter inputKeyCounter,
4.                            Counter inputValueCounter,
5.                            RecordWriter<KEYOUT,VALUEOUT> output,
6.                            OutputCommitter committer,
7.                            StatusReporter reporter,
8.                            RawComparator<KEYIN> comparator,
9.                            Class<KEYIN> keyClass,
10.                            Class<VALUEIN> valueClass
11.                           ) throws InterruptedException,IOException{
12.     super(conf, taskid, output, committer, reporter);
13.     this.input = input;
14.     this.inputKeyCounter = inputKeyCounter;
15.     this.inputValueCounter = inputValueCounter;
16.     this.comparator = comparator;
17.     this.serializationFactory =newSerializationFactory(conf);
18.     this.keyDeserializer = serializationFactory.getDeserializer(keyClass);
19.     this.keyDeserializer.open(buffer);
20.     this.valueDeserializer = serializationFactory.getDeserializer(valueClass);
21.     this.valueDeserializer.open(buffer);
22.     hasMore = input.next();
23.     this.keyClass = keyClass;
24.     this.valueClass = valueClass;
25.     this.conf = conf;
26.     this.taskid = taskid;
27.   }

 

在ReduceContextImpl类内查找comparator

1. /**
2.    * Advance to the next key/value pair.
3.    */
4.   @Override
5.   public boolean nextKeyValue() throws IOException,InterruptedException{
6.     if(!hasMore){
7.       key = null;
8.       value = null;
9.       returnfalse;
10.     }
11.     firstValue =!nextKeyIsSame;
12.     DataInputBuffer nextKey = input.getKey();
13.     currentRawKey.set(nextKey.getData(), nextKey.getPosition(),
14.                       nextKey.getLength()- nextKey.getPosition());
15.     buffer.reset(currentRawKey.getBytes(),0, currentRawKey.getLength());
16.     key = keyDeserializer.deserialize(key);
17.     DataInputBuffer nextVal = input.getValue();
18.     buffer.reset(nextVal.getData(), nextVal.getPosition(), nextVal.getLength()
19.         - nextVal.getPosition());
20.     value = valueDeserializer.deserialize(value);
21.  
22.     currentKeyLength = nextKey.getLength()- nextKey.getPosition();
23.     currentValueLength = nextVal.getLength()- nextVal.getPosition();
24.  
25.     if(isMarked){
26.       backupStore.write(nextKey, nextVal);
27.     }
28.  
29.     hasMore = input.next();
30.    if(hasMore){
31.       nextKey = input.getKey();
32.       nextKeyIsSame = comparator.compare(currentRawKey.getBytes(),0,
33.                                      currentRawKey.getLength(),
34.                                      nextKey.getData(),
35.                                      nextKey.getPosition(),
36.                                      nextKey.getLength()- nextKey.getPosition()
37.                                          )==0;
38.     }else{
39.       nextKeyIsSame =false;
40.     }
41.     inputValueCounter.increment(1);
42.     returntrue;
43.   }

这个compare方法，调用的是接口RawComparator中的

public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2); 

而一般如Text、IntWritable这些都实现了该方法

 

（一）未设置

1. if(theClass == null){
2.       return getOutputKeyComparator();
3.     }

1. /**
2.    * Get the {@link RawComparator} comparator used to compare keys.
3.    *
4.    * @return the {@link RawComparator} comparator used to compare keys.
5.    */
6.   publicRawComparator getOutputKeyComparator(){
7.     Class<? extends RawComparator> theClass = getClass(
8.       JobContext.KEY_COMPARATOR, null,RawComparator.class);
9.     if(theClass != null)
10.       returnReflectionUtils.newInstance(theClass,this);
11.     returnWritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class),this);
12.   }

 

没有job.setGroupingComparatorClass(xxxG.class);的时候，即使用默认的，调用Map输出的时候的key所属的类中的comparae，比如Text中的

原来默认情况下，调用的是比较器啊（更准确说是那个比较方法）

（这里比较器又分两种：

           1    key的类类型中的compareTo方法

           2    自定义比较器类中的compare方法

）

无论我们使用1还是2哪种方式，显然，分组和比较要么都用1 ，要么都用2，这样都是同一套规则，显然也不怎么合适。

所以我们一般是在自定义比较器类的同时又自定义分组类

 

（二）设置了

 

2.     returnReflectionUtils.newInstance(theClass,this);

如果我们job.setGroupingComparatorClass(xxxG.class),则是创建我们自定义的这个分组类的这个xxxG

这个xxxG得继承WritableComparator类，复写compare方法

如：

public static class SelfGroupComparator extends WritableComparator{

复写compare方法即可

这样，调用逻辑和compare的一样。

 

 

我更推荐方法2

 

 

alt+左箭头，返回上一次查看源码的地方

来自为知笔记(Wiz)