Hadoop中WritableComparable 和 comparator

1.WritableComparable

查看HadoopAPI,如图所示:

WritableComparable继承自Writable和java.lang.Comparable接口,是一个Writable也是一个Comparable,也就是说,既可以序列化,也可以比较!

再看看它的实现类,发现BooleanWritable, BytesWritable, ByteWritable, DoubleWritable, FloatWritable, IntWritable, LongWritable, MD5Hash, NullWritable, Record, RecordTypeInfo, Text, VIntWritable, VLongWritable都实现了WritableComparable类!

WritableComparable的实现类之间相互来比较,在Map/Reduce中,任何用作键来使用的类都应该实现WritableComparable接口!

 

Example:

 1 package cn.roboson.writable;
 2 
 3 import java.io.DataInput;
 4 import java.io.DataOutput;
 5 import java.io.IOException;
 6 
 7 import org.apache.hadoop.io.WritableComparable;
 8 
 9 /**
10  * 1.自定义一个类,继承WritableComparable
11  * 2.发现有三个未实现的方法,两个是Writable接口的(序列化),一个是Comparable接口的(用来比较)
12  * 3.自定义比较,这里以counter来作为比较
13  * @author roboson
14  *
15  */
16 public class MyWritableComparable implements WritableComparable<MyWritableComparable>{
17     
18     private int counter;
19     private long timestamp;
20     public MyWritableComparable() {
21         // TODO Auto-generated constructor stub
22     }
23 
24     public MyWritableComparable(int counter,long timestamp) {
25         // TODO Auto-generated constructor stub
26         this.counter = counter;
27         this.timestamp = timestamp;
28     }
29     
30     @Override
31     public void readFields(DataInput in) throws IOException {
32         // TODO Auto-generated method stub
33         
34         //将输入流中的字节流数据转化为结构化数据
35         counter = in.readInt();
36         timestamp = in.readLong();
37     }
38 
39     @Override
40     public void write(DataOutput out) throws IOException {
41         // TODO Auto-generated method stub
42         
43         //讲结构化数据写入输出流
44         out.writeInt(counter);
45         out.writeLong(timestamp);
46     }
47 
48     @Override
49     public int compareTo(MyWritableComparable other) {
50         // TODO Auto-generated method stub
51         int thisValue = this.counter;
52         int otherValue = other.counter;
53         return (thisValue < otherValue ? -1 : (thisValue == otherValue ? 0 : 1));
54     }
55 
56     public int getCounter() {
57         return counter;
58     }
59 
60     public void setCounter(int counter) {
61         this.counter = counter;
62     }
63 
64     public long getTimestamp() {
65         return timestamp;
66     }
67 
68     public void setTimestamp(long timestamp) {
69         this.timestamp = timestamp;
70     }
71     
72 
73     public static void main(String[] args) {
74         MyWritableComparable comparable = new MyWritableComparable(3,4);
75         MyWritableComparable otherComparable = new MyWritableComparable(4, 5);
76         int value = comparable.compareTo(otherComparable);
77         if(value==-1){
78             System.out.println("comparable<otherComparable");
79         }else if(value==0){
80             System.out.println("comparable=otherComparable");
81         }else{
82             System.out.println("comparable>otherComparable");
83         }
84     }
85 }

 

运行结果:

 

2.RawComparator

对于MapReduce来说,因为中间有个基于键的排序阶段,所以类型的比较是非常重要的。Hadoop中提供了原生的比较接口RawComparator,该接口继承子Java Comparator接口。RawComparator接口允许其实现直接比较数据流中的记录,无需先把数据流饭序列化为对象,这样便避免了新建对象的额外开销。

 1 package org.apache.hadoop.io;
 2 
 3 import java.util.Comparator;
 4 
 5 public interface RawComparator<T> extends Comparator<T>{
 6     
 7     //自己的方法
 8     public int compare(byte[] b1, int s1, int l1, byte[] b2,int s2, int l2);
 9     
10     //继承自Comparator的方法
11     @Override
12     public int compare(T o1, T o2);
13     
14     @Override
15     public boolean equals(Object obj);
16 }

 

查看HadoopAPI:

该类并非被多数的衍生类所实现,其具体的子类为WritableComparator,多数情况下是作为实现Writable接口的类的内置类,提供序列化字节的比较。如下图说所示:BooleanWritable, BytesWritable, ByteWritable, org.apache.hadoop.io.serializer.DeserializerComparator, DoubleWritable, FloatWritable, IntWritable, JavaSerializationComparator, LongWritable, LongWritable, MD5Hash, NullWritable, RecordComparator, Text, UTF8,都实现了RawComparator,作为其内部类。

而WritableComparator则是其的具体子类。

3.WritableComparator

在《Hadoop权威指南》中,说到这儿,很模糊,只说WritableComparator是对继承自WritableComparable类的RawCompartor类的一个通用实现。让人看着很迷惑,这句话什么意思呢?

首先、在第二个小标题RawComparator中,我门都知道WritableComparator实现了RawComparator这个接口,也就是说,WritableComparator是RawComparator的实现。

其次、是对继承自WritableComparable类的RawComparator的一个通用实现。那么继承自WritableComparable类的RawComparator都有哪些呢?也就是说那些类,继承自WritableComparator,并且实现了RawComparator?在第二个小标题RawComparator中有也都说明清楚了,上面的红色部分!同理,实现了WritableComparable类的在第一个小标题WritableComparable中也有说明,红色部分字体!也就谁说WritableComparator是对BooleanWritable.Comparator, BytesWritable.Comparator, ByteWritable.Comparator, DoubleWritable.Comparator, FloatWritable.Comparator, IntWritable.Comparator, LongWritable.Comparator, MD5Hash.Comparator, NullWritable.Comparator, RecordComparator, Text.Comparator, UTF8.Comparator这些类的一个通用实现!这句话就引出了WritableComparator的两个功能:第一,它提供了对原始compare()方法的一个默认实现。该方法能够饭序列化将流中进行比较的对象,并调用对象的compara()方法。第二,它充当的是RawComparator实例的工厂(已注册Writable的实现)。例如,为了获得IntWratable的comparator,我们直接如下调用:

RawComparator<IntWritable> comparator = WritableComparator.get(IntWratable.class);

再来看看WritableComparator这个类是如何定义的,如下图所示:

WritableComparator类类似于一个注册表,里面记录了所有Comparator类的集合。Comparators成员用一张Hash表记录Key=Classvalue=WritableComprator的注册信息.这就是它能够充当RawComparator实例工厂的原因!因为它本省的实现中有意个HashMap集合,HashMap<Class,WritableComparator>根据对应的Class,就能返回一个响应的WritableComparator!

 

Example:

 1 package cn.roboson.writable;
 2 
 3 import java.io.ByteArrayInputStream;
 4 import java.io.ByteArrayOutputStream;
 5 import java.io.DataInputStream;
 6 import java.io.DataOutputStream;
 7 import java.io.IOException;
 8 
 9 import org.apache.hadoop.io.IntWritable;
10 import org.apache.hadoop.io.RawComparator;
11 import org.apache.hadoop.io.Writable;
12 import org.apache.hadoop.io.WritableComparator;
13 
14 /**
15  * 1.通过WritableComparator获得IntWritable类的RawComparator实例
16  * 2.通过两种方式来比较
17  * @author roboson
18  *
19  */
20 
21 public class ComparableFinish {
22     
23     public static void main(String[] args) throws IOException {
24         
25         //创建两个IntWritable来比较
26         IntWritable writable1 = new IntWritable(163);
27         IntWritable writable2 = new IntWritable(165);
28         
29         //获得IntWritable的RawComparator实例
30         RawComparator<IntWritable> intRawComparator = WritableComparator.get(IntWritable.class);
31         
32         //直接比较对象
33         int value1 =intRawComparator.compare(writable1, writable2);
34         
35         if(value1==-1){
36             System.out.println("writable1<writable2");
37         }else if(value1==0){
38             System.out.println("writable1=writable2");
39         }else{
40             System.out.println("writable1>writable2");
41         }
42         
43         //序列化两个对象,获得其字节流
44         byte[] byte1 = serizlize(writable1);
45         byte[] byte2 = serizlize(writable2);
46         
47         //直接通过字符流比较大小
48         int value2 = intRawComparator.compare(byte1, 0, 4, byte2, 0, 4);
49         if(value2==-1){
50             System.out.println("writable1<writable2");
51         }else if(value2==0){
52             System.out.println("writable1=writable2");
53         }else{
54             System.out.println("writable1>writable2");
55         }
56     }
57     
58     public static byte[] serizlize(Writable writable) throws IOException{
59         
60         //创建一个输出字节流对象
61         ByteArrayOutputStream out = new ByteArrayOutputStream();
62         DataOutputStream dataout = new DataOutputStream(out);
63         
64         //将结构化数据的对象writable写入到输出字节流。
65         writable.write(dataout);
66         return out.toByteArray();
67     }
68     
69     public static byte[] deserizlize(Writable writable,byte[] bytes) throws IOException{
70         
71         //创建一个输入字节流对象,将字节数组中的数据,写入到输入流中
72         ByteArrayInputStream in = new ByteArrayInputStream(bytes);
73         DataInputStream datain = new DataInputStream(in);
74         
75         //将输入流中的字节流数据反序列化
76         writable.readFields(datain);
77         return bytes;
78         
79     }
80 }

 

运行结果:

关于序列化方面的知识,可以参考我的博客《Hadoop序列化》地址如下:

http://www.cnblogs.com/robert-blue/p/4157768.html

参考博文:

http://blog.csdn.net/keda8997110/article/details/8518255

http://www.360doc.com/content/12/0827/09/9318309_232551844.shtml

posted @ 2014-12-12 15:31  就像你一样回不来  阅读(7474)  评论(0编辑  收藏  举报