文本挖掘之文本聚类（MapReduce）

简介

　　针对大数量的文本数据，采用单线程处理时，一方面消耗较长处理时间，另一方面对大量数据的I/O操作也会消耗较长处理时间，同时对内存空间的消耗也是很大，因此，本文引入MapReduce计算模型，以分布式方式处理文本数据，以期提高数据处理速率。本文结合Kmeans和DBSCAN算法，对上述算法进行改进，其中借鉴Kmeans聚类方法（类别个数的确定性）以及DBSCAN聚类方法（基于密度），并在数据处理过程中引入多个Reducer对数据进行归并处理。测试结果表明：在文本个数为457条，迭代次数为50次时，该算法具有可行性；但是在数据规模较小时，其处理速率较单线程处理存在一定的劣势，但是当数据量继续增大（数据量达到一定规模）时，基于分布式的算法，其速率优势会更加明显。

基于MapReduce的改进算法实现

图-1 基于MapReduce的改进算法框架图

　　如图-1所示，为基于MapReduce的改进算法，单次执行框架图。对该框架中部分核心内容解释如下：

　　1) 在Mapper端，借鉴Kmeans算法确定K个类别及其初始质心，然后根据该质心，将所有文本进行一次聚类，以相似度与哪个质心相近，则属于质心属于该类别。

　　2）在Reducer端，借鉴DBSCAN算法，计算某个所属类别的e领域中所含个数，并以该e领域类所含个数，即minPts个数最多者为新的质心，即密度最大者为新的质心。

　　3）在Reducer端，为加快程序访问速率，采用5个Reducer来重新计算类别质心。

　　4）在Mapper端，通过读取缓存文件来获取每次迭代所需的类别新质心。

　　以下将本次设计中Mapper和Reducer端各自的输入和输出介绍一下：

　　Mapper : <Object，Text>--><IntWritable, Text>

　　输入：key未使用， value为Web文本数据；

　　输出：key为类别ID，value为Web文本数据。

　　Mapper设计的目标：给每篇文本计算出其所属类别，即类别ID。

　　Reducer: <IntWritable, Text>--><NullWritable, Text>

　　输入：key为文本类别ID， value为Web文本数据；

　　输出：key为Null， value为Web文本数据，即新的质心。

　　Reducer设计的目标：给每类数据确定新的质心。

测试结果与性能分析

　　由于本次测试目的，在于判别基于MapReduce的文本聚类算法可行性，因此数据规模并未设置很大。测试数据集为随机从网络上抓取的457篇Web标题，并迭代50次来展开测试，迭代的目的在于使每个类别的质心收敛。

表-1 改进的Kmeans和DBSCAN文本聚类算法测试结果

　　从表-1测试结果可知：在数据规模较小时，单线程处理的速率明显要优于MapReduce。主要原因在于，基于MapReduce框架，其每次迭代需要重新加载词典，同时读/写缓存文件，以获取质心或者修改质心，因此在数据规模较小时，处理数据的时间甚至不及上述文件的I/O时间，因此其优势并未发挥出来。本文作者曾尝试采用Java反射机制，加载数据对象以期解决上述问题，但收效甚微。

　　但是，采用MapReduce框架，在计算新的质心时，采用多个Reduer，明显能够改善数据规约的速率，较单线程处理来说，不仅能节省存储空间，同时处理简单、便捷。考虑到后期文本数据规模日益增大的趋势，引入分布式处理框架，对海量文本数据展开处理，已趋于一种潮流趋势，因此本文提出的算法有一定的实践意义。

　　程序源代码：

 1 public class ElementDict {
 2     private String term;
 3     private int freq;
 4     
 5     public ElementDict(String term, int freq) {
 6         this.term = term;
 7         this.freq = freq;
 8     }
 9     
10     
11     public void setFreq (int freq) {
12         this.freq = freq;
13     }
14 
15     
16     public String getTerm() {
17         return term;
18     }
19 
20     
21     public int getFreq() {
22         return freq;
23     }
24 
25     
26     public boolean equals(ElementDict e) {
27         boolean ret = false;
28         if (term.equals(e.getTerm()) && freq == e.getFreq())
29         {
30             ret = true;
31         }
32         
33         return ret;
34     }
35 }

Class ElementDict

  1 import java.io.BufferedReader;
  2 import java.io.IOException;
  3 import java.io.InputStreamReader;
  4 import java.net.URI;
  5 import java.util.HashMap;
  6 import java.util.List;
  7 import java.util.ArrayList;
  8 import java.util.Map;
  9 import org.apache.lucene.analysis.TokenStream;
 10 import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
 11 import org.wltea.analyzer.lucene.IKAnalyzer;
 12 import org.apache.hadoop.conf.Configuration;
 13 import org.apache.hadoop.fs.FSDataInputStream;
 14 import org.apache.hadoop.fs.FileSystem;
 15 import org.apache.hadoop.fs.Path;
 16 import org.apache.logging.log4j.LogManager;
 17 import org.apache.logging.log4j.Logger;
 18 
 19 public class TextCosine {
 20     private Map<String, String> map= null;
 21     private double common;
 22     private double special;
 23     private static final String PATH = "hdfs://10.1.130.10:9000";
 24     private static Logger logger = LogManager.getLogger(TextCosine.class);
 25     
 26     public TextCosine() {
 27         map = new HashMap<String, String>();
 28         try {
 29             Configuration conf = new Configuration();
 30             FileSystem fs = FileSystem.get(URI.create(PATH), conf);
 31             Path path = new Path("/user/hadoop/doc/synonyms.dict");
 32             FSDataInputStream is = fs.open(path);
 33             BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"));
 34             String s = null;
 35             while ((s = br.readLine()) != null) {
 36                 String []synonymsEnum = s.split("→");
 37                 map.put(synonymsEnum[0], synonymsEnum[1]);
 38             }
 39             br.close();
 40         } catch (IOException e) {
 41             logger.error("TextCosine IOException!");
 42         }
 43     }
 44     
 45     
 46     public TextCosine(double common, double special) {
 47         map = new HashMap<String, String>();
 48         try {
 49             Configuration conf = new Configuration();
 50             FileSystem fs = FileSystem.get(URI.create(PATH), conf);
 51             Path path = new Path("/user/hadoop/doc/synonyms.dict");
 52             FSDataInputStream is = fs.open(path);
 53             BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"));
 54             String s = null;
 55             while ((s = br.readLine()) != null) {
 56                 String []synonymsEnum = s.split("→");
 57                 map.put(synonymsEnum[0], synonymsEnum[1]);
 58             }
 59             br.close();
 60         } catch (IOException e) {
 61             logger.error("TextCosine IOException!");
 62         }
 63         
 64         this.common = common;
 65         this.special = special;
 66     }
 67     
 68     
 69     public void setCommon(double common) {
 70         this.common = common;
 71     }
 72     
 73     
 74     public void setSpecial(double special) {
 75         this.special = special;
 76     }
 77     
 78     // get the word with IK Analyzer
 79     public List<ElementDict> tokenizer(String str) {
 80         List<ElementDict> list = new ArrayList<ElementDict>();
 81         IKAnalyzer analyzer = new IKAnalyzer(true);
 82         try {
 83             TokenStream stream = analyzer.tokenStream("", str);
 84             CharTermAttribute cta = stream.addAttribute(CharTermAttribute.class);
 85             stream.reset();
 86             int index = -1;
 87             while (stream.incrementToken()) {
 88                 if ((index = isContain(cta.toString(), list)) >= 0) {
 89                     list.get(index).setFreq(list.get(index).getFreq() + 1);
 90                 }
 91                 else {
 92                     list.add(new ElementDict(cta.toString(), 1));
 93                 }
 94             }
 95             analyzer.close();
 96         } catch (IOException e) {
 97             e.printStackTrace();
 98         } 
 99         return list;
100     }
101     
102     // assert one term is in the List
103     public int isContain(String str, List<ElementDict> list) {
104         for (ElementDict ed : list) {
105             if (ed.getTerm().equals(str)) {
106                 return list.indexOf(ed);
107             } else if (map.get(ed.getTerm())!= null && map.get(ed.getTerm()).equals(str)) {
108                 return list.indexOf(ed);
109             }
110         }
111         return -1;
112     }
113     
114     // merge the two List to align the vector
115     public List<String> mergeTerms(List<ElementDict> list1, List<ElementDict> list2) {
116         List<String> list = new ArrayList<String>();
117         for (ElementDict ed : list1) {
118             if (!list.contains(ed.getTerm())) {
119                 list.add(ed.getTerm());
120             } else if (!list.contains(map.get(ed.getTerm()))) {
121                 list.add(ed.getTerm());
122             }
123         }
124         
125         for (ElementDict ed : list2) {
126             if (!list.contains(ed.getTerm())) {
127                 list.add(ed.getTerm());
128             } else if (!list.contains(map.get(ed.getTerm()))) {
129                 list.add(ed.getTerm());
130             }
131         }
132         return list;
133     }
134     
135     // get the max cosine 
136     public double analysisText(List<ElementDict> list1, List<ElementDict> list2) {
137         int len1 = list1.size();
138         int len2 = list2.size();
139         double ret = 0;
140         if (len2 >= len1 * 1.5) {
141             List<ElementDict> newList = new ArrayList<ElementDict>();
142             for (int i = 0; i + len1 <= len2; i++) {
143                 for (int j = 0; j < len1; j++) 
144                     newList.add(list2.get(i+j));
145                 
146                 newList = adjustList(newList, list2, len2, len1, i);
147                 double tmp = analysis(list1, newList);
148                 if (tmp > ret) 
149                     ret = tmp;
150             }
151         } else if (len1 >= len2 * 1.5) {
152             List<ElementDict> newList = new ArrayList<ElementDict>();
153             for (int i = 0; i + len2 <= len1; i++) {
154                 for (int j = 0; j < len2; j++)
155                     newList.add(list1.get(i+j));
156                 
157                 newList = adjustList(newList, list1, len1, len2, i);
158                 double tmp = analysis(list1, newList);
159                 if (tmp > ret) 
160                     ret = tmp;
161             }
162         } else {
163             ret = analysis(list1, list2);
164         }
165         return ret;
166     }
167     
168     // adjust the new List with the length about the original List 
169     public List<ElementDict> adjustList(List<ElementDict> newList, List<ElementDict> list, int lenBig, int lenSmall, int index) {
170         int gap = lenBig -lenSmall;
171         int size = (gap/2 > 2) ? 2: gap/2;
172         if (index < gap/2) {
173             for (int i = 0; i < size; i++) {
174                 newList.add(list.get(lenSmall+index+i));
175             }
176         } else {
177             for (int i = 0; i > size; i++) {
178                 newList.add(list.get(lenBig-index-i));
179             }
180         }
181         return newList;
182     }
183     
184     // analysis the cosine for two vectors
185     public double analysis(List<ElementDict> list1, List<ElementDict> list2) {
186         List<String> list = mergeTerms(list1, list2);
187         List<Integer> weightList1 = assignWeight(list, list1);
188         List<Integer> weightList2 = assignWeight(list, list2);
189         return countCosSimilarity(weightList1, weightList2);
190     }
191     
192     // according the frequency to assign the weight
193     public List<Integer> assignWeight(List<String> list, List<ElementDict> list1) {
194         List<Integer> vecList = new ArrayList<Integer>(list.size());
195         boolean isEqual = false;
196         for (String str : list) {
197             for (ElementDict ed : list1) {
198                 if (ed.getTerm().equals(str)) {
199                     isEqual = true;
200                     vecList.add(new Integer(ed.getFreq()));
201                 } else if (map.get(ed.getTerm())!= null && map.get(ed.getTerm()).equals(str)) {
202                     isEqual = true;
203                     vecList.add(new Integer(ed.getFreq()));
204                 }
205             }
206             
207             if (!isEqual) {
208                 vecList.add(new Integer(0));
209             }
210             isEqual = false;
211         }
212         return vecList;
213     }
214     
215     // count the cosine about the two vectors
216     public double countCosSimilarity(List<Integer> list1, List<Integer> list2) {
217         double countScores = 0;
218         int element = 0;
219         int denominator1 = 0;
220         int denominator2 = 0;
221         int index = -1;
222         for (Integer it : list1) {
223             index ++;
224             int left = it.intValue();
225             int right = list2.get(index).intValue();
226             element += left * right;
227             denominator1 += left * left;
228             denominator2 += right * right;
229         }
230         try {
231             countScores = (double)element / Math.sqrt(denominator1 * denominator2);
232         } catch (ArithmeticException e) {
233             e.printStackTrace();
234         }
235         return countScores;
236     }
237     
238     
239     public boolean isSimilarity(double param, double score) {
240         boolean ret = false;
241         if (score >= param)
242             ret = true;
243         return ret;
244     }
245     
246     
247     public boolean assertSimilarity(List<ElementDict> list1, List<ElementDict> list2) 
248     {
249         int len1 = list1.size();
250         int len2 = list2.size();
251         if (len2 >= len1 * 1.5) {
252             List<ElementDict> newList = new ArrayList<ElementDict>();
253             for (int i = 0; i + len1 <= len2; i++) {
254                 for (int j = 0; j < len1; j++) 
255                     newList.add(list2.get(i+j));
256                 
257                 newList = adjustList(newList, list2, len2, len1, i);
258                 if (isSimilarity(special, analysis(list1, newList))) 
259                     return true;
260             }
261         } else if (len1 >= len2 * 1.5) {
262             List<ElementDict> newList = new ArrayList<ElementDict>();
263             for (int i = 0; i + len2 <= len1; i++) {
264                 for (int j = 0; j < len2; j++)
265                     newList.add(list1.get(i+j));
266                 
267                 newList = adjustList(newList, list1, len1, len2, i);
268                 if (isSimilarity(special, analysis(list1, newList))) 
269                     return true;
270             }
271         } else {
272             if (isSimilarity(common, analysis(list1, list2))) 
273                 return true;
274         }
275         return false;
276     }
277 }

Class TextCosine

 1 import java.util.Collections;
 2 import java.util.List;
 3 
 4 import org.apache.logging.log4j.LogManager;
 5 import org.apache.logging.log4j.Logger;
 6 
 7 import com.gta.cosine.TextCosine;
 8 import com.gta.cosine.ElementDict;
 9 
10 public class DensityCenter {
11     private Logger logger = LogManager.getLogger(DensityCenter.class);
12     private double eps;
13     private TextCosine cosine;
14     
15     public DensityCenter(double eps, TextCosine cosine) {
16         this.eps = eps;
17         this.cosine = cosine;
18     }
19     
20     
21     public double cosineDistance(String src, String dst) 
22     {
23         List<ElementDict> vec1 = cosine.tokenizer(src);
24         List<ElementDict> vec2 = cosine.tokenizer(dst);
25         return cosine.analysisText(vec1, vec2);
26     }
27     
28     
29     public int getNeighbors(String src, List<String> dst) {
30         int ret = 0;
31         double score = 0;
32         for (String s : dst) {
33             score = cosineDistance(src, s);
34             if (score >= eps)
35                 ret++;
36         }
37         return ret;
38     }
39     
40     
41     public String getDensityCenter(List<String> text) {
42         int max = 0;
43         int i = 0;
44         int index = 0;
45         for (String s : text) {
46             int ret = getNeighbors(s, text);
47             if (ret > max) {
48                 index = i;
49                 max = ret;
50             }
51             i++;
52         }
53         return text.get(index);
54     }
55     
56     
57     public boolean compareCenters(List<String> oldCenters, List<String> newCenters) 
58     {
59         boolean ret = false;
60         Collections.sort(oldCenters);
61         Collections.sort(newCenters);
62         int oldSize = oldCenters.size();
63         int newSize = newCenters.size();
64         logger.info("oldSize : " + oldSize);
65         logger.info("newSize : " + newSize);
66         int size = oldSize > newSize ? newSize : oldSize;
67         int index = 0;
68         int count = 0;
69         for (String s : oldCenters) {
70             if (s.equals(newCenters.get(index))) 
71                 count++;
72             
73             index++;
74             if (index >= size)  // Avoid the size of two List is not the same
75                 break;
76         }
77         logger.info("count : " + count);
78         if (count == index)
79             ret = true;
80         
81         return ret;
82     }
83 }

Class DensityCenter

  1 import java.io.BufferedReader;
  2 import java.io.InputStreamReader;
  3 import java.io.IOException;
  4 import java.net.URI;
  5 import java.util.ArrayList;
  6 import java.util.List;
  7 import org.apache.hadoop.fs.FileSystem;
  8 import org.apache.hadoop.fs.FSDataInputStream;
  9 import org.apache.hadoop.fs.Path;
 10 import org.apache.hadoop.io.IntWritable;
 11 import org.apache.hadoop.io.NullWritable;
 12 import org.apache.hadoop.io.Text;
 13 import org.apache.hadoop.mapreduce.Mapper;
 14 import org.apache.hadoop.mapreduce.Reducer;
 15 import org.apache.logging.log4j.LogManager;
 16 import org.apache.logging.log4j.Logger;
 17 import com.gta.cosine.TextCosine;
 18 import com.gta.cosine.ElementDict;
 19 import com.gta.util.DensityCenter;
 20 
 21 public class KMeansProcess {
 22     
 23     public static class TextMapper extends Mapper<Object, Text, IntWritable, Text> {
 24         private static Logger logger = LogManager.getLogger(TextMapper.class);
 25         public static List<String> centersList = new ArrayList<String>();
 26         public static TextCosine cosine = new TextCosine();
 27         
 28         public void setup(Context context)
 29         {
 30             int iteration = context.getConfiguration().getInt("ITERATION", 100);
 31             if (iteration == 0) {
 32                 int task = context.getConfiguration().getInt("TASK", 0);
 33                 try {
 34                     URI[] caches = context.getCacheFiles();
 35                     if (caches == null || caches.length <= 0) {
 36                         System.exit(1);
 37                     }
 38                     for (int i = 0; i < task; i++) {
 39                         FileSystem fs = FileSystem.get(caches[i], context.getConfiguration());
 40                         FSDataInputStream is = fs.open(new Path(caches[i].toString()));
 41                         BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"));
 42                         String s = null;
 43                         while ((s = br.readLine()) != null) {
 44                             centersList.add(s);    
 45                         }
 46                         br.close();
 47                     }
 48                 } catch (IOException e) {
 49                     logger.error(e.getMessage());
 50                 }
 51             }
 52         }
 53         
 54         
 55         public void map(Object key, Text value, Context context)  
 56         {
 57             try {
 58                 String str = value.toString();
 59                 double score = 0;
 60                 double countTmp = 0;
 61                 int clusterID = 0;
 62                 int index = 0;
 63                 List<ElementDict> vec1 = cosine.tokenizer(str);
 64                 for (String s : centersList) {
 65                     List<ElementDict> vec2 = cosine.tokenizer(s);
 66                     countTmp = cosine.analysisText(vec1, vec2);
 67                     if (countTmp > score) {
 68                         clusterID = index;
 69                         score = countTmp;
 70                     }
 71                     index++;
 72                 }
 73                 context.write(new IntWritable(clusterID), new Text(str));
 74             } catch (IOException e) {
 75                 logger.error(e.getMessage());
 76             } catch (InterruptedException e) {
 77                 logger.error(e.getMessage());
 78             }
 79         }
 80     }
 81     
 82     
 83     public static class TextReducer extends Reducer<IntWritable, Text, NullWritable, Text> {
 84         private static Logger logger = LogManager.getLogger(TextReducer.class);
 85         public static DensityCenter center = new DensityCenter(0.75, KMeansProcess.TextMapper.cosine);
 86         
 87         public void reduce(IntWritable key, Iterable<Text> values, Context context) {
 88             try {
 89                 List<String> list = new ArrayList<String>();
 90                 for (Text val : values) {
 91                     list.add(val.toString());
 92                 }
 93                 context.write(NullWritable.get(), new Text(center.getDensityCenter(list)));
 94             } catch (IOException e) {
 95                 logger.error(e.getMessage());
 96             } catch (InterruptedException e) {
 97                 logger.error(e.getMessage());
 98             }
 99         }
100     }
101 }

Class KMeansProcess

 1 import java.io.BufferedReader;
 2 import java.io.IOException;
 3 import java.io.InputStreamReader;
 4 import java.net.URI;
 5 import java.util.List;
 6 import org.apache.hadoop.fs.FSDataInputStream;
 7 import org.apache.hadoop.fs.FileSystem;
 8 import org.apache.hadoop.fs.Path;
 9 import org.apache.hadoop.io.IntWritable;
10 import org.apache.hadoop.io.Text;
11 import org.apache.hadoop.mapreduce.Mapper;
12 import org.apache.hadoop.mapreduce.Reducer;
13 import org.apache.logging.log4j.LogManager;
14 import org.apache.logging.log4j.Logger;
15 import com.gta.cosine.TextCosine;
16 import com.gta.cosine.ElementDict;
17 
18 public class KMeans {
19         
20     public static class KMeansMapper extends Mapper<Object, Text, IntWritable, Text> {
21         private List<String> centersList = KMeansProcess.TextMapper.centersList;
22         private static Logger logger = LogManager.getLogger(KMeans.KMeansMapper.class);
23         private TextCosine cosine = KMeansProcess.TextMapper.cosine;
24         
25         public void setup(Context context)
26         {
27             int task = context.getConfiguration().getInt("TASK", 0);
28             try {
29                 URI[] caches = context.getCacheFiles();
30                 if (caches == null || caches.length <= 0) {
31                     System.exit(1);
32                 }
33                 for (int i = 0; i < task; i++) {
34                     FileSystem fs = FileSystem.get(caches[i], context.getConfiguration());
35                     FSDataInputStream is = fs.open(new Path(caches[i].toString()));
36                     BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"));
37                     String s = null;
38                     while ((s = br.readLine()) != null)
39                         centersList.add(s);
40                     br.close();
41                 }
42             } catch (IOException e) {
43                 logger.error(e.getMessage());
44             }
45         }
46         
47         
48         public void map(Object key, Text value, Context context) {
49             try {
50                 String str = value.toString();
51                 double score = 0;
52                 double countTmp = 0;
53                 int clusterID = 0;
54                 int index = 0;
55                 List<ElementDict> vec1 = cosine.tokenizer(str);
56                 for (String s : centersList) {
57                     List<ElementDict> vec2 = cosine.tokenizer(s);
58                     countTmp = cosine.analysisText(vec1, vec2);
59                     if (countTmp > score) {
60                         clusterID = index;
61                         score = countTmp;
62                     }
63                     index++;
64                 }
65                 context.write(new IntWritable(clusterID), new Text(str));
66             } catch (IOException e) {
67                 logger.error(e.getMessage());
68             } catch (InterruptedException e) {
69                 logger.error(e.getMessage());
70             }
71         }
72         
73         
74         public void cleanup(Context context)
75         {
76             centersList.clear();
77         }
78     }
79     
80     
81     public static class KMeansReducer extends Reducer<IntWritable, Text, IntWritable, Text> {
82         private static Logger logger = LogManager.getLogger(KMeans.KMeansReducer.class);
83         
84         public void ruduce(IntWritable key, Iterable<Text> values, Context context) {
85             try {
86                 for (Text val : values) {
87                     context.write(key, val);
88                 }
89             } catch (IOException e) {
90                 logger.error(e.getMessage());
91             } catch (InterruptedException e) {
92                 logger.error(e.getMessage());
93             }
94         }
95     }
96 
97 }

Class KMeans

  1 import java.io.BufferedReader;
  2 import java.io.IOException;
  3 import java.io.InputStreamReader;
  4 import java.util.List;
  5 import java.util.ArrayList;
  6 import java.net.URI;
  7 import org.apache.hadoop.conf.Configuration;
  8 import org.apache.hadoop.fs.FSDataInputStream;
  9 import org.apache.hadoop.fs.FileSystem;
 10 import org.apache.hadoop.fs.Path;
 11 import org.apache.hadoop.io.IntWritable;
 12 import org.apache.hadoop.io.Text;
 13 import org.apache.hadoop.mapreduce.Job;
 14 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
 15 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 
 16 import com.gta.cluster.KMeans.KMeansMapper;
 17 import com.gta.cluster.KMeans.KMeansReducer;
 18 import com.gta.cluster.KMeansProcess.TextMapper;
 19 import com.gta.cluster.KMeansProcess.TextReducer;
 20 import org.apache.logging.log4j.LogManager;
 21 import org.apache.logging.log4j.Logger;
 22 
 23 public class Cluster {
 24     public static final int MAX = 50;
 25     public static final String INPUT_PATH = "hdfs://10.1.130.10:9000/user/hadoop/input/";
 26     public static final String OUTPUT_PATH = "hdfs://10.1.130.10:9000/user/hadoop/output/";
 27     public static final String TMP_PATH = "hdfs://10.1.130.10:9000/user/hadoop/tmp/";
 28     public static final int TASK = 5;
 29     public static Logger logger = LogManager.getLogger(Cluster.class);
 30     private Configuration conf;
 31     private int iteration = 0;
 32     
 33     public Cluster() 
 34     {
 35         this.conf = new Configuration();
 36         conf.setInt("TASK", TASK);
 37     }
 38     
 39     
 40     public void run() throws IOException, InterruptedException, ClassNotFoundException
 41     {
 42         while (iteration < MAX) {
 43             logger.info("次数   ：  " + (iteration+1));
 44             conf.setInt("ITERATION", iteration);
 45             Job job = Job.getInstance(conf, "KMeans Process");
 46             if (iteration == 0) {
 47                 String cacheFile = TMP_PATH + iteration + "/part-r-0000";
 48                 for (int i = 0; i < TASK; i++)
 49                     job.addCacheFile(URI.create(cacheFile+i));
 50             }
 51             job.setJarByClass(KMeansProcess.class);
 52             job.setMapperClass(TextMapper.class);
 53             job.setNumReduceTasks(TASK);
 54             job.setReducerClass(TextReducer.class);
 55             job.setOutputKeyClass(IntWritable.class);
 56             job.setOutputValueClass(Text.class);
 57             iteration++;
 58             String outFile = TMP_PATH + iteration;
 59             FileInputFormat.addInputPath(job, new Path(INPUT_PATH));
 60             FileOutputFormat.setOutputPath(job, new Path(outFile));
 61             job.waitForCompletion(true);
 62             conf.unset("ITERATION");
 63             List<String> tmpList = getCenterList(outFile);
 64             if (KMeansProcess.TextReducer.center.compareCenters(KMeansProcess.TextMapper.centersList, tmpList)) 
 65                 break;
 66             else {
 67                 KMeansProcess.TextMapper.centersList.clear();
 68                 for (String s : tmpList) {
 69                     KMeansProcess.TextMapper.centersList.add(s);
 70                 }
 71             }
 72         }
 73     }
 74     
 75     
 76     public void lastRun() throws IOException, InterruptedException, ClassNotFoundException
 77     {
 78         String cacheFile = TMP_PATH + iteration + "/part-r-0000";
 79         Job job = Job.getInstance(conf, "KMeans");
 80         for (int i = 0; i < TASK; i++)
 81             job.addCacheFile(URI.create(cacheFile+i));
 82         job.setJarByClass(KMeans.class);
 83         job.setMapperClass(KMeansMapper.class);
 84         job.setReducerClass(KMeansReducer.class);
 85         job.setOutputKeyClass(IntWritable.class);
 86         job.setOutputValueClass(Text.class);    
 87         FileInputFormat.addInputPath(job, new Path(INPUT_PATH));
 88         FileOutputFormat.setOutputPath(job, new Path(OUTPUT_PATH));
 89         job.waitForCompletion(true);
 90     }
 91     
 92     
 93     public List<String> getCenterList(String outFile) 
 94     {
 95         List<String> centerList = new ArrayList<String>();
 96         String fileName = outFile + "/part-r-0000";
 97         try {
 98             for (int i = 0; i < TASK; i++) {
 99                 FileSystem fs = FileSystem.get(URI.create((fileName+i)), conf);
100                 FSDataInputStream is = fs.open(new Path((fileName+i).toString()));
101                 BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"));
102                 String s = null;
103                 while ((s = br.readLine()) != null)
104                     centerList.add(s);
105                 br.close();
106             }
107         } catch (IOException e) {
108             logger.info(e.getMessage());
109         }
110         
111         return centerList;
112     }
113         
114     
115     public static void main(String[] args) {
116         Cluster cluster = new Cluster();     
117         try {
118             long start = System.currentTimeMillis();
119             cluster.run();
120             cluster.lastRun();
121             long end = System.currentTimeMillis();
122             Cluster.logger.info(end-start);
123         } catch (ClassNotFoundException e) {
124             e.printStackTrace();
125         } catch (IOException e) {
126             e.printStackTrace();
127         } catch (InterruptedException e) {
128             e.printStackTrace();
129         }                
130     }
131 }

Class Cluster

　　鉴于在分布式环境下，多次迭代需要多次读取缓存文件，因此本文引入静态变量，以减少对TextCosine等初始化，以达到提升文本处理速率的目的。本文作者一直试图将对象实体传入Job中，但是经过多次实践，均以失败告终，若是有更好的解决方案，请联系我。

　　作者：志青云集
　　出处：http://www.cnblogs.com/lyssym
　　如果，您认为阅读这篇博客让您有些收获，不妨点击一下右下角的【推荐】。
　　如果，您希望更容易地发现我的新博客，不妨点击一下左下角的【关注我】。
　　如果，您对我的博客所讲述的内容有兴趣，请继续关注我的后续博客，我是【志青云集】。
　　本文版权归作者和博客园共有，欢迎转载，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文连接。

posted on 2015-11-12 23:56 志青云集阅读(2477) 评论(1) 编辑收藏举报

刷新页面返回顶部

志青云集