hello world!!!!!

写下自己的一些心得,写下自己问题的方式,写下程序之路的艰辛,希望能够有朝一日成为大牛。
  博客园  :: 首页  :: 新随笔  :: 联系 :: 订阅 订阅  :: 管理

Combiner—local reduce (在本地reduce,最后再做一次reduce)

Posted on 2011-12-17 02:35  陈力  阅读(236)  评论(0编辑  收藏  举报

In many situations with MapReduce applications, we may wish to perform a “local
reduce ” before we distribute the mapper results. Consider the WordCount example of

 

The MapReduce data flow, with an emphasis on partitioning and
shuffling. Each icon is a key/value pair. The shapes represents keys, whereas
the inner patterns represent values. After shuffling, all icons of the same shape
(key) are in the same reducer. Different keys can go to the same reducer, as seen
in the rightmost reducer. The partitioner decides which key goes where. Note that
the leftmost reducer has more load due to more data under the “ellipse” key.

 

 

3.2.6 Word counting with predefined mapper and reducer classes

3.3 Reading and writing