RunningMapReduceExampleTFIDF - hadoop-clusternet - This document describes how to run the TF-IDF MapReduce example against ascii books. - This project is for those who wants to experiment hadoop as a skunkworks in a small cluster (1-10 nodes) - Google Pro

RunningMapReduceExampleTFIDF - hadoop-clusternet - This document describes how to run the TF-IDF MapReduce example against ascii books. - This project is for those who wants to experiment hadoop as a skunkworks in a small cluster (1-10 nodes) - Google Project Hosting

Introduction

The first application written with Clusternet was an example to produce weights for terms in a corpus of ascii books. The example is developed using 3 steps to transform the results in separate MapReduce Jobs. This example can actually be run in any Hadoop cluster.

TF-IDF Algorithm

The term frequency, inverted document frequency is a well-known information retrieval algorithm for giving weights for terms of documents. It is important to note that the algorithm must not consider stop words, and for this reason, we will be using filtering the terms in the input for the algorithm with the Google Stop words. The algorithm is described as follows:

weight(t,d) = tf * idf
tf = # terms t in (d) / total # of terms in (d) = tf