MIT-6.824 lab1

github：https://github.com/haoweiz/MIT-6.824

Part1：

　　第一部分比较简单，我们只需要修改doMap和doReduce函数即可，主要涉及Go语言对Json文件的读写。简单说说part1的测试流程吧，Sequential部分代码如下

func TestSequentialSingle(t *testing.T) {
    mr := Sequential("test", makeInputs(1), 1, MapFunc, ReduceFunc)
    mr.Wait()
    check(t, mr.files)
    checkWorker(t, mr.stats)
    cleanup(mr)
}

func TestSequentialMany(t *testing.T) {
    mr := Sequential("test", makeInputs(5), 3, MapFunc, ReduceFunc)
    mr.Wait()
    check(t, mr.files)
    checkWorker(t, mr.stats)
    cleanup(mr)

　　makeInputs（M int）对于0-100000个数字平均分成了M个文件写入，，根据题目要求，我们需要将每个文件分成N个文件写出，因此doMap过程总共产生M*N个文件，我们可以先将文件的所有键值对通过mapF函数（本质上是test_test.go中的MapFunc函数）存储在数组keyvalue中，然后创建nReduce个文件并对每个键值对用ihash()%nReduce得到该键值对应存储的文件位置，利用Encoder写入文件即可

// doMap manages one map task: it reads one of the input files
// (inFile), calls the user-defined map function (mapF) for that file's
// contents, and partitions the output into nReduce intermediate files.
func doMap(
    jobName string, // the name of the MapReduce job
    mapTaskNumber int, // which map task this is
    inFile string,
    nReduce int, // the number of reduce task that will be run ("R" in the paper)
    mapF func(file string, contents string) []KeyValue,
) {
    //
    // You will need to write this function.
    //
    // The intermediate output of a map task is stored as multiple
    // files, one per destination reduce task. The file name includes
    // both the map task number and the reduce task number. Use the
    // filename generated by reduceName(jobName, mapTaskNumber, r) as
    // the intermediate file for reduce task r. Call ihash() (see below)
    // on each key, mod nReduce, to pick r for a key/value pair.
    //
    // mapF() is the map function provided by the application. The first
    // argument should be the input file name, though the map function
    // typically ignores it. The second argument should be the entire
    // input file contents. mapF() returns a slice containing the
    // key/value pairs for reduce; see common.go for the definition of
    // KeyValue.
    //
    // Look at Go's ioutil and os packages for functions to read
    // and write files.
    //
    // Coming up with a scheme for how to format the key/value pairs on
    // disk can be tricky, especially when taking into account that both
    // keys and values could contain newlines, quotes, and any other
    // character you can think of.
    //
    // One format often used for serializing data to a byte stream that the
    // other end can correctly reconstruct is JSON. You are not required to
    // use JSON, but as the output of the reduce tasks *must* be JSON,
    // familiarizing yourself with it here may prove useful. You can write
    // out a data structure as a JSON string to a file using the commented
    // code below. The corresponding decoding functions can be found in
    // common_reduce.go.
    //
    //   enc := json.NewEncoder(file)
    //   for _, kv := ... {
    //     err := enc.Encode(&kv)
    //
    // Remember to close the file after you have written all the values!
    //
    
    // Read from inFile and save all keys and values in keyvalue 
    var keyvalue []KeyValue
    fi, err := os.Open(inFile)
    if err != nil {
        log.Fatal("doMap Open: ", err)
    }
    defer fi.Close()
    br := bufio.NewReader(fi)
    for {
        a, _, c := br.ReadLine()
        if c == io.EOF {
            break
        }
        kv := mapF(inFile, string(a))
        for i := 0; i != len(kv); i++ {
            keyvalue = append(keyvalue, kv[i])
        }
    }

    // Create nReduce files and create encoder for each of them
    var names []string
    files := make([]*os.File, 0, nReduce)
    enc := make([]*json.Encoder, 0, nReduce)
    for r := 0; r != nReduce; r++ {
        names = append(names, fmt.Sprintf("mrtmp.%s-%d-%d", jobName, mapTaskNumber, r))
        file, err := os.Create(names[r])
        if err != nil {
            log.Fatal("doMap Create: ", err)
        }
        files = append(files, file)
        enc = append(enc, json.NewEncoder(file))
    }
    
    // Choose which file to store for each keyvalue
    for _, kv := range keyvalue {
        index := ihash(kv.Key)%nReduce
        enc[index].Encode(kv)
    }

    // Close all files
    for _, f := range files {
        f.Close()
    }
}

 

　　对于doReduce函数我们需要读取nMap个文件，将所有键值对解码并重新编码写出到outFile中

// doReduce manages one reduce task: it reads the intermediate
// key/value pairs (produced by the map phase) for this task, sorts the
// intermediate key/value pairs by key, calls the user-defined reduce function
// (reduceF) for each key, and writes the output to disk.
func doReduce(
    jobName string, // the name of the whole MapReduce job
    reduceTaskNumber int, // which reduce task this is
    outFile string, // write the output here
    nMap int, // the number of map tasks that were run ("M" in the paper)
    reduceF func(key string, values []string) string,
) {
    //
    // You will need to write this function.
    //
    // You'll need to read one intermediate file from each map task;
    // reduceName(jobName, m, reduceTaskNumber) yields the file
    // name from map task m.
    //
    // Your doMap() encoded the key/value pairs in the intermediate
    // files, so you will need to decode them. If you used JSON, you can
    // read and decode by creating a decoder and repeatedly calling
    // .Decode(&kv) on it until it returns an error.
    //
    // You may find the first example in the golang sort package
    // documentation useful.
    //
    // reduceF() is the application's reduce function. You should
    // call it once per distinct key, with a slice of all the values
    // for that key. reduceF() returns the reduced value for that key.
    //
    // You should write the reduce output as JSON encoded KeyValue
    // objects to the file named outFile. We require you to use JSON
    // because that is what the merger than combines the output
    // from all the reduce tasks expects. There is nothing special about
    // JSON -- it is just the marshalling format we chose to use. Your
    // output code will look something like this:
    //
    // enc := json.NewEncoder(file)
    // for key := ... {
    //     enc.Encode(KeyValue{key, reduceF(...)})
    // }
    // file.Close()
    //

    // Read all mrtmp.xxx-m-reduceTaskNumber and write to outFile
    var names []string
    file, err := os.Create(outFile)
    if err != nil {
        log.Fatal("doReduce Create: ", err)
    }
    enc := json.NewEncoder(file)
    defer file.Close()

    // Read all contents from mrtmp.xxx-m-reduceTaskNumber
    kvs := make(map[string][]string)
    for m := 0; m != nMap; m++ {
        names = append(names,  fmt.Sprintf("mrtmp.%s-%d-%d", jobName, m, reduceTaskNumber))
        fi, err := os.Open(names[m])
        if err != nil {
            log.Fatal("doReduce Open: ", err)
        }
        dec := json.NewDecoder(fi)
        for {
            var kv KeyValue
            err = dec.Decode(&kv)
            if err != nil {
                break
            }
            kvs[kv.Key] = append(kvs[kv.Key], kv.Value)
        }
        fi.Close()
    }
    for k, v := range kvs {
        enc.Encode(KeyValue{k, reduceF(k, v)})
    }
}

　　通过测试

 

 

Part2：

　　第二部分建立在第一部分的基础上，要进行词频统计，就很简单，对于mapF函数我们产生键值对，对于reduceF函数我们返回值出现的次数即可

//
// The map function is called once for each file of input. The first
// argument is the name of the input file, and the second is the
// file's complete contents. You should ignore the input file name,
// and look only at the contents argument. The return value is a slice
// of key/value pairs.
//
func mapF(filename string, contents string) []mapreduce.KeyValue {
    // TODO: you have to write this function
    f := func(c rune) bool {
        return !unicode.IsLetter(c)
    }
    words := strings.FieldsFunc(contents, f)
    var keyvalue []mapreduce.KeyValue
    for _, word := range words {
        keyvalue = append(keyvalue, mapreduce.KeyValue{word,""})
    }
    return keyvalue
}

//
// The reduce function is called once for each key generated by the
// map tasks, with a list of all the values created for that key by
// any map task.
//
func reduceF(key string, values []string) string {
    // TODO: you also have to write this function
    return strconv.Itoa(len(values))
}

　　通过测试

 

Part3：

　　这一部分有点难，主要是对Go的并发机制要有一些了解才能做，阅读源代码，这个测试将100000以内的数字生成了100个文件，即100个Map任务，并产生50个Reduce任务，同时产生了两个worker，当全部map结束后进入reduce，所以schedule被调用了两次，先整体把握下该怎么写，由于这次需要分布式计算，所以我们可以分配ntasks个go程每次从registerChan中得到一个空闲的worker后调用call函数分配任务，然后结束后重新将其放到registerChan中（这里是第一个小坑，原本以为call函数调用结束后会自动将worker重新放入registerChan中，后来发现registerChan根本没有作为参数传入call函数，自然不能实现这个操作）

func TestBasic(t *testing.T) {
    mr := setup()
    for i := 0; i < 2; i++ {
        go RunWorker(mr.address, port("worker"+strconv.Itoa(i)),
            MapFunc, ReduceFunc, -1)
    }
    mr.Wait()
    check(t, mr.files)
    checkWorker(t, mr.stats)
    cleanup(mr)
}

　　阅读文档可知这一部分的核心是call函数，用于任务的分配。

// call() sends an RPC to the rpcname handler on server srv
// with arguments args, waits for the reply, and leaves the
// reply in reply. the reply argument should be the address
// of a reply structure.
//
// call() returns true if the server responded, and false
// if call() was not able to contact the server. in particular,
// reply's contents are valid if and only if call() returned true.
//
// you should assume that call() will time out and return an
// error after a while if it doesn't get a reply from the server.
//
// please use call() to send all RPCs, in master.go, mapreduce.go,
// and worker.go.  please don't change this function.
//
func call(srv string, rpcname string,
    args interface{}, reply interface{}) bool {
    c, errx := rpc.Dial("unix", srv)
    if errx != nil {
        return false
    }
    defer c.Close()

    err := c.Call(rpcname, args, reply)
    if err == nil {
        return true
    }

    fmt.Println(err)
    return false
}

　　首先我们需要明确四个参数的值，文档中说了，第一个参数从registerChan中获得，第二个参数是给定的字符串"Worker.DoTask"，第四个参数是nil，第三个参数是个结构体DoTaskArgs，看看这个结构体长啥样

// What follows are RPC types and methods.
// Field names must start with capital letters, otherwise RPC will break.

// DoTaskArgs holds the arguments that are passed to a worker when a job is
// scheduled on it.
type DoTaskArgs struct {
    JobName    string
    File       string   // only for map, the input file
    Phase      jobPhase // are we in mapPhase or reducePhase?
    TaskNumber int      // this task's index in the current phase

    // NumOtherPhase is the total number of tasks in other phase; mappers
    // need this to compute the number of output bins, and reducers needs
    // this to know how many input files to collect.
    NumOtherPhase int
}

　　再对照schedule的参数赋值即可，所有参数均有对应。基本上了解了以上几点可以得到下面的代码，和上面的思路一样，创建ntasks个go程，然后将doTaskArgs参数设置好，连同registerChan一并传入，一旦channel中有空闲的worker，随机找一个被阻塞的go程运行即可，因为都是并发的，所以谁先执行都没关系，然后用waitGroup阻塞主程，看起来很完美，但运行一半时会卡住，这是这个部分最大的坑，现在来分析下原因。

　　咱们先反复运行一下测试，发现得到下面的信息，仔细观察可以发现，这90和91号任务已经是最后两个任务了，为什么到最后会卡住，这是因为channel是阻塞式的（Distributed函数中定义channel时并未为其分配缓存），当倒数第二个任务完成后重新加到channel中时，已经没有go程会取出这个worker了，此时最后一个go程再想将worker添加到channel中时就会阻塞，所以这个代码会阻塞在最后一个go程的下面代码的第八行。

 

    var wg sync.WaitGroup
    wg.Add(ntasks)
    for i := 0; i != ntasks; i++ {
        doTaskArgs := DoTaskArgs{jobName, mapFiles[i] , phase, i, n_other}
        go func(doTaskArgs DoTaskArgs, registerChan chan string) {
            address := <-registerChan
            call(address, "Worker.DoTask", doTaskArgs, nil)
            registerChan <- address
            wg.Done()
            return
        }(doTaskArgs, registerChan)
    }
    wg.Wait()

　　最简单的做法就是为第八行创建一个新协程，这样最后一个任务虽然无法添加到channel中，但由于它是另一个go程，所以当主程退出后，这个子程虽然还阻塞着，但也会强制退出。修改方式修改后得到如下代码

//
// schedule() starts and waits for all tasks in the given phase (Map
// or Reduce). the mapFiles argument holds the names of the files that
// are the inputs to the map phase, one per map task. nReduce is the
// number of reduce tasks. the registerChan argument yields a stream
// of registered workers; each item is the worker's RPC address,
// suitable for passing to call(). registerChan will yield all
// existing registered workers (if any) and new ones as they register.
//
func schedule(jobName string, mapFiles []string, nReduce int, phase jobPhase, registerChan chan string) {
    var ntasks int
    var n_other int // number of inputs (for reduce) or outputs (for map)
    switch phase {
    case mapPhase:
        ntasks = len(mapFiles)
        n_other = nReduce
    case reducePhase:
        ntasks = nReduce
        n_other = len(mapFiles)
    }

    fmt.Printf("Schedule: %v %v tasks (%d I/Os)\n", ntasks, phase, n_other)

    // All ntasks tasks have to be scheduled on workers, and only once all of
    // them have been completed successfully should the function return.
    // Remember that workers may fail, and that any given worker may finish
    // multiple tasks.
    //
    // TODO TODO TODO TODO TODO TODO TODO TODO TODO TODO TODO TODO TODO
    //
    var wg sync.WaitGroup
    wg.Add(ntasks)
    for i := 0; i != ntasks; i++ {
        doTaskArgs := DoTaskArgs{jobName, mapFiles[i] , phase, i, n_other}
        go func(doTaskArgs DoTaskArgs, registerChan chan string) {
            address := <-registerChan
            call(address, "Worker.DoTask", doTaskArgs, nil)
            go func() {
                registerChan <- address
            }()
            wg.Done()
            return
        }(doTaskArgs, registerChan)
    }
    wg.Wait()
    fmt.Printf("Schedule: %v phase done\n", phase)
}

 　　通过测试

 

Part4：

　　有了第三部分铺垫，这一部分就非常简单了，如果worker失效，那么call函数会返回false，此时我们需要从channel中获得下一个worker直到call返回true为止，因此将获取worker和call函数放在一个for循环中即可。

//
// schedule() starts and waits for all tasks in the given phase (Map
// or Reduce). the mapFiles argument holds the names of the files that
// are the inputs to the map phase, one per map task. nReduce is the
// number of reduce tasks. the registerChan argument yields a stream
// of registered workers; each item is the worker's RPC address,
// suitable for passing to call(). registerChan will yield all
// existing registered workers (if any) and new ones as they register.
//
func schedule(jobName string, mapFiles []string, nReduce int, phase jobPhase, registerChan chan string) {
    var ntasks int
    var n_other int // number of inputs (for reduce) or outputs (for map)
    switch phase {
    case mapPhase:
        ntasks = len(mapFiles)
        n_other = nReduce
    case reducePhase:
        ntasks = nReduce
        n_other = len(mapFiles)
    }

    fmt.Printf("Schedule: %v %v tasks (%d I/Os)\n", ntasks, phase, n_other)

    // All ntasks tasks have to be scheduled on workers, and only once all of
    // them have been completed successfully should the function return.
    // Remember that workers may fail, and that any given worker may finish
    // multiple tasks.
    //
    // TODO TODO TODO TODO TODO TODO TODO TODO TODO TODO TODO TODO TODO
    //
    var wg sync.WaitGroup
    wg.Add(ntasks)
    for i := 0; i != ntasks; i++ {
        doTaskArgs := DoTaskArgs{jobName, mapFiles[i] , phase, i, n_other}
        go func(doTaskArgs DoTaskArgs, registerChan chan string) {
            success := false
            var address string
            for success==false {
                address = <-registerChan
                success = call(address, "Worker.DoTask", doTaskArgs, nil)
            }
            go func() {
                registerChan <- address
            }()
            wg.Done()
        }(doTaskArgs, registerChan)
    }
    wg.Wait()
    fmt.Printf("Schedule: %v phase done\n", phase)
}

　　通过测试

 

Part5：

　　第五部分让我们实现一个inverted index，说白了就是每个出现过的单词->所有出现过该单词的文件，所以map过程输出单词->文件名，reduce过程构建所有独一无二的文件个数（利用哈希表，因为一个单词可能在同一文件出现多次，所以要去除重复文件名）以及所有文件名组成的长字符串（注意需要进行排序）。注意那个源代码上的注释有点误导人，忽略即可。

// The mapping function is called once for each piece of the input.
// In this framework, the key is the name of the file that is being processed,
// and the value is the file's contents. The return value should be a slice of
// key/value pairs, each represented by a mapreduce.KeyValue.
func mapF(document string, value string) (res []mapreduce.KeyValue) {
    // TODO: you should complete this to do the inverted index challenge
    f := func(c rune) bool {
        return !unicode.IsLetter(c)
    }
    words := strings.FieldsFunc(value, f)
    for _, word := range words {
        res = append(res, mapreduce.KeyValue{word, document})
    }
    return
}

// The reduce function is called once for each key generated by Map, with a
// list of that key's string value (merged across all inputs). The return value
// should be a single output value for that key.
func reduceF(key string, values []string) string {
    // TODO: you should complete this to do the inverted index challenge
    m := make(map[string]string)
    for _, value := range values {
        m[value] = value
    }
    var uniqueValues []string
    for _, value := range m {
        uniqueValues = append(uniqueValues, value)
    }
    sort.Strings(uniqueValues)
    s := strconv.Itoa(len(m))
    s += " "
    s += uniqueValues[0]
    for i := 1; i != len(uniqueValues); i++ {
        s += ","
        s += uniqueValues[i]
    }
    return s
}

　　通过所有测试