python版 mapreduce 矩阵相乘
参考张老师的mapreduce 矩阵相乘。
转载请注明：来自chybot的学习笔记http://i.cnblogs.com/EditPosts.aspx?postid=4541939
下面是我用python版本的mapreduce 矩阵相乘。
矩阵相乘的思路详见张老师的博客，对于两个矩阵m1和m2，mapreduce的计算过程如下：
这里面最主要的地方是key的构成，map输出的key是相乘后的矩阵的下标，比如c[i][j] = sum(A[i][:]*B[:][j])。
注意：该实现知识矩阵相乘的一个思路的实现，并不适合真实场景，这里面map task只能为2(对应两个输入矩阵的文件)，reduce task只能为1。
主要原因是由于这里面每个map程序都使用了全局变量，而每个reduce程序则默认矩阵相乘结果所需的值均在一个分片。
输入文件：
matrixA.txt
A#-1,0,2 A#1,3,1
matrixB.txt
B#3,1 B#2,1 B#1,0
maper程序:
#!/usr/bin/python # -*-coding:utf-8 -*- import sys rowNum = 2 colNum = 2 rowIndexA = 1 rowIndexB = 1 def read_inputdata(splitstr): for line in sys.stdin:
#分割出矩阵名和矩阵的一行元素 yield line.split(splitstr) if __name__ == '__main__': for matrix, matrixline in read_inputdata('#'): if matrix == 'A':
# 分割出矩阵元素(使用,分隔)，并用key，value输出 for i in range(rowNum): key = str(rowIndexA) + ',' + str(i+1) value = matrix + ':' j = 1 for element in matrixline.split(','): print '%s\t%s%s,%s' % (key, value, j, element) j += 1 rowIndexA += 1 elif matrix == 'B': for i in range(colNum): value = matrix + ':' j = 1 for element in matrixline.split(','): print '%s,%s\t%s%s,%s' % (i+1, j, value, rowIndexB, element) j = j+1 rowIndexB += 1 else: continue
reduce程序：
#!/usr/bin/python # -*- coding:utf-8 -*- import sys from itertools import groupby from operator import itemgetter def read_input(splitstr): for line in sys.stdin: line = line.strip() if len(line) == 0: continue yield line.split(splitstr, 1) def run(): data = read_input('\t') for current_element, group in groupby(data, itemgetter(0)): try: matrixA = {} matrixB = {} result = 0
#获取A的一行和b的一列 for current_element, elements in group: matrix, index_value = elements.split(':') index, value = index_value.split(',') if matrix == 'A': matrixA[index] = int(value) else: matrixB[index] = int(value)
#计算相乘结果，注意一定要用下标，如果依赖mapreduce的sort可能会出错 for key in matrixA: result += matrixA[key]*matrixB[key] print '%s\t%s' % (current_element, result) except Exception: pass if __name__ == '__main__': run()
本地测试是否可行：
bogon:program xuguoqiang$ cat matrixA.txt matrixB.txt |python matrix_mapper.py |sort |python matrix_reducer.py
1,1 -1
1,2 -1
2,1 10
2,2 4
使用hadoop streaming 运行mapred程序，结果如下：
bogon:hadoop-1.2.1 xuguoqiang$ bin/hadoop jar contrib/streaming/hadoop-streaming-1.2.1.jar -D mapred.map.tasks=2 -D mapred.reduce.tasks=1 \ > -mapper /Users/xuguoqiang/hadoop-1.2.1/program/matrix_mapper.py \ > -reducer /Users/xuguoqiang/hadoop-1.2.1/program/matrix_reducer.py \ > -input /matrix/* \ > -output output5 packageJobJar: [/tmp/hadoop-xuguoqiang/hadoop-unjar2547149142116420858/] [] /var/folders/7_/jmj1yhgx7b1_2cg9w74h0q5r0000gn/T/streamjob1502134034482177499.jar tmpDir=null 15/05/31 16:37:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/05/31 16:37:06 WARN snappy.LoadSnappy: Snappy native library not loaded 15/05/31 16:37:06 INFO mapred.FileInputFormat: Total input paths to process : 2 15/05/31 16:37:06 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-xuguoqiang/mapred/local] 15/05/31 16:37:06 INFO streaming.StreamJob: Running job: job_201505311232_0019 15/05/31 16:37:06 INFO streaming.StreamJob: To kill this job, run: 15/05/31 16:37:06 INFO streaming.StreamJob: /Users/xuguoqiang/hadoop-1.2.1/libexec/../bin/hadoop job -Dmapred.job.tracker=hdfs://localhost:9001 -kill job_201505311232_0019 15/05/31 16:37:06 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201505311232_0019 15/05/31 16:37:07 INFO streaming.StreamJob: map 0% reduce 0% 15/05/31 16:37:11 INFO streaming.StreamJob: map 100% reduce 0% 15/05/31 16:37:20 INFO streaming.StreamJob: map 100% reduce 100% 15/05/31 16:37:22 INFO streaming.StreamJob: Job complete: job_201505311232_0019 15/05/31 16:37:22 INFO streaming.StreamJob: Output: output5 bogon:hadoop-1.2.1 xuguoqiang$ bin/hadoop fs -cat output5/* 1,1 -1 1,2 -1 2,1 10 2,2 4
可以看出，结果和在本地运行结果是相同的。
二、稀疏矩阵乘法
稀疏矩阵和矩阵乘法思想类似，只不过把之前一行的数据变成了多行来体现。
输入：
矩阵A
A#1,1,1 A#1,4,3 A#2,1,2 A#2,2,5 A#2,4,4 A#3,4,1 A#4,1,4 A#4,2,7 A#4,3,1 A#4,4,2
矩阵B
B#1,1,5 B#2,2,2 B#4,1,3 B#4,2,1
mapper程序：
#!/usr/bin/python # -*-coding:utf-8 -*- import sys rowNum = 2 colNum = 4 def read_inputdata(splitstr): for line in sys.stdin: yield line.strip().split(splitstr) if __name__ == '__main__': for matrix, matrixline in read_inputdata('#'): if matrix == 'A': for i in range(rowNum): index1, index2, element = matrixline.split(',') print '%s,%s\t%s:%s,%s' % (index1, (i+1), matrix, index2, element) elif matrix == 'B': for i in range(colNum): index1, index2, element = matrixline.split(',') print '%s,%s\t%s:%s,%s' % (i+1, index2, matrix,index1, element) else: continue
reduce程序：
#!/usr/bin/python # -*- coding:utf-8 -*- import sys from itertools import groupby from operator import itemgetter def read_input(splitstr): for line in sys.stdin: line = line.strip() if len(line) == 0: continue yield line.split(splitstr, 1) def run(): data = read_input('\t') for current_element, group in groupby(data, itemgetter(0)): try: matrixA = {} matrixB = {} result = 0 for current_element, elements in group: matrix, index_value = elements.split(':') index, value = index_value.split(',') if matrix == 'A': matrixA[index] = int(value) else: matrixB[index] = int(value) for key in matrixA: if key in matrixB: result += matrixA[key]*matrixB[key] print '%s\t%s' % (current_element, result) except Exception: pass if __name__ == '__main__': run()
本地程序测试结果：
bogon:program xuguoqiang$ cat sparsematrixB.txt sparsematrixA.txt | python sparsematrix_mapper.py |sort |python sparsematrix_reduce.py
1,1 14
1,2 3
2,1 22
2,2 14
3,1 3
3,2 1
4,1 26
4,2 16
hadoop测试结果：
bogon:hadoop-1.2.1 xuguoqiang$ bin/hadoop jar contrib/streaming/hadoop-streaming-1.2.1.jar -D mapred.map.tasks=2 -D mapred.reduce.tasks=1 -mapper /Users/xuguoqiang/hadoop-1.2.1/program/sparsematrix_mapper.py -reducer /Users/xuguoqiang/hadoop-1.2.1/program/sparsematrix_reduce.py -input /sparsematrix/* -output output packageJobJar: [/tmp/hadoop-xuguoqiang/hadoop-unjar2334049571009138288/] [] /var/folders/7_/jmj1yhgx7b1_2cg9w74h0q5r0000gn/T/streamjob7964024689233782754.jar tmpDir=null 15/05/31 16:31:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/05/31 16:31:11 WARN snappy.LoadSnappy: Snappy native library not loaded 15/05/31 16:31:11 INFO mapred.FileInputFormat: Total input paths to process : 2 15/05/31 16:31:11 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-xuguoqiang/mapred/local] 15/05/31 16:31:11 INFO streaming.StreamJob: Running job: job_201505311232_0018 15/05/31 16:31:11 INFO streaming.StreamJob: To kill this job, run: 15/05/31 16:31:11 INFO streaming.StreamJob: /Users/xuguoqiang/hadoop-1.2.1/libexec/../bin/hadoop job -Dmapred.job.tracker=hdfs://localhost:9001 -kill job_201505311232_0018 15/05/31 16:31:11 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201505311232_0018 15/05/31 16:31:12 INFO streaming.StreamJob: map 0% reduce 0% 15/05/31 16:31:16 INFO streaming.StreamJob: map 67% reduce 0% 15/05/31 16:31:19 INFO streaming.StreamJob: map 100% reduce 0% 15/05/31 16:31:25 INFO streaming.StreamJob: map 100% reduce 33% 15/05/31 16:31:26 INFO streaming.StreamJob: map 100% reduce 100% 15/05/31 16:31:27 INFO streaming.StreamJob: Job complete: job_201505311232_0018 15/05/31 16:31:27 INFO streaming.StreamJob: Output: output
刚开始学习hadoop，加油！坚持！希望同道的人能给出建议。
参考：
粉丝日志：http://blog.fens.me/hadoop-mapreduce-matrix/
