mongo-mapreduce测试(9)——python调用

mongo-mapreduce测试(1)——count/sum/where条件

mongo-mapreduce测试(2)——列转行与finalize函数

mongo-mapreduce测试(3)——group by having

mongo-mapreduce测试(4)——avg

mongo-mapreduce测试(5)——max/min

mongo-mapreduce测试(6)——综合测试

mongo-mapreduce测试(7)——使用js存储过程

mongo-mapreduce测试(8)——阶段总结(1)

mongo-mapreduce测试(9)——python调用

mongo-mapreduce测试(10)——阶段总结(2)

mongo-mapreduce测试(11)——跟踪调试

可以在python中执行mapreduce(使用count功能进行测试)

1. 测试数据如下:
> db.tianyc_test3.find()
{ "_id" : ObjectId("51258e47ff13bc6383473137"), "name" : "xtt", "dic" : 1 }
{ "_id" : ObjectId("51258e49ff13bc6383473138"), "name" : "xtt", "dic" : 2 }
{ "_id" : ObjectId("51258e4bff13bc6383473139"), "name" : "xtt", "dic" : 3 }
{ "_id" : ObjectId("51258e4eff13bc638347313a"), "name" : "xtt", "dic" : 4 }
{ "_id" : ObjectId("51258e50ff13bc638347313b"), "name" : "xtt", "dic" : 5 }
{ "_id" : ObjectId("51258e5aff13bc638347313c"), "name" : "yct", "dic" : 1 }
{ "_id" : ObjectId("51258e5cff13bc638347313d"), "name" : "yct", "dic" : 2 }
{ "_id" : ObjectId("51258e62ff13bc638347313e"), "name" : "neu", "dic" : 1 }
{ "_id" : ObjectId("51258e64ff13bc638347313f"), "name" : "neu", "dic" : 2 }
{ "_id" : ObjectId("51258e65ff13bc6383473140"), "name" : "neu", "dic" : 3 }
{ "_id" : ObjectId("51258e68ff13bc6383473141"), "name" : "neu", "dic" : 4 }

2. 实现python调用

方法1:将MR的js脚本写在python字符串中。
>>> from pymongo import Connection
>>> from bson.code import Code
>>> connection = Connection('localhost', 20000)
>>> db = connection.test
>>>
>>> mapfun = Code("function () {emit(this.name, {cnt:1});}")
>>> reducefun = Code("""
... function(key, values) {
... var count = 0;
... for (var i=0;i<values.length;i++){
... count+=values[i].cnt;
... }
... return count;
... }
... """)
>>> result = db.tianyc_test3.map_reduce(mapfun, reducefun, "tianyc_test3_result_py")
>>> for doc in result.find():
... print doc
...
{u'_id': u'neu', u'value': 4.0}
{u'_id': u'xtt', u'value': 5.0}
{u'_id': u'yct', u'value': 2.0}

方法2:将MR的js脚本写在文件中,使用python打开文件进行调用。

 

参考:MapReduce with MongoDB and Python

posted @ 2013-02-25 11:12  醇酒醉影  阅读(1355)  评论(0编辑  收藏  举报