MongoDB聚合管道

通过上一篇文章中，认识了MongoDB中四个聚合操作，提供基本功能的count、distinct和group，还有可以提供强大功能的mapReduce。

在MongoDB的2.2版本以后，聚合框架中多了一个新的成员，聚合管道，数据进入管道后就会经过一级级的处理，直到输出。

对于数据量不是特别大，逻辑也不是特别复杂的聚合操作，聚合管道还是比mapReduce有很多优势的：

1. 相比mapReduce，聚合管道比较容易理解和使用
2. 可以直接使用管道表达式操作符，省掉了很多自定义js function，一定程度上提高执行效率
3. 和mapReduce一样，它也可以作用于分片集合

但是，对于数据量大，逻辑复杂的聚合操作，还是要使用mapReduce实现。

 

聚合管道

在聚合管道中，每一步操作（管道操作符）都是一个工作阶段（stage），所有的stage存放在一个array中。MongoDB文档中的描述如下：

db.collection.aggregate( [ { <stage> }, ... ] )

在聚合管道中，每一个stage都对应一个管道操作符，根据MongoDB文档，聚合管道可以支持以下管道操作符：

Name	Description
$geoNear	Returns an ordered stream of documents based on the proximity to a geospatial point. Incorporates the functionality of $match, $sort, and $limit for geospatial data. The output documents include an additional distance field and can include a location identifier field.
$group	Groups input documents by a specified identifier expression and applies the accumulator expression(s), if specified, to each group. Consumes all input documents and outputs one document per each distinct group. The output documents only contain the identifier field and, if specified, accumulated fields.
$limit	Passes the first n documents unmodified to the pipeline where n is the specified limit. For each input document, outputs either one document (for the first n documents) or zero documents (after the first n documents).
$match	Filters the document stream to allow only matching documents to pass unmodified into the next pipeline stage. $match uses standard MongoDB queries. For each input document, outputs either one document (a match) or zero documents (no match).
$out	Writes the resulting documents of the aggregation pipeline to a collection. To use the $out stage, it must be the last stage in the pipeline.
$project	Reshapes each document in the stream, such as by adding new fields or removing existing fields. For each input document, outputs one document.
$redact	Reshapes each document in the stream by restricting the content for each document based on information stored in the documents themselves. Incorporates the functionality of $project and $match. Can be used to implement field level redaction. For each input document, outputs either one or zero document.
$skip	Skips the first n documents where n is the specified skip number and passes the remaining documents unmodified to the pipeline. For each input document, outputs either zero documents (for the first n documents) or one document (if after the first n documents).
$sort	Reorders the document stream by a specified sort key. Only the order changes; the documents remain unmodified. For each input document, outputs one document.
$unwind	Deconstructs an array field from the input documents to output a document for each element. Each output document replaces the array with an element value. For each input document, outputs n documents where n is the number of array elements and can be zero for an empty array.

 

下面通过具体的例子看看聚合管道的基本用法：

首先，通过以下代码准备测试数据：

var dataList = [
    { "name" : "Will0", "gender" : "Female", "age" : 22 , "classes": ["MongoDB", "C#", "C++"]},
    { "name" : "Will1", "gender" : "Female", "age" : 20 , "classes": ["Node", "JavaScript"]},
    { "name" : "Will2", "gender" : "Male", "age" : 24 , "classes": ["Java", "WPF", "C#"]},
    { "name" : "Will3", "gender" : "Male", "age" : 23 , "classes": ["WPF", "C",]},
    { "name" : "Will4", "gender" : "Male", "age" : 21 , "classes": ["SQL", "HTML"]},
    { "name" : "Will5", "gender" : "Male", "age" : 20 , "classes": ["DOM", "CSS", "HTML5"]},
    { "name" : "Will6", "gender" : "Female", "age" : 20 , "classes": ["PPT", "Word", "Excel"]},
    { "name" : "Will7", "gender" : "Female", "age" : 24 , "classes": ["HTML5", "C#"]},
    { "name" : "Will8", "gender" : "Male", "age" : 21 , "classes": ["Java", "VB", "BASH"]},
    { "name" : "Will9", "gender" : "Female", "age" : 24 , "classes": ["CSS"]}
]              

for(var i = 0; i < dataList.length; i++){
    db.school.students.insert(dataList[i]);
}

 

$project

$project主要用于数据投影，实现字段的重命名、增加和删除。下面例子中，重命名了"name"字段，增加了"birthYear"字段，删除了"_id"字段。

> db.runCommand({
...     "aggregate": "school.students",
...     "pipeline": [
...                         {"$match": {"age": {"$lte": 22}}},
...                         {"$project": {"_id": 0, "studentName": "$name", "gender": 1, "birthYear": {"$subtract": [2014, "$age"]}}},
...                         {"$sort": {"birthYear":1}}
...                     ]
... })
{
        "result" : [
                {
                        "gender" : "Female",
                        "studentName" : "Will0",
                        "birthYear" : 1992
                },
                {
                        "gender" : "Male",
                        "studentName" : "Will4",
                        "birthYear" : 1993
                },
                {
                        "gender" : "Male",
                        "studentName" : "Will8",
                        "birthYear" : 1993
                },
                {
                        "gender" : "Female",
                        "studentName" : "Will1",
                        "birthYear" : 1994
                },
                {
                        "gender" : "Male",
                        "studentName" : "Will5",
                        "birthYear" : 1994
                },
                {
                        "gender" : "Female",
                        "studentName" : "Will6",
                        "birthYear" : 1994
                }
        ],
        "ok" : 1
}
>

 

$unwind

$unwind用来将数组拆分为独立字段。

> db.runCommand({
...     "aggregate": "school.students",
...     "pipeline": [
...                         {"$match": {"age": 23}},
...                         {"$project": {"name": 1, "gender": 1, "age": 1, "classes": 1}},
...                         {"$unwind": "$classes"}
...                     ]
... })
{
        "result" : [
                {
                        "_id" : ObjectId("54805220e31c9e1578ed0ccc"),
                        "name" : "Will3",
                        "gender" : "Male",
                        "age" : 23,
                        "classes" : "WPF"
                },
                {
                        "_id" : ObjectId("54805220e31c9e1578ed0ccc"),
                        "name" : "Will3",
                        "gender" : "Male",
                        "age" : 23,
                        "classes" : "C"
                }
        ],
        "ok" : 1
}
>

 

$group

跟单个的group用法类似，用作分组操作。

使用$group时，必须要指定一个_id域，同时也可以包含一些算术类型的表达式操作符。

下面的例子就是用聚合管道实现得到男生和女生的平均年龄。

> db.runCommand({
...     "aggregate": "school.students",
...     "pipeline": [
...                     {"$group":{"_id":"$gender", "avg": { "$avg": "$age" } }}
...                  ]
... })
{
        "result" : [
                {
                        "_id" : "Male",
                        "avg" : 21.8
                },
                {
                        "_id" : "Female",
                        "avg" : 22
                }
        ],
        "ok" : 1
}
>

 

查询男生和女生的最大年龄

> db.runCommand({
...     "aggregate": "school.students",
...     "pipeline": [
...                         {"$group": {"_id": "$gender", "max": {"$max": "$age"}}}
...                     ]
... })
{
        "result" : [
                {
                        "_id" : "Male",
                        "max" : 24
                },
                {
                        "_id" : "Female",
                        "max" : 24
                }
        ],
        "ok" : 1
}
>

 

 

管道操作符、管道表达式和表达式操作符

上面的例子中，已经接触了这几个概念，下面进行进一步介绍。

管道操作符作为"键"，所对应的"值"叫做管道表达式，例如"{"$group": {"_id": "$gender", "max": {"$max": "$age"}}}"

• $group是管道操作符
• {"_id": "$gender", "max": {"$max": "$age"}}是管道表达式
• $gender在文档中叫做"field path"，通过"$"符号来获得特定字段，是管道表达式的一部分
• $max是表达式操作符，正是因为聚合管道中提供了很多表达式操作符，我们可以省去很多自定义js函数

下面列出了常用的表达式操作符，更详细的信息，请参阅MongoDB文档

• Boolean Expressions
  • $and, $or, $ not
• Comparison Expressions
  • $cmp, $eq, $gt, $gte, $lt, $lte, $ne
• Arithmetic Expressions
  • $add, $divide, $mod, $multiply, $subtract
• String Expressions
  • $concat, $strcasecmp, $substr, $toLower, $toUpper

 

总结

聚合管道提供了一种mapReduce 的替代方案，mapReduce使用相对来说比较复杂，而聚合管道的拥有固定的接口，一系列可选择的表达式操作符，对于大多数的聚合任务，聚合管道一般来说是首选方法。

Ps: 文章中使用的例子可以通过以下链接查看

https://files.cnblogs.com/wilber2013/pipeline.js