NoSQL Databases - MongoDB

MongoDB被称为最象RDBMS的NoSQL, 确实是, 因为跟RDBMS相比, 它最大的改动其实就是在数据模型上有所不同.
其次就是MongoDB不支持事务, MongoDB只支持单文件的原子性修改, 和HBase的单row原子性修改一个级别

MongoDB使用Bson(类Json的二进制版), 即灵活的文档类型, 以树型的嵌套关系来替代严格范式的二维关系.
好处,
便于web场景, 即很多嵌套数据的case, 使用mysql需要把嵌套关系拆分成多个二维表, 而使用MongoDB可以直接将字典数据(比如,从一个page爬取下来的所有数据)直接存成文档, 很方便
方便的改变DB Schema, 不会影响之前的数据
可以较好的支持大文件存储, GridFS

文档类型, 等同于字典类型(Python), 在python中, 可用很方便的在json文件和字典之间进行load和store. 他应该算是Key-Value的最复杂的形式, KV –> Sorted KV –> Column based KV(HBase) –> Json
当然文档类型并不是MongoDB发明的, 在Lucene API中, 文档就是以这种形式存储的, 所以有人说 Search Index是document DB的一种特殊形式, 只是他们的索引方式不一样, search Index用的inverted index以用于search场景. 而MongoDB用的是和RDBMS完全相同的索引实现, B+ Tree, 来对于document里面的fields经行索引.

 

除了数据模型外, MongoDB在水平扩展和备份方面也做了不少工作, 不过似乎在auto-sharding上面还有不少问题.

Replica and Sharding

对于Replica, MongoDB除了可以使用最传统的M/S方式, 还可用使用Replica Sets机制, 这个机制就是单Master多Slaver机制, 并通过Paxos协议, 支持新Master的选举(当master dead的时候).

而且MongoDB在这儿采取了保守设计, 对于Replica Sets,

不支持多master, 而且写操作必须通过master, 所以不存在并发写冲突, 牺牲可用性, 来获取一致性和设计的简单性

不像CouchDB支持多数据版本, 而采用传统的Update来对应数据变化, 而Update的效率不会很高, 尤其不在内存中时

对于sharding, 是MongoDB一直引以为豪的功能, 相对于传统RDBMS的基于手动的水平sharding, MongoDB支持自动Sharding(只需指明依赖的field), 自动的load balance, 自动的Failover. 并且Sharding机制也是基于replica sets, 每个shard都是一个独立的replica set.

但是个人认为, 这块和RDBMS所采取的方式, 并没有本质的不同, 对于Mysql, 也可以采用这样的方式进行replica, 和进行sharding.
只是说MongoDB在这块做了不少提高, 以方便用户使用.

 

在同样牺牲复杂的关系型操作(Join)和复杂的trasaction的情况下, MongoDB比使用MySQL Cluster具有更好的数据模型的扩展性和水平数据扩展性(auto sharding), 更好的读写效率(足够多的ram), 并且方便部署,学习门槛低容易上手(但不支持SQL).

所以MongoDB, 非常适合如下的场景

1. 对NoSQL刚刚入门, 想尝试从MySQL转向NoSQL

2. 对关系型事务要求不太高, 并想以最小代价提升MySQL的扩展性的场景

3. Web领域的快速开发, schema经常变化, 很多嵌套结构数据

4. 不适合, 海量数据, 对水平扩展, 和可用性, 并发写很高的场景, MongoDB更适合中小型企业(Google没有类似系统), 对于海量数据, 更适合用于存储中间数据或buffer数据.

 

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

5.2. MongoDB


5.2.1. Overview

MongoDB is a schema-free document database written in C++ and developed in an open-source project which is mainly driven by the company 10gen Inc that also offers professional services around MongoDB.

http://www.mongodb.org/display/DOCS/Manual
According to its developers the main goal of MongoDB is to close the gap between the fast and highly scalable key-/value-stores and feature-rich traditional RDBMSs relational database management systems.
MongoDBs name is derived from the adjective humongous (cf. [10g10]). Prominent users of MongoDB include SourceForge.net, foursquare, the New York Times, the URL-shortener bit.ly and the distributed social network DIASPORA* (cf. [Cop10], [Hey10], [Mah10], [Rid10], [Ric10]).

这个图挺好, 总结的不错, 不是说MongoDB的主要目的就是减少KV和RDBMS之间的Gap吗, 号称最象SQL的NoSQL, 就来比较一下看看...
关注一下Horizontal partitioning scheme, Replication, Atomicity

 

5.2.2. Databases and Collections

MongoDB databases reside on a MongoDB server that can host more than one of such databases which are independent and stored separately by the MongoDB server.

A database contains one or more collections consisting of documents. In order to control access to the database a set of security credentials may be defined for databases

Collections inside databases are referred to by the MongoDB manual as “named groupings of documents”.

As MongoDB is schema-free the documents within a collection may be heterogeneous although the MongoDB manual suggests to create “one database collection for each of your top level
objects” (cf. [MMM+10b]).

 

5.2.3. Documents

The abstraction and unit of data storable in MongoDB is a document, a data structure comparable to an XML document, a Python dictionary, a Ruby hash or a JSON document.

In fact, MongoDB persists documents by a format called BSON which is very similar to JSON but in a binary representation for reasons of efficiency and because of additional datatypes compared to JSON.

Documents in MongoDB are limited in size by 4 megabytes (cf. [SM10]).

Datatypes for Document Fields

MongoDB provides the following datatypes for document fields (cf. [CHM10a], [MC09], [MMM+10a]):

  • scalar types: boolean, integer, double
  • character sequence types: string (for character sequences encoded in UTF-8), regular expression, code (JavaScript)
  • object (for BSON-objects)
  • object id is a data type for 12 byte long binary values used by MongoDB and all officially supported programming language drivers for a field named _id that uniquely identifies documents within collections.
  • null
  • array
  • date

 

References between Documents

MongoDB does not provide a foreign key mechanism so that references between documents have to be resolved by additional queries issued from client applications.

References may be set manually by assigning some reference field the value of the _id field of the referenced document. In addition, MongoDB provides a more formal way to specify references called DBRef (“Database Reference”). The advantages of using DBRefs are that documents in other collections can be referenced by them and that some programming
language drivers dereference them automatically (cf. [MDC+10], [MMM+10d, Database References]).

 

5.2.4. Database Operations

Transaction Properties

Regarding transactions, MongoDB only provides atomicity for update and delete operations by setting the $atomic flag to true and adding it to the selection criteria. Transactional locking and complex transactions are not supported for the following reasons discussed in the MongoDB manual.

  • Performance as “in sharded environments, distributed locks could be expensive and slow”. MongoDB is intended to be “leightweight and fast”.
  • Avoidance of deadlocks
  • Keeping database operations simple and predictable
  • MongoDB shall “work well for realtime problems”. Therefore, locking of large amounts of data which “might stop some small light queries for an extended period of time [. . . ] would make it even harder” to achieve this goal.

 

Server-Side Code Execution

MongoDB—like relational databases with their stored procedures—allows to execute code locally on database nodes. Server-side code execution comprises three different approaches in MongoDB:
1. Execution arbitrary code on a single database node via the eval operation (cf. [MMC+10b])
2. Aggregation via the operations count, group and distinct (cf. [MMM+10g])
3. MapReduce-fashioned code execution on multiple database nodes (cf. [HMC+10])

 

The eval-operation To execute arbitrary blocks of code locally on a database server, the code has to be enclosed by a anonymous JavaScript function and passed to MongoDB’s generic eval operation (cf.[MMC+10b]):


db. eval ( function (< formal parameters >) { ... }, <actual parameters >);

有两个缺点,

a write lock is held during execution

not supported in sharded setups

 

Aggregation To accomplish the aggregation of query results MongoDB provides the count, distinct and group operation that may be invoked via programming language libraries but executed on the database servers (cf. [MMM+10g]).

 

MapReduce A third approach to server-side code execution especially suited for batch manipulation and aggregation of data in sharded setups is MapReduce (cf. [HMC+10]). MongoDB’s MapReduce implementation is similar to the concepts described in Google’s MapReduce paper (cf. [DG04]) and its open-source implementation Hadoop: there are two phases—map and the reduce—in which code written in JavaScript is executed on the database servers and results in a temporary or permanent collection containing the outcome.

The MongoDB manual points out that—in contrast to CouchDB—MapReduce is neither used for basic queries nor indexing in MongoDB; it is only be used if explicitly invoked by the mapreduce operation. As mentioned in the sections before, the MapReduce approach is especially useful or even has to be used (e. g. for server-side aggregation) in sharded setups. Only in these scenarios MapReduce jobs can be executed in parallel as “ jobs on a single mongod process are single threaded [. . . ] due to a design limitation in current JavaScript engines” (cf. [MMM+10g, Map/Reduce], [HMC+10]).

MongoDB的mapreduce是和Google的mapreduce比较象的, 用于多shard node的数据分析和查询, 虽然说是由于JavaScript engines的原因, 在单个mongod进程上只能单线程, 意思是说如果要真正的并发处理, 需要多mongod进程? 但是, 这个比CouchDB强的多, CouchDB不支持sharding, 查询时使用mapreduce也只能在单节点上使用...

 

5.2.5. Indexing

Like relational database systems, MongoDB allows to specify indexes on document fields of a collection. The information gathered about these fields is stored in B-Trees and utilized by the query optimizing component to “to quickly sort through and order the documents in a collection” thereby enhancing read performance.

他的索引原理和relational database 没有区别, B-Trees

MongoDB manual concludes that “indexes are best for collections where the number of reads is much greater than the number of writes. For collections which are write-intensive, indexes, in some cases, may be counterproductive”

 

5.2.7. Distribution Aspects

Concurrency and Locking

The MongoDB architecture is described to be “concurrency friendly” although “some work with respect to granular locking and latching is not yet done. This means that some operations can block others.”

 

Replication

For redundancy and the failover of a MongoDB cluster in the presence of unavailable database nodes, MongoDB provides asynchronous replication. In such a setup only one database node is in charge of write operations at any given time (called primary server/node). Read operations may go to this same server for strong consistency semantics or to any of its replica peers if eventual consistency is sufficient (cf.[MMG+10]).

Master-Slave is a setup consisting of two servers out of one which takes the role of a master handling write requests and replicating those operations to the second server, the slave (cf. [MJS+10]).

Replica Sets are groups of MongoDB nodes “that work together to provide automated failover” (cf.[BCM+10]). They are described as an “an elaboration on the existing master/slave replication, adding automatic failover and automatic recovery of member nodes” (cf. [MCD+10]).

 image

这个方法比传统的MS要灵活, 你可以把一堆nodes, 设为replica sets, 会elect一个primary, 你可以增加和减少node, 当primary fail的时候, 还提供了自动failover的机制, 细节就不介绍了

 

Sharding

Since Version 1.6 MongoDB supports horizontal scaling via an automatic sharding architecture to distribute data across “thousands of nodes” with automatic balancing of load and data as well as automatic failover (cf. [MHC+10b], [MDH+10]).

Sharding is understood to be “the partitioning of data among multiple machines in an order-preserving manner” by the MongoDB documentation.

http://www.mongodb.org/display/DOCS/Sharding+Introduction

 

The MongoDB mentions Yahoo!’s PNUTS ([CRS+08]) and Google’s Bigtable ([CDG+06]) as important influences for the partitioning scheme implemented in MongoDB (cf. [MDH+10, MongoDB’s Auto-Sharding, Scaling Model]).

Sharding in MongoDB “occurs on a per-collection basis, not on the database as a whole”.
In a setup configured for sharding, MongoDB automatically detects which collections grow much faster than the average so that they become subject to sharding while the other collections may still reside on single nodes. MongoDB also detects imbalances in the load different shards have to handle and can automatically rebalance data to reduce disproportionate load distribution (cf. [MDH+10, MongoDB’s Auto-Sharding]).

 

Sharding in MongoDB is built on top of replica sets which have been discussed above. This means that for each partition of data a number of nodes forming a replica set is in charge: at any time one of these servers is primary and handles all write requests which then get propagated to the secondary servers to replicate changes and keep the set in-sync; if the primary node fails the remaining nodes elect a new primary via consensus so that the operation of the replica set is continued. This way, automated failover is provided for each shard (cf. [MDH+10, MongoDB’s Auto-Sharding, Balancing and Failover]).

Sharding Architecture A MongoDB shard cluster is built up by three components as depicted in figure 5.2 (cf. [MDH+10, Architectural Overview]):

 image

Shards consisting of servers that run mongod processes and store data. To ensure availability and automated failover in production systems, each shard typically consists of multiple servers comprising a replica set.

Config Servers “store the cluster’s metadata, which includes basic information on each shard server and the chunks contained therein”.

Routing Services are server-side mongos-processes executing read and write requests on behalf of client applications. They are in charge of looking up the shards to be read from or written to via the config servers, connecting to these shards, executing the requested operation and returning the results to client applications, also merging results of the request execution on different shards. This makes a distributed MongoDB setup look like a single server towards client applications which do not have to be aware of sharding

Shards, config and routing servers can be organized in different ways on physical or virtual servers as described by the MongoDB documentation (cf. [MDH+10, Server Layout]). A minimal sharding setup requires at least two shards, one config and one routing server (cf. [MHB+10a, Introduction]).

 

Limitations and Renounced Features Regarding Distribution

MongoDB does not provide multiversion-storage, multi-master replication setups or any support for version conflict reconciliation. The rationale for taking a different approach than many other NoSQL stores and not providing these features is explained by the MongoDB documentation as follows:


“Merging back old operations later, after another node has accepted writes, is a hard problem. One then has multi-master replication, with potential for conflicting writes. Typically that is handled in other products by manual version reconciliation code by developers. We think that is too much work : we want MongoDB usage to be less developer work, not more. Multi-master also can make atomic operation semantics problematic. It is possible (as mentioned above) to manually recover these events, via manual DBA effort, but we believe in large system with many, many nodes that such efforts become impractical.” (cf. [MD10, Rationale])

不具备多版本storage, 多主复本, 不支持版本冲突的reconciliation, 对于一份数据只有primary node可以写, 所以不存在版本冲突, 理由其实就是那样太复杂了

其实决定因素是适用场景, MongoDB选择牺牲可用性来保证一致性和原子性

而couchDB却是选择放弃一致性而保持高可用性, 所以CouchDB更像KV,nosql, 而MongoDB太象关系型数据库了...

 

5.2.8. Security and Authentication

According to its documentation, as of version 1.6 “Mongo supports only very basic security”.

 

5.2.9. Special Features

Capped Collections

MongoDB allows to specify collections of a fixed size that are called capped collections and can be processed highly performant by MongoDB according to its manual.

The MongoDB mentions the following use cases for capped collections (cf. [MDM+10, Applications]):
Logging is an application for capped collections as inserts of documents are processed at a speed comparable to file system writes while the space required for such a log is held constant.
Caching of small or limited numbers of precalculated objects in a LRU-manner can be done via capped collections conveniently.
Auto Archiving is understood by the MongoDB manual as a situation in which aged data needs to be dropped from the database. While for other databases and regular MongoDB collections scripts have to be written and scheduled to remove data by age capped collections already provide such behavior and no further effort is needed.

 

GridFS

BSON objects that allow to store binary data in MongoDB are limited in size by 4MB.
Therefore, MongoDB implements a specification named GridFS to store large binary data and provide operations on large data objects—such as videos or audios—like retrieving a limited number bytes of such an object. The GridFS specification is implemented by MongoDB transparent to client applications by dividing large data objects among multiple documents maintained in a collection and maintaining a collection containing metadata on these documents. Most programming language drivers also support the GridFS specification (cf. [CDM+10]).
The MongoDB manual considers GridFS also as an alternative to file system storage when large numbers of files have to be maintained, replicated and backed up as well as if files are expected to change often. In contrast, if a large number of small and static files has to be stored and processed, storage in MongoDB by GridFS is not regarded reasonable (cf. [Hor10]).

 

Geospatial Indexes

To support location based queries like “find the closest n items to a specific location” MongoDB provides two-dimensional geospatial indexing (cf. [HMS+10]). The latitude and longitude values have to be saved in a document field that is either an object or an array with the first two elements representing coordinates,

 

auto-sharding 无用论：auto-sharding vs. manual-sharding

http://blog.nosqlfan.com/html/841.html

 

MongoDB的局限性与不足

http://blog.nosqlfan.com/html/3166.html

  • 在32位系统上，不支持大于2.5G的数据。详见这里
  • 单个文档大小限制为 4 M/16 M（1.8版本后升为16M）
  • 锁粒度太粗，MongoDB使用的是一把全局的读写锁，详见这里
  • 不支持join操作和事务机制，这个确实是非MongoDB要做的领域
  • 对内存要求比较大，至少要保证热数据（索引，数据及系统其它开销）都能装进内存
  • 用户权限方面比较弱，这一点MongoDB官方推荐的是将机器部署在安全的内网环境中，尽量不要用权限，详见这里
  • MapReduce在单个实例上无法并行，只有采用Auto-Sharding才能并行。这是由JS引擎的限制造成的
  • MapReduce的结果无法写入到一个被Sharding的Collection中，2.0版本对这个问题的解决好像也不彻底
  • 对于数组型的数据操作不够丰富
  • Auto-Sharding还存在很多问题，所谓的水平扩展也不是那么理想

 

http://mkv.cn/1906/is-mongodb-the-next-big-thing

原文标题是《Is MongoDB the next big thing?》，请允许我把big thing等同于目前应用最广泛的开源数据库 MySQL。作为当下最热门的NoSQL之一，下面我们就来看看它具体的优点与不足。

MongoDB的特性

  • 简单的查询语句，没有Join操作
  • 文档型存储，其数据是用二进制的Json格式Bson存储的。其数据就像Ruby的hashes，或者Python的字典，或者PHP的数组
  • Sharding，MongoDB提供auto-sharding实现数据的扩展性
  • GridFS，MongoDB的提供的文件存储API
  • 数组索引，你可以对文档中的某个数组属性建立索引
  • MapReduce，可以用于进行复杂的统计和并行计算
  • 高性能，通过使用mmap和定时fsync的方法，避免了频繁IO，使其性能更高

MongoDB的优点

  • 高性能，速度非常快（如果你的内存足够的话）
  • 没有固定的表结构，不用为了修改表结构而进行数据迁移
  • 查询语言简单，容易上手
  • 使用Sharding实现水平扩展
  • 部署方便

使用MongoDB，你得记住以下几点：

  • MongoDB 假设你有大磁盘空间
  • MongoDB 假设你的内存也足够大于放下你的热数据
  • MongoDB 假设你是部署在64位系统上的（32位有2G的限制，试用还可以）
  • MongoDB 假设你的系统是little-endian的
  • MongoDB 假设你有多台机器（并不专注于单机可靠性）
  • MongoDB 假设你希望用安全换性能，同时允许你用性能换安全

MongoDB在下面领域不太擅长

  • 不太稳定，特别是auto-sharding目前还有很多问题
  • 不支持SQL，这意味着你很多通过SQL接口的工具不再适用
  • 持久化，MongoDB单机可靠性不太好，宕机可能丢失一段时间的数据
  • 相关文档比较少，新功能都有这个问题
  • 相关人才比较难找，这也是新功能的问题之一

posted on 2012-07-02 17:33  fxjwind  阅读(976)  评论(0)  编辑  收藏  举报