实时分布式搜索引擎比较（senseidb、Solr、elasticsearch）

  • 1 1. Solr
    • 1.1 Features
    • 1.2 Pros & Cons
    • 1.3 References
  • 2 2. Senseidb
    • 2.1 Features
    • 2.2 Pros & Cons
    • 2.3 为何没有直接用Solr？
    • 2.4 References
  • 3 3. elasticsearch
    • 3.1 Features
    • 3.2 Pros & Cons
    • 3.3 References
  • 4 4. Conclusion
  • 5 其它参考文献

比较的时候，主要关注以下几个方面：

  1. Clustering
    • Scalability on Storage and Service
    • High Availability Considerations
  2. Features
  3. Flexibility

1. Solr

很显然， Solr跟Lucene是一家，所以，对Lucene做了很多扩展，与lucene的集成也比较好，而且，业界貌似求稳的都会选择Solr来构建他们的搜索体系。

但SolrCloud才是最终的理想解决方案，而SolrCloud还没有production-ready。

下面是Solr相关的架构图：

 image solr architecture

1.1 Features

Solr的首页上对自己的特性罗列阐述的很详细了，这里不再赘述。

1.2 Pros & Cons

  • Pros
    • 成熟且验证过的方案
    • 文档资料丰富
    • 社区活跃
    • plugin extension points
  • Cons
    • 貌似体系比较庞杂， replication的架构扩展有稍许问题？！

1.3 References

  1. New SolrCloud Design
  2. Scaling Lucene and Solr
  3. Turbocharging Solr Index Replication with BitTorrent
    • funny and sparkling idea by introducing BitTorrent replication mechanism *****
  4. Distributed Searching
  5. Carrot2-OSS framework for building search clustering engine
    • Solr search results clustering is based on the Carrot2 real-time document clustering engine.
  6. Clustering Component
    • 结果集的分类
  7. New SolrCloud Design
  8. SolrCloud
  9. UniqueKey
  10. Solr Near Realtime Search
    • will be added in Solr4, currently available in trunk
  11. Scaling Solr Indexing with SolrCloud, Hadoop and Behemoth

2. Senseidb

architecture of sensei

architecture of sensei

2.1 Features

  1. 主要解决高速索引更新的问题;
    • 底层是zoie的“2-swapping-in-memory-index + 1-on-disk-index”索引结构支持
  2. 需要定义schema;
  3. 通过Gateway可以接入多种数据源;
  4. 通过BQL或者REST API，甚至各种语言bindings进行数据查询；
  5. 支持通过hadoop MR job批量更新数据索引；

2.2 Pros & Cons

  • Pros
    • 高速索引更新
    • 多数据源接入
    • 灵活的访问接口
    • 与hadoop生态的集成
    • 优秀的分布式扩展能力
  • Cons
    • static schema
    • application side versioning maitaining

2.3 为何没有直接用Solr？

摘录在John Wang的访谈片段：

Sensei leverages Lucene.

We weren’t able to leverage Solr because of the following requirements:

    * High update requirement, 10’s of thousands updates per second in to the system
    * Real distributed solution, current Solr’s distributed story has a SPOF at the master, and Solr Cloud is not yet completed.
    * Complex faceting support. Not just your standard terms based faceting. We needed to facet on social graph, dynamic time ranges and many other interesting faceting scenarios. Faceting behavior also needs to be highly customizable, which is not available via Solr.

2.4 References

  1. Introducing SenseiDB 1.0: an open-source, distributed, realtime, semi-structured database
  2. Sensei: distributed, realtime, semi-structured database

3. elasticsearch

很新，当前0.19RC3版本， 文档缺乏。不过， ES确实有很多值得喝彩的地方。

image

3.1 Features

  1. Schema-Free | Schemaless
  2. feed index engine with JSON formatted documents
  3. Query by Lucene based query string or JSON based query DSL over HTTP or Native Java;
  4. shards and replicas, LB and routings
  5. cloud integration
  6. multiple search types
  7. multiple data sources integration with River
  8. many more...

3.2 Pros & Cons

  • Pros
    • 许多灵活， 优秀的特性（见features列表）
    • 作者拥有多年在搜索领域的涉猎经验
    • senseidb的pros它也基本都有
  • Cons
    • 文档不足
    • 后端没有大的商业机构支持

3.3 References

  1. quick intro to elastic search
  2. Flume, Hive and realtime indexing via ElasticSearch
  3. The Future of Compass & ElasticSearch
  4. Elastic Search: Distributed, Lucene-based Search Engine
  5. ElasticSearch at berlinbuzzwords 2010
  6. Elastic Search Vs. Apache Solr
    • 这篇貌似倾向于ES比较多一些
  7. Your Data, Your Search
  8. Search Engine Time Machine
    • transient状态与持久化状态的结合， write behind策略
  9. NoSQL, Yes Search
    • 多种数据源类型的平滑接入
  10. Geo Location and Search
    • 基于geo进行排序的特性很新颖
  11. Zero Conf Persistency
    • Local Gateway (Local Storage | Local FileSystem)
  12. The River
    • ES里River的概念跟Senseidb里Gateway的概念相近， 是数据源通道的意思，可以根据不同的数据源给出不同的River实现，比如基于MysqlBinlog的River， 基于Hbase的River，或者RabbitMQ River，CouchDB River etc.
  13. Percolator
    • 这个Percolator是ES里的概念，不要跟Google的Percolator混淆
  14. Versioning
    • Optimistic Concurrency Control
  15. New Search Types
    • Introduce count and scan search types, the latter can be used to scroll large result set
  16. Data Visualization with ElasticSearch and Protovis
  17. Distributed Diagram (Video)
  18. Road to a Distributed Search Engine (Video)*****

4. Conclusion

  1. All are based on Lucene.
  2. All are distributed.
    • senseidb shards with multi-write?!
    • solr shards with master-slaves and slave pull strategy;
    • elasticsearch shards with primary-secondary push strategy;
  3. All do partitioning in document granularity, All require some unique key for each document(optional for some situations);
  4. Sensei is good at real-time index update; Solr is good at stable and wide adoption; Elasticsearch is good at flexible and good ideas;

5 其它参考文献

  1. Lily架构简介
    • 在自己的lily node里实现了multiwrite + wal+ message queue的数据分发， 没有充分利用现有系统中各个组件/系统的能力(虽然是基于hbase的table实现的)， 部分上来讲把事情搞复杂了。

引自：http://afoo.me/notes-on-senseidb-solr-and-elasticsearch.html

posted @ 2013-03-22 13:15  爱开卷360  阅读(10838)  评论(2)  编辑  收藏  举报