Solr data import 中XML/HTTP 数据源的使用

Solr data import 中XML/HTTP 数据源的使用

参考自：http://wiki.apache.org/solr/DataImportHandler

DataImportHandler 可以通过datasource数据源索引来自于HTTP的数据。包括REST/XML和RSS/ATOM 。

在版本1.4中，推荐使用URLDataSource。

配置示例如下：

Xml代码
  1. <dataSource name="b" type="HttpDataSource" baseUrl="http://host:port/" encoding="UTF-8" connectionTimeout="5000" readTimeout="10000"/>  
  2. <!-- or in Solr 1.4-->  
  3. <dataSource name="a" type="URLDataSource" baseUrl="http://host:port/" encoding="UTF-8" connectionTimeout="5000" readTimeout="10000"/>  
<dataSource name="b" type="HttpDataSource" baseUrl="http://host:port/" encoding="UTF-8" connectionTimeout="5000" readTimeout="10000"/> <!-- or in Solr 1.4--> <dataSource name="a" type="URLDataSource" baseUrl="http://host:port/" encoding="UTF-8" connectionTimeout="5000" readTimeout="10000"/>
 datasource的属性有：

baseUrl(可选): you should use it when the host/port changes between Dev/QA/Prod environments. Using this attribute isolates the changes to be made to the solrconfig.xml

encoding（可选）:定义响应头里面的编码方式。这个属性可以替换掉服务器的默认编码方式。

connectionTimeout(可选)：默认时间是5000ms

readTimeout（可选）：默认是10000ms

 

下面是一个data-config.xml示例的配置：

这是一个Slashdot RSS feed的例子。

Xml代码
  1. <dataConfig>  
  2.         <dataSource type="HttpDataSource" />  
  3.         <document>  
  4.                 <entity name="slashdot"  
  5.                         pk="link"  
  6.                         url="http://rss.slashdot.org/Slashdot/slashdot"  
  7.                         processor="XPathEntityProcessor"  
  8.                         forEach="/RDF/channel | /RDF/item"  
  9.                         transformer="DateFormatTransformer">  
  10.   
  11.                         <field column="source"       xpath="/RDF/channel/title"   commonField="true" />  
  12.                         <field column="source-link"  xpath="/RDF/channel/link"    commonField="true" />  
  13.                         <field column="subject"      xpath="/RDF/channel/subject" commonField="true" />  
  14.   
  15.                         <field column="title"        xpath="/RDF/item/title" />  
  16.                         <field column="link"         xpath="/RDF/item/link" />  
  17.                         <field column="description"  xpath="/RDF/item/description" />  
  18.                         <field column="creator"      xpath="/RDF/item/creator" />  
  19.                         <field column="item-subject" xpath="/RDF/item/subject" />  
  20.   
  21.                         <field column="slash-department" xpath="/RDF/item/department" />  
  22.                         <field column="slash-section"    xpath="/RDF/item/section" />  
  23.                         <field column="slash-comments"   xpath="/RDF/item/comments" />  
  24.                         <field column="date" xpath="/RDF/item/date" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss" />  
  25.                 </entity>  
  26.         </document>  
  27. </dataConfig>  
<dataConfig>         <dataSource type="HttpDataSource" />         <document>                 <entity name="slashdot"                         pk="link"                         url="http://rss.slashdot.org/Slashdot/slashdot"                         processor="XPathEntityProcessor"                         forEach="/RDF/channel | /RDF/item"                         transformer="DateFormatTransformer">                          <field column="source"       xpath="/RDF/channel/title"   commonField="true" />                         <field column="source-link"  xpath="/RDF/channel/link"    commonField="true" />                         <field column="subject"      xpath="/RDF/channel/subject" commonField="true" />                          <field column="title"        xpath="/RDF/item/title" />                         <field column="link"         xpath="/RDF/item/link" />                         <field column="description"  xpath="/RDF/item/description" />                         <field column="creator"      xpath="/RDF/item/creator" />                         <field column="item-subject" xpath="/RDF/item/subject" />                          <field column="slash-department" xpath="/RDF/item/department" />                         <field column="slash-section"    xpath="/RDF/item/section" />                         <field column="slash-comments"   xpath="/RDF/item/comments" />                         <field column="date" xpath="/RDF/item/date" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss" />                 </entity>         </document> </dataConfig>
 看到这个东西的确有点迷糊，因此下面是我们索引wikipedia的例子：

下面这个data-config.xml是用于索引http://dumps.wikimedia.org/enwiki/20100312/ 这个网页下的pages-articles.xml.bz2文件，我们需要去这个网页下载该文件，解压以后放到conf/data目录下，这个文件不压缩的大小是1.50GB。

Xml代码
  1. <dataConfig>  
  2.         <dataSource type="FileDataSource" encoding="UTF-8" />  
  3.         <document>  
  4.         <entity name="page"  
  5.                 processor="XPathEntityProcessor"  
  6.                 stream="true"  
  7.                 forEach="/mediawiki/page/"  
  8.                 url="/data/enwiki-20100312-pages-articles.xml"  
  9.                 transformer="RegexTransformer,DateFormatTransformer"  
  10.                 >  
  11.             <field column="id"        xpath="/mediawiki/page/id" />  
  12.             <field column="title"     xpath="/mediawiki/page/title" />  
  13.             <field column="revision"  xpath="/mediawiki/page/revision/id" />  
  14.             <field column="user"      xpath="/mediawiki/page/revision/contributor/username" />  
  15.             <field column="userId"    xpath="/mediawiki/page/revision/contributor/id" />  
  16.             <field column="text"      xpath="/mediawiki/page/revision/text" />  
  17.             <field column="timestamp" xpath="/mediawiki/page/revision/timestamp" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss'Z'" />  
  18.             <field column="$skipDoc"  regex="^#REDIRECT .*" replaceWith="true" sourceColName="text"/>  
  19.        </entity>  
  20.         </document>  
  21. </dataConfig>  
<dataConfig>         <dataSource type="FileDataSource" encoding="UTF-8" />         <document>         <entity name="page"                 processor="XPathEntityProcessor"                 stream="true"                 forEach="/mediawiki/page/"                 url="/data/enwiki-20100312-pages-articles.xml"                 transformer="RegexTransformer,DateFormatTransformer"                 >             <field column="id"        xpath="/mediawiki/page/id" />             <field column="title"     xpath="/mediawiki/page/title" />             <field column="revision"  xpath="/mediawiki/page/revision/id" />             <field column="user"      xpath="/mediawiki/page/revision/contributor/username" />             <field column="userId"    xpath="/mediawiki/page/revision/contributor/id" />             <field column="text"      xpath="/mediawiki/page/revision/text" />             <field column="timestamp" xpath="/mediawiki/page/revision/timestamp" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss'Z'" />             <field column="$skipDoc"  regex="^#REDIRECT .*" replaceWith="true" sourceColName="text"/>        </entity>         </document> </dataConfig>
 注意一下上面的url最好写绝对路径，我的相对路径老是出错-_-。

相关的schema.xml文件如下：

Xml代码
  1. <field name="id"        type="integer" indexed="true" stored="true" required="true"/>  
  2. <field name="title"     type="string"  indexed="true" stored="false"/>  
  3. <field name="revision"  type="sint"    indexed="true" stored="true"/>  
  4. <field name="user"      type="string"  indexed="true" stored="true"/>  
  5. <field name="userId"    type="integer" indexed="true" stored="true"/>  
  6. <field name="text"      type="text"    indexed="true" stored="false"/>  
  7. <field name="timestamp" type="date"    indexed="true" stored="true"/>  
  8. <field name="titleText" type="text"    indexed="true" stored="true"/>  
  9. ...  
  10. <uniqueKey>id</uniqueKey>  
  11. <copyField source="title" dest="titleText"/>  
<field name="id"        type="integer" indexed="true" stored="true" required="true"/> <field name="title"     type="string"  indexed="true" stored="false"/> <field name="revision"  type="sint"    indexed="true" stored="true"/> <field name="user"      type="string"  indexed="true" stored="true"/> <field name="userId"    type="integer" indexed="true" stored="true"/> <field name="text"      type="text"    indexed="true" stored="false"/> <field name="timestamp" type="date"    indexed="true" stored="true"/> <field name="titleText" type="text"    indexed="true" stored="true"/> ... <uniqueKey>id</uniqueKey> <copyField source="title" dest="titleText"/>
 下面会花费数小时的时间去索引数据，并且内存使用率将持续最高，请注意很多wikipedia的文章都仅仅是指向其它文章$skipDoc（1.4）的使用可以使得这些文章被忽略，当然前提是正则表达式匹配。

请注意：由于目前唯一支持delta import的EntityProcessor是SqlEntityProcessor，而XPathEntityProcessor并现在没有实现delta import。如果你想实现这些方法的话，你需要修改EntityProcessor.java        

posted @ 2010-04-27 21:29  searchDM  阅读(669)  评论(0)  编辑  收藏  举报