IKAnalyzer With Solr 5
中文分詞裡IKAnalyzer和結巴是大家比較常用的分詞器, 不過IKAnalyzer已經很久沒有更新了, IKAnalyzer中文分词器V2012使用手册也跟IK Analyer 2012-FF Hotfix 1對不起來。我自己觀察的結果是
IKAnalyzer中文分词器V2012使用手册是IK Analyer 2012 upgrade 6的使用手册, 不是IK Analyer 2012-FF Hotfix 1的使用手册
IK Analyer 2012 upgrade 6支援Lucene 3.X API, 不支援Lucene 4.X API
IK Analyer 2012-FF Hotfix 1支援Lucene 4.X API, 不支援Lucene 5.X API
如果你硬要在Solr 5.1.0上使用IK Analyer 2012-FF Hotfix 1, 會產生下列錯誤訊息
1 java.lang.AbstractMethodError 2 at org.apache.lucene.analysis.AnalyzerWrapper.createComponents(AnalyzerWrapper.java:101) 3 at org.apache.lucene.analysis.AnalyzerWrapper.createComponents(AnalyzerWrapper.java:101) 4 at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:179) 5 at org.apache.lucene.document.Field.tokenStream(Field.java:556) 6 at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:606) 7 at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344) 8 at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300) 9 at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232) 10 at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458) 11 at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1350) 12 at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239) 13 at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163) 14 at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69) 15 at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51) 16 at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:947) 17 at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1102) 18 at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:703) 19 at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104) 20 at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:250) 21 at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:177) 22 at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:103) 23 at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) 24 at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) 25 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984) 26 at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829) 27 at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446) 28 at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220) 29 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) 30 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) 31 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) 32 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) 33 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) 34 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) 35 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) 36 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) 37 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) 38 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) 39 at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) 40 at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) 41 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) 42 at org.eclipse.jetty.server.Server.handle(Server.java:364) 43 at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) 44 at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) 45 at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) 46 at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) 47 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953) 48 at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) 49 at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) 50 at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) 51 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) 52 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) 53 at java.lang.Thread.run(Thread.java:745)
另外在IK Analyer 2012-FF Hotfix 1, IKTokenizerFactory被拿掉了, 如果你要在schema.xml裡使用filter element, 就會用到IKTokenizerFactory, 語法類似
<fieldType name="text_ik" class="solr.TextField">
<analyzer type="index">
<tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="false" />
<filter class="solr.LowerCaseFilterFactory" min="0" max="32764" />
</analyzer>
<analyzer type="query">
<tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="true" />
<filter class="solr.LengthFilterFactory" min="2" max="7" />
</analyzer>
</fieldType>
當然如果你沒有要使用任何的filters, 是可以不需要IKTokenizerFactory, 語法可以簡化成
<fieldType name="text_ik" class="solr.TextField"> <analyzer type="index" useSmart="false" class="org.wltea.analyzer.lucene.IKAnalyzer"/> <analyzer type="query" useSmart="true" class="org.wltea.analyzer.lucene.IKAnalyzer"/> </fieldType>
在程式碼的差異上, Tokenizer的constructor不需要再提供input reader當參數, Analyzer class的createComponents method也不需要再提供input reader當參數，TokenizerFactory也是不再使用input reader當參數。
