nutch2.3命令参数解析

nutch中可执行的命令列表

[root@ewanalysis ~]# nutch
Usage: nutch COMMAND
where COMMAND is one of:
 inject         inject new urls into the database
 hostinject     creates or updates an existing host table from a text file
 generate       generate new batches to fetch from crawl db
 fetch          fetch URLs marked during generate
 parse          parse URLs marked during fetch
 updatedb       update web table after parsing
 updatehostdb   update host table after parsing
 readdb         read/dump records from page database
 readhostdb     display entries from the hostDB
 index          run the plugin-based indexer on parsed batches
 elasticindex   run the elasticsearch indexer - DEPRECATED use the index command                        instead
 solrindex      run the solr indexer on parsed batches - DEPRECATED use the inde                       x command instead
 solrdedup      remove duplicates from solr
 solrclean      remove HTTP 301 and 404 documents from solr - DEPRECATED use the                        clean command instead
 clean          remove HTTP 301 and 404 documents and duplicates from indexing b                       ackends configured via plugins
 parsechecker   check the parser for a given url
 indexchecker   check the indexing filters for a given url
 plugin         load a plugin and run one of its classes main()
 nutchserver    run a (local) Nutch server on a user defined port
 webapp         run a local Nutch web application
 junit          runs the given JUnit test
 or
 CLASSNAME      run the class named CLASSNAME
Most commands print help when invoked w/o parameters.

 

crawl

Usage: crawl <seedDir> <crawlID> [<solrUrl>] <numberOfRounds>

参数说明：

<seedDir>：包括URL列表的文本文件，它是一个已存在的文件夹。

<crawlID>：爬取的ID编号

[<solrUrl>]：solr解析的建立索引的地址

<numberOfRounds>：爬取的轮次

nutch inject

Usage: InjectorJob <url_dir> [-crawlId <id>]

参数说明：

<url_dir>：包括URL列表的文本文件，它是一个已存在的文件夹。 

nutch generate

Usage: GeneratorJob [-topN N] [-crawlId id] [-noFilter] [-noNorm] [-adddays numDays]

参数说明：

[-topN N]：选取前多少个链接，默认值为Long.MAX_VALUE 

[-noFilter]：不激活过滤器插件过滤url，默认是true

[-noNorm] ：不激活normalizer插件规范化的url，默认是true

[-adddays numDays]: 添加 <numDays>到当前时间，配置crawling urls ，以将很快被爬取db.default.fetch.interval默认值为0。爬取结束时间在当前时间以前的。 

nutch fetch

Usage: FetcherJob (<batchId> | -all) [-crawlId <id>] [-threads N] [-resume] [-numTasks N]

参数说明：

[-crawlId <id>]：

[-threads N]：运行的fetcher线程数默认值为 Configuration Key -> fetcher.threads.fetch -> 10 

[-resume]：恢复中断的工作

[-numTasks N]：如果N>0，则使用设定的N减少抓取任务（默认值: mapred.map.tasks）

nutch parse

Usage: ParserJob (<batchId> | -all) [-crawlId <id>] [-resume] [-force]

参数说明：

[-crawlId <id>]：

[-resume]：恢复之前中断的任务

[-force]：强制重新解析这个页面，即使这个页面已经被解析过了

nutch updatedb

Usage: DbUpdaterJob (<batchId> | -all) [-crawlId <id>] <batchId> - crawl identifier returned by Generator, or -all for all
generated batchId-s
-crawlId <id> - the id to prefix the schemas to operate on,
(default: storage.crawl.id)

参数说明：

nutch index

Usage: IndexingJob (<batchId> | -all | -reindex) [-crawlId <id>]

参数说明：