nutch2.2.1+mysql抓取数据
基本环境：linux centos6.5 nutch2.2.1 源码包， mysql 5.5 ，elasticsearch1.1.1, jdk1.7
1、下载地址http://mirror.bjtu.edu.cn/apache/nutch/2.2.1/ 解压
2、修改数据存储方式是mysql
修改nutch根目录/ivy/ivy.xml文件，原来mysql数据存储是注释的。
<dependency org="org.apache.gora" name="gora-core" rev="0.2.1" conf="*->default"/> 104 <!-- Uncomment this to use SQL as Gora backend. It should be noted that the 105 gora-sql 0.1.1-incubating artifact is NOT compatable with gora-core 0.3. Users should 106 downgrade to gora-core 0.2.1 in order to use SQL as a backend. --> 107 108 <dependency org="org.apache.gora" name="gora-sql" rev="0.1.1-incubating" conf="*->default" /> 109 110 <!-- Uncomment this to use MySQL as database with SQL as Gora store. --> 111 112 <dependency org="mysql" name="mysql-connector-java" rev="5.1.18" conf="*->default">
#gora.sqlstore.jdbc.driver=org.hsqldb.jdbc.JDBCDriver #gora.sqlstore.jdbc.url=jdbc:hsqldb:hsql://localhost/nutchtest #gora.sqlstore.jdbc.user=sa #gora.sqlstore.jdbc.password= # MySQL properties # ############################### gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver gora.sqlstore.jdbc.url=jdbc:mysql://ip:3306/nutch? useUnicode=true&characterEncoding=utf8&autoReconnect=true&zeroDateTimeBehavior=convertToNull gora.sqlstore.jdbc.user=user gora.sqlstore.jdbc.password=pwd
4、修改修改conf的nutch-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>http.agent.name</name> <value>My Spider</value> </property> <property> <name>http.accept.language</name> <value>ja-jp,zh-cn,en-us,en-gb,en;q=0.7,*;q=0.3</value> </property> <property> <name>parser.character.encoding.default</name> <value>utf-8</value> <description>The character encoding to fall back to when no other information is available</description> </property> <property> <name>storage.data.store.class</name> <value>org.apache.gora.sql.store.SqlStore</value> </property> <property> <name>plugin.includes</name> <value>protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|urlnormalizer-(pass|regex|basic)|scoring-opic</value> </property> </configuration>
5、使用ant 编译源码
在nutch 目录下执行 ant
job: [jar] Building jar: /home/hadoop/nutch221/build/apache-nutch-2.2.1.job runtime: [mkdir] Created dir: /home/hadoop/nutch221/runtime [mkdir] Created dir: /home/hadoop/nutch221/runtime/local [mkdir] Created dir: /home/hadoop/nutch221/runtime/deploy [copy] Copying 1 file to /home/hadoop/nutch221/runtime/deploy [copy] Copying 2 files to /home/hadoop/nutch221/runtime/deploy/bin [copy] Copying 1 file to /home/hadoop/nutch221/runtime/local/lib [copy] Copying 1 file to /home/hadoop/nutch221/runtime/local/lib/native [copy] Copying 26 files to /home/hadoop/nutch221/runtime/local/conf [copy] Copying 2 files to /home/hadoop/nutch221/runtime/local/bin [copy] Copying 100 files to /home/hadoop/nutch221/runtime/local/lib [copy] Copying 106 files to /home/hadoop/nutch221/runtime/local/plugins [copy] Copied 2 empty directories to 2 empty directories under /home/hadoop/nutch221/runtime/local/test BUILD SUCCESSFUL Total time: 41 seconds 编译成功。
6 创建数据库
CREATE DATABASE nutch DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE TABLE `webpage` (`id` varchar(767) CHARACTER SET latin1 NOT NULL, `headers` blob, `text` mediumtext DEFAULT NULL, `status` int(11) DEFAULT NULL, `markers` blob, `parseStatus` blob, `modifiedTime` bigint(20) DEFAULT NULL, `score` float DEFAULT NULL, `typ` varchar(32) CHARACTER SET latin1 DEFAULT NULL, `baseUrl` varchar(512) CHARACTER SET latin1 DEFAULT NULL, `content` mediumblob, `title` varchar(2048) DEFAULT NULL, `reprUrl` varchar(512) CHARACTER SET latin1 DEFAULT NULL, `fetchInterval` int(11) DEFAULT NULL, `prevFetchTime` bigint(20) DEFAULT NULL, `inlinks` mediumblob, `prevSignature` blob, `outlinks` mediumblob, `fetchTime` bigint(20) DEFAULT NULL, `retriesSinceFetch` int(11) DEFAULT NULL, `protocolStatus` blob, `signature` blob, `metadata` blob, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
bin/nutch crawl urls -depth 3bin/nutch elasticindex clustername -allhadoop@master bin]$ nutch crawl urls -depth 3 Exception in thread "main" java.lang.ClassNotFoundException: org.apache.gora.sql.store.SqlStore at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:190) at org.apache.nutch.storage.StorageUtils.getDataStoreClass(StorageUtils.java:89) at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:73) at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:221) at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68) at org.apache.nutch.crawl.Crawler.run(Crawler.java:136) at org.apache.nutch.crawl.Crawler.run(Crawler.java:250) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.nutch.crawl.Crawler.main(Crawler.java:257)
#####################
参照网上资料：http://blog.sina.com.cn/s/blog_3c9872d00101p4f0.html 还是没有解决。
#官方解决办法：
#http://mail-archives.apache.org/mod_mbox/nutch-user/201307.mbox/%3CCAErFeLSwoZ2UhxMA1iYi7H-L52Ojo-j9KoWT7xDittBzvB0F0A@mail.gmail.com%3E
######################
20141103
问题解决办法：重新编译一下即可
又出现一个新的问题：
./nutch crawl ../urls -depth 3
InjectorJob: Using class org.apache.gora.sql.store.SqlStore as the Gora storage class.
Exception in thread "main" java.lang.RuntimeException: job failed: name=inject ../urls, jobid=job_local713211278_0001
at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:54)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:233)
at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68)
./nutch crawl ../urls -depth 3 -topN 5
InjectorJob: Using class org.apache.gora.sql.store.SqlStore as the Gora storage class.
Exception in thread "main" java.lang.RuntimeException: job failed: name=inject ../urls, jobid=job_local1302478362_0001
at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:54)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:233)
文章参考：
官网资料：http://nlp.solutions.asia/?p=362
https://issues.apache.org/jira/browse/NUTCH-1473
