nutch0.8配置
1. \cygwin\nutch-0.7.2\conf下的nutch-default.xml替换掉nutch-site.xml
2． 在 crawl-urlfilter.txt 设置正则匹
# skip URLs containing certain characters as probable queries, etc.
-[!]（除掉可能出现的问号等）
正则表达式，修改为 +^http://(\.*)*
3. nutch 0.7.2中，urls.txt中不用加
4. Tomat的中文问题，修改server.xml下段，
<Connector port="8080" maxHttpHeaderSize="8192"
maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" redirectPort="8443" acceptCount="100"
connectionTimeout="20000"
disableUploadTimeout="true"
URIEncoding="UTF-8" useBodyEncodingForURI="true" /> //添加这行
5. 在文件regex-normalize.xml中增加下述描述
<regex>
<pattern>(\?|\&|\& *([a-zA-Z0-9]*\.)*</pattern>
<substitution>$1$3</substitution>
</regex>
5. 修改nite-site.xml
<property>
<name>http.content.limit</name>
<value>9065536</value>//如不能正确抓取，在这里把容量设置大些
<description>The length limit for downloaded content, in bytes.
If this value is nonnegative (>=0), content longer than it will be truncated;
otherwise, no truncation at all.
</description>
</property>
把nutch-site.xml的内容 copy 到nutch-default.xml 一份，保持文件一致。
1. \cygwin\nutch-0.7.2\conf下的nutch-default.xml替换掉nutch-site.xml
2． 在 crawl-urlfilter.txt 设置正则匹
# skip URLs containing certain characters as probable queries, etc.
-[!]（除掉可能出现的问号等）
正则表达式，修改为 +^http://(\.*)*
3. nutch 0.7.2中，urls.txt中不用加
4. Tomat的中文问题，修改server.xml下段，
<Connector port="8080" maxHttpHeaderSize="8192"
maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" redirectPort="8443" acceptCount="100"
connectionTimeout="20000"
disableUploadTimeout="true"
URIEncoding="UTF-8" useBodyEncodingForURI="true" /> //添加这行
5. 在文件regex-normalize.xml中增加下述描述
<regex>
<pattern>(\?|\&|\& *([a-zA-Z0-9]*\.)*</pattern>
<substitution>$1$3</substitution>
</regex>
5. 修改nite-site.xml
<property>
<name>http.content.limit</name>
<value>9065536</value>//如不能正确抓取，在这里把容量设置大些
<description>The length limit for downloaded content, in bytes.
If this value is nonnegative (>=0), content longer than it will be truncated;
otherwise, no truncation at all.
</description>
</property>
把nutch-site.xml的内容 copy 到nutch-default.xml 一份，保持文件一致。
