Nutch关于robot.txt的处理


在nutch中,默认情况下尊重robot.txt的配置,同时不提供配置项以忽略robot.txt。
以下是其中一个解释。即作为apache的一个开源项目,必须遵循某些规定,同时由于开放了源代码,可以简单的通过修改源代码来忽略robot.txt的限制。

From the point of view of research and crawling certain pieces of the web, and i strongly agree with you that it should be configurable. But because Nutch being an Apache project, i dismiss it (arguments available upon request). We should adhere to some ethics, it is bad enough that we can just DoS a server by setting some options to a high level. We publish source code, it leaves the option open to everyone to change it, and i think the current situation is balanced enough.
Patching it is simple, i think we should keep it like that :)

以下为修改源代码的方法:【未验证】
修改类org.apache.nutch.fetcher.FetcherReducer.java
将以下内容注释掉:

       if (!rules.isAllowed(fit.u.toString())) {
              // unblock
              fetchQueues.finishFetchItem(fit, true);
              if (LOG.isDebugEnabled()) {
                LOG.debug("Denied by robots.txt: " + fit.url);
              }
              output(fit, null, ProtocolStatusUtils.STATUS_ROBOTS_DENIED,
                  CrawlStatus.STATUS_GONE);
              continue;
            }




posted @ 2015-01-28 11:20  eagleGeek  阅读(162)  评论(0编辑  收藏  举报