crawler4j 学习(二)
crawler4j 学习(二)
实现控制器类以制定抓取的种子(seed)、中间数据存储的文件夹、并发线程的数目:
public class Controller { public static void main(String[] args) throws Exception { String crawlStorageFolder = "/data/crawl/root"; int numberOfCrawlers = 7; CrawlConfig config = new CrawlConfig(); config.setCrawlStorageFolder(crawlStorageFolder); /* * Instantiate the controller for this crawl. */ PageFetcher pageFetcher = new PageFetcher(config); RobotstxtConfig robotstxtConfig = new RobotstxtConfig(); RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher); CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer); /* * For each crawl, you need to add some seed urls. These are the first * URLs that are fetched and then the crawler starts following links * which are found in these pages */ controller.addSeed("http://www.ics.uci.edu/~lopes/"); controller.addSeed("http://www.ics.uci.edu/~welling/"); controller.addSeed("http://www.ics.uci.edu/"); /* * Start the crawl. This is a blocking operation, meaning that your code * will reach the line after this only when crawling is finished. */ controller.start(MyCrawler.class, numberOfCrawlers); } }
配置介绍
控制器类必须传一个类型为CrawlConfig的参数,用于配置crawler4j。下面描述了一些关于配置的细节。
抓取深度
默认情况下没有抓取深度的限制。可以通过配置来限制深度,比如,你有个种子页面A连接到B,B又连接到C,C又连接到D。结构如下:
A --> B --> C --> D
A是种子页面深度为0,B为1,C、D以此类推。如:当设置抓取深度是2是,就不会抓取页面D。抓取最大深度通过以下代码配置:
crawlConfig.setMaxDepthOfCrawling(maxDepthOfCrawling);
页面抓取的最大数量
默认情况下没有抓取数量限制,可以通过以下代码配置:
crawlConfig.setMaxPagesToFetch(maxPagesToFetch);
其他限制
crawler4j是高效的,有着极快的抓取能力(比如:每秒可以抓取200个Wikipedia页面)。然而,这会给服务器带来很大的负荷(而服务器可能会阻断你的请求!)。所以,从1.3版开始,默认情况下,crawler4j每次请求前等待200毫秒。但是这个参数可以修改:
crawlConfig.setPolitenessDelay(politenessDelay);
代理
使用下代码配置爬虫通过代理:
crawlConfig.setProxyHost("proxyserver.example.com");
crawlConfig.setProxyPort(8080);
如果你的代理需要认证:
crawlConfig.setProxyUsername(username);
crawlConfig.getProxyPassword(password);
抓取恢复
有时爬虫需要运行很长时间,但中途可能意外终止了。这种情况下,可以通过以下配置恢复停止/崩溃的爬虫:
crawlConfig.setResumableCrawling(true);