本文要分析的是Heritrix3.1.0的Frontier组件，先熟悉一下相关的UML类图
通过浏览该图，我们可以清楚的看出Frontier组件的相关接口和类的继承和调用关系，不必我再文字描述了
在分析BdbFrontier类的相关方法之前，有必要熟悉一下在系统环境中该对象的初始状态，以及状态的改变
这部分内容需要回顾 Heritrix 3.1.0 源码解析（六）一文，本人就不在这里重复了
BdbFrontier类重要的有几个方法，分别是添加CrawlURI,获取CrawlURI，完成CrawlURI
void schedule(CrawlURI curi)
CrawlURI next()
void finished(CrawlURI curi)
下面首先来分析void schedule(CrawlURI curi)方法，在它的父类WorkQueueFrontier里面
org.archive.crawler.frontier.BdbFrontier
org.archive.crawler.frontier.WorkQueueFrontier
/** * Arrange for the given CrawlURI to be visited, if it is not * already enqueued/completed. * * Differs from superclass in that it operates in calling thread, rather * than deferring operations via in-queue to managerThread. TODO: settle * on either defer or in-thread approach after testing. * * @see org.archive.crawler.framework.Frontier#schedule(org.archive.modules.CrawlURI) */ @Override public void schedule(CrawlURI curi) { sheetOverlaysManager.applyOverlaysTo(curi); try { KeyedProperties.loadOverridesFrom(curi); if(curi.getClassKey()==null) { // remedial processing preparer.prepare(curi); } processScheduleIfUnique(curi); } finally { KeyedProperties.clearOverridesFrom(curi); } }
org.archive.crawler.frontier.BdbFrontier
org.archive.crawler.frontier.WorkQueueFrontier
/** * Arrange for the given CrawlURI to be visited, if it is not * already scheduled/completed. * * @see org.archive.crawler.framework.Frontier#schedule(org.archive.modules.CrawlURI) */ protected void processScheduleIfUnique(CrawlURI curi) { // assert Thread.currentThread() == managerThread; assert KeyedProperties.overridesActiveFrom(curi); // Canonicalization may set forceFetch flag. See // #canonicalization(CrawlURI) javadoc for circumstance. String canon = curi.getCanonicalString(); if (curi.forceFetch()) { uriUniqFilter.addForce(canon, curi); } else { uriUniqFilter.add(canon, curi); } }
void processScheduleIfUnique(CrawlURI curi)方法调用了BdbUriUniqFilter类成员的void add(String key, CrawlURI value)方法
在分析BdbUriUniqFilter类成员的void add(String key, CrawlURI value)方法前，首先有必要查看一下该对象的状态
org.archive.crawler.util.BdbUriUniqFilter类在它的初始化方法里面初始化下面成员变量，前者是BDB数据库，后者是BDB数据库的数据对象（在BdbUriUniqFilter类方法里面我还没找发现该变量的作用）
protected transient Database alreadySeen = null;
protected transient DatabaseEntry value = null;
public void start() { if(isRunning()) { return; } boolean isRecovery = (recoveryCheckpoint != null); try { BdbModule.BdbConfig config = getDatabaseConfig(); config.setAllowCreate(!isRecovery); initialize(bdb.openDatabase(DB_NAME, config, isRecovery)); } catch (DatabaseException e) { throw new IllegalStateException(e); } if(isRecovery) { JSONObject json = recoveryCheckpoint.loadJson(beanName); try { count.set(json.getLong("count")); } catch (JSONException e) { throw new RuntimeException(e); } } isRunning = true; }
进一步调用 void initialize(Database db)
/** * Method shared by constructors. * @param env Environment to use. * @throws DatabaseException */ protected void initialize(Database db) throws DatabaseException { open(db); }
初始化数据库Database alreadySeen,void open(final Database db)
protected void open(final Database db) throws DatabaseException { this.alreadySeen = db; this.value = new DatabaseEntry("".getBytes()); }
另外在BdbFrontier对象的初始化方法里面我们发现执行了BdbUriUniqFilter类成员的void setDestination(CrawlUriReceiver receiver)方法，这里是设置BdbFrontier对象本身，在BdbUriUniqFilter对象的相关方法里面调用了BdbFrontier对象的void receive(CrawlURI item)方法（BdbFrontier类实现了CrawlUriReceiver接口），这里类似于回调接口
org.archive.crawler.frontier.BdbFrontier
org.archive.crawler.frontier.WorkQueueFrontier
public void start() { if(isRunning()) { return; } uriUniqFilter.setDestination(this); super.start(); try { initInternalQueues(); } catch (Exception e) { throw new IllegalStateException(e); } }
BdbUriUniqFilter对象的状态分析完毕，现在我们再来分析上面uriUniqFilter.add(canon, curi)方法，在BdbUriUniqFilter的父类里面
org.archive.crawler.util.BdbUriUniqFilter
org.archive.crawler.util.SetBasedUriUniqFilter
public void add(String key, CrawlURI value) { profileLog(key); if (setAdd(key)) { this.receiver.receive(value); if (setCount() % 50000 == 0) { LOGGER.log(Level.FINE, "count: " + setCount() + " totalDups: " + duplicateCount + " recentDups: " + (duplicateCount - duplicatesAtLastSample)); duplicatesAtLastSample = duplicateCount; } } else { duplicateCount++; } }
进一步调用BdbUriUniqFilter本身的boolean setAdd(CharSequence uri)方法
org.archive.crawler.util.BdbUriUniqFilter
protected boolean setAdd(CharSequence uri) { DatabaseEntry key = new DatabaseEntry(); LongBinding.longToEntry(createKey(uri), key); long started = 0; OperationStatus status = null; try { if (logger.isLoggable(Level.INFO)) { started = System.currentTimeMillis(); } status = alreadySeen.putNoOverwrite(null, key, ZERO_LENGTH_ENTRY); if (logger.isLoggable(Level.INFO)) { aggregatedLookupTime += (System.currentTimeMillis() - started); } } catch (DatabaseException e) { logger.severe(e.getMessage()); } if (status == OperationStatus.SUCCESS) { count.incrementAndGet(); if (logger.isLoggable(Level.FINE)) { final int logAt = 10000; if (count.get() > 0 && ((count.get() % logAt) == 0)) { logger.fine("Average lookup " + (aggregatedLookupTime / logAt) + "ms."); aggregatedLookupTime = 0; } } } if(status == OperationStatus.KEYEXIST) { return false; // not added } else { return true; } }
这里有意思的是，我们可以看到Heritrix是怎么实现URL排重的
**
* Create fingerprint.
* Pubic access so test code can access createKey.
* @param uri URI to fingerprint.
* @return Fingerprint of passed <code>url</code>.
*/
public static long createKey(CharSequence uri) {
String url = uri.toString();
int index = url.indexOf(COLON_SLASH_SLASH);
if (index > 0) {
index = url.indexOf('/', index + COLON_SLASH_SLASH.length());
}
CharSequence hostPlusScheme = (index == -1)? url: url.subSequence(0, index);
long tmp = FPGenerator.std24.fp(hostPlusScheme);
return tmp | (FPGenerator.std40.fp(url) >>> 24);
}
这里需要注意的是Database alreadySeen数据库并没用真正存储URL链接地址，而是URL链接的Fingerprint值
BdbUriUniqFilter本身的boolean setAdd(CharSequence uri)方法的真正意义在于判断URL链接是否添加过，除此之外，大概没有其他含义
接下来我们看到void add(String key, CrawlURI value)方法后面调用了this.receiver.receive(value)方法
我们前面分析BdbUriUniqFilter初始化状态的时候已经提到了，BdbUriUniqFilter对象是通过回调BdbFrontier对象的void receive(CrawlURI item)方法（BdbFrontier类实现了CrawlUriReceiver接口）
在BdbFrontier类的父类的父类AbstractFrontier我们发现该方法void receive(CrawlURI curi)
org.archive.crawler.frontier.BdbFrontier
org.archive.crawler.frontier.AbstractFrontier
/** * Accept the given CrawlURI for scheduling, as it has * passed the alreadyIncluded filter. * * Choose a per-classKey queue and enqueue it. If this * item has made an unready queue ready, place that * queue on the readyClassQueues queue. * @param caUri CrawlURI. */ public void receive(CrawlURI curi) { sheetOverlaysManager.applyOverlaysTo(curi); // prefer doing asap if already in manager thread try { KeyedProperties.loadOverridesFrom(curi); processScheduleAlways(curi); } finally { KeyedProperties.clearOverridesFrom(curi); } }
继续调用BdbFrontier类的void processScheduleAlways(CrawlURI curi)方法，在它的父类WorkQueueFrontier里面
org.archive.crawler.frontier.BdbFrontier
org.archive.crawler.frontier.WorkQueueFrontier
/** * Accept the given CrawlURI for scheduling, as it has * passed the alreadyIncluded filter. * * Choose a per-classKey queue and enqueue it. If this * item has made an unready queue ready, place that * queue on the readyClassQueues queue. * @param caUri CrawlURI. */ protected void processScheduleAlways(CrawlURI curi) { // assert Thread.currentThread() == managerThread; assert KeyedProperties.overridesActiveFrom(curi); prepForFrontier(curi); sendToQueue(curi); }
void prepForFrontier(CrawlURI curi)方法是设置curi的序号
void sendToQueue(CrawlURI curi)方法添加CrawlURI curi到工作队列里面（该方法是由BdbFrontier类的void schedule(CrawlURI curi)发起的调用的方法里面的实质性方法），在BdbFrontier类的父类WorkQueueFrontier里面
org.archive.crawler.frontier.BdbFrontier
org.archive.crawler.frontier.WorkQueueFrontier
/** * Send a CrawlURI to the appropriate subqueue. * * @param curi */ protected void sendToQueue(CrawlURI curi) { // assert Thread.currentThread() == managerThread; WorkQueue wq = getQueueFor(curi.getClassKey()); synchronized(wq) { int originalPrecedence = wq.getPrecedence(); wq.enqueue(this, curi); // always take budgeting values from current curi // (whose overlay settings should be active here) wq.setSessionBudget(getBalanceReplenishAmount()); wq.setTotalBudget(getQueueTotalBudget()); if(!wq.isRetired()) { incrementQueuedUriCount(); int currentPrecedence = wq.getPrecedence(); if(!wq.isManaged() || currentPrecedence < originalPrecedence) { // queue newly filled or bumped up in precedence; ensure enqueuing // at precedence level (perhaps duplicate; if so that's handled elsewhere) deactivateQueue(wq); } } } // Update recovery log. doJournalAdded(curi); wq.makeDirty(); largestQueues.update(wq.getClassKey(), wq.getCount()); }
void sendToQueue(CrawlURI curi)方法首先是根据CrawlURI curi的ClassKey得到工作队列，这里是BdbWorkQueue（后面还设置了BdbWorkQueue对象的相关属性，涉及到Heritrix3.1.0工作队列的调度，后文再分析），
然后调用它的long enqueue(final WorkQueueFrontier frontier,CrawlURI curi)方法，在BdbWorkQueue类的父类WorkQueue里面
org.archive.crawler.frontier.BdbWorkQueue
org.archive.crawler.frontier.WorkQueue
/** * Add the given CrawlURI, noting its addition in running count. (It * should not already be present.) * * @param frontier Work queues manager. * @param curi CrawlURI to insert. */ protected synchronized long enqueue(final WorkQueueFrontier frontier, CrawlURI curi) { try { insert(frontier, curi, false); } catch (IOException e) { //FIXME better exception handling e.printStackTrace(); throw new RuntimeException(e); } count++; enqueueCount++; return count; }
继续调用BdbWorkQueue对象的void insert(final WorkQueueFrontier frontier, CrawlURI curi,boolean overwriteIfPresent)方法，也在在BdbWorkQueue类的父类WorkQueue里面
org.archive.crawler.frontier.BdbWorkQueue
org.archive.crawler.frontier.WorkQueue
/** * Insert the given curi, whether it is already present or not. * @param frontier WorkQueueFrontier. * @param curi CrawlURI to insert. * @throws IOException */ private void insert(final WorkQueueFrontier frontier, CrawlURI curi, boolean overwriteIfPresent) throws IOException { insertItem(frontier, curi, overwriteIfPresent); lastQueued = curi.toString(); }
进一步调用BdbWorkQueue对象的void insertItem(final WorkQueueFrontier frontier, final CrawlURI curi, boolean overwriteIfPresent)方法
org.archive.crawler.frontier.BdbWorkQueue
protected void insertItem(final WorkQueueFrontier frontier, final CrawlURI curi, boolean overwriteIfPresent) throws IOException { try { final BdbMultipleWorkQueues queues = ((BdbFrontier) frontier) .getWorkQueues(); queues.put(curi, overwriteIfPresent); if (LOGGER.isLoggable(Level.FINE)) { LOGGER.fine("Inserted into " + getPrefixClassKey(this.origin) + " (count " + Long.toString(getCount())+ "): " + curi.toString()); } } catch (DatabaseException e) { throw new IOException(e); } }
我们从这里可以看到，最终是调用BdbFrontier的成员变量BdbMultipleWorkQueues pendingUris的void put(CrawlURI curi, boolean overwriteIfPresent) 方法，将CrawlURI curi放入待采集队列
现在总结一下：
BdbFrontier对象的void schedule(CrawlURI curi)方法首先通过调用BdbUriUniqFilter对象void add(String key, CrawlURI value)方法，
里面先判断当前CrawlURI curi是否已经添加过，然后回调BdbFrontier对象的void receive(CrawlURI curi)方法，
而BdbFrontier对象的void receive(CrawlURI curi)方法：根据当前CrawlURI curi的ClassKey获取相应的BdbWorkQueue对象，然后调用BdbWorkQueue对象的long enqueue(final WorkQueueFrontier frontier,CrawlURI curi)方法，最后调用将CrawlURI curi添加到BdbFrontier的成员变量BdbMultipleWorkQueues pendingUris的void put(CrawlURI curi, boolean overwriteIfPresent) 方法，归入pendingUris队列
---------------------------------------------------------------------------
本系列Heritrix 3.1.0 源码解析系本人原创
转载请注明出处 博客园 刺猬的温驯
本文链接 http://www.cnblogs.com/chenying99/archive/2013/04/16/3023659.html
