Hdfs Active Namenode向JournalNode同步editlog流程

FSEditLog

Active namenode进程上使用的类,达到条件后执行logSync()向journalnode同步最新的EditLog段
调用过程

FSEditLog.logSync(long mytxid)
->EditLogOutputStream.flush()
->EditLogOutputStream.flush(boolean durable)
->EditLogOutputStream.flushAndSync(boolean durable)
->JournalSet.flushAndSync(boolean durable)
->JournalSet.mapJournalsAndReportErrors(
      JournalClosure closure, String status)

此过程最终的调用函数为JournalSet中的

/**
   * Apply the given operation across all of the journal managers, disabling
   * any for which the closure throws an IOException.
   * @param closure {@link JournalClosure} object encapsulating the operation.
   * @param status message used for logging errors (e.g. "opening journal")
   * @throws IOException If the operation fails on all the journals.
   */
  private void mapJournalsAndReportErrors(
      JournalClosure closure, String status) throws IOException{

    List<JournalAndStream> badJAS = Lists.newLinkedList();
    for (JournalAndStream jas : journals) {
      try {
        closure.apply(jas);
      } catch (Throwable t) {
        if (jas.isRequired()) {
          final String msg = "Error: " + status + " failed for required journal ("
            + jas + ")";
          LOG.fatal(msg, t);
          // If we fail on *any* of the required journals, then we must not
          // continue on any of the other journals. Abort them to ensure that
          // retry behavior doesn't allow them to keep going in any way.
          abortAllJournals();
          // the current policy is to shutdown the NN on errors to shared edits
          // dir. There are many code paths to shared edits failures - syncs,
          // roll of edits etc. All of them go through this common function 
          // where the isRequired() check is made. Applying exit policy here 
          // to catch all code paths.
          terminate(1, msg);
        } else {
          LOG.error("Error: " + status + " failed for (journal " + jas + ")", t);
          badJAS.add(jas);          
        }
      }
    }
    disableAndReportErrorOnJournals(badJAS);
    if (!NameNodeResourcePolicy.areResourcesAvailable(journals,
        minimumRedundantJournals)) {
      String message = status + " failed for too many journals";
      LOG.error("Error: " + message);
      throw new IOException(message);
    }
  }

mapJournalsAndReportErrors中如果成功个数没有达到minimumRedundantJournals,则namenode会主动shutdown。

posted @ 2021-12-08 01:33  Yu\.W  阅读(240)  评论(0编辑  收藏  举报