Region Compact请求是在Region MemStore Flush之后被触发的：
boolean shouldCompact = region.flushcache(); // We just want to check the size boolean shouldSplit = region.checkSplit() != null; if (shouldSplit) { this.server.compactSplitThread.requestSplit(region); } else if (shouldCompact) { server.compactSplitThread.requestCompaction(region, getName()); } server.getMetrics().addFlush(region.getRecentFlushInfo());
region.flushcache方法负责该Region实例的Flush操作，该操作的返回值shouldCompact如果为true，则表示该Region达到了Region Compact的要求，但此时并不一定会触发Compact请求，因为shouldCompact之后会有shouldSplit的判断，只有当shouldSplit为false且shouldCompact为true时才会触发Compact请求。
注：某一Region的Flush、Split、Compact是分别在不同的线程中进行的，三者之间的协调会在后续详细介绍，在此仅仅关注Compact的触发条件，以及Compact进行时如何选取StoreFile进行合并即可。
Region Compact触发（即shouldCompact为true）的判断条件是该Region中各个Store中的StoreFile数目：
/** * See if there's too much store files in this store * * @return true if number of store files is greater than the number defined * in minFilesToCompact */ public boolean needsCompaction() { return (storefiles.size() - filesCompacting.size()) > minFilesToCompact; }
在某一Region做Flush操作时，内部是逐个对该Region的多个Store依次进行Flush的，每一个Store Flush完成之后，就会调用上述代码进行判断：
storefiles：表示Store Flush之后，StoreFile的总数目；
filesCompacting：表示Store中正处于Compact状态的StoreFile的数目，即表示该Store中的某些StoreFile已被选择进行Compact，这次判断需要忽略这些StoreFile；
minFilesToCompact：表示进行Store Compact时Store中的StoreFile数目（不包含已被选择进行Compact的StoreFile）所需要达到的下限值，它的值是由以下代码计算而来的：
// By default, compact if storefile.count >= minFilesToCompact this.minFilesToCompact = Math.max(2, conf.getInt( "hbase.hstore.compaction.min", /* old name */conf .getInt("hbase.hstore.compactionThreshold", 3)));
注意：hbase.hstore.compactionThreshold是一个旧的配置项，当配置项中出现hbase.hstore.compaction.min时，它将失效。
由上所示可知，shouldCompact为true的前提条件是当前Region中的某个Store中的StoreFile数目需要达到下限要求（需要去除那些已处于合并状态的StoreFile），如果满足条件，则通过requestCompaction发出具体的Compact请求，requestCompaction有多个重载方法，最终执行流程会被导向下述这些重载方法：
@Override public synchronized List<CompactionRequest> requestCompaction( final HRegion r, final String why, int pri, final List<CompactionRequest> requests) throws IOException { List<CompactionRequest> ret; // not a special compaction request, so make out own list if (requests == null) { ret = new ArrayList<CompactionRequest>(r.getStores().size()); for (Store s : r.getStores().values()) { ret.add(requestCompaction(r, s, why, pri, null)); } } else { ret = new ArrayList<CompactionRequest>(requests.size()); for (CompactionRequest request : requests) { ret.add(requestCompaction(r, request.getStore(), why, pri, request)); } } return ret; }
可以看出，实际的Compact请求也是以Store为单位发出的，如下代码所示：
@Override public synchronized CompactionRequest requestCompaction(final HRegion r, final Store s, final String why, int priority, CompactionRequest request) throws IOException { if (this.server.isStopped()) { return null; } CompactionRequest cr = s.requestCompaction(priority, request); if (cr != null) { cr.setServer(server); if (priority != Store.NO_PRIORITY) { cr.setPriority(priority); } ThreadPoolExecutor pool = s.throttleCompaction(cr.getSize()) ? largeCompactions : smallCompactions; pool.execute(cr); if (LOG.isDebugEnabled()) { String type = (pool == smallCompactions) ? "Small " : "Large "; LOG.debug(type + "Compaction requested: " + cr + (why != null && !why.isEmpty() ? "; Because: " + why : "") + "; " + this); } } else { if (LOG.isDebugEnabled()) { LOG.debug("Not compacting " + r.getRegionNameAsString() + " because compaction request was cancelled"); } } return cr; }
首先会根据Store的状态形成一个CompactionRequest对象（实现了Runnable接口，所谓的请求实际上就是一个Runnable对象，以线程的形式执行），然后根据当前Store需要进行Compact操作的StoreFile的总大小（不是文件数目）判断是何种类型的Compact（LargeCompaction或者SmallCompaction），从而将CompactionRequest提交至不同的线程池中执行。
LargeCompaction与SmallCompaction的选择过程如下：
boolean throttleCompaction(long compactionSize) { long throttlePoint = conf.getLong( "hbase.regionserver.thread.compaction.throttle", 2 * this.minFilesToCompact * this.region.memstoreFlushSize); return compactionSize > throttlePoint; }
至此，CompactRequest请求已被提交至相应的线程池中，根据线程池的执行策略，在适当的时机被执行。
CompactionRequest
如何根据Store的状态生成相应的CompactionRequest对象，即选择Store中的哪些StoreFile进行Compact，该过程是由Store requestCompaction方法完成的，下面对该方法的核心代码进行讲述。
代码主要是在一个同步代码块中完成的，
synchronized (filesCompacting) { ...... }
随着MemStore的不断Flush操作，可能会造成同一个Region的多次Compact Request，为了避免同一StoreFile出现在多个Compact中，所以需要对filesCompacting（表示已被选取进行Compact的StoreFile）进行互斥访问。
接下来的所有代码都是出现在上述同步代码块中的。
// candidates = all storefiles not already in compaction queue List<StoreFile> candidates = Lists.newArrayList(storefiles);
选取Store中的所有StoreFile作为备选对象，storefiles是依据StoreFile.Comparators.FLUSH_TIME进行排序的，此时candidates以及后续的filesCompacting也是有序的（older –> newer）。
if (!filesCompacting.isEmpty()) { // exclude all files older than the newest file we're // currently // compacting. this allows us to preserve contiguity // (HBASE-2856) StoreFile last = filesCompacting .get(filesCompacting.size() - 1); int idx = candidates.indexOf(last); Preconditions.checkArgument(idx != -1); candidates.subList(0, idx + 1).clear(); }
如果filesCompacting不为空，则需要从candidates中移除一些StoreFile：
（1）选中filesCompacting中的最后一个StoreFile保存至变量last中，即filesCompacting中FlushTime最新的那个StoreFile；
（2）移除candidates中所有FlushTime比last旧的StoreFile。
这样做的目的主要是为了Compact操作的持续性，即每次尽可以多的合并新产生的StoreFile，不要重复去Compact那些已经被合并过或正在进行Compact的StoreFile。
boolean override = false; if (region.getCoprocessorHost() != null) { override = region.getCoprocessorHost().preCompactSelection( this, candidates, request); } CompactSelection filesToCompact; if (override) { // coprocessor is overriding normal file selection filesToCompact = new CompactSelection(conf, candidates); } else { filesToCompact = compactSelection(candidates, priority); } if (region.getCoprocessorHost() != null) { region.getCoprocessorHost().postCompactSelection( this, ImmutableList.copyOf(filesToCompact .getFilesToCompact()), request); }
这里不讨论Coprocessor对Compact的影响，所以override为false，执行compactSelection算法（详细算法描述见后）进行选取，选取结果会被保存至filesToCompact中。
// no files to compact if (filesToCompact.getFilesToCompact().isEmpty()) { return null; }
经过相应算法选取之后，如果选取结果为空，则表示不需要进行Compact，返回null即可。
// basic sanity check: do not try to compact the same StoreFile // twice. if (!Collections.disjoint(filesCompacting, filesToCompact.getFilesToCompact())) { // TODO: change this from an IAE to LOG.error after // sufficient testing Preconditions.checkArgument(false, "%s overlaps with %s", filesToCompact, filesCompacting); }
这是一个对选取结果进行核验的工作，主要是为了保证选取结果不会出现在filesCompacting中，即不会对同一个StoreFile进行两次Compact操作。
filesCompacting.addAll(filesToCompact.getFilesToCompact());
Collections.sort(filesCompacting,
StoreFile.Comparators.FLUSH_TIME);
将选取结果添加至filesCompacting中，并对更新后的filesCompacting依据FlushTime重新排序。
// major compaction iff all StoreFiles are included boolean isMajor = (filesToCompact.getFilesToCompact().size() == this.storefiles .size()); if (isMajor) { // since we're enqueuing a major, update the compaction wait // interval this.forceMajor = false; }
如果选取结果包含该Store中所有的StoreFile，则表示我们应该进行一个Major Compaction，并取消Force Major Compaction。
// everything went better than expected. create a compaction // request int pri = getCompactPriority(priority);
计算本次Compact的优先级：
/** * @return The priority that this store should have in the compaction queue * @param priority */ public int getCompactPriority(int priority) { // If this is a user-requested compaction, leave this at the highest // priority if (priority == PRIORITY_USER) { return PRIORITY_USER; } else { return this.blockingStoreFileCount - this.storefiles.size(); } }
如果Compact Request是由用户发起的，则需要给该Compaction赋于最高的优先级PRIORITY_USER（1），否则需要进行差值计算得出，其中blockingStoreFileCount来源于如下代码：
this.blockingStoreFileCount = conf.getInt( "hbase.hstore.blockingStoreFiles", 7);
以下代码开始生成具体的CompactionRequest的对象，在我们讲述的流程中request为null，所以通过相应的构造函数创建对象即可。
// not a special compaction request, so we need to make one if (request == null) { request = new CompactionRequest(region, this, filesToCompact, isMajor, pri); } else { // update the request with what the system thinks the // request should be // its up to the request if it wants to listen request.setSelection(filesToCompact); request.setIsMajor(isMajor); request.setPriority(pri); }
代码至此完成了一个CompactRequest的形成。
CompactSelection
该算法由方法compactSelection实现：
/** * Algorithm to choose which files to compact * * Configuration knobs: * * "hbase.hstore.compaction.ratio" normal case: minor compact when file <= * sum(smaller_files) * ratio * * "hbase.hstore.compaction.min.size" unconditionally compact individual * files below this size * * "hbase.hstore.compaction.max.size" never compact individual files above * this size (unless splitting) * * "hbase.hstore.compaction.min" min files needed to minor compact * * "hbase.hstore.compaction.max" max files to compact at once (avoids OOM) * * @param candidates * candidate files, ordered from oldest to newest * @return subset copy of candidate list that meets compaction criteria * @throws IOException */ CompactSelection compactSelection(List<StoreFile> candidates, int priority) throws IOException { ...... }
方法开始处有如下注释:
// ASSUMPTION!!! filesCompacting is locked when calling this function /* * normal skew: * * older ----> newer _ | | _ | | | | _ --|-|- |-|- * |-|---_-------_------- minCompactSize | | | | | | | | _ | | | | | | | * | | | | | | | | | | | | | | | | | | | */
该算法的选取主要从旧到新依次进行，minCompactSize 的应用见后面代码所示。
CompactSelection compactSelection = new CompactSelection(conf, candidates);
创建一个CompactSelection对象，在构造方法中主要是初始化了一些参数。
boolean forcemajor = this.forceMajor && filesCompacting.isEmpty();
根据forceMajor（在compactSelection方法调用之前已经计算出该值）与filesCompacting的状态计算forcemajor的值。如果forecemajor为false则执行下述代码：
if (!forcemajor) { // Delete the expired store files before the compaction selection. if (conf.getBoolean("hbase.store.delete.expired.storefile", true) && (ttl != Long.MAX_VALUE) && (this.scanInfo.minVersions == 0)) { CompactSelection expiredSelection = compactSelection .selectExpiredStoreFilesToCompact(EnvironmentEdgeManager .currentTimeMillis() - this.ttl); // If there is any expired store files, delete them by // compaction. if (expiredSelection != null) { return expiredSelection; } } // do not compact old files above a configurable threshold // save all references. we MUST compact them int pos = 0; while (pos < compactSelection.getFilesToCompact().size() && compactSelection.getFilesToCompact().get(pos) .getReader().length() > maxCompactSize && !compactSelection.getFilesToCompact().get(pos) .isReference()) { ++pos; } if (pos != 0) { compactSelection.clearSubList(0, pos); } }
其中
// Delete the expired store files before the compaction selection. if (conf.getBoolean("hbase.store.delete.expired.storefile", true) && (ttl != Long.MAX_VALUE) && (this.scanInfo.minVersions == 0)) { CompactSelection expiredSelection = compactSelection .selectExpiredStoreFilesToCompact(EnvironmentEdgeManager .currentTimeMillis() - this.ttl); // If there is any expired store files, delete them by // compaction. if (expiredSelection != null) { return expiredSelection; } }
这部分代码主要用于删除那些过期的StoreFile，如果存在有过期的StoreFile，则算法的选取结果即为这些过期的StoreFile，不再需要进行进一步的选取，在此先忽略这一步。
// do not compact old files above a configurable threshold // save all references. we MUST compact them int pos = 0; while (pos < compactSelection.getFilesToCompact().size() && compactSelection.getFilesToCompact().get(pos) .getReader().length() > maxCompactSize && !compactSelection.getFilesToCompact().get(pos) .isReference()) { ++pos; } if (pos != 0) { compactSelection.clearSubList(0, pos); }
compactSelection.getFilesToCompact()的返回值即为candidates，该代码的作用是从前到后淘汰一些文件大小超过配置大小的StoreFile，直至在此过程中遇到下述两种情况：
（1）某StoreFile的文件大小小于或等于maxCompactSize（hbase.hstore.compaction.max.size）；
（2）某StoreFile的文件类型为Reference（该文件类型在Split的过程中产生）。
if (compactSelection.getFilesToCompact().isEmpty()) { LOG.debug(this.getHRegionInfo().getEncodedName() + " - " + this + ": no store files to compact"); compactSelection.emptyFileList(); return compactSelection; }
如果经过前一步的淘汰过程之后，compactSelection.getFilesToCompact()的返回结果为空，则表示没有相应的StoreFile可以进行Compact，选取结束，直接返回即可。
// Force a major compaction if this is a user-requested major // compaction, // or if we do not have too many files to compact and this was requested // as a major compaction boolean majorcompaction = (forcemajor && priority == PRIORITY_USER) || (forcemajor || isMajorCompaction(compactSelection .getFilesToCompact())) && (compactSelection.getFilesToCompact().size() < this.maxFilesToCompact);
majorcompaction为true，需要达到下述两个条件之一：
（1）用户主动请求进行Major Compaction；
（2）forcemajor为true或者满足系统主动进行Major Compaction的条件（主要根据StoreFile时间戳进行判断），而且此时待选取的StoreFile数目没有超过临界值（maxFilesToCompact：hbase.hstore.compaction.max）。
关于用户主动请求或系统主动进行Major Compaction的情况，后续再进行讨论。
接下来的代码会根据条件的不同出现两个大的代码块，分别进行讲述。
if (!majorcompaction && !hasReferences(compactSelection.getFilesToCompact())) { ...... } else { ...... }
（1）如果majorcompaction为false且剩余的待选取StoreFile中不包含Reference类型的文件，则执行如下代码：
// we're doing a minor compaction, let's see what files are // applicable int start = 0; double r = compactSelection.getCompactSelectionRatio();
r的值即为hbase.hstore.compaction.ratio，不考虑配置hbase.offpeak.start.hour、hbase.offpeak.end.hour情况。
// remove bulk import files that request to be excluded from minors compactSelection.getFilesToCompact().removeAll( Collections2.filter(compactSelection.getFilesToCompact(), new Predicate<StoreFile>() { public boolean apply(StoreFile input) { return input.excludeFromMinorCompaction(); } }));
在待选取的StoreFile中移除那些已经被设置从Minor Compaction中移除的StoreFile。
// skip selection algorithm if we don't have enough files if (compactSelection.getFilesToCompact().size() < this.minFilesToCompact) { if (LOG.isDebugEnabled()) { LOG.debug("Not compacting files because we only have " + compactSelection.getFilesToCompact().size() + " files ready for compaction. Need " + this.minFilesToCompact + " to initiate."); } compactSelection.emptyFileList(); return compactSelection; }
如果此时我们选取的StoreFile数目小于minFilesToCompact（配置时注意该值不包含等于的情况），则本次Compact的操作取消，直接返回即可，其中minFilesToCompact的计算代码如下：
// By default, compact if storefile.count >= minFilesToCompact this.minFilesToCompact = Math.max(2, conf.getInt( "hbase.hstore.compaction.min", /* old name */conf .getInt("hbase.hstore.compactionThreshold", 3)));
继续进行选取算法。
// get store file sizes for incremental compacting selection. int countOfFiles = compactSelection.getFilesToCompact().size(); long[] fileSizes = new long[countOfFiles]; long[] sumSize = new long[countOfFiles]; for (int i = countOfFiles - 1; i >= 0; --i) { StoreFile file = compactSelection.getFilesToCompact().get(i); fileSizes[i] = file.getReader().length(); // calculate the sum of fileSizes[i,i+maxFilesToCompact-1) for // algo int tooFar = i + this.maxFilesToCompact - 1; sumSize[i] = fileSizes[i] + ((i + 1 < countOfFiles) ? sumSize[i + 1] : 0) - ((tooFar < countOfFiles) ? fileSizes[tooFar] : 0); }
上述代码实际是做一些准备工作，主要生成了两个重要的数组：fileSizes、sumSize。
fileSizes：依次保存着各个StoreFile的文件大小；
sumSize：对应fileSizes，依次保存着fileSizes[i,i+maxFilesToCompact-1)之和，即sumSize[i]中保存着compactSelection.getFilesToCompact()中第i个StoreFile（包含）之后最多maxFilesToCompact个StoreFile的总大小。
根据这两个数组，继续选举算法：
/* * Start at the oldest file and stop when you find the first file * that meets compaction criteria: * * (1) a recently-flushed, small file (i.e. <= minCompactSize) OR * (2) within the compactRatio of sum(newer_files) * * Given normal skew, any newer files will also meet this criteria * * Additional Note: If fileSizes.size() >> maxFilesToCompact, we * will recurse on compact(). Consider the oldest files first to * avoid a situation where we always compact [end-threshold,end). * Then, the last file becomes an aggregate of the previous * compactions. */ while (countOfFiles - start >= this.minFilesToCompact && fileSizes[start] > Math.max(minCompactSize, (long) (sumSize[start + 1] * r))) { ++start; } int end = Math.min(countOfFiles, start + this.maxFilesToCompact); long totalSize = fileSizes[start] + ((start + 1 < countOfFiles) ? sumSize[start + 1] : 0); compactSelection = compactSelection.getSubList(start, end);
从前到后（根据FlushTime从旧到新）依次淘汰相应的StoreFile，直至遇到某一StoreFile的大小满足条件：
fileSizes[start] <= Math.max(minCompactSize, (long) (sumSize[start + 1] * r))
minCompactSize：hbase.hstore.compaction.min.size
理解其中的一种情况：fileSizes[start] > (long) (sumSize[start + 1] * r) > minCompactSize，它表示当前StoreFile的大小比它之后maxFilesToCompact范围内的所有StoreFile大小之和还要大，我们认为当前StoreFile太大，应淘汰掉，优先合并它之后那些较小的StoreFile。
根据注释可知，HBase希望最近Flush、Small的StoreFile文件大小满足下述两个条件之一：
（1）小于等于minCompactSize；
（2）小于等于the compactRatio of sum(newer_files)。
上述代码最后会更新compactSelection对象的状态。
// if we don't have enough files to compact, just wait if (compactSelection.getFilesToCompact().size() < this.minFilesToCompact) { if (LOG.isDebugEnabled()) { LOG.debug("Skipped compaction of " + this + ". Only " + (end - start) + " file(s) of size " + StringUtils.humanReadableInt(totalSize) + " have met compaction criteria."); } compactSelection.emptyFileList(); return compactSelection; }
如果此时没有足够的StoreFile进行Compact（StoreFile数目小于minFilesToCompact），则跳过这次Compact，直接返回。
（2）如果majorcompaction为true或者待选取的StoreFile中包含Reference文件，则执行如下代码：
if (majorcompaction) { if (compactSelection.getFilesToCompact().size() > this.maxFilesToCompact) { LOG.debug("Warning, compacting more than " + this.maxFilesToCompact + " files, probably because of a user-requested major compaction"); if (priority != PRIORITY_USER) { LOG.error("Compacting more than max files on a non user-requested compaction"); } } } else if (compactSelection.getFilesToCompact().size() > this.maxFilesToCompact) { // all files included in this compaction, up to max int pastMax = compactSelection.getFilesToCompact().size() - this.maxFilesToCompact; compactSelection.getFilesToCompact().subList(0, pastMax) .clear(); }
如果majorcompaction为true，仅需要在选取进行Compact的StoreFile数目大于maxFilesToCompact（hbase.hstore.compaction.max）时，打印一些信息即可；
如果majorcompaction为false，且在选取进行Compact的StoreFile数目大于maxFilesToCompact（hbase.hstore.compaction.max）时，移除掉后面多余的StoreFile。
return compactSelection;
至此，选举算法结束。
