六TCC-transaction的事务恢复

六 TCC-transaction的事务恢复

五 TCC-transaction的事务恢复

由于2.X提供了embedded和server的部署模式，在embedded模式下，事务恢复仍旧由tcc-client实现；在server模式时，事务恢复交给tcc-server模块实现。这里，我仅以tcc-client端（embedded模式）的事务模式，研究Tcc-transaction的事务恢复过程。

5.1 TCC-Client的启动

public class SpringTccClient extends TccClient implements TransactionManagerFactory {

    public SpringTccClient() {
        super(new ClientConfig());
    }

    public SpringTccClient(ClientConfig clientConfig) {
        super(clientConfig);
    }
}

SpringTccClient类在spring配置类中，实例化进入spring容器。

@Configuration
@EnableAspectJAutoProxy(proxyTargetClass = true)
@ComponentScan(value = "org.mengyun.tcctransaction", excludeFilters = {@ComponentScan.Filter(type = FilterType.ASSIGNABLE_TYPE, classes = {XmlTccTransactionConfiguration.class})})
public class AnnotationTccTransactionConfiguration {

     @Bean("springBeanFactory")
    public SpringBeanFactory getSpringBeanFactory() {
        return new SpringBeanFactory();
    }
  
    @Bean
    @DependsOn({"springBeanFactory"})
    public SpringTccClient getTccClient() {
        return new SpringTccClient(clientConfig);
    }
    
    }

随后，在TccClient的初始化过程中：

  @Override
    @PostConstruct
    //spring容器启动的时候，执行该方法
    public void start() throws Exception {
        this.isStarting = true;

        //如果TCC-Transaction的客户端client的存储模式为REMOTING，表示使用server的部署结构
        if (this.clientConfig.getStorageType() == StorageType.REMOTING) {
            try {
                this.registryService.start();
                this.registryService.subscribe();
            } catch (Exception e) {
                logger.error("failed to initialize registryService, stop the application!", e);
                StopUtils.stop();
            }
            initializeRemotingClient(); //初始化远程client
          
        } else { //此时，使用本地存储，事务对象的持久化和事务恢复，都在本地

            registerDomain(this.clientConfig.getDomain());
            if (transactionRepository.supportRecovery() && this.clientConfig.isRecoveryEnabled()) {
							//TAG1 scheduler
              //TAG2 scheduler.registerScheduleAndStartIfNotPresent
                scheduler.registerScheduleAndStartIfNotPresent(this.clientConfig.getDomain());
            }
        }

        this.isStarting = false;
    }

tccclient.start方法在spring容器启动加载该对象时，执行。

TccClient.start():
		1 StorageType.REMOTING 当使用remoting存储模式时，需要向remotingClient注册当前client；
    2 其他情况，使用本地的持久化和事务恢复
      scheduler.registerScheduleAndStartIfNotPresent

TAG1 scheduler

当前scheduler对象在实例化时创建：

        this.scheduler = new RecoveryScheduler(this.clientConfig);

public class RecoveryScheduler {

    public static final String JOB_NAME = "TCC_JOB_%s"; //任务名称
    public static final String TRIGGER_NAME = "TCC_TRIGGER_%s"; //触发任务的名称
    private static final Logger logger = LoggerFactory.getLogger(RecoveryScheduler.class.getSimpleName());
    private RecoveryConfig recoveryConfig;
		//缓存domain和Scheduler---一个domain一个Scheduler
    private Map<String, Scheduler> schedulers = new ConcurrentHashMap<>();

    public RecoveryScheduler(RecoveryConfig recoveryConfig) {
        this.recoveryConfig = recoveryConfig;
    }

TAG2 scheduler.registerScheduleAndStartIfNotPresent

向RecoveryScheduler定时恢复任务器中创建、缓存scheduler并启动

    public void registerScheduleAndStartIfNotPresent(String domain) {
        Scheduler scheduler = registerScheduleIfNotPresent(domain); //创建、缓存scheduler
        start(scheduler);//启动scheduler
    }

    public Scheduler registerScheduleIfNotPresent(String domain) {
      //如果没有创建过切没有缓存入schedulers的本地缓存中
        if (!schedulers.containsKey(domain)) {
            synchronized (RecoveryScheduler.class) {
              //如果没有创建和缓存过Scheduler
                if (!schedulers.containsKey(domain)) { 
                  //TAG2.1 createScheduler
                    Scheduler scheduler = createScheduler(domain);
                  //TAG2.2 scheduleJob
                    scheduleJob(scheduler, domain);
                    schedulers.put(domain, scheduler);
                }
            }
        }
        return schedulers.get(domain);
    }

TAG2.1 createScheduler 创建scheduler内的任务JobDetail

该方法创建scheduler内执行的任务，并设置定时任务的触发器

   private void scheduleJob(Scheduler scheduler, String domain) {

        String jobName = String.format(JOB_NAME, domain);
        String triggerName = String.format(TRIGGER_NAME, domain);

     //创建Quartz定时任务执行器的JobDetail
     //TAG2.1.1 QuartzRecoveryTask任务
        JobDetail jobDetail = JobBuilder.newJob(QuartzRecoveryTask.class).withIdentity(jobName).build();
        jobDetail.getJobDataMap().put(MixAll.DOMAIN, domain);
        //定义触发器
        CronTrigger cronTrigger = TriggerBuilder.newTrigger().withIdentity(triggerName)
                //TCC配置的操作间隔：0/30 * * * * ? 每30秒一次
                .withSchedule(CronScheduleBuilder.cronSchedule(recoveryConfig.getCronExpression())
                        //如果任务时间完成-下一次触发时间<misfireThreshold，立即执行；否则，等待下一次触发时间
                        .withMisfireHandlingInstructionDoNothing()).build();

        try {
          //检查当前job任务是否存在于scheduler中
            if (!scheduler.checkExists(JobKey.jobKey(jobName))) {
              
              //TAG2.1.2 scheduler.scheduleJob
              //如果不存在，就按JobDetail、触发器cronTrigger执行定时任务
                scheduler.scheduleJob(jobDetail, cronTrigger);
            } else {
                if (recoveryConfig.isUpdateJobForcibly()) {
                  //如果存在，就先删除scheduler内的job，再传入并执行新的job
                    scheduler.deleteJob(JobKey.jobKey(jobName));
                    scheduler.scheduleJob(jobDetail, cronTrigger);
                }
            }
        } catch (SchedulerException se) {
            try {
                scheduler.shutdown();
            } catch (Exception ignore) {
                //ignore
            }
            throw new SystemException(String.format("register recovery task for domain<%s> failed", domain), se);
        }
    }

Quartz每30秒执行QuartzRecoveryTask任务，进行事务的恢复重试。

TAG2.1.1 QuartzRecoveryTask任务

这是TCC-Transaction创建的Quartz定时任务执行类

@DisallowConcurrentExecution
public class QuartzRecoveryTask implements Job {

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        String domain = context.getJobDetail().getJobDataMap().getString(MixAll.DOMAIN);
        logger.info("start recovery {}", domain);
      //RTask1 TransactionStoreRecovery.startRecover(domain)
        FactoryBuilder.factoryOf(TccService.class).getInstance().getTransactionStoreRecovery().startRecover(domain);
    }
}

RTask1 TransactionStoreRecovery.startRecover(domain)执行任务恢复

 public class TransactionStoreRecovery implements Closeable {

    public void startRecover(String domain) {

        try {
            String offset = null;

            int totalCount = 0;
            do {

              //RT1.1 loadErrorTransactionsByPage分页加载错误的事务transaction
                Page<TransactionStore> page = loadErrorTransactionsByPage(domain, offset);

                if (!page.getData().isEmpty()) {
                  //RT1.2 并发恢复错误事务transaction
                    concurrentRecoveryErrorTransactions(page.getData());
                    offset = page.getNextOffset();
                    totalCount += page.getData().size();
                } else {
                    break;
                }
            } while (true);

            // 告警
            AlertManager.tryAlert(domain, totalCount, transactionStorage);

            logger.debug("total recovery count {} from repository:{}", totalCount, transactionStorage.getClass().getName());
        } catch (Throwable e) {
            logger.error("recovery failed from repository:{}.", transactionStorage.getClass().getName(), e);
        }
    }
    
    }

RT1.1 loadErrorTransactionsByPage分页加载错误的事务transaction

    private Page<TransactionStore> loadErrorTransactionsByPage(String domain, String offset) {

        long currentTimeInMillis = Instant.now().toEpochMilli();
            //事务恢复间隔默认设置30s
      //RT1.1.1 transactionStorage).findAllUnmodifiedSince
        return ((StorageRecoverable) transactionStorage).findAllUnmodifiedSince(domain, new Date(currentTimeInMillis - recoveryConfig.getRecoverDuration() * 1000), offset, recoveryConfig.getFetchPageSize());
    }

注意这里对于异常事务的定义：

new Date(currentTimeInMillis - recoveryConfig.getRecoverDuration() * 1000)

就是当前时间-事务恢复间隔>事务变更时间。也就是事务的修改事件，在超过事务恢复间隔时间后，仍旧未被执行，就被定义为异常的事务。对于这部分事务需要进行恢复、重试。

RT1.1.1 transactionStorage).findAllUnmodifiedSince 调用数据库查询transaction

找到所有未修改的事务信息。

public class JdbcTransactionStorage extends AbstractTransactionStorage implements StorageRecoverable {

    @Override
    public Page<TransactionStore> findAllUnmodifiedSince(String domain, Date date, String offset, int pageSize) {
        return pageList(domain, date, offset, pageSize, false);
    }
  
      private Page<TransactionStore> pageList(String domain, Date date, String offset, int pageSize, boolean isMarkDeleted) {

        List<TransactionStore> transactions = new ArrayList<>();

        Connection connection = null;
        PreparedStatement stmt = null;

        int currentOffset = StringUtils.isEmpty(offset) ? 0 : Integer.parseInt(offset);

        try {
          //获取数据库连接
            connection = this.getConnection();
/**………………………………………………………………………………………………………创建查询语句…………………………………………………………………………………………………………………………… */
            StringBuilder builder = new StringBuilder();
            builder.append(SQL_SELECT_PREFIX_FOR_TCC_TRANSACTION + getTableName() + " WHERE LAST_UPDATE_TIME < ?");
            builder.append(" AND IS_DELETE = ?");
            builder.append(StringUtils.isNotEmpty(domain) ? " AND DOMAIN = ?" : "");
            builder.append(" ORDER BY TRANSACTION_ID ASC");
            builder.append(String.format(" LIMIT %s, %d", currentOffset, pageSize));

            stmt = connection.prepareStatement(builder.toString());
						//设置sql语句内的参数
            stmt.setTimestamp(1, new Timestamp(date.getTime()));
            stmt.setInt(2, isMarkDeleted ? MARK_DELETED_YES : MARK_DELETED_NO);

            if (StringUtils.isNotEmpty(domain)) {
                stmt.setString(3, domain);
            }
						//执行sql语句
            ResultSet resultSet = stmt.executeQuery();

            this.constructTransactions(resultSet, transactions);
        } catch (Throwable e) {
            throw new TransactionIOException(e);
        } finally {
            closeStatement(stmt);
            this.releaseConnection(connection);
        }

        return new Page<>(String.valueOf(currentOffset + transactions.size()), transactions);
    }
  
}

这里，对在持久存储数据库查询的语句：

SELECT DOMAIN,ROOT_XID,XID,CONTENT,STATUS,TRANSACTION_TYPE,CREATE_TIME,LAST_UPDATE_TIME,RETRIED_COUNT,VERSION,IS_DELETE,ROOT_DOMAIN,REQUEST_ID 
FROM TableName
 WHERE LAST_UPDATE_TIME < TIME
 AND IS_DELETE =0 #未被标记删除
 AND DOMAIN =domain
 ORDER BY TRANSACTION_ID ASC
  currentOffset, pageSize

从数据库查询error的transaction

RT1.2 concurrentRecoveryErrorTransactions 并发恢复错误事务transaction

  private void concurrentRecoveryErrorTransactions(List<TransactionStore> transactions) throws InterruptedException, ExecutionException {

        initLogStatistics();
				//创建恢复transaction的task
        List<RecoverTask> tasks = new ArrayList<>();
        for (TransactionStore transaction : transactions) {
          //RT1.2.1 RecoverTask error任务的task
            tasks.add(new RecoverTask(transaction));
        }
			//RT1.2.2 recoveryExecutorService.invokeAll批量执行任务
        List<Future<Void>> futures = recoveryExecutorService.invokeAll(tasks, CONCURRENT_RECOVERY_TIMEOUT, TimeUnit.SECONDS);

        for (Future future : futures) {
            future.get();
        }
    }

RT1.2.1 RecoverTask error任务的task

TransactionStoreRecovery的内部类

   class RecoverTask implements Callable<Void> {

        TransactionStore transaction;

        public RecoverTask(TransactionStore transaction) {
            this.transaction = transaction;
        }

        @Override
        public Void call() throws Exception {
          
            recoverErrorTransaction(transaction);
            return null;
        }
    }

  private void recoverErrorTransaction(TransactionStore transactionStore) {
				
    //如果当前重试次数，超过recoveryConfig.getMaxRetryCount()
        if (transactionStore.getRetriedCount() > recoveryConfig.getMaxRetryCount()) { //1

            logSync.lock();
            try {
              //当triggerMaxRetryPrintCount的log输出不大于max，继续输出重试信息
                if (triggerMaxRetryPrintCount.get() < logMaxPrintCount) {
                    logger.error(
                            "recover failed with max retry count,will not try again. domain:{}, xid:{}, rootDomain:{}, rootXid:{}, status:{},retried count:{}",
                            transactionStore.getDomain(),
                            transactionStore.getXid(),
                            transactionStore.getRootDomain(),
                            transactionStore.getRootXid(),
                            transactionStore.getStatusId(),
                            transactionStore.getRetriedCount());
                    triggerMaxRetryPrintCount.incrementAndGet();
                } else if (triggerMaxRetryPrintCount.get() == logMaxPrintCount) {
                    logger.error("Too many transactionStore's retried count max then MaxRetryCount during one page transactions recover process , will not print errors again!");
                }

            } finally {
                logSync.unlock();
            }

            return;
        } //1
    
/** …………………………………………………………………………………………………未超过重试次数时……………………………………………………………………………………………………………………………*/
        try {
				/**……………………………………………………………………………………………………根事务的恢复逻辑…………………………………………………………………… */
            if (transactionStore.getTransactionTypeId() == TransactionType.ROOT.getId()) {

                switch (TransactionStatus.valueOf(transactionStore.getStatusId())) {
                    case CONFIRMING:
                        commitTransaction(transactionStore); 
                        break;
                    case CANCELLING:
                        rollbackTransaction(transactionStore);
                        break;
                    case TRYING:
                        tryTreatAsFailed(transactionStore, TransactionStatus.CANCELLING);
                        break;
                    default:
                        //ignore it.
                        break;

                }

            } 
				/**……………………………………………………………………………………………………branch分支事务的恢复逻辑…………………………………………………………………… */
          else {

                //transactionStore type is BRANCH
                switch (TransactionStatus.valueOf(transactionStore.getStatusId())) {
                    case CONFIRMING:
                    //ERR1 commitTransaction
                        commitTransaction(transactionStore);
                        break;
                    case CANCELLING:
                    case TRY_FAILED:
                    //ERR2 rollbackTransaction
                        rollbackTransaction(transactionStore);
                        break;
                    case TRY_SUCCESS:

                        if (storageMode == StorageMode.CENTRAL) {

                            //check the root transactionStore
                            TransactionStore rootTransaction = transactionStorage.findByXid(transactionStore.getRootDomain(), transactionStore.getRootXid());

                            if (rootTransaction == null) {
                                // In this case means the root transactionStore is already rollback.
                                // Need cancel this branch transactionStore.
                                rollbackTransaction(transactionStore);
                            } else {
                                switch (TransactionStatus.valueOf(rootTransaction.getStatusId())) {
                                    case CONFIRMING:
                                        commitTransaction(transactionStore);
                                        break;
                                    case CANCELLING:
                                        rollbackTransaction(transactionStore);
                                        break;
                                    default:
                                        break;
                                }
                            }
                        }
                        break;
                    case TRYING:
                    //ERR3 tryTreatAsFailed
                        tryTreatAsFailed(transactionStore, TransactionStatus.TRY_FAILED);
                        break;
                    default:
                        //ignore it.
                        break;
                }

            }

        } catch (Throwable throwable) {

            if (throwable instanceof TransactionOptimisticLockException
                    || ExceptionUtils.getRootCause(throwable) instanceof TransactionOptimisticLockException) {

                logger.warn(
                        "optimisticLockException happened while recover. txid:{}, status:{},retried count:{}",
                        transactionStore.getXid(),
                        transactionStore.getStatusId(),
                        transactionStore.getRetriedCount());
            } else {

                logSync.lock();
                try {
                    if (recoveryFailedPrintCount.get() < logMaxPrintCount) {
                        try {
                            logger.error("recover failed, txid:{}, status:{},retried count:{},transactionStore content:{}",
                                    transactionStore.getXid(),
                                    transactionStore.getStatusId(),
                                    transactionStore.getRetriedCount(),
                                    jackson.writeValueAsString(transactionStore), throwable);
                        } catch (JsonProcessingException e) {
                            logger.error("failed to serialize transactionStore {}", transactionStore.toString(), e);
                        }
                        recoveryFailedPrintCount.incrementAndGet();
                    } else if (recoveryFailedPrintCount.get() == logMaxPrintCount) {
                        logger.error("Too many transactionStore's recover error during one page transactions recover process , will not print errors again!");
                    }
                } finally {
                    logSync.unlock();
                }
            }
        }
    }

这里，对error事务的恢复逻辑如下：

事务状态主要有：CONFIRMING、CANCELLING、TRYING、TRY_FAILED、TRY_SUCCESSED。

根事务主要需要对CONFIRMING、CANCELLING、TRYING进行处理；

分支事务，需要对所有情况处理，此外，对于TRY_SUCCESSED的事务状态，要根据根事务root的事务状态：如果store模式为central中心存储，仅用于客户端，当取值为CENTRAL，分支事务补偿时会对TRY_SUCCESS状态的事件，进行处理。

如果此时root事务不存在，则表示根事务已经rollback，此时branch需要rollback；

如果root事务存在，处于confirming，就confirm；cancelling，就cnacel；其他情况，不处理。

transactionStore.getTransactionTypeId() ROOT根事务时
  事务当前状态：
  	 1 case CONFIRMING:
       commitTransaction(transactionStore); //重新commit
     2 case CANCELLING:
       rollbackTransaction(transactionStore);//重新rollback
     3 case TRYING:
       tryTreatAsFailed(transactionStore, TransactionStatus.CANCELLING); //将其视作失败，并设置状态为cancelling
		其他情况：
      忽略不处理

transactionStore.getTransactionTypeId() BRANCH分支事务时
  事务当前状态：
     1 case CONFIRMING:
       commitTransaction(transactionStore);

     2 case CANCELLING:
     case TRY_FAILED:
       rollbackTransaction(transactionStore);

		3 case TRY_SUCCESS:
		if (storageMode == StorageMode.CENTRAL) {
      if (rootTransaction == null)
        rollbackTransaction(transactionStore)
      else
        case CONFIRMING:
            commitTransaction(transactionStore); //重新commit
     		case CANCELLING:
       rollbackTransaction(transactionStore);//重新rollback
      其他情况：
        不处理
        
     4 case TRYING:
       tryTreatAsFailed(transactionStore, TransactionStatus.TRY_FAILED); //将其视作失败，并设置状态为TRY_FAILED

ERR1 commitTransaction

   private void commitTransaction(TransactionStore transactionStore) {
        recoveryExecutor.commit(transactionStore);
    }

public class ClientRecoveryExecutor implements RecoveryExecutor {

    public void commit(TransactionStore transactionStore) {

        Transaction transaction = TransactionConvertor.getTransaction(transactionSerializer, transactionStore);

      //将事务的重试次数+1
        transaction.setRetriedCount(transaction.getRetriedCount() + 1);
        transaction.setStatus(TransactionStatus.CONFIRMING); //设置事务状态confirming
        try {
            transactionRepository.update(transaction); //更新
        } catch (TransactionOptimisticLockException e) {
            logger.debug("multiple instances try to recovery<commit> the same transaction<{}>, this instance ignore the recovery.", transactionStore.getXid());
            return;
        }
        transaction.commit(); //重新执行commit提交
        transactionRepository.delete(transaction); //删除
    }

ERR2 rollbackTransaction

rollback的恢复处理，和commitTransaction相同

ERR3 tryTreatAsFailed

    private void tryTreatAsFailed(TransactionStore transactionStore, TransactionStatus transactionStatus) {
        Date lastUpdateTime = transactionStore.getLastUpdateTime();
        Date currentTime = new Date();
      //trying阶段的恢复，视作failed，同样具有最大次数
        int maxTimeTreatTryingAsFailed = recoveryConfig.getMaxTimeTreatTryingAsFailed();
        if (maxTimeTreatTryingAsFailed > 0
                && (currentTime.getTime() - lastUpdateTime.getTime()) > maxTimeTreatTryingAsFailed * 1000) {
            //update the status to cancel or try failed, waiting for the recovery task to recover
          //版本号+1
            transactionStore.setVersion(transactionStore.getVersion() + 1);
          //TransactionStatus.TRY_FAILED设置transaction的状态为try-failed
            transactionStore.setStatusId(transactionStatus.getId());
            transactionStorage.update(transactionStore);
        }
    }

这里的操作，是将transaction的状态设置为TransactionStatus.TRY_FAILED。等待重试时候，进行rollback。

注意：

对于root根事务的事务恢复，只对trying的视作cancelling，然后再下一次的失败重试时，会对root为cancelling的直接rollback；

对于branch的分支事务，对于trying的视作try-failed。然后等待下一次再次的重试，在branch的再次恢复中，对于try-failed的会直接rollback；

RT1.2.2 recoveryExecutorService.invokeAll批量执行任务

调用恢复transaction的线程池，调用invokeAll执行批量任务。

TAG2.2 Scheduler.scheduleJob

这里调用quartz定时任务处理器的类，执行前述的定时恢复任务。

到这里，TCC-client端的事务恢复就执行完毕。

5.2 TCC-Server的启动(todo)

基本同tcc-server，等随后贴源码过程