Orchestrator global recovery disable 全局开关

目录

• 1. DB 层
• 2. raft 同步层
• 3. API 层
• 4. snapshot 层
• 5. 自动故障恢复
• 6. Dashboard 页面

Orchestrator中，在MySQL集群粒度，有故障自动恢复开关，在全局粒度，也有一个全局的开关(global recovery disable)。

本文主要介绍全局开关(global recovery disable)的基本实现方式。

下面分别从几个层面阐述。

1. DB 层

在DB层定义一个数据表，用于存储全局开关状态。

global_recovery_disable 数据表:

CREATE TABLE IF NOT EXISTS global_recovery_disable (
			disable_recovery tinyint unsigned NOT NULL COMMENT 'Insert 1 to disable recovery globally',
			PRIMARY KEY (disable_recovery)
		) ENGINE=InnoDB DEFAULT CHARSET=ascii

相应的，有操作DB的相关接口，查询、插入记录（关闭开关）、删除记录（打开开关）。

func IsRecoveryDisabled() (disabled bool, err error) {}

func DisableRecovery() error {}

unc EnableRecovery() error {}

2. raft 同步层

为了在Orchestrator 集群节点之间同步全局开关状态，定义raft同步接口：

func (applier *CommandApplier) disableGlobalRecoveries(value []byte) interface{} {}

func (applier *CommandApplier) enableGlobalRecoveries(value []byte) interface{} {}

3. API 层

在HTTP 层，提供相关接口，供外部使用。

查询开关状态

// CheckGlobalRecoveries checks whether
func (this *HttpAPI) CheckGlobalRecoveries(params martini.Params, r render.Render, req *http.Request) {}

关闭全局开关

// DisableGlobalRecoveries globally disables recoveries
func (this *HttpAPI) DisableGlobalRecoveries(params martini.Params, r render.Render, req *http.Request, user auth.User) {}

打开全局开关

// EnableGlobalRecoveries globally enables recoveries
func (this *HttpAPI) EnableGlobalRecoveries(params martini.Params, r render.Render, req *http.Request, user auth.User) {}

4. snapshot 层

从数据表中查询记录，写入snapshot：

func CreateSnapshotData() *SnapshotData {
	snapshotData := NewSnapshotData()
    ... ...
	snapshotData.RecoveryDisabled, _ = IsRecoveryDisabled()
    ... ...
}

从snapshot中恢复到数据表中：

func (this *SnapshotDataCreatorApplier) Restore(rc io.ReadCloser) error {
	snapshotData := NewSnapshotData()
	... ...
	// recovery disable
	{
		SetRecoveryDisabled(snapshotData.RecoveryDisabled)
	}
	... ...

5. 自动故障恢复

自动故障恢复发起前，检查全局开关状态，如果全局禁用，则直接返回，不继续执行：

func executeCheckAndRecoverFunction(analysisEntry inst.ReplicationAnalysis, candidateInstanceKey *inst.InstanceKey, forceInstanceRecovery bool, skipProcesses bool) (recoveryAttempted bool, topologyRecovery *TopologyRecovery, err error) {
    ... ...
    	// Check for recovery being disabled globally
	if recoveryDisabledGlobally, err := IsRecoveryDisabled(); err != nil {
		// Unexpected. Shouldn't get this
		log.Errorf("Unable to determine if recovery is disabled globally: %v", err)
	} else if recoveryDisabledGlobally {
		if !forceInstanceRecovery {
			log.Infof("CheckAndRecover: Analysis: %+v, InstanceKey: %+v, candidateInstanceKey: %+v, "+
				"skipProcesses: %v: NOT Recovering host (disabled globally)",
				analysisEntry.Analysis, analysisEntry.AnalyzedInstanceKey, candidateInstanceKey, skipProcesses)

			return false, nil, err
		}
		log.Infof("CheckAndRecover: Analysis: %+v, InstanceKey: %+v, candidateInstanceKey: %+v, "+
			"skipProcesses: %v: recoveries disabled globally but forcing this recovery",
			analysisEntry.Analysis, analysisEntry.AnalyzedInstanceKey, candidateInstanceKey, skipProcesses)
	}
    
    ... ...
}

6. Dashboard 页面

在Dashboard上，通过调用HTTP接口查看和操作。
如下图所示：