03_Sentinel规则持久化之拉模式

sentinel规则持久化部分源码分析

Sentinel规则推送模式

Sentinel规则的推送有下面三种模式:

原始模式

如果不做任何修改，Dashboard 的推送规则方式是通过 API 将规则推送至客户端并直接更新到内存中： 

这种做法的好处是简单，无依赖；坏处是应用重启规则就会消失，仅用于简单测试，不能用于生产环境。 

拉模式

pull 模式的数据源（如本地文件、RDBMS 等）一般是可写入的。使用时需要在客户端注册数据源：将对应的读数据源注册至对应的 RuleManager，将写数据源注册至 transport 的 WritableDataSourceRegistry 中。

拉模式工作流程：

首先 Sentinel 控制台通过 API 将规则推送至客户端并更新到内存中，接着注册的写数据源会将新的规则保存到本地的文件中。使用 pull 模式的数据源时一般不需要对 Sentinel 控制台进行改造。这种实现方法好处是简单，坏处是无法保证监控数据的一致性。

sentinel控制台和微服务端通信整体流程

控制台设置规则（以流控规则为例）controller层com.alibaba.csp.sentinel.dashboard.controller.FlowControllerV1：

    @PostMapping("/rule")
    @AuthAction(PrivilegeType.WRITE_RULE)
    public Result<FlowRuleEntity> apiAddFlowRule(@RequestBody FlowRuleEntity entity) {
        Result<FlowRuleEntity> checkResult = checkEntityInternal(entity);
        if (checkResult != null) {
            return checkResult;
        }
        entity.setId(null);
        Date date = new Date();
        entity.setGmtCreate(date);
        entity.setGmtModified(date);
        entity.setLimitApp(entity.getLimitApp().trim());
        entity.setResource(entity.getResource().trim());
        try {
            entity = repository.save(entity);

            publishRules(entity.getApp(), entity.getIp(), entity.getPort()).get(5000, TimeUnit.MILLISECONDS);
            return Result.ofSuccess(entity);
        } catch (Throwable t) {
            Throwable e = t instanceof ExecutionException ? t.getCause() : t;
            logger.error("Failed to add new flow rule, app={}, ip={}", entity.getApp(), entity.getIp(), e);
            return Result.ofFail(-1, e.getMessage());
        }
    }

RuleRepository接口实现类将规则实体保存到内存中的Map：allRules、machineRules、appRules中

com.alibaba.csp.sentinel.dashboard.repository.rule.InMemoryRuleRepositoryAdapter#save

@Override
    public T save(T entity) {
        if (entity.getId() == null) {
            entity.setId(nextId());
        }
        T processedEntity = preProcess(entity);
        if (processedEntity != null) {
            allRules.put(processedEntity.getId(), processedEntity);
            machineRules.computeIfAbsent(MachineInfo.of(processedEntity.getApp(), processedEntity.getIp(),
                processedEntity.getPort()), e -> new ConcurrentHashMap<>(32))
                .put(processedEntity.getId(), processedEntity);
            appRules.computeIfAbsent(processedEntity.getApp(), v -> new ConcurrentHashMap<>(32))
                .put(processedEntity.getId(), processedEntity);
        }

        return processedEntity;
    }

将规则发送到微服务端：com.alibaba.csp.sentinel.dashboard.controller.FlowControllerV1#publishRules最终执行到com.alibaba.csp.sentinel.dashboard.client.SentinelApiClient#setRulesAsync

发送类型：

private static final String FLOW_RULE_TYPE = "flow";

private CompletableFuture<Void> setRulesAsync(String app, String ip, int port, String type, List<? extends RuleEntity> entities) {
    try {
        AssertUtil.notNull(entities, "rules cannot be null");
        AssertUtil.notEmpty(app, "Bad app name");
        AssertUtil.notEmpty(ip, "Bad machine IP");
        AssertUtil.isTrue(port > 0, "Bad machine port");
        String data = JSON.toJSONString(
            entities.stream().map(r -> r.toRule()).collect(Collectors.toList()));
        Map<String, String> params = new HashMap<>(2);
        params.put("type", type);
        params.put("data", data);
        return executeCommand(app, ip, port, SET_RULES_PATH, params, true)
            .thenCompose(r -> {
                if ("success".equalsIgnoreCase(r.trim())) {
                    return CompletableFuture.completedFuture(null);
                }
                return AsyncUtils.newFailedFuture(new CommandFailedException(r));
            });
    } catch (Exception e) {
        logger.error("setRulesAsync API failed, type={}", type, e);
        return AsyncUtils.newFailedFuture(e);
    }
}

微服务端：

通信模块中监听端口8719（默认）被占用则+1，核心代码：

com.alibaba.csp.sentinel.command.handler.ModifyRulesCommandHandler#handle:

@Override
public CommandResponse<String> handle(CommandRequest request) {
    // XXX from 1.7.2, force to fail when fastjson is older than 1.2.12
    // We may need a better solution on this.
    if (VersionUtil.fromVersionString(JSON.VERSION) < FASTJSON_MINIMAL_VER) {
        // fastjson too old
        return CommandResponse.ofFailure(new RuntimeException("The \"fastjson-" + JSON.VERSION
                + "\" introduced in application is too old, you need fastjson-1.2.12 at least."));
    }
    String type = request.getParam("type");
    // rule data in get parameter
    String data = request.getParam("data");
    if (StringUtil.isNotEmpty(data)) {
        try {
            data = URLDecoder.decode(data, "utf-8");
        } catch (Exception e) {
            RecordLog.info("Decode rule data error", e);
            return CommandResponse.ofFailure(e, "decode rule data error");
        }
    }

    RecordLog.info("Receiving rule change (type: {}): {}", type, data);

    String result = "success";

    if (FLOW_RULE_TYPE.equalsIgnoreCase(type)) {
        List<FlowRule> flowRules = JSONArray.parseArray(data, FlowRule.class);
        FlowRuleManager.loadRules(flowRules);
        if (!writeToDataSource(getFlowDataSource(), flowRules)) {
            result = WRITE_DS_FAILURE_MSG;
        }
        return CommandResponse.ofSuccess(result);
    } else if (AUTHORITY_RULE_TYPE.equalsIgnoreCase(type)) {
        List<AuthorityRule> rules = JSONArray.parseArray(data, AuthorityRule.class);
        AuthorityRuleManager.loadRules(rules);
        if (!writeToDataSource(getAuthorityDataSource(), rules)) {
            result = WRITE_DS_FAILURE_MSG;
        }
        return CommandResponse.ofSuccess(result);
    } else if (DEGRADE_RULE_TYPE.equalsIgnoreCase(type)) {
        List<DegradeRule> rules = JSONArray.parseArray(data, DegradeRule.class);
        DegradeRuleManager.loadRules(rules);
        if (!writeToDataSource(getDegradeDataSource(), rules)) {
            result = WRITE_DS_FAILURE_MSG;
        }
        return CommandResponse.ofSuccess(result);
    } else if (SYSTEM_RULE_TYPE.equalsIgnoreCase(type)) {
        List<SystemRule> rules = JSONArray.parseArray(data, SystemRule.class);
        SystemRuleManager.loadRules(rules);
        if (!writeToDataSource(getSystemSource(), rules)) {
            result = WRITE_DS_FAILURE_MSG;
        }
        return CommandResponse.ofSuccess(result);
    }
    return CommandResponse.ofFailure(new IllegalArgumentException("invalid type"));
}

可以看到有扩展点writeToDataSource(getFlowDataSource(), flowRules)，可以实现接口com.alibaba.csp.sentinel.datasource.WritableDataSource#write方法并通过com.alibaba.csp.sentinel.transport.util.WritableDataSourceRegistry#registerFlowDataSource方法注册写数据源之后便可实现规则持久化。

拉模式定时读取文件中的持久化规则的实现：

参考sentinel官方提供的demo:

com.alibaba.csp.sentinel.demo.file.rule.FileDataSourceDemo核心代码：

private void listenRules() throws Exception {
    ClassLoader classLoader = getClass().getClassLoader();
    String flowRulePath = URLDecoder.decode(Objects.requireNonNull(classLoader.getResource("FlowRule.json")).getFile(), "UTF-8");
    String degradeRulePath = URLDecoder.decode(classLoader.getResource("DegradeRule.json").getFile(), "UTF-8");
    String systemRulePath = URLDecoder.decode(classLoader.getResource("SystemRule.json").getFile(), "UTF-8");

    // Data source for FlowRule
    FileRefreshableDataSource<List<FlowRule>> flowRuleDataSource = new FileRefreshableDataSource<>(
        flowRulePath, flowRuleListParser);
    FlowRuleManager.register2Property(flowRuleDataSource.getProperty());

    // Data source for DegradeRule
    FileRefreshableDataSource<List<DegradeRule>> degradeRuleDataSource
        = new FileRefreshableDataSource<>(
        degradeRulePath, degradeRuleListParser);
    DegradeRuleManager.register2Property(degradeRuleDataSource.getProperty());

    // Data source for SystemRule
    FileRefreshableDataSource<List<SystemRule>> systemRuleDataSource
        = new FileRefreshableDataSource<>(
        systemRulePath, systemRuleListParser);
    SystemRuleManager.register2Property(systemRuleDataSource.getProperty());
}

com.alibaba.csp.sentinel.datasource.FileRefreshableDataSource的构造方法中会开启一个线程定时(默认3s)监听文件变化从文件中拉取规则：

判断文件是否发生变com.alibaba.csp.sentinel.datasource.FileRefreshableDataSource#isModified：

protected boolean isModified() {
    long curLastModified = file.lastModified();
    if (curLastModified != this.lastModified) {
        this.lastModified = curLastModified;
        return true;
    }
    return false;
}

开启轮询任务com.alibaba.csp.sentinel.datasource.AutoRefreshDataSource#startTimerService：

@SuppressWarnings("PMD.ThreadPoolCreationRule")
private void startTimerService() {
    service = Executors.newScheduledThreadPool(1,
        new NamedThreadFactory("sentinel-datasource-auto-refresh-task", true));
    service.scheduleAtFixedRate(new Runnable() {
        @Override
        public void run() {
            try {
                if (!isModified()) {
                    return;
                }
                T newValue = loadConfig();
                getProperty().updateValue(newValue);
            } catch (Throwable e) {
                RecordLog.info("loadConfig exception", e);
            }
        }
    }, recommendRefreshMs, recommendRefreshMs, TimeUnit.MILLISECONDS);
}

加载规则信息com.alibaba.csp.sentinel.datasource.AbstractDataSource#loadConfig()：

@Override
public T loadConfig() throws Exception {
    return loadConfig(readSource());
}

public T loadConfig(S conf) throws Exception {
    T value = parser.convert(conf);
    return value;
}

更新到内存com.alibaba.csp.sentinel.property.DynamicSentinelProperty#updateValue：

 

@Override
public boolean updateValue(T newValue) {
    if (isEqual(value, newValue)) {
        return false;
    }
    RecordLog.info("[DynamicSentinelProperty] Config will be updated to: {}", newValue);

    value = newValue;
    for (PropertyListener<T> listener : listeners) {
        listener.configUpdate(newValue);
    }
    return true;
}

通过监听器com.alibaba.csp.sentinel.property.PropertyListener来配置规则configUpdate:

所以需要向property中注册监听器

FlowRuleManager.register2Property(flowRuleDataSource.getProperty());

com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager#register2Property：

public static void register2Property(SentinelProperty<List<FlowRule>> property) {
    AssertUtil.notNull(property, "property cannot be null");
    synchronized (LISTENER) {
        RecordLog.info("[FlowRuleManager] Registering new property to flow rule manager");
        currentProperty.removeListener(LISTENER);
        property.addListener(LISTENER);
        currentProperty = property;
    }
}

监听器实例com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager.FlowPropertyListener更新内存中规则方法#configUpdate：

@Override
public synchronized void configUpdate(List<FlowRule> value) {
    Map<String, List<FlowRule>> rules = FlowRuleUtil.buildFlowRuleMap(value);
    if (rules != null) {
        flowRules = rules;
    }
    RecordLog.info("[FlowRuleManager] Flow rules received: {}", rules);
}

怎么将拉模式核心代码整合到springClound中进行初始化

可以通过以下方式：

spring提供的扩展点：

beanPostProcessor 、beanFactoryPostProcessor、SmartInitializingSingleton、ApplicationListener、FactoryBean#getObject

springBoot的初始化：

ApplicationRunner

但以上方式都对spring环境有强依赖关系。

所以通过查看源码发现sentinel的初始化是通过提供的spi机制进行初始化：

webMVC的拦截器中进行初始化：

com.alibaba.csp.sentinel.adapter.spring.webmvc.AbstractSentinelInterceptor#preHandle

Entry entry = SphU.entry(resourceName, ResourceTypeConstants.COMMON_WEB, EntryType.IN);

会调用：

Env.sph.entryWithType(name, resourceType, trafficType, 1, OBJECTS0);

com.alibaba.csp.sentinel.Env的实体类中有一个静态代码块：

static {
    // If init fails, the process will exit.
    InitExecutor.doInit();
}

通过SPI机制去加载sentinel相关依赖包下META-INF/services/下所有的InitFunc实例类：

public static void doInit() {
    if (!initialized.compareAndSet(false, true)) {
        return;
    }
    try {
        List<InitFunc> initFuncs = SpiLoader.of(InitFunc.class).loadInstanceListSorted();
        List<OrderWrapper> initList = new ArrayList<OrderWrapper>();
        for (InitFunc initFunc : initFuncs) {
            RecordLog.info("[InitExecutor] Found init func: {}", initFunc.getClass().getCanonicalName());
            insertSorted(initList, initFunc);
        }
        for (OrderWrapper w : initList) {
            w.func.init();
            RecordLog.info("[InitExecutor] Executing {} with order {}",
                w.func.getClass().getCanonicalName(), w.order);
        }
    } catch (Exception ex) {
        RecordLog.warn("[InitExecutor] WARN: Initialization failed", ex);
        ex.printStackTrace();
    } catch (Error error) {
        RecordLog.warn("[InitExecutor] ERROR: Initialization failed with fatal error", error);
        error.printStackTrace();
    }
}

CommandCenterlnitFunc#init就加载了通信模块的socket监听控制台发送的通信，还有注册机器等初始化操作入口：com.alibaba.csp.sentinel.transport.init.HeartbeatSenderInitFunc#scheduleHeartbeatTask

所以拉模式可以通过此方式进行初始化：

在sentinel-extension包下新增一个拉模式模块sentinel-datasource-extension-file-pull：

核心代码：com.yyj.sentinel.extension.filepull.FileDataSourceInit

/*
 * Copyright 1999-2018 Alibaba Group Holding Ltd.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.yyj.sentinel.extension.filepull;

import com.alibaba.csp.sentinel.command.handler.ModifyParamFlowRulesCommandHandler;
import com.alibaba.csp.sentinel.datasource.FileRefreshableDataSource;
import com.alibaba.csp.sentinel.datasource.FileWritableDataSource;
import com.alibaba.csp.sentinel.datasource.ReadableDataSource;
import com.alibaba.csp.sentinel.datasource.WritableDataSource;
import com.alibaba.csp.sentinel.init.InitFunc;
import com.alibaba.csp.sentinel.slots.block.authority.AuthorityRule;
import com.alibaba.csp.sentinel.slots.block.authority.AuthorityRuleManager;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRule;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRuleManager;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRule;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager;
import com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowRule;
import com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowRuleManager;
import com.alibaba.csp.sentinel.slots.system.SystemRule;
import com.alibaba.csp.sentinel.slots.system.SystemRuleManager;
import com.alibaba.csp.sentinel.transport.util.WritableDataSourceRegistry;

import java.io.FileNotFoundException;
import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;
import java.util.List;
import java.util.Objects;

/**
 * InitFunc实现类，处理dataSource初始化逻辑
 */
public class FileDataSourceInit implements InitFunc {

    @Override
    public void init() throws Exception {
        //创建文件存储目录
        RuleFileUtils.mkdirIfNotExits(PersistenceRuleConstant.storePath);

        //创建规则文件
        RuleFileUtils.createFileIfNotExits(PersistenceRuleConstant.rulesMap);

        //处理流控规则逻辑  配置读写数据源
        dealFlowRules();
        // 处理降级规则
        dealDegradeRules();
        // 处理系统规则
        dealSystemRules();
        // 处理热点参数规则
        dealParamFlowRules();
        // 处理授权规则
        dealAuthRules();
    }


    private void dealFlowRules() throws FileNotFoundException {
        String ruleFilePath = PersistenceRuleConstant.rulesMap.get(PersistenceRuleConstant.FLOW_RULE_PATH);

        //创建流控规则的可读数据源
        ReadableDataSource<String, List<FlowRule>> flowRuleRDS = new FileRefreshableDataSource(ruleFilePath, RuleListConverterUtils.flowRuleListParser);

        // 将可读数据源注册至FlowRuleManager 这样当规则文件发生变化时，就会更新规则到内存
        FlowRuleManager.register2Property(flowRuleRDS.getProperty());

        WritableDataSource<List<FlowRule>> flowRuleWDS = new FileWritableDataSource<>(ruleFilePath, RuleListConverterUtils.flowRuleEnCoding);

        // 将可写数据源注册至 transport 模块的 WritableDataSourceRegistry 中.
        // 这样收到控制台推送的规则时，Sentinel 会先更新到内存，然后将规则写入到文件中.
        WritableDataSourceRegistry.registerFlowDataSource(flowRuleWDS);
    }

    private void dealDegradeRules() throws FileNotFoundException {
        //获取规则文件路径
        String degradeRuleFilePath = PersistenceRuleConstant.rulesMap.get(PersistenceRuleConstant.DEGRAGE_RULE_PATH).toString();

        //创建流控规则的可读数据源
        ReadableDataSource<String, List<DegradeRule>> degradeRuleRDS = new FileRefreshableDataSource(
                degradeRuleFilePath, RuleListConverterUtils.degradeRuleListParse
        );

        // 将可读数据源注册至FlowRuleManager 这样当规则文件发生变化时，就会更新规则到内存
        DegradeRuleManager.register2Property(degradeRuleRDS.getProperty());


        WritableDataSource<List<DegradeRule>> degradeRuleWDS = new FileWritableDataSource<>(
                degradeRuleFilePath, RuleListConverterUtils.degradeRuleEnCoding
        );

        // 将可写数据源注册至 transport 模块的 WritableDataSourceRegistry 中.
        // 这样收到控制台推送的规则时，Sentinel 会先更新到内存，然后将规则写入到文件中.
        WritableDataSourceRegistry.registerDegradeDataSource(degradeRuleWDS);
    }

    private void dealSystemRules() throws FileNotFoundException {
        //获取规则文件路径
        String systemRuleFilePath = PersistenceRuleConstant.rulesMap.get(PersistenceRuleConstant.SYSTEM_RULE_PATH).toString();

        //创建流控规则的可读数据源
        ReadableDataSource<String, List<SystemRule>> systemRuleRDS = new FileRefreshableDataSource(
                systemRuleFilePath, RuleListConverterUtils.sysRuleListParse
        );

        // 将可读数据源注册至FlowRuleManager 这样当规则文件发生变化时，就会更新规则到内存
        SystemRuleManager.register2Property(systemRuleRDS.getProperty());


        WritableDataSource<List<SystemRule>> systemRuleWDS = new FileWritableDataSource<>(
                systemRuleFilePath, RuleListConverterUtils.sysRuleEnCoding
        );

        // 将可写数据源注册至 transport 模块的 WritableDataSourceRegistry 中.
        // 这样收到控制台推送的规则时，Sentinel 会先更新到内存，然后将规则写入到文件中.
        WritableDataSourceRegistry.registerSystemDataSource(systemRuleWDS);
    }


    private void dealParamFlowRules() throws FileNotFoundException {
        //获取规则文件路径
        String paramFlowRuleFilePath = PersistenceRuleConstant.rulesMap.get(PersistenceRuleConstant.HOT_PARAM_RULE).toString();

        //创建流控规则的可读数据源
        ReadableDataSource<String, List<ParamFlowRule>> paramFlowRuleRDS = new FileRefreshableDataSource(
                paramFlowRuleFilePath, RuleListConverterUtils.paramFlowRuleListParse
        );

        // 将可读数据源注册至FlowRuleManager 这样当规则文件发生变化时，就会更新规则到内存
        ParamFlowRuleManager.register2Property(paramFlowRuleRDS.getProperty());


        WritableDataSource<List<ParamFlowRule>> paramFlowRuleWDS = new FileWritableDataSource<>(
                paramFlowRuleFilePath, RuleListConverterUtils.paramRuleEnCoding
        );

        // 将可写数据源注册至 transport 模块的 WritableDataSourceRegistry 中.
        // 这样收到控制台推送的规则时，Sentinel 会先更新到内存，然后将规则写入到文件中.
        ModifyParamFlowRulesCommandHandler.setWritableDataSource(paramFlowRuleWDS);
    }

    private void dealAuthRules() throws FileNotFoundException {
        //获取规则文件路径
        String authFilePath = PersistenceRuleConstant.rulesMap.get(PersistenceRuleConstant.AUTH_RULE_PATH).toString();

        //创建流控规则的可读数据源
        ReadableDataSource<String, List<AuthorityRule>> authRuleRDS = new FileRefreshableDataSource(
                authFilePath, RuleListConverterUtils.authorityRuleParse
        );

        // 将可读数据源注册至FlowRuleManager 这样当规则文件发生变化时，就会更新规则到内存
        AuthorityRuleManager.register2Property(authRuleRDS.getProperty());

        //创建流控规则的写数据源
        WritableDataSource<List<AuthorityRule>> authRuleWDS = new FileWritableDataSource<>(
                authFilePath, RuleListConverterUtils.authorityEncoding
        );

        // 将可写数据源注册至 transport 模块的 WritableDataSourceRegistry 中.
        // 这样收到控制台推送的规则时，Sentinel 会先更新到内存，然后将规则写入到文件中.
        WritableDataSourceRegistry.registerAuthorityDataSource(authRuleWDS);
    }

}

在resources资源目录下新增文件META-INF/services/com.alibaba.csp.sentinel.init.InitFunc文件内容：

com.yyj.sentinel.extension.filepull.FileDataSourceInit

之后通过maven 打包安装到自己本地的maven仓库：

在微服务端直接引入该模块依赖即完成了对拉模式的整合：

<!--   引入自己安装到仓库的拉模式的依赖  -->
<dependency>
    <groupId>com.alibaba.csp</groupId>
    <artifactId>sentinel-datasource-extension-file-pull</artifactId>
    <version>1.8.4</version>
</dependency>