03_Sentinel规则持久化之拉模式
sentinel规则持久化部分源码分析
Sentinel规则推送模式
Sentinel规则的推送有下面三种模式:
原始模式
如果不做任何修改，Dashboard 的推送规则方式是通过 API 将规则推送至客户端并直接更新到内存中：
这种做法的好处是简单，无依赖；坏处是应用重启规则就会消失，仅用于简单测试，不能用于生产环境。
拉模式
pull 模式的数据源（如本地文件、RDBMS 等）一般是可写入的。使用时需要在客户端注册数据源：将对应的读数据源注册至对应的 RuleManager，将写数据源注册至 transport 的 WritableDataSourceRegistry 中。
拉模式工作流程：
首先 Sentinel 控制台通过 API 将规则推送至客户端并更新到内存中，接着注册的写数据源会将新的规则保存到本地的文件中。使用 pull 模式的数据源时一般不需要对 Sentinel 控制台进行改造。这种实现方法好处是简单，坏处是无法保证监控数据的一致性。
sentinel控制台和微服务端通信整体流程
控制台设置规则（以流控规则为例）controller层com.alibaba.csp.sentinel.dashboard.controller.FlowControllerV1：
@PostMapping("/rule")
@AuthAction(PrivilegeType.WRITE_RULE)
public Result<FlowRuleEntity> apiAddFlowRule(@RequestBody FlowRuleEntity entity) {
Result<FlowRuleEntity> checkResult = checkEntityInternal(entity);
if (checkResult != null) {
return checkResult;
}
entity.setId(null);
Date date = new Date();
entity.setGmtCreate(date);
entity.setGmtModified(date);
entity.setLimitApp(entity.getLimitApp().trim());
entity.setResource(entity.getResource().trim());
try {
entity = repository.save(entity);
publishRules(entity.getApp(), entity.getIp(), entity.getPort()).get(5000, TimeUnit.MILLISECONDS);
return Result.ofSuccess(entity);
} catch (Throwable t) {
Throwable e = t instanceof ExecutionException ? t.getCause() : t;
logger.error("Failed to add new flow rule, app={}, ip={}", entity.getApp(), entity.getIp(), e);
return Result.ofFail(-1, e.getMessage());
}
}
RuleRepository接口实现类将规则实体保存到内存中的Map：allRules、machineRules、appRules中
com.alibaba.csp.sentinel.dashboard.repository.rule.InMemoryRuleRepositoryAdapter#save
@Override public T save(T entity) { if (entity.getId() == null) { entity.setId(nextId()); } T processedEntity = preProcess(entity); if (processedEntity != null) { allRules.put(processedEntity.getId(), processedEntity); machineRules.computeIfAbsent(MachineInfo.of(processedEntity.getApp(), processedEntity.getIp(), processedEntity.getPort()), e -> new ConcurrentHashMap<>(32)) .put(processedEntity.getId(), processedEntity); appRules.computeIfAbsent(processedEntity.getApp(), v -> new ConcurrentHashMap<>(32)) .put(processedEntity.getId(), processedEntity); } return processedEntity; }
将规则发送到微服务端：com.alibaba.csp.sentinel.dashboard.controller.FlowControllerV1#publishRules最终执行到com.alibaba.csp.sentinel.dashboard.client.SentinelApiClient#setRulesAsync
发送类型：
private static final String FLOW_RULE_TYPE = "flow";
private CompletableFuture<Void> setRulesAsync(String app, String ip, int port, String type, List<? extends RuleEntity> entities) { try { AssertUtil.notNull(entities, "rules cannot be null"); AssertUtil.notEmpty(app, "Bad app name"); AssertUtil.notEmpty(ip, "Bad machine IP"); AssertUtil.isTrue(port > 0, "Bad machine port"); String data = JSON.toJSONString( entities.stream().map(r -> r.toRule()).collect(Collectors.toList())); Map<String, String> params = new HashMap<>(2); params.put("type", type); params.put("data", data); return executeCommand(app, ip, port, SET_RULES_PATH, params, true) .thenCompose(r -> { if ("success".equalsIgnoreCase(r.trim())) { return CompletableFuture.completedFuture(null); } return AsyncUtils.newFailedFuture(new CommandFailedException(r)); }); } catch (Exception e) { logger.error("setRulesAsync API failed, type={}", type, e); return AsyncUtils.newFailedFuture(e); } }
微服务端：
通信模块中监听端口8719（默认）被占用则+1，核心代码：
com.alibaba.csp.sentinel.command.handler.ModifyRulesCommandHandler#handle:
@Override public CommandResponse<String> handle(CommandRequest request) { // XXX from 1.7.2, force to fail when fastjson is older than 1.2.12 // We may need a better solution on this. if (VersionUtil.fromVersionString(JSON.VERSION) < FASTJSON_MINIMAL_VER) { // fastjson too old return CommandResponse.ofFailure(new RuntimeException("The \"fastjson-" + JSON.VERSION + "\" introduced in application is too old, you need fastjson-1.2.12 at least.")); } String type = request.getParam("type"); // rule data in get parameter String data = request.getParam("data"); if (StringUtil.isNotEmpty(data)) { try { data = URLDecoder.decode(data, "utf-8"); } catch (Exception e) { RecordLog.info("Decode rule data error", e); return CommandResponse.ofFailure(e, "decode rule data error"); } } RecordLog.info("Receiving rule change (type: {}): {}", type, data); String result = "success"; if (FLOW_RULE_TYPE.equalsIgnoreCase(type)) { List<FlowRule> flowRules = JSONArray.parseArray(data, FlowRule.class); FlowRuleManager.loadRules(flowRules); if (!writeToDataSource(getFlowDataSource(), flowRules)) { result = WRITE_DS_FAILURE_MSG; } return CommandResponse.ofSuccess(result); } else if (AUTHORITY_RULE_TYPE.equalsIgnoreCase(type)) { List<AuthorityRule> rules = JSONArray.parseArray(data, AuthorityRule.class); AuthorityRuleManager.loadRules(rules); if (!writeToDataSource(getAuthorityDataSource(), rules)) { result = WRITE_DS_FAILURE_MSG; } return CommandResponse.ofSuccess(result); } else if (DEGRADE_RULE_TYPE.equalsIgnoreCase(type)) { List<DegradeRule> rules = JSONArray.parseArray(data, DegradeRule.class); DegradeRuleManager.loadRules(rules); if (!writeToDataSource(getDegradeDataSource(), rules)) { result = WRITE_DS_FAILURE_MSG; } return CommandResponse.ofSuccess(result); } else if (SYSTEM_RULE_TYPE.equalsIgnoreCase(type)) { List<SystemRule> rules = JSONArray.parseArray(data, SystemRule.class); SystemRuleManager.loadRules(rules); if (!writeToDataSource(getSystemSource(), rules)) { result = WRITE_DS_FAILURE_MSG; } return CommandResponse.ofSuccess(result); } return CommandResponse.ofFailure(new IllegalArgumentException("invalid type")); }
可以看到有扩展点writeToDataSource(getFlowDataSource(), flowRules)，可以实现接口com.alibaba.csp.sentinel.datasource.WritableDataSource#write方法并通过com.alibaba.csp.sentinel.transport.util.WritableDataSourceRegistry#registerFlowDataSource方法注册写数据源之后便可实现规则持久化。
拉模式定时读取文件中的持久化规则的实现：
参考sentinel官方提供的demo:
com.alibaba.csp.sentinel.demo.file.rule.FileDataSourceDemo核心代码：
private void listenRules() throws Exception { ClassLoader classLoader = getClass().getClassLoader(); String flowRulePath = URLDecoder.decode(Objects.requireNonNull(classLoader.getResource("FlowRule.json")).getFile(), "UTF-8"); String degradeRulePath = URLDecoder.decode(classLoader.getResource("DegradeRule.json").getFile(), "UTF-8"); String systemRulePath = URLDecoder.decode(classLoader.getResource("SystemRule.json").getFile(), "UTF-8"); // Data source for FlowRule FileRefreshableDataSource<List<FlowRule>> flowRuleDataSource = new FileRefreshableDataSource<>( flowRulePath, flowRuleListParser); FlowRuleManager.register2Property(flowRuleDataSource.getProperty()); // Data source for DegradeRule FileRefreshableDataSource<List<DegradeRule>> degradeRuleDataSource = new FileRefreshableDataSource<>( degradeRulePath, degradeRuleListParser); DegradeRuleManager.register2Property(degradeRuleDataSource.getProperty()); // Data source for SystemRule FileRefreshableDataSource<List<SystemRule>> systemRuleDataSource = new FileRefreshableDataSource<>( systemRulePath, systemRuleListParser); SystemRuleManager.register2Property(systemRuleDataSource.getProperty()); }
com.alibaba.csp.sentinel.datasource.FileRefreshableDataSource的构造方法中会开启一个线程定时(默认3s)监听文件变化从文件中拉取规则：
判断文件是否发生变com.alibaba.csp.sentinel.datasource.FileRefreshableDataSource#isModified：
protected boolean isModified() { long curLastModified = file.lastModified(); if (curLastModified != this.lastModified) { this.lastModified = curLastModified; return true; } return false; }
开启轮询任务com.alibaba.csp.sentinel.datasource.AutoRefreshDataSource#startTimerService：
@SuppressWarnings("PMD.ThreadPoolCreationRule")
private void startTimerService() {
service = Executors.newScheduledThreadPool(1,
new NamedThreadFactory("sentinel-datasource-auto-refresh-task", true));
service.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
if (!isModified()) {
return;
}
T newValue = loadConfig();
getProperty().updateValue(newValue);
} catch (Throwable e) {
RecordLog.info("loadConfig exception", e);
}
}
}, recommendRefreshMs, recommendRefreshMs, TimeUnit.MILLISECONDS);
}
加载规则信息com.alibaba.csp.sentinel.datasource.AbstractDataSource#loadConfig()：
@Override public T loadConfig() throws Exception { return loadConfig(readSource()); } public T loadConfig(S conf) throws Exception { T value = parser.convert(conf); return value; }
更新到内存com.alibaba.csp.sentinel.property.DynamicSentinelProperty#updateValue：
@Override public boolean updateValue(T newValue) { if (isEqual(value, newValue)) { return false; } RecordLog.info("[DynamicSentinelProperty] Config will be updated to: {}", newValue); value = newValue; for (PropertyListener<T> listener : listeners) { listener.configUpdate(newValue); } return true; }
通过监听器com.alibaba.csp.sentinel.property.PropertyListener来配置规则configUpdate:
所以需要向property中注册监听器
FlowRuleManager.register2Property(flowRuleDataSource.getProperty());
com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager#register2Property：
public static void register2Property(SentinelProperty<List<FlowRule>> property) { AssertUtil.notNull(property, "property cannot be null"); synchronized (LISTENER) { RecordLog.info("[FlowRuleManager] Registering new property to flow rule manager"); currentProperty.removeListener(LISTENER); property.addListener(LISTENER); currentProperty = property; } }
监听器实例com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager.FlowPropertyListener更新内存中规则方法#configUpdate：
@Override public synchronized void configUpdate(List<FlowRule> value) { Map<String, List<FlowRule>> rules = FlowRuleUtil.buildFlowRuleMap(value); if (rules != null) { flowRules = rules; } RecordLog.info("[FlowRuleManager] Flow rules received: {}", rules); }
怎么将拉模式核心代码整合到springClound中进行初始化
可以通过以下方式：
spring提供的扩展点：
beanPostProcessor 、beanFactoryPostProcessor、SmartInitializingSingleton、ApplicationListener、FactoryBean#getObject
springBoot的初始化：
ApplicationRunner
但以上方式都对spring环境有强依赖关系。
所以通过查看源码发现sentinel的初始化是通过提供的spi机制进行初始化：
webMVC的拦截器中进行初始化：
com.alibaba.csp.sentinel.adapter.spring.webmvc.AbstractSentinelInterceptor#preHandle
Entry entry = SphU.entry(resourceName, ResourceTypeConstants.COMMON_WEB, EntryType.IN);
会调用：
Env.sph.entryWithType(name, resourceType, trafficType, 1, OBJECTS0);
com.alibaba.csp.sentinel.Env的实体类中有一个静态代码块：
static { // If init fails, the process will exit. InitExecutor.doInit(); }
通过SPI机制去加载sentinel相关依赖包下META-INF/services/下所有的InitFunc实例类：
public static void doInit() { if (!initialized.compareAndSet(false, true)) { return; } try { List<InitFunc> initFuncs = SpiLoader.of(InitFunc.class).loadInstanceListSorted(); List<OrderWrapper> initList = new ArrayList<OrderWrapper>(); for (InitFunc initFunc : initFuncs) { RecordLog.info("[InitExecutor] Found init func: {}", initFunc.getClass().getCanonicalName()); insertSorted(initList, initFunc); } for (OrderWrapper w : initList) { w.func.init(); RecordLog.info("[InitExecutor] Executing {} with order {}", w.func.getClass().getCanonicalName(), w.order); } } catch (Exception ex) { RecordLog.warn("[InitExecutor] WARN: Initialization failed", ex); ex.printStackTrace(); } catch (Error error) { RecordLog.warn("[InitExecutor] ERROR: Initialization failed with fatal error", error); error.printStackTrace(); } }
CommandCenterlnitFunc#init就加载了通信模块的socket监听控制台发送的通信，还有注册机器等初始化操作入口：com.alibaba.csp.sentinel.transport.init.HeartbeatSenderInitFunc#scheduleHeartbeatTask
所以拉模式可以通过此方式进行初始化：
在sentinel-extension包下新增一个拉模式模块sentinel-datasource-extension-file-pull：
核心代码：com.yyj.sentinel.extension.filepull.FileDataSourceInit
/* * Copyright 1999-2018 Alibaba Group Holding Ltd. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package com.yyj.sentinel.extension.filepull; import com.alibaba.csp.sentinel.command.handler.ModifyParamFlowRulesCommandHandler; import com.alibaba.csp.sentinel.datasource.FileRefreshableDataSource; import com.alibaba.csp.sentinel.datasource.FileWritableDataSource; import com.alibaba.csp.sentinel.datasource.ReadableDataSource; import com.alibaba.csp.sentinel.datasource.WritableDataSource; import com.alibaba.csp.sentinel.init.InitFunc; import com.alibaba.csp.sentinel.slots.block.authority.AuthorityRule; import com.alibaba.csp.sentinel.slots.block.authority.AuthorityRuleManager; import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRule; import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRuleManager; import com.alibaba.csp.sentinel.slots.block.flow.FlowRule; import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager; import com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowRule; import com.alibaba.csp.sentinel.slots.block.flow.param.ParamFlowRuleManager; import com.alibaba.csp.sentinel.slots.system.SystemRule; import com.alibaba.csp.sentinel.slots.system.SystemRuleManager; import com.alibaba.csp.sentinel.transport.util.WritableDataSourceRegistry; import java.io.FileNotFoundException; import java.io.UnsupportedEncodingException; import java.net.URLDecoder; import java.util.List; import java.util.Objects; /** * InitFunc实现类，处理dataSource初始化逻辑 */ public class FileDataSourceInit implements InitFunc { @Override public void init() throws Exception { //创建文件存储目录 RuleFileUtils.mkdirIfNotExits(PersistenceRuleConstant.storePath); //创建规则文件 RuleFileUtils.createFileIfNotExits(PersistenceRuleConstant.rulesMap); //处理流控规则逻辑 配置读写数据源 dealFlowRules(); // 处理降级规则 dealDegradeRules(); // 处理系统规则 dealSystemRules(); // 处理热点参数规则 dealParamFlowRules(); // 处理授权规则 dealAuthRules(); } private void dealFlowRules() throws FileNotFoundException { String ruleFilePath = PersistenceRuleConstant.rulesMap.get(PersistenceRuleConstant.FLOW_RULE_PATH); //创建流控规则的可读数据源 ReadableDataSource<String, List<FlowRule>> flowRuleRDS = new FileRefreshableDataSource(ruleFilePath, RuleListConverterUtils.flowRuleListParser); // 将可读数据源注册至FlowRuleManager 这样当规则文件发生变化时，就会更新规则到内存 FlowRuleManager.register2Property(flowRuleRDS.getProperty()); WritableDataSource<List<FlowRule>> flowRuleWDS = new FileWritableDataSource<>(ruleFilePath, RuleListConverterUtils.flowRuleEnCoding); // 将可写数据源注册至 transport 模块的 WritableDataSourceRegistry 中. // 这样收到控制台推送的规则时，Sentinel 会先更新到内存，然后将规则写入到文件中. WritableDataSourceRegistry.registerFlowDataSource(flowRuleWDS); } private void dealDegradeRules() throws FileNotFoundException { //获取规则文件路径 String degradeRuleFilePath = PersistenceRuleConstant.rulesMap.get(PersistenceRuleConstant.DEGRAGE_RULE_PATH).toString(); //创建流控规则的可读数据源 ReadableDataSource<String, List<DegradeRule>> degradeRuleRDS = new FileRefreshableDataSource( degradeRuleFilePath, RuleListConverterUtils.degradeRuleListParse ); // 将可读数据源注册至FlowRuleManager 这样当规则文件发生变化时，就会更新规则到内存 DegradeRuleManager.register2Property(degradeRuleRDS.getProperty()); WritableDataSource<List<DegradeRule>> degradeRuleWDS = new FileWritableDataSource<>( degradeRuleFilePath, RuleListConverterUtils.degradeRuleEnCoding ); // 将可写数据源注册至 transport 模块的 WritableDataSourceRegistry 中. // 这样收到控制台推送的规则时，Sentinel 会先更新到内存，然后将规则写入到文件中. WritableDataSourceRegistry.registerDegradeDataSource(degradeRuleWDS); } private void dealSystemRules() throws FileNotFoundException { //获取规则文件路径 String systemRuleFilePath = PersistenceRuleConstant.rulesMap.get(PersistenceRuleConstant.SYSTEM_RULE_PATH).toString(); //创建流控规则的可读数据源 ReadableDataSource<String, List<SystemRule>> systemRuleRDS = new FileRefreshableDataSource( systemRuleFilePath, RuleListConverterUtils.sysRuleListParse ); // 将可读数据源注册至FlowRuleManager 这样当规则文件发生变化时，就会更新规则到内存 SystemRuleManager.register2Property(systemRuleRDS.getProperty()); WritableDataSource<List<SystemRule>> systemRuleWDS = new FileWritableDataSource<>( systemRuleFilePath, RuleListConverterUtils.sysRuleEnCoding ); // 将可写数据源注册至 transport 模块的 WritableDataSourceRegistry 中. // 这样收到控制台推送的规则时，Sentinel 会先更新到内存，然后将规则写入到文件中. WritableDataSourceRegistry.registerSystemDataSource(systemRuleWDS); } private void dealParamFlowRules() throws FileNotFoundException { //获取规则文件路径 String paramFlowRuleFilePath = PersistenceRuleConstant.rulesMap.get(PersistenceRuleConstant.HOT_PARAM_RULE).toString(); //创建流控规则的可读数据源 ReadableDataSource<String, List<ParamFlowRule>> paramFlowRuleRDS = new FileRefreshableDataSource( paramFlowRuleFilePath, RuleListConverterUtils.paramFlowRuleListParse ); // 将可读数据源注册至FlowRuleManager 这样当规则文件发生变化时，就会更新规则到内存 ParamFlowRuleManager.register2Property(paramFlowRuleRDS.getProperty()); WritableDataSource<List<ParamFlowRule>> paramFlowRuleWDS = new FileWritableDataSource<>( paramFlowRuleFilePath, RuleListConverterUtils.paramRuleEnCoding ); // 将可写数据源注册至 transport 模块的 WritableDataSourceRegistry 中. // 这样收到控制台推送的规则时，Sentinel 会先更新到内存，然后将规则写入到文件中. ModifyParamFlowRulesCommandHandler.setWritableDataSource(paramFlowRuleWDS); } private void dealAuthRules() throws FileNotFoundException { //获取规则文件路径 String authFilePath = PersistenceRuleConstant.rulesMap.get(PersistenceRuleConstant.AUTH_RULE_PATH).toString(); //创建流控规则的可读数据源 ReadableDataSource<String, List<AuthorityRule>> authRuleRDS = new FileRefreshableDataSource( authFilePath, RuleListConverterUtils.authorityRuleParse ); // 将可读数据源注册至FlowRuleManager 这样当规则文件发生变化时，就会更新规则到内存 AuthorityRuleManager.register2Property(authRuleRDS.getProperty()); //创建流控规则的写数据源 WritableDataSource<List<AuthorityRule>> authRuleWDS = new FileWritableDataSource<>( authFilePath, RuleListConverterUtils.authorityEncoding ); // 将可写数据源注册至 transport 模块的 WritableDataSourceRegistry 中. // 这样收到控制台推送的规则时，Sentinel 会先更新到内存，然后将规则写入到文件中. WritableDataSourceRegistry.registerAuthorityDataSource(authRuleWDS); } }
在resources资源目录下新增文件META-INF/services/com.alibaba.csp.sentinel.init.InitFunc文件内容：
com.yyj.sentinel.extension.filepull.FileDataSourceInit
之后通过maven 打包安装到自己本地的maven仓库：
在微服务端直接引入该模块依赖即完成了对拉模式的整合：
<!-- 引入自己安装到仓库的拉模式的依赖 -->
<dependency>
<groupId>com.alibaba.csp</groupId>
<artifactId>sentinel-datasource-extension-file-pull</artifactId>
<version>1.8.4</version>
</dependency>
