sentinel dashboard 持久化规则，监控数据+nacos和mysql
2019-03-08 14:51 乌托邦的美梦 阅读(5295) 评论(0) 编辑 收藏 举报- Sentinel是阿里巴巴开源的分布式系统的流量防卫组件，Sentinel 把流量作为切入点，从流量控制，熔断降级，系统负载保护等多个维度保护服务的稳定性。
- 结合目前项目做了sentinel-dashboard数据持久化的改造，参考生产环境使用sentinel。https://github.com/alibaba/Sentinel/wiki/%E5%9C%A8%E7%94%9F%E4%BA%A7%E7%8E%AF%E5%A2%83%E4%B8%AD%E4%BD%BF%E7%94%A8-Sentinel。
- 克隆代码地址 https://github.com/alibaba/Sentinel.git。选择1.4.0版本（其他版本修改比较大）。
- 规则持久化
<dependency>
<groupId>com.alibaba.csp</groupId>
<artifactId>sentinel-datasource-nacos</artifactId>
</dependency>
原来的<scope>test</scope>去掉
- 自定义规则数据持久化
将项目测试下面的代码移动到正式代码里。如
添加com.taobao.csp.sentinel.dashboard.config.NacosConfig的配置
@Bean
public Converter<String, List<SystemRuleEntity>> systemRuleEntityDecoder() {
return s -> JSON.parseArray(s, SystemRuleEntity.class);
}
@Bean
public ConfigService nacosConfigService() throws Exception {
Properties properties = new Properties();
properties.put("serverAddr",address);
// 命名空间暂时不用，还是使用默认的public，
// 因为客服端sentinel-datasource-nacos的jar包目前只能设置地址，还不能设置命名空间
// properties.put("namespace","938ed9f5-b24b-4491-a41c-a2560dc42919");
return ConfigFactory.createConfigService(properties);
}
添加com.taobao.csp.sentinel.dashboard.config.NacosConfigUtil的配置
主要是一些静态属性添加，目前添加了限流文件和系统规则文件。
- 持久化限流规则接口重新实现
a) 推送数据
@Component("flowRuleNacosPublisher")
public class FlowRuleNacosPublisher implements DynamicRulePublisher<List<FlowRuleEntity>> {
@Autowired
private ConfigService configService;
@Autowired
private Converter<List<Map<String,Object>>,String> convert;
@Override
public void publish(String app, List<FlowRuleEntity> rules) throws Exception {
AssertUtil.notEmpty(app, "app name cannot be empty");
if (rules == null) {
return;
}
List<Map<String,Object>> ruleList= new ArrayList<>();
rules.forEach(obj->{
Map<String,Object> map = new HashMap<>();
map.put("resource",obj.getResource());
map.put("controlBehavior",obj.getControlBehavior());
map.put("count",obj.getCount());
map.put("grade",obj.getGrade());
map.put("limitApp",obj.getLimitApp());
map.put("strategy",obj.getStrategy());
ruleList.add(map);
});
configService.publishConfig(app+NacosConfigUtil.FLOW_DATA_ID_POSTFIX,NacosConfigUtil.GROUP_ID,convert.convert(ruleList));
}
}
b) 从nacos获取数据
@Component("flowRuleNacosProvider")
public class FlowRuleNacosProvider implements DynamicRuleProvider<List<FlowRuleEntity>> {
@Autowired
private ConfigService configService;
@Autowired
private Converter<String, List<FlowRuleEntity>> converter;
@Override
public List<FlowRuleEntity> getRules(String appName) throws Exception {
String rules = configService.getConfig(appName + NacosConfigUtil.FLOW_DATA_ID_POSTFIX,
NacosConfigUtil.GROUP_ID, 3000);
if (StringUtil.isEmpty(rules)) {
return new ArrayList<>();
}
return converter.convert(rules);
}
}
c) 修改com.taobao.csp.sentinel.dashboard.view.FlowControllerV2
- 系统规则的修改与上面限流规则修改差多。
- 系统监控的数据持久化采（采用mysql+jpa）
1) 添mysql和jpa所需要的jar包
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.11</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
<version>2.0.4.RELEASE</version>
</dependency>
2) application.properties文件的修改
spring.datasource.url=jdbc:mysql://localhost:3307/edu_operation_platform?useSSL=false&useUnicode=true&characterEncoding=UTF8&allowMultiQueries=true
spring.datasource.username=root
spring.datasource.password=Abcd1234
spring.datasource.driver-class-name= com.mysql.jdbc.Driver
# spring data jpa
spring.jpa.hibernate.ddl-auto=none
spring.jpa.hibernate.use-new-id-generator-mappings=false
spring.jpa.database-platform=org.hibernate.dialect.MySQLDialect
spring.jpa.show-sql=true
3) 自定义监控数据存储
com.taobao.csp.sentinel.dashboard.repository.metric;
import com.alibaba.csp.sentinel.util.StringUtil;
import com.taobao.csp.sentinel.dashboard.datasource.entity.MetricEntity;
import com.taobao.csp.sentinel.dashboard.datasource.entity.MetricPO;
import org.springframework.beans.BeanUtils;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.util.CollectionUtils;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import javax.persistence.Query;
@Transactional
@Component("jpaMetricsRepository")
public class JpaMetricsRepository implements MetricsRepository<MetricEntity> {
@PersistenceContext
private EntityManager em;
@Override
public void save(MetricEntity entity) {
if (entity == null || StringUtil.isBlank(entity.getApp())) {
return;
}
MetricPO metricPO = new MetricPO();
BeanUtils.copyProperties(entity, metricPO);
metricPO.setTimestamp(entity.getTimestamp());
metricPO.setGmtCreate(entity.getGmtCreate());
metricPO.setGmtModified(entity.getGmtModified());
em.persist(metricPO);
}
@Override
public synchronized void saveAll(Iterable<MetricEntity> metrics) {
if (metrics == null) {
return;
}
metrics.forEach(this::save);
}
@Override
public List<MetricEntity> queryByAppAndResourceBetween(String app, String resource, long startTime, long endTime) {
List<MetricEntity> results = new ArrayList<MetricEntity>();
if (StringUtil.isBlank(app)) {
return results;
}
if (StringUtil.isBlank(resource)) {
return results;
}
StringBuilder hql = new StringBuilder();
hql.append("FROM MetricPO");
hql.append(" WHERE app=:app");
hql.append(" AND resource=:resource");
hql.append(" AND timestamp>=:startTime");
hql.append(" AND timestamp<=:endTime");
Query query = em.createQuery(hql.toString());
query.setParameter("app", app);
query.setParameter("resource", resource);
query.setParameter("startTime", Date.from(Instant.ofEpochMilli(startTime)));
query.setParameter("endTime", Date.from(Instant.ofEpochMilli(endTime)));
List<MetricPO> metricPOs = query.getResultList();
if (CollectionUtils.isEmpty(metricPOs)) {
return results;
}
for (MetricPO metricPO : metricPOs) {
MetricEntity metricEntity = new MetricEntity();
BeanUtils.copyProperties(metricPO, metricEntity);
results.add(metricEntity);
}
return results;
}
@Override
public List<String> listResourcesOfApp(String app) {
List<String> results = new ArrayList<>();
if (StringUtil.isBlank(app)) {
return results;
}
StringBuilder hql = new StringBuilder();
hql.append("FROM MetricPO");
hql.append(" WHERE app=:app");
hql.append(" AND timestamp>=:startTime");
long startTime = System.currentTimeMillis() - 1000 * 60;
Query query = em.createQuery(hql.toString());
query.setParameter("app", app);
query.setParameter("startTime", Date.from(Instant.ofEpochMilli(startTime)));
List<MetricPO> metricPOs = query.getResultList();
if (CollectionUtils.isEmpty(metricPOs)) {
return results;
}
List<MetricEntity> metricEntities = new ArrayList<MetricEntity>();
for (MetricPO metricPO : metricPOs) {
MetricEntity metricEntity = new MetricEntity();
BeanUtils.copyProperties(metricPO, metricEntity);
metricEntities.add(metricEntity);
}
Map<String, MetricEntity> resourceCount = new HashMap<>(32);
for (MetricEntity metricEntity : metricEntities) {
String resource = metricEntity.getResource();
if (resourceCount.containsKey(resource)) {
MetricEntity oldEntity = resourceCount.get(resource);
oldEntity.addPassQps(metricEntity.getPassQps());
oldEntity.addRtAndSuccessQps(metricEntity.getRt(), metricEntity.getSuccessQps());
oldEntity.addBlockQps(metricEntity.getBlockQps());
oldEntity.addExceptionQps(metricEntity.getExceptionQps());
oldEntity.addCount(1);
} else {
resourceCount.put(resource, MetricEntity.copyOf(metricEntity));
}
}
return resourceCount.entrySet()
.stream()
.sorted((o1, o2) -> {
MetricEntity e1 = o1.getValue();
MetricEntity e2 = o2.getValue();
int t = e2.getBlockQps().compareTo(e1.getBlockQps());
if (t != 0) {
return t;
}
return e2.getPassQps().compareTo(e1.getPassQps());
})
.map(Map.Entry::getKey)
.collect(Collectors.toList());
}
}
4) 修改com.taobao.csp.sentinel.dashboard.metric. MetricFetcher类对监控数据吃久化的实现类的注册：
修改com.taobao.csp.sentinel.dashboard.view. MetricController:
添加实体类：
@Entity
@Table(name = "sentinel_metric")
public class MetricPO implements Serializable {
private static final long serialVersionUID = 7200023615444172715L;
/**id，主键*/
@Id
@GeneratedValue
@Column(name = "id")
private Long id;
/**创建时间*/
@Column(name = "gmt_create")
private Date gmtCreate;
/**修改时间*/
@Column(name = "gmt_modified")
private Date gmtModified;
/**应用名称*/
@Column(name = "app")
private String app;
/**统计时间*/
@Column(name = "timestamp")
private Date timestamp;
/**资源名称*/
@Column(name = "resource")
private String resource;
/**通过qps*/
@Column(name = "pass_qps")
private Long passQps;
/**成功qps*/
@Column(name = "success_qps")
private Long successQps;
/**限流qps*/
@Column(name = "block_qps")
private Long blockQps;
/**发送异常的次数*/
@Column(name = "exception_qps")
private Long exceptionQps;
/**所有successQps的rt的和*/
@Column(name = "rt")
private Double rt;
/**本次聚合的总条数*/
@Column(name = "_count")
private Integer count;
/**资源的hashCode*/
@Column(name = "resource_code")
private Integer resourceCode;
public static long getSerialVersionUID() {
return serialVersionUID;
}
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public Date getGmtCreate() {
return gmtCreate;
}
public void setGmtCreate(Date gmtCreate) {
this.gmtCreate = gmtCreate;
}
public Date getGmtModified() {
return gmtModified;
}
public void setGmtModified(Date gmtModified) {
this.gmtModified = gmtModified;
}
public String getApp() {
return app;
}
public void setApp(String app) {
this.app = app;
}
public Date getTimestamp() {
return timestamp;
}
public void setTimestamp(Date timestamp) {
this.timestamp = timestamp;
}
public String getResource() {
return resource;
}
public void setResource(String resource) {
this.resource = resource;
}
public Long getPassQps() {
return passQps;
}
public void setPassQps(Long passQps) {
this.passQps = passQps;
}
public Long getSuccessQps() {
return successQps;
}
public void setSuccessQps(Long successQps) {
this.successQps = successQps;
}
public Long getBlockQps() {
return blockQps;
}
public void setBlockQps(Long blockQps) {
this.blockQps = blockQps;
}
public Long getExceptionQps() {
return exceptionQps;
}
public void setExceptionQps(Long exceptionQps) {
this.exceptionQps = exceptionQps;
}
public Double getRt() {
return rt;
}
public void setRt(Double rt) {
this.rt = rt;
}
public Integer getCount() {
return count;
}
public void setCount(Integer count) {
this.count = count;
}
public Integer getResourceCode() {
return resourceCode;
}
public void setResourceCode(Integer resourceCode) {
this.resourceCode = resourceCode;
}
// getter setter省略
}
数据库创建表sql：
-- 创建监控数据表
CREATE TABLE `sentinel_metric` (
`id` INT NOT NULL AUTO_INCREMENT COMMENT 'id，主键',
`gmt_create` DATETIME COMMENT '创建时间',
`gmt_modified` DATETIME COMMENT '修改时间',
`app` VARCHAR(100) COMMENT '应用名称',
`timestamp` DATETIME COMMENT '统计时间',
`resource` VARCHAR(500) COMMENT '资源名称',
`pass_qps` INT COMMENT '通过qps',
`success_qps` INT COMMENT '成功qps',
`block_qps` INT COMMENT '限流qps',
`exception_qps` INT COMMENT '发送异常的次数',
`rt` DOUBLE COMMENT '所有successQps的rt的和',
`_count` INT COMMENT '本次聚合的总条数',
`resource_code` INT COMMENT '资源的hashCode',
INDEX app_idx(`app`) USING BTREE,
INDEX resource_idx(`resource`) USING BTREE,
INDEX timestamp_idx(`timestamp`) USING BTREE,
PRIMARY KEY (`id`)
) ENGINE=INNODB DEFAULT CHARSET=utf8;
- 客服端改造
- 引入nacos对数据源支持的jar
- 重新实现InitFunc接口。
<dependency>
<groupId>com.alibaba.csp</groupId>
<artifactId>sentinel-datasource-nacos</artifactId>
<version>1.4.0</version>
</dependency>
@Component
public class NacosInit implements InitFunc {
private static String address;
@Value("${nacos.address}")
public void setAddress(String address) {
this.address = address;
}
@Override
public void init() throws Exception {
Converter<String, List<FlowRule>> flowParser = source -> JSON.parseObject(source,new TypeReference<List<FlowRule>>() {});
ReadableDataSource<String, List<FlowRule>> flowSource = new NacosDataSource<>(address, NacosConfigUtil.GROUPID, NacosConfigUtil.FLOW_RULE, flowParser);
FlowRuleManager.register2Property(flowSource.getProperty());
Converter<String, List<SystemRule>> systemParser = source -> JSON.parseObject(source,new TypeReference<List<SystemRule>>() {});
ReadableDataSource<String, List<SystemRule>> systemSource = new NacosDataSource<>(address, NacosConfigUtil.SYSTEM_RULE, NacosConfigUtil.GROUPID, systemParser);
SystemRuleManager.register2Property(systemSource.getProperty());
}
}
配置静态属性：
public final class NacosConfigUtil {
public NacosConfigUtil() {
}
public static final String GROUPID="hbd_rule";
public static final String FLOW_RULE="auth-server_flow_rule.yml";
public static final String SYSTEM_RULE="auth-server_system_rule.yml";
}
注入新实现的类
创建一个叫com.alibaba.csp.sentinel.init.InitFunc 的文件，内容是你实现InitFunc类的路径，如com.hanboard.educloud.auth.config.NacosInit 放在resources/META-INF/services
打包启动项目，就行了......
项目github地址：https://github.com/chenshijun007/sentinel-nacos-mysql.git
