prometheus数据持久化设计

一、prometheus数据持久化设计

1、需求

1分钟级别的数据保留 10天，10分钟级别的数据保留20天，1小时级别的数据保留30天，1天级别的保留3个月

2、方案

1、创建4个数据库，分别存储不同频率的数据，prometheus：原始数据 、prometheus_10m：按10分钟频率的数据、prometheus_1h：按1小时频率的数据、prometheus_1d：按天频率的数据

2、创建CQ，统计上级频率数据并写入相应频率的数据库中；10m由原始数据统计，1h由10m数据统计，1d数据由1h数据统计

3、开发influxdb remote read adapter，根据查询的开始时间来确定从哪个数据库查数据

3、db设计

prometheus , 数据保留：15d，分区：1d

prometheus_10m , 数据保留：20d，分区：1d

prometheus_1h , 数据保留：30d，分区：1d

prometheus_1d , 数据保留：730d，分区：7d

4、相应的sql

CREATE DATABASE "prometheus" WITH DURATION 15d SHARD DURATION 1d NAME "prometheus"
CREATE DATABASE "prometheus_10m" WITH DURATION 20d SHARD DURATION 1d NAME "prometheus_10m"
CREATE DATABASE "prometheus_1h" WITH DURATION 30d SHARD DURATION 1d NAME "prometheus_1h"
CREATE DATABASE "prometheus_1d" WITH DURATION 730d SHARD DURATION 7d NAME "prometheus_1d"

 

5、CQ

 

CREATE CONTINUOUS QUERY "cq_prometheus_10m" ON "prometheus" BEGIN SELECT mean(value) as "value" INTO "prometheus_10m"..:MEASUREMENT FROM prometheus../.*/ GROUP BY time(10m),* END

 

CREATE CONTINUOUS QUERY "cq_prometheus_1h" ON "prometheus_10m" BEGIN SELECT mean(value) as "value" INTO "prometheus_1h"..:MEASUREMENT FROM prometheus_10m../.*/ GROUP BY time(1h),* END

 

CREATE CONTINUOUS QUERY "cq_prometheus_1d" ON "prometheus_1h" BEGIN SELECT mean(value) as "value" INTO "prometheus_1d"..:MEASUREMENT FROM prometheus_1h../.*/ GROUP BY time(1d),* END

adapter

主要是做http请求转发，基于prometheus remote read的请求中Query的StartTimestampMs来确定/api/v1/prom/read?db=%s最终db调用哪一个

StartTimestampMs在7天内转发到/api/v1/prom/read?db=prometheus

StartTimestampMs在7-15天内转发到/api/v1/prom/read?db=prometheus_10m

StartTimestampMs在15-30天内转发到/api/v1/prom/read?db=prometheus_1h

StartTimestampMs在30天以上转发到/api/v1/prom/read?db=prometheus_1d