prometheus2.0 联邦的配置

Prometheus联邦

1.分层联邦

分层联邦允许普罗米修斯扩展到数十个数据中心和数百万个节点的环境。在这种用例中，联邦拓扑类似于一棵树，更高级别的普罗米修斯服务器从大量的从属服务器收集汇总的时间序列数据。

2.跨服务联邦

In cross-service federation, a Prometheus server of one service is configured to scrape selected data from another service's Prometheus server to enable alerting and queries against both datasets within a single server.

在跨服务联合中，一个服务的普罗米修斯服务器被配置为从另一个服务的普罗米修斯服务器中刮取选定的数据，以使得能够针对单个服务器内的两个数据集进行警报和查询。

For example, a cluster scheduler running multiple services might expose resource usage information (like memory and CPU usage) about service instances running on the cluster. On the other hand, a service running on that cluster will only expose application-specific service metrics. Often, these two sets of metrics are scraped by separate Prometheus servers. Using federation, the Prometheus server containing service-level metrics may pull in the cluster resource usage metrics about its specific service from the cluster Prometheus, so that both sets of metrics can be used within that server.

例如，运行多个服务的集群调度程序可能会暴露有关在集群上运行的服务实例的资源使用情况信息（如内存和CPU使用情况）。另一方面，在该群集上运行的服务将仅公开特定于应用程序的服务度量标准。通常，这两套指标是由单独的普罗米修斯服务器来抓取的。使用联邦，包含服务级别度量的普罗米修斯服务器可以从集群普罗米修斯拉入有关其特定服务的集群资源使用度量，以便这两组度量可以在该服务器内使用。

【举个例子：我们要监控mysqld的运行状态，可以使用1个主Prometheus+2个分片Prometheus(一个用来采集node_exporter的metrics、一个用来采集mysql_exporter的metrics)，然后在主Prometheus上做汇总】

安装prometheus和mysqld_exporter、postgres_exporter的步骤也不写了，很简单的，exporter的部署我们通常用ansible或saltstack之类的工具批量分发。

我这里实验起见，在一台机器上跑了3个shard节点，1个global节点。

Node1： 10.0.20.25 （跑了老版本的prometheus1.7、mysql_exporter、postgres_exporter，这是之前做实验搭建的环境）

Node2： 10.0.20.26 （跑了prometheus2.0、mysql_exporter、postgres_exporter）

下面开始在Node2上开始我们的联邦的配置吧。

注意：我这里做实验的时候，跨服务联邦的配置还不太规范，照官方的说法是采用多个独立的Prometheus节点分别采集node_exporter、mysql_exporter、postgres_exporter 这些metrics，然后再在GLOBAL节点做汇总。

3.文件配置

cd /usr/local/prometheus

编写存放要采集主机的文件：

配置文件一

cat mysqld.json

[
    {
        "targets": [
            "10.0.20.26:9104",
            "10.0.20.26:9100",
            "10.0.20.25:9104",
            "10.0.20.25:9100"
        ],
        "labels": {
            "services": "dba_test",
        }
    }
]

配置文件二

cat pgsql.json

3个Shard节点配置文件如下：

Prometheus配置一

节点1，搜集的是mysql的信息

cat prometheus1.yml

global:

  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
scrape_configs:
  - job_name: 'mysql'
    file_sd_configs:
      - files: ['./mysqld.json']

Prometheus配置二

节点2，搜集的是pgsql的信息

cat prometheus2.yml

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
scrape_configs:
  - job_name: 'pgsql'
    file_sd_configs:
      - files: ['./pgsql.json']

Prometheus配置三

节点3，搜集的是prometheus节点的信息

cat prometheus3.yml

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
scrape_configs:
  - job_name: 'prometheus25'
    static_configs:
      - targets: ['10.0.20.25:9090']
  - job_name: 'prometheus26'
    static_configs:
      - targets: ['10.0.20.26:9090']

然后，启动3个分片节点：

Prometheus全局配置

再来配置GLOBAL节点：

cat prometheus.yml 内容如下：

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job=~"prometheus.*"}'
        - '{job="mysql"}'
        - '{job="pgsql"}'
    static_configs:
      - targets:
        - '10.0.20.26:9091'
        - '10.0.20.26:9092'
        - '10.0.20.26:9093'

启动GLOBAL节点：

然后，在浏览器访问http://10.0.20.26:9090/targets 这个GLOBAL节点，效果如下图：