Alertmanager的路由配置

一.配置Alertmanager

1.修改Alertmanager的配置文件

[root@prometheus-server32 alertmanager-0.27.0.linux-amd64]# cat alertmanager.yml
route:
  group_by: ['alertname']
  group_wait: 1s
  group_interval: 3s
  repeat_interval: 1m
  # 默认发给"sre_system"组用户
  receiver: 'sre_system'
  continue: false
  # 配置子路由
  routes:
    - receiver: 'sre_dba'
      match_re:
        job: oldboyedu_mysqld_exporter
      # 建议将continue的值设置为true，表示当前的条件是否匹配，都将继续向下匹配规则
      # 这样做的目的是将消息发给最后的系统组(sre_system)
      continue: true
    - receiver: 'sre_bigdata'
      match_re:
        job: oldboyedu_flink_exporter
      continue: true
    - receiver: 'sre_k8s'
      match_re:
        job: oldboyedu_istio_exporter
      continue: true
    - receiver: 'sre_system'
      match_re:
        job: .*
      continue: true
receivers:
  - name: 'sre_dba'
    webhook_configs:
      - url: 'http://10.0.0.42:6666/oldboyedu/alert'
        http_config: {}
        max_alerts: 0
        send_resolved: true
  - name: 'sre_bigdata'
    webhook_configs:
      - url: 'http://10.0.0.42:7777/oldboyedu/alert'
        http_config: {}
        max_alerts: 0
        send_resolved: true
  - name: 'sre_k8s'
    webhook_configs:
      - url: 'http://10.0.0.42:8888/oldboyedu/alert'
        http_config: {}
        max_alerts: 0
        send_resolved: true
  - name: 'sre_system'
    webhook_configs:
      - url: 'http://10.0.0.42:9999/oldboyedu/alert'
        http_config: {}
        max_alerts: 0
        send_resolved: true
[root@prometheus-server32 alertmanager-0.27.0.linux-amd64]# 

2.运行Alertmanager

[root@prometheus-server32 alertmanager-0.27.0.linux-amd64]# ./alertmanager 

二.准备接受者的url

1.下载go程序

[root@node-exporter42 ~]# wget http://192.168.15.253/Resources/Prometheus/softwares/Alertmanager/oldboyedu-alertmanger-receiveData

2.添加执行权限

[root@node-exporter42 ~]# chmod +x oldboyedu-alertmanger-receiveData 
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# ll oldboyedu-alertmanger-receiveData 
-rwxr-xr-x 1 root root 27207714 Nov 14 15:39 oldboyedu-alertmanger-receiveData*
[root@node-exporter42 ~]# 

3.运行go程序

[root@node-exporter42 ~]# ./oldboyedu-alertmanger-receiveData -addr :6666
[root@node-exporter42 ~]# ./oldboyedu-alertmanger-receiveData -addr :7777
[root@node-exporter42 ~]# ./oldboyedu-alertmanger-receiveData -addr :8888
[root@node-exporter42 ~]# ./oldboyedu-alertmanger-receiveData -addr :9999

三.Prometheus添加规则

1.修改Prometheus的配置文件

[root@prometheus-server31 prometheus-2.53.3.linux-amd64]# vim prometheus.yml
...
rule_files:
  ...
  - "oldboyedu-linux95-rules.yml"
  - "oldboyedu-linux95-rules-alertmanager.yml"

scrape_configs:
  ...

  - job_name: "oldboyedu_mysqld_exporter"
    static_configs:
      - targets:
          - 10.0.0.41:9100

  - job_name: "oldboyedu_flink_exporter"
    static_configs:
      - targets:
          - 10.0.0.42:9100

  - job_name: "oldboyedu_istio_exporter"
    static_configs:
      - targets:
          - 10.0.0.43:9100

2.编写规则文件

[root@prometheus-server31 prometheus-2.53.3.linux-amd64]# cat oldboyedu-linux95-rules-alertmanager.yml
groups:
  - name: xixi
    rules:
      - alert: yinzhengjie_mysqld_exporter-alert
        expr: up{job="oldboyedu_mysqld_exporter"} == 0
        labels:
          severity: critical
          apps: mysql80
        annotations:
          summary: DBA机器异常
  
  - name: haha
    rules:
      - alert: yinzhengjie_bigdata_exporter-alert
        expr: up{job="oldboyedu_flink_exporter"} == 0
        labels:
          severity: warning
          apps: flink
        annotations:
          summary: 大数据集群机器异常

  - name: hehe
    rules:
      - alert: yinzhengjie_k8s_exporter-alert
        expr: up{job="oldboyedu_istio_exporter"} == 0
        labels:
          severity: warning
          apps: k8s
        annotations:
          summary: k8s集群机器异常
[root@prometheus-server31 prometheus-2.53.3.linux-amd64]# 

3.检查配置文件是否生效

[root@prometheus-server31 prometheus-2.53.3.linux-amd64]# ./promtool check config prometheus.yml
Checking prometheus.yml
  SUCCESS: 2 rule files found
 SUCCESS: prometheus.yml is valid prometheus config file syntax

Checking oldboyedu-linux95-rules.yml
  SUCCESS: 2 rules found

Checking oldboyedu-linux95-rules-alertmanager.yml
  SUCCESS: 3 rules found

[root@prometheus-server31 prometheus-2.53.3.linux-amd64]# 

四.热加载Prometheus

[root@prometheus-server31 prometheus-2.53.3.linux-amd64]# curl -X POST http://10.0.0.31:9090/-/reload 

五.验证配置是否生效

[root@node-exporter41 ~]# reboot 

可以查看Alertmanager的WebUI
http://10.0.0.32:9093/#/alerts

观察42节点的各个终端

不难发现，仅有sre_dba和sre_system收到了告警消息