Alertmanager简介及机制_告警抑制

测试目标：客户端node_explore停止，抑制了blackbox探测邮件告警功能。

 

客户端安装blackbox_exporter服务

[root@promethus blackbox_exporter]# ll

-rwxr-xr-x 1 3434 3434 14925964 3月  15 21:32 blackbox_exporter

-rw-r--r-- 1 3434 3434      629 3月  15 21:42 blackbox.yml

配置好blackbox.yml文件

[root@promethus blackbox_exporter]# vim blackbox.yml

modules:

  http_2xx:

    prober: http

  http_post_2xx:

    prober: http

    http:

      method: POST

  tcp_connect:

    prober: tcp

  pop3s_banner:

    prober: tcp

    tcp:

      query_response:

      - expect: "^+OK"

      tls: true

      tls_config:

        insecure_skip_verify: false

  icmp:

    prober: icmp

 

服务端主配置文件。

  - job_name: 'promeclient'

    scrape_interval: 15s

    file_sd_configs:

    - files:

      - /usr/local/prometheus/files/client.json

    relabel_configs:

      - source_labels: [__address__]

        target_label: host_ip

 

  - job_name: 'HTTP/TCP Health Check'

    scrape_interval: 15s

    metrics_path: /probe

    params:

      module: [http_2xx]

    file_sd_configs:

    - files:

      - /usr/local/prometheus/files/service_blackbox.json

    relabel_configs:

      - source_labels: [__param_target]

        target_label: instance

      - target_label: scraper

        replacement: blackbox-exporter

      - target_label: __address__

        replacement: 192.168.19.128:9115

      - target_label: env

        replacement: prod

 

rule_files:

  - "/usr/local/prometheus/host.rule"

  - "/usr/local/prometheus/blackbox.rules"

 

配置文件

[root@promethus files]# vim client.json

[

    {

        "targets": [

            "hongk:9100"

        ],

        "labels": {

            "project": "HK测试服务器",

            "app": "HK test",

            "scraper": "telegraf",

            "region": "SZD"

        }

    }

]

 

[root@promethus files]# vim service_blackbox.json

[

    {

        "targets": [

            "http://192.168.19.128:9093",

            "http://192.168.19.128:3000"

        ],

        "labels": {

            "app": "grafana_server/alertmanager",

            "project": "pass-regisrty",

            "scraper": "blackbox-export",

            "region": "SZD",

            "module": "http_2xx"

        }

    }

]

测试方式，正常情况下运行，先使得blackbox挂掉，然后node_explore挂掉。测试结果：当blackbox挂掉，正常显示blackbox相关url异常，当node_explore挂掉后，邮件只显示node_explore挂掉。达到了预期效果。

 

告警部分配置。

inhibit_rules:

  - source_match:

      job: 'promeclient'

    target_match:

      job: 'HTTP/TCP Health Check'

    equal: ['region']

 

 如上截图显示。