heml3 部署prometheus-operator（二）

添加alertmanager钉钉报警

配置alertmanager web访问

vim prom_alert.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: alert-web
  namespace: monitoring
spec:
  rules:
  - host: alert.my.com
    http:
      paths:
      - backend:
          serviceName: prometheus-operator-alertmanager
          servicePort: 9093
        path: /

kubectl apply -f rom_alert.yaml

web 访问，查看status

 命令行查看

kubectl get secrets -n monitoring  alertmanager-prometheus-operator-alertmanager -o go-template='{{ index .data "alertmanager.yaml" }}' |base64 --decode

 

1、添加钉钉机器人

https://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq

 

2、创建钉钉告警插件

[root@master alert]# vim dingtalk-webhook.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: dingtalk
  name: webhook-dingtalk
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      run: dingtalk
  template:
    metadata:
      labels:
        run: dingtalk
    spec:
      containers:
      - name: dingtalk
        image: timonwong/prometheus-webhook-dingtalk:v0.3.0
        imagePullPolicy: IfNotPresent
        # 设置钉钉群聊自定义机器人后，使用实际 access_token 替换下面 xxxxxx部分
        args:
          - --ding.profile=webhook1=https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxxxxxxxxxxxxxx18284bee528adefcae97c909b
        ports:
        - containerPort: 8060
          protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: dingtalk
  name: webhook-dingtalk
  namespace: monitoring
spec:
  ports:
  - port: 8060
    protocol: TCP
    targetPort: 8060
  selector:
    run: dingtalk

3、应用dingtalk-webhook.yaml

$ kubectl apply -f dingtalk-webhook.yaml

 

4、添加告警接收器
到安装包的路径下创建告警接收器。（alertmanager.yaml ）

# Configuration for alertmanager
alertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ["job"]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: webhook
      routes:
      - match:
          alertname: Watchdog
        receiver: "null"
    receivers:
      - name: "null"
      - name: "webhook"
        webhook_configs:
          - url: 'http://webhook-dingtalk.monitoring.svc.cluster.local:8060/dingtalk/webhook1/send'
            send_resolved: true

 

webhook-dingtalk.monitoring.svc.cluster.local 为dingtalk-webhook.yaml svc 地址

5、更新原有的secret
kubectl get secrets -n monitoring

 

 k8s更新方式，先删除再新建

kubectl delete secrets alertmanager-prometheus-operator-alertmanager  -n monitoring

kubectl create  secret generic alertmanager-prometheus-operator-alertmanager --from-file=alertmanager.yaml -n monitoring

 helm 更新

helm upgrade prometheus-operator prometheus-operator --values=alertmanager.yml -n monitoring

 

查看web访问

 

 钉钉报警