通过 API 快速创建 AlertManager silence
概述
通常我们要 silence 某个 AlertManager 的 alert 时,需要通过 UI 界面操作,如下图:
效率有点低,而且不够自动化,那么是否可以有一种办法快速创建 AlertManager silence 呢?
-- 有的,通过 API.
API Payload
v1
如下:
{
"matchers": [
{
"name": "alername1",
"value": ".*",
"isRegex": true
}
],
"startsAt": "2022-04-29T22:12:33.533Z",
"endsAt": "2022-04-29T23:11:44.603Z",
"createdBy": "api",
"comment": "Silence",
"status": {
"state": "active"
}
}
v2
{
"matchers": [
{
"name": "service",
"value": "rancher",
"isRegex": false,
"isEqual": true
},
{
"name": "alertname",
"value": "TargetDown",
"isRegex": false,
"isEqual": true
}
],
"startsAt": "2022-04-29T10:11:35.656Z",
"endsAt": "2022-04-29T12:11:35.656Z",
"createdBy": "Casey Cui",
"comment": "配置错误导致的误报",
"id": null
}
具体实现
curl 实现
📝 Notes:
以 v1 api 为例
如下:
curl https://alertmanager.ewhisper.cn/api/v1/silences -d '{
"matchers": [
{
"name": "alername1",
"value": ".*",
"isRegex": true
}
],
"startsAt": "2022-04-29T22:12:33.533Z",
"endsAt": "2022-04-29T23:11:44.603Z",
"createdBy": "api",
"comment": "Silence",
"status": {
"state": "active"
}
}'
📝Notes:
在 K8S 集群内,地址一般为:http://alertmanager:9093
python 实现
如下:
#!/usr/bin/python3
import requests
import socket
import datetime
import time
res = requests.post("http://alertmanager:9093/api/v2/silences", json={
"matchers": [
{"name": "job", "value": "myjob", "isRegex": False},
{"name": "instance", "value": "{}:1234".format(socket.gethostname()), "isRegex": False},
],
"startsAt": datetime.datetime.utcfromtimestamp(time.time()).isoformat(),
"endsAt": datetime.datetime.utcfromtimestamp(time.time() + 4*3600).isoformat(),
"comment": "Backups on {}".format(socket.gethostname()),
"createdBy": "My backup script",
},
)
res.raise_for_status()
silenceId = res.json()["silenceID"]
移除 silence 的脚本:
res = requests.delete("http://alertmanager:9093/api/v2/silence/{}".format(silenceId))
res.raise_for_status()
EOF