基于consul服务发现监控prometheus数据采集
业务服务使用micrometer + prometheus + grafana模式进行监控,如果服务机器列表发生变化,需要改动prometheus.yml机器域名或者IP,如果发生机器迁移或者扩缩容则容易遗忘,因此,我们可以通过prometheus与consul集成来实现动态监测机器列表变动来获取对应的监控服务机器信息。
1、业务服务目前基本是基于consul做的服务发现,可以直接在promethus.yml里配置,注意通过drop过滤非业务服务,并进行标签的重命名;
- job_name: 'consul-prometheus' metrics_path: /actuator/prometheus scheme: http consul_sd_configs: - server: 'consul_ip:8500' services: [] relabel_configs: - source_labels: - __meta_consul_service regex: 'consul|nginx|comment' action: drop - source_labels: - __meta_consul_service target_label: group
2、将nginx服务注册到consul上,在promethus.yml里配置,注意通过nginx服务,并进行标签的重命名;
- job_name: 'nginx-consul-prometheus' metrics_path: /metrics consul_sd_configs: - server: 'consul_ip:8500' services: - nginx relabel_configs: - source_labels: ["__meta_consul_service"] target_label: "group"
nginx服务机器上需安装consul客户端,模板文件nginx.json,启动consul客户端时记得要额外加载该配置文件
{
"service":{
"id": "nginx",
"name": "nginx",
"address": "{{hostIp['stdout']}}",
"port": 9145,
"meta": {
"sogou_host": "{{hostIp['stdout']}}"
},
"tags": ["nginx"],
"checks": [
{
"http": "http://{{hostIp['stdout']}}:9145/metrics",
"interval": "30s"
}
]
}
}
3、ansible脚本如下,将nginx服务注册consul:
- name: register hostIp shell: /sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:" |head -n 1 register: hostIp ignore_errors: True tags: - consul - name: register nginx_metrics shell: netstat -ntlp | grep 9145 register: nginx_metrics ignore_errors: True failed_when: nginx_metrics.rc != 1 and nginx_metrics.rc != 0 - name: Copy consul.json template: src=consul.hcl.j2 dest=/search/odin/consul/conf/consul.hcl - name: Copy nginx.json template: src=nginx.json.j2 dest=/search/odin/consul/conf/nginx.json when: nginx_metrics.rc == 0 tags: - consul - name: Copy systemd script template: src=consul-promethus.service dest=/etc/systemd/system/consul-promethus.service tags: - consul - name: chown /search/odin file: path: /search/odin/consul owner: odin group: odin mode: 0755 recurse: yes tags: - consul - name: Service consul started systemd: daemon_reload: yes enabled: yes name: consul-promethus state: restarted tags: - consul
4、安装consul太麻烦,可以直接使用api将服务注册到consul server,可以忽略上面的了
curl http://consulip:8500/v1/agent/service/register -X PUT -i -H "Content-Type:application/json" -d '{ "ID": "id", "Name": "name", "Tags": [ "nginx" ], "Address": "ip", "Port": 9145, "Check": { "DeregisterCriticalServiceAfter": "30s", "HTTP": "http://ip:9145/metrics", "Interval": "30s" } }'
。。。
向上吧,少年