prometheus监控之自动发现

prometheus监控之自动发现,这里采用服务端添加配置文件,具体操作如下,目前prometheus server只有如下节点:

 

 

 现在开始添加配置文件:

1.首先创建存放配置文件的目录:

# mkdir /usr/local/prometheus/target/node/ -p

2.然后在prometheus server配置文件中定义file类型:

  - job_name: 'host_discovery'
    file_sd_configs:
    - files:
      - "/usr/local/prometheus/target/node/host_discovery.json"
      refresh_interval: 6s

3.添加targets到json配置文件中:

[root@master node]# cat host_discovery.json 
[{
    "targets": ["172.16.23.121:9100"],
    "labels": {
        "instance": "172.16.23.121",
        "role": "node1"
    }
}]

上面双引号一定要注意不要使用单引号,不然日志会出现如下报错:

[root@master node]# tail -f /var/log/messages
Sep 22 23:06:50 master prometheus: level=error ts=2020-09-22T15:06:50.849Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"
Sep 22 23:06:50 master prometheus: level=error ts=2020-09-22T15:06:50.849Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"
Sep 22 23:06:50 master prometheus: level=error ts=2020-09-22T15:06:50.849Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"
Sep 22 23:06:54 master prometheus: level=error ts=2020-09-22T15:06:54.378Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"
Sep 22 23:06:54 master prometheus: level=error ts=2020-09-22T15:06:54.866Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"
Sep 22 23:07:00 master prometheus: level=error ts=2020-09-22T15:07:00.378Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"

4.配置完成后,最后校验prometheus server配置文件,校验没问题后就加载配置文件:

# /usr/local/prometheus/promtool check config /usr/local/prometheus/prometheus.yml
# curl -X POST http://172.16.23.120:9090/-/reload

5.刷新控制台,查看targets节点:

 

 接下来通过向json文件host_discovery.json增加节点,然后不加载prometheus,刷新控制台看效果:

[root@master node]# cat host_discovery.json 
[{
    "targets": ["172.16.23.121:9100","172.16.23.122:9100"],
    "labels": {
        "instance": "172.16.23.121",
        "role": "node1"
    }
}]

刷新prometheus控制台:

 

 可以看见新增了122节点,但是label属性都是相同,这种明显不是我们希望看见的,所以继续修改:

[root@master node]# cat host_discovery.json 
[{
    "targets": ["172.16.23.121:9100"],
    "labels": {
        "instance": "172.16.23.121",
        "role": "node1"
    }
},
{
    "targets": ["172.16.23.122:9100"],
    "labels": {
        "instance": "172.16.23.122",
        "role": "node2"
    }
}]

上面配置就是一个targets对应自己私有的label,然后刷新控制台:

 

 当然基于配置文件发现的方式除了json文件,yaml文件也是可以,操作如下:

[root@master node]# cat host_discovery.yml 
- targets:
  - "172.16.23.121:9100"
  labels:
    instance: "172.16.23.121"

然后重载prometheus server:

# curl -X POST http://172.16.23.120:9090/-/reload

刷新控制台:

 

 继续向yml配置文件添加节点:

[root@master node]# cat host_discovery.yml 
- targets:
  - "172.16.23.121:9100"
  labels:
    instance: "172.16.23.121"
- targets:
  - "172.16.23.122:9100"
  labels:
    instance: "172.16.23.122"

然后刷新控制台: