Prometheus 与 Grafana

Prometheus Grafana

⭐⭐⭐

一、Prometheus简介

Prometheus 收集其指标并将其存储为时间序列数据，即指标信息与记录它的时间戳一起存储，以及称为标签的可选键值对。

Prometheus 的主要特点是：

• 具有由指标名称和键/值对标识的时间序列数据的多维数据模型
• PromQL，一种灵活的查询语言，可利用此维度
• 不依赖分布式存储;单服务器节点是自治的
• 时序收集通过 HTTP 上的拉取模型进行，通过中间网关支持推送时间序列
• 通过服务发现或静态配置发现目标，多种图形和仪表板支持模式

Download | Prometheus

一文看懂 YAML

二、安装配置

# Prometheus :9090
[root@www_001 system]# cat prometheus-server.service
	[Unit]
	Description=Prometheus
	After=network.target
	[Service]
	Type=simple
	ExecStart=/home/application/prometheus/prometheus --config.file=/home/application/prometheus/prometheus.yml --storage.tsdb.path=/home/application/prometheus/data --web.listen-address=127.0.0.1:9090
	[Install]
	WantedBy=multi-user.target

# 反向代理(Prometheus+Nginx 设置认证)
[root@www_001 nginx]# printf "xy:$(openssl passwd -1 xy)\n" >/etc/nginx/htpasswd
[root@www_001 nginx]# cat ./conf.d/prometheus.conf
	upstream prometheus {
	  server 127.0.0.1:9090;
	}

	server {
	  listen          19090;
	  server_name     localhost;
	  root            /usr/local/nginx/html;
	  location / {
		  auth_basic "xy auth";
		  auth_basic_user_file /etc/nginx/htpasswd;
		  proxy_pass http://prometheus;    # 反向代理到:9090
	  }
	  error_page      500 502 503 504 /50x.html;
	  location = /50x.html {
		  root   html;
	  }
	}

[root@www_001 prometheus]# cat prometheus.yml
	....
	scrape_configs:
	  - job_name: "prometheus"
		static_configs:
		  - targets: ["localhost:9090"]
	  # node_exporter
	  - job_name: "xy_node"
		basic_auth:
		  username: xy
		  password: xy
		static_configs:
		  - targets: ["192.168.56.133:9100"]   # node_exporter

# node_exporter :9100
[root@www_001 system]# cat ./node-exporter.service 
	[Unit]
	Description=Node-exporter
	After=network.target
	[Service]
	Type=simple
	ExecStart=/home/application/node_exporter/node_exporter --web.config.file=/home/application/node_exporter/config.yml
	[Install]
	WantedBy=multi-user.target

[root@www_001 node_exporter]# htpasswd -nBC 12 ''
	New password:
	Re-type new password:
	:$2y$12$ZK0Q5edJUbHcVZqR0Rbvm.HQYSjZ1PtCVIRfgh3kKaR3M/d0lCTf/Wm

[root@www_001 node_exporter]# more config.yml
	basic_auth_users:
	  xy: $2y$12$ZK0Q5edJUbHcVZqR0Rbvm.HQYSjZ1PtCVIRfgh3kKaR3M/d0lCTf/Wm

在线YAML转JSON

三、案例

3.1. Grafana+node_exporter

Metric:

node_load1 node_load5 node_load15 rate(node_cpu_seconds_total{mode="idle"}[2m])

node_memory_MemFree_bytes node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

node_filesystem_free_bytes{fstype="xfs"} / node_filesystem_size_bytes{fstype="xfs"} rate(node_network_receive_bytes_total{device="ens33"}[5m])*8

3.2. AlertManager 告警

Introduction

Prometheus监控神器-Alertmanager篇(1)

# Grafana 实现告警
[root@www_001 alertmanager]# cat /usr/lib/systemd/system/alertmanager.service 
	[Unit]
	Description=Alertmanager
	After=network.target
	[Service]
	Type=simple
	ExecStart=/home/application/alertmanager/alertmanager --config.file=/home/application/alertmanager/alertmanager.yml --cluster.listen-address=127.0.0.1:9094 --web.listen-address=127.0.0.1:9093
	[Install]
	WantedBy=multi-user.target

[root@www_001 alertmanager]# cat ./alertmanager.yml
	global:
	  resolve_timeout: 5m
	  smtp_from: 'xxxx'
	  smtp_smarthost: 'smtp.163.com:465'
	  smtp_auth_username: 'xxxx'
	  smtp_auth_password: 'xxxx'
	  smtp_require_tls: false
	  smtp_hello: '163.com'
	route:
	  group_by: ['alertname']
	  group_wait: 30s
	  group_interval: 5m
	  repeat_interval: 1h
	  receiver: 'email'
	receivers:
	  - name: 'email'
		email_configs:
		  - to: 'xxxx'
			send_resolved: true
	inhibit_rules:
	  - source_match:
		  severity: 'critical'
		target_match:
		  severity: 'warning'
		equal: ['alertname', 'dev', 'instance']

# Alertmanager :9093
# Prometheus + Alertmanager实现告警
# Prometheus Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 127.0.0.1:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "rules/*_rules.yml"
  - "rules/*_alerts.yml"
[root@www_001 prometheus]# more ./rules/xy_alerts.yml 
	groups:
	- name: xy_system_alert
	  rules:
	  - alert: system load15 alert
		expr: node_load15 > 0.09
		for: 1m
	  - alert: system load5 alert
		expr: node_load5 > 0.1
		for: 1m

{
  "rule_files": [
    "rules/*_rules.yml",
    "rules/*_alerts.yml"
  ]
}

四、Prometheus监控主机发现实践

4.1. 发现机制

1. 静态文件

   1. 基于JSON配置文件
   2. 不用重启Prometheus

2. Consul

   1. 实现服务发现、配置共享
   2. Install | Consul | HashiCorp Developer

4.2. 具体配置

1. JSON文件配置实现

   # Prometheus
   [root@www_001 prometheus]# vim prometheus.yml
   	- job_name: xy_discovery
   	  basic_auth:
   		username: xy
   		password: xy
   	  file_sd_config:
   		- files:
   			- /usr/local/prometheus/xy_discovery.json
   		  refresh_interval: 15s
   

   # xy_discovery.json
   [
   	{
   		"targets": [
   			"172.29.105.156:9100"
   		]
   	}
   ]
   

   

2. Consul实现

1. 安装：

   unzip consul_1.15.1_linux_amd64.zip && mkdir -pv /usr/local/consul
   mv consul /usr/local/consul/
   

2. 配置服务(/usr/lib/system/systemd/consul.service)：

   [Unit]
   Description=consul
   After=network.target
   [Service]
   Type=simple
   ExecStart=/usr/local/consul/consul agent -dev -data-dir=/data/consul/
   [Install]
   WantedBy=multi-user.target
   

3. 使用Nginx代理

   # Nginx
   upstream consul{
      server 127.0.0.1:8500;
   }
   server {
      listen 18500;
      root /usr/share/nginx/html;
      # Load configuration files for the default server block.
      include /etc/nginx/default.d/*.conf;
      location / {
   	   auth_basic "xy auth";
   	   auth_basic_user_file /etc/nginx/htpasswd;
   	   proxy_pass http://consul;
      }
   }
   
   # Consul
   curl -X PUT -d '{"id": "xy","name":"xy_01","address": "172.29.105.156","port":9100}' http://127.0.0.1:8500/v1/agent/service/register
   
   # Prometheus配置
   - job_name: 'xy_consul'
     basic_auth:
   	username: xy
   	password: xy
     consul_sd_configs:
   	- server: '127.0.0.1:8500'
   	services: []