监控之Prometheus+grafana+node_exporter配置

一、安装环境说明

环境阿里云ECS centos7.6 x86_64 最小化安装

涉及到的服务端口提前在安全组内放开允许内网访问

三台服务器服务部署情况:

tidb04  192.10.0.59 安装 prometheus,grafana,node_exporter,mysqld_exporter
tidb05  192.10.0.246 安装 node_exporter,mysqld_exporter
tidb06  192.10.0.247  安装 node_exporter 
特别提示：三台服务器主机名都绑定本地hosts文件
1.
2.
3.
4.

二、prometheus安装

2.1 安装包下载

mkdir -p /data/soft/prometheus/data
cd /data/soft/prometheus/
wget https://github-releases.githubusercontent.com/6838921/4063cb00-da9b-11eb-8735-e43628dda2bc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210712%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210712T041623Z&X-Amz-Expires=300&X-Amz-Signature=47d660187c9a5410afb6acb2ae3da60ce26fcc42a63cb36a94f02d388ccf2266&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=6838921&response-content-disposition=attachment%3B%20filename%3Dprometheus-2.28.1.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
tar -xvf prometheus-2.28.1.linux-amd64.tar.gz
mv prometheus-2.28.1.linux-amd64 prometheus
cd prometheus; mkdir data;
1.
2.
3.
4.
5.
6.

2.2 启动脚本

cat </etc/systemd/system/prometheus.service
[Unit]
Description="prometheus"
Documentation=https://prometheus.io/
After=network.target
 
[Service]
Type=simple
ExecStart=/data/soft/prometheus/prometheus  --config.file=/data/soft/prometheus/prometheus.yml --storage.tsdb.path=/data/soft/prometheus/data --web.enable-lifecycle --enable-feature=remote-write-receiver --query.lookback-delta=2m
 
Restart=on-failure
RestartSecs=5s
SuccessExitStatus=0
LimitNOFILE=65536
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=prometheus
 
[Install]
WantedBy=multi-user.target
EOF
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.

2.3 服务器启动命令

服务加载
systemctl daemon-reload
systemctl enable prometheus
启动服务
systemctl restart prometheus
服务状态查看
systemctl status prometheus
1.
2.
3.
4.
5.
6.
7.

2.4 测试服务是否启动

浏览器访问 http://192.10.0.59:9090/targets

截图如下:代表启动成功

三、node_exporter安装

node_exporter 是prometheus的系统资源监控组件

3.1 node_exporter服务启动脚本

cat <<EOF >/etc/systemd/system/node_exporter.service

[Unit]
Description="node_exporter"
Documentation=https://github.com/prometheus/node_exporter
After=network.target

[Service]
WorkingDirectory=/opt/server/exporter
Type=simple
ExecStart=/opt/server/exporter/node_exporter/node_exporter --web.listen-address=:19100
Restart=on-failure
RestartSec=10s
SuccessExitStatus=0
LimitNOFILE=300000

[Install]
WantedBy=multi-user.target
EOF
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

3.2 服务器启动命令

服务加载
systemctl daemon-reload
systemctl enable node_exporter
启动服务
systemctl restart node_exporter
服务状态查看
systemctl status node_exporter
1.
2.
3.
4.
5.
6.
7.

四、node_exporter 接入Prometheus

在scrape_configs块下添加如下内容：(注意参数格式缩进,否则导致服务启动失败)

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['192.10.0.59:9090']
  - job_name: 'node_exporter'
    static_configs:
    - targets: ['192.10.0.59:19100','192.10.0.246:19100','192.10.0.247:19100']
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.

完整配置文件如下:

[root@tidb04 ~]# cat /data/soft/prometheus/prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['192.10.0.59:9090']
  - job_name: 'node_exporter'
    static_configs:
    - targets: ['192.10.0.59:19100','192.10.0.246:19100','192.10.0.247:19100']
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.

重启服务：

systemctl restart prometheus

验证node_exporter是否接入到prometheus:

​ ​http://192.10.0.59:19100/metrics​​

添加的node_exporter的服务器都可以查到

五、Grafana安装

官网地址:

​ ​https://grafana.com/grafana/download?edition=oss ​​

Red Hat, CentOS, RHEL, and Fedora(64 Bit)系统采用下面的2种方式安装:

rpm包安装：
wget https://dl.grafana.com/oss/release/grafana-8.3.3-1.x86_64.rpm
sudo yum install grafana-8.3.3-1.x86_64.rpm

下载二进制包安装:
wget https://dl.grafana.com/oss/release/grafana-8.3.3.linux-amd64.tar.gz
tar -zxvf grafana-8.3.3.linux-amd64.tar.gz

本次演示安装采用的是rpm包安装
1.
2.
3.
4.
5.
6.
7.
8.
9.

配置文件:可以指定管理账户名称和pass,可以接入ldap账户

[root@tidb04 soft]# egrep "ldap|admin_user|admin_password" /etc/grafana/grafana.ini  
;admin_user = admin

;admin_password = admin

[auth.ldap]

;config_file = /etc/grafana/ldap.toml
1.
2.
3.
4.
5.
6.
7.
8.

启动命令：

systemctl status grafana-server
 systemctl stop grafana-server
/etc/init.d/grafana-server status/stop/start/restart
1.
2.
3.

grafana接入prometheus数据源:

登录http://192.10.0.59:3000/login

输入URL，点击save&test即可

六、导入node_exporter dashboard模版

登录 grafana官网查找node_exporter  dashboard的模板 导入到自建的grafana系统  

dashboard的模板查找地址:

 https://grafana.com/grafana/dashboards

具体导入node_exporter dashboard模版配置过程可以参考：https://www.cnblogs.com/Pycainiao/p/14568910.html

点击选中的中文版node_exporter dashboard 进入到详情中可以看到这个模版到ID为8919

登录grafana控制台 选择左侧+号 选择下拉框Import:

过几秒中，发现dashboard出现图，代表添加成功:

七、删除已经存在的dashboard模板

点击左上角General/Home  进入搜索dashboard页面:

​