CentOS 操作系统下 Prometheus + node_exporter + Grafana 安装部署

安装 prometheus

直接下载了LTS的版本
网速快的可以直接用wget
操作系统CentOS9

cd /tmp/
dnf install wget -y
# 连不上github的话可以使用镜像之类的下下来再继续
wget https://github.com/prometheus/prometheus/releases/download/v2.37.5/prometheus-2.37.5.linux-amd64.tar.gz
umask 022
mkdir /usr/local/monitor
cd /usr/local/monitor
# 如果不是直接在系统内下载的，可以下载后再上传
# dnf install -y lrzsz
# rz -E # 选择下载的文件
dnf install -y tar
tar zxvf prometheus-2.37.5.linux-amd64.tar.gz
ln -s prometheus-2.37.5.linux-amd64 prometheus
cd prometheus
ll
mkdir -p /data/prometheus_data
echo "
[Unit]
Description=prometheus
[Service]
ExecStart=/usr/local/monitor/prometheus/prometheus --config.file=/usr/local/monitor/prometheus/prometheus.yml --storage.tsdb.path=/data/prometheus_data
ExecReload=/bin/kill -HUP $MAINPID
User=root
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
" > /usr/lib/systemd/system/prometheus.service
# systemctl daemon-reload # 如果中间修改过 prometheus.service 需要执行
systemctl start prometheus
systemctl status prometheus

上面systemctl中的参数可以添加更多，具体可以使用 ./prometheus --help 查看（参考https://www.cnblogs.com/zhoujinyi/p/11934062.html）

systemctl status prometheus 执行结果

● prometheus.service - prometheus
     Loaded: loaded (/usr/lib/systemd/system/prometheus.service; disabled; vendor preset: disabled)
     Active: active (running) since Tue 2023-01-10 18:03:44 CST; 4s ago
   Main PID: 3156 (prometheus)
      Tasks: 6 (limit: 5909)
     Memory: 19.5M
        CPU: 50ms
     CGroup: /system.slice/prometheus.service
             └─3156 /usr/local/monitor/prometheus/prometheus --config.file=/usr/local/monitor/prometheus/prometheus.yml --storage.tsdb.path=/data/prometheus_data

打开web页面 http://hostip:9090

此时选择Graph在搜索框中输入"cpu"或者"mem"等关键词，选择一个指标，然后点击Execute即可看到曲线图。

安装node_exporter

umask 022
dnf install -y wget
mkdir -p /usr/local/monitor
cd /usr/local/monitor
wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz
tar zxf node_exporter-1.5.0.linux-amd64.tar.gz
ln -s node_exporter-1.5.0.linux-amd64 node_exporter
echo "
[Unit]
Description=node_exporter
[Service]
ExecStart=/usr/local/monitor/node_exporter/node_exporter
ExecReload=/bin/kill -HUP $MAINPID
User=root
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
" > /usr/lib/systemd/system/node_exporter.service
systemctl start node_exporter
systemctl status node_exporter

通过浏览器打开http://hostip:9100，显示"Node Exporter - Metrics"，点击“Metrics”显示对应的指标等信息。

将 node_exporter 添加到Prometheus

cd /usr/local/monitor/prometheus
cp prometheus.yml prometheus.yml.bak$(date +%Y%m%d%H%M%S)
sed   's/\["localhost:9090"\]/\["localhost:9090","[这里填安装nodeexporter的IP地址]:9100"\]/g' prometheus.yml
systemctl restart prometheus

可以查图，例如process_cpu_seconds_total这个指标是prometheus和node_exporter都有的。

Grafana安装

# 参考 https://grafana.com/grafana/download?pg=get&plcmt=selfmanaged-box1-cta1&edition=oss
wget https://dl.grafana.com/oss/release/grafana-9.3.2-1.x86_64.rpm
yum install -y systemctlgrafana-9.3.2-1.x86_64.rpm
# https://grafana.com/docs/grafana/latest/setup-grafana/installation/rpm/#2-start-the-server
systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server
# 默认web http://ip:3000/
# 默认用户名密码 admin admin

grafana中增加 Prometheus的dashboard

先增加datasource
configuration - datasource - add data source
选择Prometheus
在 URL 中输入http://ip:9090/
其他选项都不懂，点击 Save&test
提示 Data source is working 说明data source 添加成功。
从Grafana官方找Dashboard配置，访问https://grafana.com/grafana/dashboards/
通过Filter by进行筛选，在Collector Types中选择node exporter
点击node-exporter-full，在右侧找到Download JSON (https://grafana.com/api/dashboards/1860/revisions/29/download)
点击Dashboard，然后点击New旁边的下拉箭头，选择import，点击Upload JSON file
在prometheus中选择刚刚配置的data source，然后点击import
既可看到前面添加的主机的信息。

异常

页面提示：

Warning: Error fetching server time: Detected 47.81699991226196 seconds time difference between your browser and the server. Prometheus relies on accurate time and time drift might cause unexpected query results.

原因是服务器时间和终端时间不一致。
服务器时间可以使用timedatectl查看时区和时间。
终端和服务端时间都可以直接找ntp服务器进行同步。
我这里终端时间有问题，直接使用的是腾讯的ntp服务ntp.tencent.com。

需要对服务器时间进行同步的话，可以参考这篇文章https://www.cnblogs.com/lizhaoxian/p/11260041.html。

F5刷新页面后就没有错误提示了。

参考资料：
https://zhuanlan.zhihu.com/p/117719823
https://www.cnblogs.com/minseo/p/13395851.html
https://www.prometheus.wang/
https://www.cnblogs.com/zhoujinyi/p/11934062.html
https://blog.csdn.net/weixin_42230348/article/details/107720416
https://blog.csdn.net/tototuzuoquan/article/details/119718255