prometheus过滤采集node_exporter数据
参考链接:
https://github.com/prometheus/node_exporter?tab=readme-ov-file
1.修改Prometheus server的配置文件
[root@prometheus-server31 ~]
...
30 - job_name: "yanshier-node-exporter"
31 metrics_path: "/metrics"
32 scheme: "http"
33
34 params:
35 collect[]:
36 - meminfo
37 static_configs:
38 - targets: ["10.0.0.41:9100","10.0.0.42:9100","10.0.0.43:9100"]
2.重新加载配置配置
[root@prometheus-server31 ~]
[root@prometheus-server31 ~]
3.验证测试
发现无法查询CPU相关的指标，但是可以查询内存的指标。
node_memory_MemTotal_bytes
node_cpu_seconds_total
4.底层原理说明
node_exporter是采集到数据了，这一点可以在相应节点测试。
http://10.0.0.43:9100/metrics
搜索： node_cpu_seconds_total
之所以Prometheus采集不到CPU指标，本质是是修改了访问的URL参数。
http://10.0.0.43:9100/metrics?collect[]=meminfo
编码后:
http://10.0.0.43:9100/metrics?collect%5B%5D=meminfo
搜索： node_cpu_seconds_total
prometheus监控zookeeper集群
1.修改zookeeper集群开启metric指标暴露
[root@elk91 ~]
...
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpHost=0.0.0.0
metricsProvider.httpPort=7000
metricsProvider.exportJvmInfo=true
[root@elk91 ~]
[root@elk91 ~]
[root@elk91 ~]
[root@elk91 ~]
2.启动zookeeper集群
[root@elk91 ~]
[root@elk92 ~]
[root@elk93 ~]
3.检查zookeeper集群是否正常
[root@elk91 ~]
ZooKeeper JMX enabled by default
Using config: /yanshier/softwares/apache-zookeeper-3.8.4-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
[root@elk91 ~]
[root@elk92 ~]
ZooKeeper JMX enabled by default
Using config: /yanshier/softwares/apache-zookeeper-3.8.4-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
[root@elk92 ~]
[root@elk93 ~]
ZooKeeper JMX enabled by default
Using config: /yanshier/softwares/apache-zookeeper-3.8.4-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader
[root@elk93 ~]
4.访问prometheus的metric接口
http://10.0.0.91:7000/metrics
http://10.0.0.92:7000/metrics
http://10.0.0.93:7000/metrics
5.修改Prometheus的配置文件
[root@prometheus-server31 ~]
...
- job_name: "yanshier-zookeeper"
static_configs:
- targets:
- 10.0.0.91:7000
- 10.0.0.92:7000
- 10.0.0.93:7000
6.热加载配置
[root@prometheus-server31 ~]
[root@prometheus-server31 ~]
7.验证是否监控成功
http://10.0.0.31:9090/targets
8.导入grafana的ID
10465
prometheus监控kafka集群
1.启动kafka集群
[root@elk91 ~]
[root@elk92 ~]
[root@elk93 ~]
2.启动生产者
[root@elk91 ~]
>111111111111111
[2024-11-12 02:56:55,457] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 6 : {yanshier-linux94=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
2>22222222222222
>333333333333333333333
>
3.启动消费者
[root@elk93 ~]
111111111111111
222222222222222
333333333333333333333
4.下载kafka_exporter
[root@elk91 ~]
5.解压软件包
[root@elk91 ~]
6.启动kafka_exporter
[root@elk91 ~]
7.访问kafka_exporter的metrics接口
http://10.0.0.91:9308/metrics
8.Prometheus监控kafka_exporter
[root@prometheus-server31 ~]
...
- job_name: "yanshier-kafka"
static_configs:
- targets:
- 10.0.0.91:9308
9.热加载配置
[root@prometheus-server31 ~]
[root@prometheus-server31 ~]
10.检查配置是否生效
http://10.0.0.31:9090/targets
11.grafna导入模板ID
12460
10122
Prometheus监控elasticSearch集群
1.下载elastcsearch_exporter
wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.8.0/elasticsearch_exporter-1.8.0.linux-amd64.tar.gz
2.解压软件包
[root@elk91 ~]
3.启动elasticsearch_exporter
[root@elk91 ~]
level=info ts=2024-11-12T03:56:10.458466996Z caller=clusterinfo.go:214 msg="triggering initial cluster info call"
level=info ts=2024-11-12T03:56:10.458879438Z caller=clusterinfo.go:183 msg="providing consumers with updated cluster info label"
level=info ts=2024-11-12T03:56:10.520366346Z caller=main.go:244 msg="started cluster info retriever" interval=5m0s
level=info ts=2024-11-12T03:56:10.520832298Z caller=tls_config.go:313 msg="Listening on" address=[::]:9114
level=info ts=2024-11-12T03:56:10.520844867Z caller=tls_config.go:316 msg="TLS is disabled." http2=false address=[::]:9114
...
4.访问elasticsearch_exporter的metrics
http://10.0.0.91:9114/metrics
5.Prometheus修改配置文件
[root@prometheus-server31 ~]
...
- job_name: "yanshier-elasticsearch"
static_configs:
- targets:
- 10.0.0.91:9114
6.热加载配置
[root@prometheus-server31 ~]
[root@prometheus-server31 ~]
7.检查配置是否生效
http://10.0.0.31:9090/targets
8.grafna导入模板ID
14191
Prometheus监控mysql
1.下载mysql_exporter
wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.16.0/mysqld_exporter-0.16.0.linux-amd64.tar.gz
2.解压软件包
[root@elk92 ~]
3.启动MySQL数据库
[root@elk93 ~]
[root@elk93 ~]
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.36 MySQL Community Server - GPL
Copyright (c) 2000, 2024, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
mysql> SELECT user,host,plugin FROM mysql.user;
+------------------+-----------+-----------------------+
| user | host | plugin |
+------------------+-----------+-----------------------+
| linux94 | % | caching_sha2_password |
| root | % | caching_sha2_password |
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session | localhost | caching_sha2_password |
| mysql.sys | localhost | caching_sha2_password |
| root | localhost | caching_sha2_password |
+------------------+-----------+-----------------------+
6 rows in set (0.00 sec)
mysql>
mysql> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO linux94;
Query OK, 0 rows affected (0.00 sec)
mysql>
mysql> SHOW GRANTS FOR linux94;
+-------------------------------------------------------------------+
| Grants for linux94@% |
+-------------------------------------------------------------------+
| GRANT SELECT, PROCESS, REPLICATION CLIENT ON *.* TO `linux94`@`%` |
+-------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql>
mysql>
4.启动mysql_exporter
[root@elk92 ~]
[client]
user=linux94
password=yanshier
[root@elk92 ~]
[root@elk92 ~]
5.访问metrics测试
http://10.0.0.92:9104/metrics
6.修改Prometheus的配置文件
[root@prometheus-server31 ~]
...
- job_name: "yanshier-mysqld"
static_configs:
- targets:
- 10.0.0.92:9104
7.热加载配置
[root@prometheus-server31 ~]
8.检查配置是否生效
http://10.0.0.31:9090/targets
9.grafana导入模板ID
17320
18949
Prometheus监控Redis
1.部署redis环境
[root@elk93 ~]
[root@elk93 ~]
[root@elk93 ~]
[root@elk93 ~]
[root@elk93 ~]
[root@elk93 ~]
LISTEN 0 511 0.0.0.0:6379 0.0.0.0:*
[root@elk93 ~]
[root@elk93 ~]
127.0.0.1:6379[5]> KEYS *
127.0.0.1:6379[5]>
127.0.0.1:6379[5]> set school yanshier
OK
127.0.0.1:6379[5]>
127.0.0.1:6379[5]> LPUSH hobby K8S docker prometheus
3
127.0.0.1:6379[5]>
127.0.0.1:6379[5]> KEYS *
hobby
school
127.0.0.1:6379[5]>
127.0.0.1:6379[5]> TYPE hobby
list
127.0.0.1:6379[5]>
127.0.0.1:6379[5]> TYPE school
string
127.0.0.1:6379[5]>
2.下载redis_exporter
wget https://github.com/oliver006/redis_exporter/releases/download/v1.66.0/redis_exporter-v1.66.0.linux-amd64.tar.gz
3.解压软件包
[root@elk92 ~]
4.启动redis_exporter
[root@elk92 ~]
5.访问metrics测试
http://10.0.0.92:9121/metrics
6.修改Prometheus的配置文件
[root@prometheus-server31 ~]
...
- job_name: "yanshier-redis"
static_configs:
- targets:
- 10.0.0.92:9121
7.热加载配置
[root@prometheus-server31 ~]
8.检查配置是否生效
http://10.0.0.31:9090/targets
9.grafana导入模板ID
763
14091
14615
grafana实现插件安装
1.安装插件
[root@grafana51 ~]
✔ Downloaded and extracted natel-discrete-panel v0.1.1 zip successfully to /var/lib/grafana/plugins/natel-discrete-panel
Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.
[root@grafana51 ~]
[root@grafana51 ~]
total 12
drwxr-xr-x 3 root root 4096 Nov 12 15:29 ./
drwxr-xr-x 6 grafana grafana 4096 Nov 12 15:29 ../
drwxr-xr-x 4 root root 4096 Nov 12 15:29 natel-discrete-panel/
[root@grafana51 ~]
2.重启服务使得配置生效
[root@grafana51 ~]
[root@grafana51 ~]
[root@grafana51 ~]
3.访问WebUI测试
Prometheus监控nginx
1 编译安装nginx
1.1 安装编译工具
[root@elk92 ~]
1.2 克隆nginx-module-vts模块
git clone git://github.com/vozlt/nginx-module-vts.git
如果上述连接不好用，可以执行下面代码:
git clone https://gitee.com/jasonyin2020/nginx-module-vts.git
1.3 下载nginx软件包
wget https://nginx.org/download/nginx-1.27.2.tar.gz
1.4 解压nginx
tar xf nginx-1.27.2.tar.gz
1.5 配置nginx
[root@elk92 ~]
[root@elk92 nginx-1.27.2]
[root@elk92 nginx-1.27.2]
1.6 编译并安装nginx
make -j 2 && make install
1.7 修改nginx的配置文件
vim /yanshier/softwares/nginx/conf/nginx.conf
...
http {
vhost_traffic_status_zone;
upstream yanshier-grafana {
server 10.0.0.51:3000;
}
...
server {
...
location / {
root html;
proxy_pass http://yanshier-grafana;
}
location /status {
vhost_traffic_status_display;
vhost_traffic_status_display_format html;
}
}
}
1.8 检查配置文件语法
[root@elk92 ~]
nginx: the configuration file /yanshier/softwares/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /yanshier/softwares/nginx/conf/nginx.conf test is successful
[root@elk92 ~]
1.9 启动nginx
[root@elk92 ~]
[root@elk92 ~]
[root@elk92 ~]
LISTEN 0 511 0.0.0.0:80 0.0.0.0:*
[root@elk92 ~]
1.10 访问nginx的状态页面
http://10.0.0.92/status
http://10.0.0.92/status/format/prometheus
2 安装nginx-vtx-exporter
2.1 下载nginx-vtx-exporter
CentOS：(不支持更高版本)
wget https://github.com/hnlq715/nginx-vts-exporter/releases/download/v0.10.3/nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
温馨提示:
不建议下载更高版本，因为其会提升对GLIC的版本要求，可以通过" strings /lib64/libc.so.6 | grep GLIBC_ "查看默认是2.17版本，若使用较高版本则需要2.32+
Ubuntu可以采用：
[root@elk93 ~]
2.2 解压软件包到path路径
[root@elk93 ~]
[root@elk93 ~]
2.3 运行nginx-vtx-exporter
[root@elk93 ~]
...
2024/11/12 16:34:50 Starting nginx_vts_exporter (version=, branch=, revision=)
2024/11/12 16:34:50 Build context (go=go1.20.5, user=, date=)
2024/11/12 16:34:50 Starting Server at : :9913
2024/11/12 16:34:50 Metrics endpoint: /metrics
2024/11/12 16:34:50 Metrics namespace: nginx
2024/11/12 16:34:50 Scraping information from : http://10.0.0.92/status/format/json
3 配置prometheus采集nginx数据
3.1 修改配置文件
[root@prometheus-server31 ~]
...
scrape_configs:
...
- job_name: "yanshier-nginx"
static_configs:
- targets:
- 10.0.0.93:9913
3.2 重新加载配置
[root@prometheus-server31 ~]
[root@prometheus-server31 ~]
3.3 检查配置是否生效
http://10.0.0.31:9090/targets
3.4 导入grafana模板
2949
Prometheus监控tomcat
1.下载
[root@elk93 ~]
2.解压软件包
[root@elk93 ~]
[root@elk93 ~]
[root@elk93 ~]
[root@elk93 ~]
3.编译镜像
[root@elk93 ~]
[root@elk93 tomcat]
[root@elk93 tomcat]
[root@elk93 tomcat]
[root@elk93 tomcat]
[root@elk93 tomcat]
[root@elk93 tomcat]
[root@elk93 tomcat]
[root@elk93 tomcat]
4.启动tomcat实例
[root@elk93 tomcat]
dc07d5e0222150bab62e340f8ca495541413e902e6e0a1d947e69e529c5ebc19
[root@elk93 tomcat]
[root@elk93 tomcat]
LISTEN 0 100 *:8080 *:*
[root@elk93 tomcat]
5.访问测试metrics
http://10.0.0.93:8080/myapp/
http://10.0.0.93:8080/metrics/
6.修改Prometheus的配置文件
[root@prometheus-server31 ~]
...
- job_name: "yanshier-tomcat"
static_configs:
- targets:
- 10.0.0.93:8080
7.热加载配置
[root@prometheus-server31 ~]
8.检查配置是否生效
http://10.0.0.31:9090/targets
9.grafana导入模板ID
自行导入模板文件。
参考链接：
https://github.com/nlighten/tomcat_exporter/blob/master/dashboard/example.json
Prometheus监控docker容器
1 cAdvisor概述
cAdvisor（Container Advisor）让容器用户了解其运行容器的资源使用情况和性能特征。
它是一个正在运行的守护进程，用于收集、聚合、处理和导出有关正在运行的容器的信息。
具体来说，对于每个容器，它保留资源隔离参数、历史资源使用情况、完整历史资源使用的直方图和网络统计信息。
官网地址:
https://github.com/google/cadvisor
2 部署cAdvisor容器
docker run --volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=18080:8080 \
--detach=true \
--name=yanshier_cadvisor \
registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/cadvisor:v0.47.2
3.访问测试
http://10.0.0.93:18080/metrics
4.修改Prometheus的配置文件
[root@prometheus-server31 ~]
...
scrape_configs:
...
- job_name: "yanshier-docker"
static_configs:
- targets:
- 10.0.0.93:18080
5.重载配置
[root@prometheus-server31 ~]
[root@prometheus-server31 ~]
6 验证配置是否生效
http://10.0.0.31:9090/targets
7.导入grafana模板
315
10619
- Prometheus监控服务流程
- 1.验证被监控端服务是否启动; 【mysql】
- 2.检测服务对应的exporter是否正常启动; (/metrics)【mysql_exporter】
- 3.prometheus server配置监控exporter
- 4.热加载配置并验证是否生效;
- 5.grafana导入模板ID
MongoDB配置Prometheus监控
下载镜像
[root@elk93:~]
部署容器
[root@elk93:~]
上传mongodb_exporter-0.42.0.linux-amd64.tar.gz--"F:\Prometheus监控\windows_exporter-0.29.2-amd64.exe"
tar xf mongodb_exporter-0.42.0.linux-amd64.tar.gz -C /usr/local/bin
cd /usr/local/bin
移动目录文件到当前
最后运行
root@elk93:~
修改Prometheus配置文件
vim /yanshier/softwares/prometheus-2.53.3.linux-amd64/prometheus.yml
- job_name: "yanshier-mongdb"
static_configs:
- targets:
- 10.0.0.93:9216
浏览器访问测试
-------------------------------------
Prometheus监控windows系统
直接点击文件运行cmd查看端口
"F:\Prometheus监控\windows_exporter-0.29.2-amd64.exe"
修改Prometheus配置文件
vim /yanshier/softwares/prometheus-2.53.3.linux-amd64/prometheus.yml
- job_name: "yanshier-wendins"
static_configs:
- targets:
- 10.0.0.1:9182
- prometheus监控windows server
1.下载windows的exporter
https://github.com/prometheus-community/windows_exporter/releases/download/v0.28.2/windows_exporter-0.28.2-amd64.exe
2.启动服务
windows_exporter-0.28.2-amd64.exe --web.listen-address=:9182 --telemetry.path="/metrics"
3.访问metrics测试
http://10.0.0.1:9182/metrics
4.修改Prometheus的配置文件
[root@prometheus-server31 ~]
...
- job_name: "oldboyedu-windows"
static_configs:
- targets:
- 10.0.0.1:9182
5.热加载配置文件
[root@prometheus-server31 ~]
6.验证服务是否生效
http://10.0.0.31:9090/targets
7.导入grafana的模板ID
20763
14694
- prometheus监控mongoDB
1.部署MongoDB
[root@node-exporter43 ~]
2dc7c724eb169a3046bf18ebe79d9075b44d8d46db3e30bf032b3acb5c5a827a
[root@node-exporter43 ~]
[root@node-exporter43 ~]
LISTEN 0 4096 0.0.0.0:27017 0.0.0.0:*
[root@node-exporter43 ~]
2.部署mongoDB的exporter
wget https://github.com/percona/mongodb_exporter/releases/download/v0.42.0/mongodb_exporter-0.42.0.linux-amd64.tar.gz
3.解压软件包
[root@node-exporter42 ~]
4.启动MongoDB_exporter
[root@node-exporter42 ~]
5.验证mongoDB的metrics
http://10.0.0.42:9216/metrics
6.prometheus配置MongoDB的监控
[root@prometheus-server31 ~]
...
- job_name: "oldboyedu-mongoDB"
static_configs:
- targets:
- 10.0.0.42:9216
7.热加载配置文件
[root@prometheus-server31 ~]
8.验证服务是否生效
http://10.0.0.31:9090/targets
9.导入grafana的模板ID
12079
16490
10.后续优化点【ToDO添加mangoDB的授权配置】
[root@node-exporter42 ~]
ERRO[0000] Cannot connect to MongoDB: cannot connect to MongoDB: connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism "SCRAM-SHA-1": (AuthenticationFailed) Authentication failed.
...
参考文档:
https://www.mongodb.com/zh-cn/docs/v5.0/faq/sharding/
db.createUser({user:"jasonyin",pwd:"yinzhengjie",roles:[{role:"clusterMonitor",db:"admin"},{role:"read",db:"local"}]})
db.grantRolesToUser("jasonyin",[{role:"clusterMonitor",db:"admin"}])
【推荐】国内首个AI IDE，深度理解中文开发场景，立即下载体验Trae
【推荐】编程新体验，更懂你的AI，立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包，你的智能百科全书，全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell：AI 加持，快人一步
· TypeScript + Deepseek 打造卜卦网站：技术与玄学的结合
· Manus的开源复刻OpenManus初探
· 三行代码完成国际化适配，妙~啊~
· .NET Core 中如何实现缓存的预热？
· 如何调用 DeepSeek 的自然语言处理 API 接口并集成到在线客服系统