中小型系统必要可行的运维监控实践--搭建promtetheus+grafana性能监控环境

环境准备

CentOS Linux release 7.7.1908 (Core)
prometheus-2.36.2
grafana-enterprise-9.0.2-1
node_exporter-1.3.1
mysqld_exporter-0.14.0

备注：软件可以去清华镜像站下载：https://mirrors.tuna.tsinghua.edu.cn/github-release/prometheus/prometheus/
或者：https://download.csdn.net/download/cac2020/87904471
jmx_prometheus_javaagent：https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent

监控服务器

主机列表	功能	安装软件列表
192.168.6.101	监控服务器	prometheus-2.36.2.linux-amd64.tar.gz、grafana-enterprise-9.0.2-1.x86_64.rpm
192.168.6.102	应用服务器	node_exporter-1.3.1.linux-amd64.tar.gz 、jmx_prometheus_javaagent-0.14.0.jar
192.168.6.103	mysql服务器	node_exporter-1.3.1.linux-amd64.tar.gz、mysqld_exporter-0.14.0.linux-amd64.tar.gz

搭建步骤

1. 192.168.6.102、192.168.6.103安装node_exporter

以192.168.6.102为例:

点击查看安装代码

[root@node102 install]# tar -zxvf node_exporter-1.4.0.linux-amd64.tar.gz -C /usr/local && cd /usr/local/node_exporter-1.4.0.linux-amd64
[root@node102 node_exporter-1.4.0.linux-amd64]# nohup ./node_exporter --web.listen-address 192.168.6.102:39100 &
[1] 5927
[root@node102 node_exporter-1.4.0.linux-amd64]# nohup: 忽略输入并把输出追加到"nohup.out"
[root@node102 node_exporter-1.4.0.linux-amd64]# ps -ef|grep node_exporter
root      5927     1  0 08:53 ?        00:00:26 /usr/local/node_exporter-1.4.0.linux-amd64/node_exporter --web.listen-address 192.168.6.102:39100
root      6101  6081  0 10:25 pts/1    00:00:00 grep --color=auto node_exporter

验证：浏览器内输入 http://192.168.6.102:39100/ 出现下面即为启动成功： Node Exporter Metrics

2. 192.168.6.103安装mysqld_exporter

点击查看安装代码

#mysql数据库创建一个监控账号
GRANT ALL PRIVILEGES ON *.* TO 'monitor'@'%' IDENTIFIED BY '123456';

#解压安装
[root@node103 install]# tar -zxvf mysqld_exporter-0.14.0.linux-amd64.tar.gz -C /usr/local && cd /usr/local/mysqld_exporter-0.14.0.linux-amd64
#创建监控配置文件
[root@node103 mysqld_exporter-0.14.0.linux-amd64]# vi my.cnf
[client]

host=192.168.6.103
port=3306
user=monitor
password=123456
#启动
[root@node103 mysqld_exporter-0.14.0.linux-amd64]# nohup ./mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter-0.14.0.linux-amd64/my.cnf &
[1] 5935
[root@node103 mysqld_exporter-0.14.0.linux-amd64]# nohup: 忽略输入并把输出追加到"nohup.out"
[root@node103 BM]# ps -ef |grep mysqld_exporter
root      5935     1  0 08:53 ?        00:00:36 /usr/local/mysqld_exporter-0.14.0.linux-amd64/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter-0.14.0.linux-amd64/my.cnf
root      6142  6081  0 10:41 pts/1    00:00:00 grep --color=auto mysqld_exporter

验证：浏览器内输入 http://192.168.6.103:9104/ 出现下面即为启动成功： MySQLd exporter Metrics

3. 192.168.6.102安装JMX-Exporter

配置jmx_exporter_config.yaml：

点击查看代码

[root@node102 ~]# cd /usr/local/jmx_exporter && ls
jmx_prometheus_javaagent-0.14.0.jar
[root@node102 jmx_prometheus]# vi jmx_exporter_config.yaml
---
lowercaseOutputLabelNames: true
lowercaseOutputName: true
whitelistObjectNames: ["java.lang:type=OperatingSystem"]
blacklistObjectNames: []
rules:
  - pattern: 'java.lang<type=OperatingSystem><>(committed_virtual_memory|free_physical_memory|free_swap_space|total_physical_memory|total_swap_space)_size:'
    name: os_$1_bytes
    type: GAUGE
    attrNameSnakeCase: true
  - pattern: 'java.lang<type=OperatingSystem><>((?!process_cpu_time)\w+):'
    name: os_$1
    type: GAUGE
    attrNameSnakeCase: true

应用启动配置：

点击查看代码

nohup java -javaagent:/usr/local/jmx_exporter/jmx_prometheus_javaagent-0.14.0.jar=8586:/usr/local/jmx_exporter/jmx_exporter_config.yaml -jar -Xms4g -Xmx4g test.jar > catalina.out  2>&1 &

4. 192.168.6.101安装prometheus

点击查看代码

[root@node101 ~]# tar -zxvf prometheus-2.36.2.linux-amd64.tar.gz -C /usr/local && mv /usr/local/prometheus-2.36.2.linux-amd64 /usr/local/prometheus && cd /usr/local/prometheus
[root@node101 prometheus]# vi prometheus.yml
# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "102node-exporter"
    static_configs:
      - targets: ['192.168.6.102:39100']
  - job_name: "java"
    static_configs:
      - targets: ['192.168.6.102:8586']
  - job_name: "103node-exporter"
    static_configs:
      - targets: ['192.168.6.102:39100']
        labels:
           instance: '192.168.6.103_3306'
  - job_name: "103mysql-exporter"
    static_configs:
      - targets: ['192.168.6.103:9104']
        labels:
           instance: '192.168.6.103_3306'
#启动 加参数--storage.tsdb.retention=30d 表示保留监控数据30天
[root@node101 prometheus]# nohup /usr/local/prometheus/prometheus &

#验证 http://192.168.6.101:9090/targets?search=

会出现102、103上的node-exporter；103上的mysql-exporter；102上的java
注意：mysql监控对应的instance要相同、jvm监控的jobname要写成java

5. 192.168.6.101安装grafana

点击查看代码

[root@101 install]# yum -y install grafana-enterprise-9.0.2-1.x86_64.rpm
[root@101 install]# systemctl daemon-reload
[root@101 install]# systemctl start grafana-server
[root@101 install]# systemctl status grafana-server
[root@101 install]# systemctl enable grafana-server.service

打开web管理端 http://192.168.6.101:3000/ 初始账户密码 admin / admin

6. prometheus添加数据源

7. prometheus导入dashboards仪表盘

服务器资源监控：常规版：1860、整体版：8919
Mysql监控：导入7362
JMeter监控：导入5496
jVM监控：导入8563

参考

监控服务器资源
centos安装prometheus+grafana
grafana提供的各种dashboards仪表盘
监控mysql资源
Prometheus+Mysqld_exporter+Grafana从0到1搭建MySQL的可视化监控
 grafana+Prometheus+mysqld_exporter
mysql监控指标：buffer pool size of total ram no data问题
监控JVM
prometheus+grafana监控jvm
监控Jmeter
Garafana+influxDB监控Jmeter数据(无界面压测中如何实时的监控)
MDC实现全局链路跟踪
利用Spring boot+LogBack+MDC实现链路追踪

posted @ 2023-06-14 11:20 cac2020 阅读(106) 评论(0) 编辑收藏举报

刷新页面返回顶部

cac2020

中小型系统必要可行的运维监控实践--搭建promtetheus+grafana性能监控环境

环境准备

监控服务器

搭建步骤

1. 192.168.6.102、192.168.6.103安装node_exporter

2. 192.168.6.103安装mysqld_exporter

3. 192.168.6.102安装JMX-Exporter

4. 192.168.6.101安装prometheus

5. 192.168.6.101安装grafana

6. prometheus添加数据源

7. prometheus导入dashboards仪表盘

参考

公告