Linux 数据采集与监控工具(Prometheus、Grafana)
一、MariaDB/MySQL集群监控(mysqld_exporter)
简介：mysql_exporter是用来收集MySQL或MariaDB数据库相关指标的，需要给mysql_exporter连接数据库的账号、密码和相关权限。
1. mysqld_exporter安装
A. 官网下载：https://github.com/prometheus/mysqld_exporter/releases/；
B. 登录MariaDB数据库后创建用户，同时赋予权限，注意max_user_connections参数用来限制exporter用户最大连接数，避免监控引起的数据库过载；
C. 在mysqld_exporter目录下，创建.my.cnf文件：
D. 创建systemctl启动方式服务：systemctl start xxx；
2. docker mysqld_exporter安装
A. 下载镜像：docker pull quintoandar/mysqld_exporter:latest；
B. docker-stack.yml部署文件
version: '3.8' networks: rhxy-monitor: external: true services: # MariaDB Galera监控 mysqld-expoter_rhxy-db1: image: quintoandar/mysqld_exporter:latest hostname: mysqld-expoter_rhxy-db1 deploy: mode: global placement: constraints: - node.hostname == rhxy-db1 networks: - rhxy-monitor - rhxy-service environment: DATA_SOURCE_NAME: root:123456@(haproxy:3300)/ mysqld-expoter_rhxy-db2: image: quintoandar/mysqld_exporter:latest hostname: mysqld-expoter_rhxy-db2 deploy: mode: global placement: constraints: - node.hostname == rhxy-db2 networks: - rhxy-monitor - rhxy-mp environment: DATA_SOURCE_NAME: root:123456@(haproxy:3300)/ mysqld-expoter_rhxy-db3: image: quintoandar/mysqld_exporter:latest hostname: mysqld-expoter_rhxy-db3 deploy: mode: global placement: constraints: - node.hostname == rhxy-db3 networks: - rhxy-monitor - rhxy-mp environment: DATA_SOURCE_NAME: root:123456@(haproxy:3300)/
3. 在prometheus中配置mysqld_exporter
A. prometheus.yml配置文件
global: scrape_interval: 15s evaluation_interval: 15s rule_files: - "/etc/prometheus/rules/*.yml" scrape_configs: - job_name: mysqld-exporter metrics_path: /metrics static_configs: - targets: [ 'mysqld-expoter_rhxy-db1:9104', 'mysqld-expoter_rhxy-db2:9104', 'mysqld-expoter_rhxy-db3:9104' ]
B. 重新加载prometheus配置：curl -X POST http://localhost:9090/-/reload；
4. 编写告警规则
5. grafana模板
A. MariaDB Galera集群模板：https://grafana.com/grafana/dashboards/13106；
B. MariaDB模板：https://grafana.com/grafana/dashboards/7362；
二、SpringBoot微服务监控
1. 引入pom.xml Maven依赖
<dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency> <dependency> <groupId>io.dropwizard.metrics</groupId> <artifactId>metrics-core</artifactId> </dependency>
2. application.xml配置文件
management: endpoints: web: base-path: /management exposure: include: "*" exclude: shutdown endpoint: health: show-details: always jhi-metrics: enabled: false info: git: mode: full health: mail: enabled: false # When using the MailService, configure an SMTP server and set this to true metrics: export: prometheus: enabled: true step: 60 binders: jvm: enabled: true processor: enabled: true uptime: enabled: true logback: enabled: true files: enabled: true integration: enabled: true distribution: percentiles-histogram: all: true percentiles: all: 0, 0.25, 0.5, 0.75, 0.95, 0.99, 1.0 web: server: auto-time-requests: true
3. prometheus.yml配置文件
global: scrape_interval: 15s evaluation_interval: 15s rule_files: - "/etc/prometheus/rules/*.yml" scrape_configs: - job_name: 'spring-cloud-exporter' metrics_path: /management/prometheus static_configs: - targets: [ 'rhxy-gateway:8899', 'rhxy-manage:8081', 'rhxy-business:8082', 'rhxy-websocket:8083' ]
4. grafana模板
A. SpringBoot1.x模板：https://grafana.com/grafana/dashboards/6756；
B. SpringBoot2.x模板：https://grafana.com/grafana/dashboards/10280；
三、服务器/主机监控(node_exporter)
简介：node_exporter是用于收集机器系统数据，监控服务器CPU、内存、磁盘及I/0信息等。
1. node_exporter安装
A. 官网下载地址：https://github.com/prometheus/node_exporter/releases/；
B. 创建systemctl启动方式服务：systemctl start xxx；
2. docker node_exporter安装
A. 下载镜像：docker pull prom/node-exporter:latest；
B. docker-stack.yml部署文件
version: '3.8'
networks:
rhxy-monitor:
external: true
services: # 集群主机监控 node-exporter_rhxy-gateway: image: prom/node-exporter:latest hostname: node-exporter_rhxy-gateway deploy: mode: global placement: constraints: - node.hostname == rhxy-gateway networks: - rhxy-monitor node-exporter_rhxy-db: image: prom/node-exporter:latest hostname: node-exporter_rhxy-db deploy: mode: global placement: constraints: - node.hostname == rhxy-db networks: - rhxy-monitor node-exporter_rhxy-cache: image: prom/node-exporter:latest hostname: node-exporter_rhxy-cache deploy: mode: global placement: constraints: - node.hostname == rhxy-cache networks: - rhxy-monitor node-exporter_rhxy-app: image: prom/node-exporter:latest hostname: node-exporter_rhxy-app deploy: mode: global placement: constraints: - node.hostname == rhxy-app networks: - rhxy-monitor
3. prometheus.yml配置文件
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "/etc/prometheus/rules/*.yml"
scrape_configs: - job_name: 'node-exporter' metrics_path: /metrics static_configs: - targets: [ 'node-exporter_rhxy-gateway:9100', 'node-exporter_rhxy-db:9100',
'node-exporter_rhxy-cache:9100',
'node-exporter_rhxy-app:9100' ]
4. grafana模板：https://grafana.com/grafana/dashboards/8919。
四、Redis集群监控(redis_exporter)
1. redis_exporter安装
A. 官网下载地址：https://github.com/oliver006/redis_exporter/releases/；
B. 启动服务：./redis_exporter -redis-only-metrics -redis.password 123456 -web.listen-address=:9121；
2. docker redis_exporter安装
A. 下载镜像：docker pull oliver006/redis_exporter:latest；
B. docker-stack.yml部署文件
version: '3.8' networks: rhxy-monitor: external: true services: redis-exporter_rhxy-redis: image: oliver006/redis_exporter:latest hostname: redis-exporter_rhxy-redis deploy: mode: global placement: constraints: - node.hostname == rhxy-cache networks: - rhxy-monitor environment: REDIS_ADDR: 10.10.10.11:6701 REDIS_PASSWORD: 123456
3. prometheus.yml配置文件
global: scrape_interval: 15s evaluation_interval: 15s rule_files: - "/etc/prometheus/rules/*.yml" scrape_configs: - job_name: redis-exporter-targets metrics_path: /scrape static_configs: - targets: [ 'redis://10.10.10.11:6701', 'redis://10.10.10.11:6702', 'redis://10.10.10.11:6703', 'redis://10.10.10.12:6701', 'redis://10.10.10.12:6702', 'redis://10.10.10.12:6703', 'redis://10.10.10.13:6701', 'redis://10.10.10.13:6702', 'redis://10.10.10.13:6703' ] relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: redis-exporter_rhxy-redis:9121 - job_name: redis-exporter metrics_path: /metrics static_configs: - targets: ['redis-exporter_rhxy-redis:9121']
4. grafana模板：https://grafana.com/grafana/dashboards/11835。
五、RabbitMQ集群监控(rabbitmq_prometheus)
1. rabbitmq_exporter安装
A. 官网下载地址：https://github.com/kbudde/rabbitmq_exporter/releases/；
B. 启动服务：RABBIT_USER=guest RABBIT_PASSWORD=guest OUTPUT_FORMAT=JSON PUBLISH_PORT=9099 RABBIT_URL=http://localhost:15672 nohup ./rabbitmq_exporter &；
2. docker rabbitmq_exporter安装
A. 下载镜像：docker pull kbudde/rabbitmq-exporter:latest；
B. docker-stack.yml部署文件
version: '3.8' networks: rhxy-monitor: external: true rhxy-service: external: true services: rabbitmq-exporter_rhxy-rabbitmq: image: kbudde/rabbitmq-exporter:latest hostname: rabbitmq-exporter_rhxy-rabbitmq deploy: mode: global placement: constraints: - node.hostname == rhxy-cache networks: - rhxy-monitor - rhxy-service environment: RABBIT_URL: http://haproxy:15672 RABBIT_USER: guest RABBIT_PASSWORD: guest
3. rabbitmq_prometheus插件
A. rabbitmq_prometheus插件是3.8.0版本新增的，用来提供对Prometheus指标数据收集，采用TCP 15692端口公布数据；
B. 启用插件：rabbitmq-plugins enable rabbitmq_prometheus；
4. prometheus.yml配置文件
global: scrape_interval: 15s evaluation_interval: 15s rule_files: - "/etc/prometheus/rules/*.yml" scrape_configs: - job_name: 'rabbitmq-exporter' metrics_path: /metrics static_configs: - targets: ['rabbitmq-exporter_rhxy-rabbitmq:9419'] - job_name: 'rabbitmq-exporter-self' metrics_path: /metrics static_configs: - targets: [ 'rabbitmq-1:15692', 'rabbitmq-2:15692', 'rabbitmq-3:15692' ]
5. grafana模板
A. 使用rabbitmq_exporter方式模板：https://grafana.com/grafana/dashboards/10568；
B. 使用rabbitmq_prometheus插件模板：https://grafana.com/grafana/dashboards/10991。
六、ElasticSearch集群监控(elasticsearch_exporter)
1. elasticsearch_exporter安装
A. 下载地址：https://github.com/prometheus-community/elasticsearch_exporter/releases/；
B. 启动服务：nohup ./elasticsearch_exporter --es.all --es.indices --es.cluster_settings --es.indices_settings --es.shards --es.snapshots --es.timeout=10s --web.listen-address=":9115" --web.telemetry-path="/metrics" --es.uri http://10.xxx.xxx.11:9200 &；
2. docker elasticsearch_exporter安装
A. 下载镜像：docker pull justwatch/elasticsearch_exporter:latest；
B. docker-stack.yml部署文件
version: '3.8' networks: rhxy-monitor: external: true rhxy-service: external: true services: elasticsearch-exporter_rhxy-elasticsearch: image: justwatch/elasticsearch_exporter:latest hostname: elasticsearch-exporter_rhxy-elasticsearch deploy: mode: global placement: constraints: - node.hostname == rhxy-cache networks: - rhxy-monitor - rhxy-service command: - "--es.uri=http://haproxy:8200"
3. prometheus.yml配置文件
global: scrape_interval: 15s evaluation_interval: 15s rule_files: - "/etc/prometheus/rules/*.yml" scrape_configs: - job_name: 'elasticsearch-exporter' metrics_path: /metrics static_configs: - targets: ['elasticsearch-exporter_rhxy-elasticsearch:9114']
4. grafana模板：https://grafana.com/grafana/dashboards/2322。
七、Kafka
