prometheus学习笔记之基于redis-exporter、mysqld-exporter、nginx-vts-exporter

一、redis_exporter

通过 redis_exporter 监控 redis 服务状态

git地址:https://github.com/oliver006/redis_exporter

docker地址：https://hub.docker.com/r/oliver006/redis_exporter

实验环境：redis k8部署 prometheus二进制部署

1.redis_exporter使用简解

二进制部署prometheus配置方式

scrape_configs:
  - job_name: redis_exporter
    static_configs:
    - targets: ['<<REDIS-EXPORTER-HOSTNAME>>:9121'] #注意 redis_exporter使用的是9121端口保留redis数据

k8s服务发现配置

通过 pod 角色抓取指标，则可以添加：
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: instance
            regex: (.*redis.*)  #只有名称中带有“redis”的 pod 会被抓取

抓取redis集群配置

scrape_configs:
  ## config for the multiple Redis targets that the exporter will scrape
  - job_name: 'redis_exporter_targets'
    static_configs:
      - targets:
        - redis://first-redis-host:6379
        - redis://second-redis-host:6379
        - redis://second-redis-host:6380
        - redis://second-redis-host:6381
    metrics_path: /scrape
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: <<REDIS-EXPORTER-HOSTNAME>>:9121

  ## config for scraping the exporter itself
  - job_name: 'redis_exporter'
    static_configs:
      - targets:
        - <<REDIS-EXPORTER-HOSTNAME>>:9121

身份验证

1.设置-redis.password-file=sample-pwd-file.json指定密码文件
2.包含密码的 URI 示例为：redis://<<username (optional)>>:<<PASSWORD>>@<<HOSTNAME>>:<<PORT>>
3.使用--redis.user并--redis.password直接向 redis_exporter 提供用户名和/或密码。

2.安装redis及redis_exporter

2.1 redis_exporter作为side-car和redis一起部署

YAML文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: registry.cn-shanghai.aliyuncs.com/zdbl-base/redis:5.0.3
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379
      - name: redis-exporter
        image: registry.cn-shanghai.aliyuncs.com/zdbl-base/redis_exporter:v1.63.0 #默认redis_exporter监听本地6379 如果不和redis在同一节点，需要指定redis地址
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 9121
---
kind: Service  #service 类型
apiVersion: v1
metadata:
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: "9121"
  name: redis-exporter-service
spec:
  selector:
    app: redis
  ports:
  - nodePort: 31082
    name: prom
    port: 9121
    protocol: TCP
    targetPort: 9121
  type: NodePort
---
kind: Service  #service 类型
apiVersion: v1
metadata:
  name: redis-redis-service
spec:
  selector:
    app: redis
  ports:
  - nodePort: 31081
    name: redis
    port: 6379
    protocol: TCP
    targetPort: 6379
  type: NodePort

2.2 redis已存在单独部署并指定redis

 

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-passwd-cm
  namespace: monitoring
data:
  redis_passwd.json: |
    {"redis:///redis-redis-service:6379":""
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-exporter
  namespace: monitoring 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-exporter
  template:
    metadata:
      labels:
        app: redis-exporter
    spec:
      containers:
      - name: redis-exporter
        image: registry-vpc.cn-shanghai.aliyuncs.com/zdbl-base/redis_exporter:v1.63.0
        env:
        - name: TZ
          value: "Asia/Shanghai"
        - name: REDIS_ADDR
          value: redis://redis-redis-service:6379 
        args:
        - -redis.password-file=/opt/redis_passwd.json
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 9121
        volumeMounts:
        - name: redis-passwd-conf-map
          mountPath: "/opt"
      volumes:
      - name: redis-passwd-conf-map
        configMap:
          name: redis-passwd-cm
---
kind: Service  #service 类型
apiVersion: v1
metadata:
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: "9121"
  name: redis-exporter-service
  namespace: monitoring 
spec:
  selector:
    app: redis-exporter
  ports:
  - nodePort: 31082
    name: redis-exporter
    port: 9121
    protocol: TCP
    targetPort: 9121
  type: NodePort

 

验证

kubectl get svc|grep redis
kubectl get pods|grep redis

访问metrics端口(注意端口为redis-exporter的端口)

3.配置prometheus

vim prometheus.yml 
  - job_name: "redis-monitor-metrics"
    static_configs:
      - targets: ["192.168.100.131:31082"]

curl -X POST http://127.0.0.1:9090/-/reload 

4.配置grafana

推荐模板：

单机：模板ID:763 17507(中文)

集群：模板ID 14615

核心指标

连接的客户端数redis_connected_clients
内存使用率100 * (redis_memory_used_bytes / redis_memory_max_bytes )
命令执行qps rate(redis_commands_processed_total[1m])
cache命中qpsirate(redis_keyspace_hits_total[5m])cache未命中qps irate(redis_keyspace_misses_total[5m])网络入流量rate(redis_net_input_bytes_total[5m])
网络出流量rate(redis_net_output_bytes_total[5m])db中的key数量sum (redis_db_keys) by (db)
db中的过期key数量sum (redis_db_keys_expiring) by (db)
每一种命令的qps topk(5, irate(redis_commands_total[1m]))

二、mysqld_exporter

项目地址：https://github.com/prometheus/mysqld_exporter

镜像地址：https://hub.docker.com/r/prom/mysqld-exporter

版本要求：

MySQL >= 5.6.
MariaDB >= 10.3
如果低于以上版本，部分指标可能不展示

1.部署MySQL

略

2.创建mysqld_exporter用户

CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'XXXXXXXX' WITH MAX_USER_CONNECTIONS 10; #根据实际情况调整客户端地址
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';

3.下载并启动mysqld_exporter

https://github.com/prometheus/mysqld_exporter/releases/download/v0.15.1/mysqld_exporter-0.15.1.linux-amd64.tar.gz
tar xf mysqld_exporter-0.15.1.linux-amd64.tar.gz
mv mysqld_exporter-0.15.1.linux-amd64/mysqld_exporter /usr/local/bin/

vim ~/.my.cnf
[client]
user=root #这里我为了方便直接使用root，生产环境不建议
password=123456
host=192.168.100.134
port=3306


vim /etc/systemd/system/mysqld_exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
ExecStart=/usr/local/bin/mysqld_exporter --config.my-cnf=/root/.my.cnf

[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl restart mysqld_exporter
systemctl enable mysqld_exporter
netstat -tnlp|grep 9104

验证数据

4.配置prometheus

vim prometheus.yml 
  - job_name: "mysql-monitor-metrics"
    static_configs:
      - targets: ["192.168.100.131:9104"]

curl -X POST http://127.0.0.1:9090/-/reload 

5.配置grafana

推荐模板：

mysql： 17320(中文)  14031

MariaDB：13106 11323

MySQL cluster：15641

三、nginx-vts-exporter

https://github.com/vozlt/nginx-module-vts

https://github.com/sysulq/nginx-vts-exporter

抓取 Nginx vts 统计信息并通过 HTTP 导出以供 Prometheus 使用的简单服务器依赖 Nginx VTS模块

1.编译安装Nginx

wget https://github.com/vozlt/nginx-module-vts/archive/refs/tags/v0.2.2.tar.gz
tar xf v0.2.2.tar.gz 
mv nginx-module-vts-0.2.2 /usr/local/src/nginx-module-vts

wget http://nginx.org/download/nginx-1.20.2.tar.gz
tar xf nginx-1.20.2.tar.gz
cd nginx-1.20.2
./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_stub_status_module --with-http_gzip_static_module --with-pcre --with-file-aio --with-stream --with-stream_ssl_module --with-stream_realip_module --add-module=/usr/local/src/nginx-module-vts/  #注意要添加VTS模块
make && make install
/usr/local/nginx/sbin/nginx -V

2.配置Nginx开启VTS

http {
    vhost_traffic_status_zone; #在Nginx conf中 http字段添加

    ...

    server {

        ...
        # 添加该location
        location /status { 
            vhost_traffic_status_display;
            vhost_traffic_status_display_format html;
        }
    }
}

/usr/local/nginx/sbin/nginx -t
/usr/local/nginx/sbin/nginx 
netstat -tnlp|grep 80

浏览器验证

3.安装nginx-vts-exporter

wget https://github.com/sysulq/nginx-vts-exporter/releases/download/v0.10.0/nginx-vtx-exporter_0.10.0_linux_arm64.tar.gz #我是centos7 安装10.0版本之上报错
tar xf nginx-vts-exporter-0.10.0.linux-amd64.tar.gz
cd nginx-vts-exporter-0.10.0.linux-amd64
./nginx-vts-exporter -h
nohup ./nginx-vts-exporter -nginx.scrape_uri http://192.168.100.134/status/format/json &
netstat -tnlp|grep 9913

浏览器验证

4.配置prometheus

vim prometheus.yml 
  - job_name: "nginx-monitor-metrics"
    static_configs:
      - targets: ["192.168.100.131:9913"]

curl -X POST http://127.0.0.1:9090/-/reload 

浏览器验证

5.配置grafana

推荐模板: ID 2949 18918

官方支持的exporter列表：https://prometheus.io/docs/instrumenting/exporters/