k8s 部署 prometheus + grafana 监控服务器

1. 部署node-exporter

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter-deploy
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      containers:
      - image: prom/node-exporter:latest
        name: node-exporter
        ports:
        - containerPort: 9100
          hostPort: 9100
          name: scrape
      hostNetwork: true
      hostPID: true
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  name: node-exporter-svc
spec:
  selector:
    app: node-exporter
  clusterIP: None
  ports:
  - name: scrape
    port: 9100

node_exporter不建议使用容器部署,最好直接部署到机器上

2. 部署prometheus-server

• 创建ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-cm
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  prometheus.yml: |
    global:
     scrape_interval: 30s
     scrape_timeout: 30s
    scrape_configs:
    - job_name: 'prometheus'
      static_configs:
        - targets: ['localhost:9090']
    - job_name: 'k8s-nodes'
      static_configs:
        - targets: 
          - 192.168.0.200:9100
          - 192.168.0.201:9100

第一个job是监控自身
第二个job是监控k8s的节点

• 部署prometheus-server

apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-data-pv
  labels:
    app: prometheus
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  storageClassName: nfs
  nfs:
    path: /var/nfs/prometheus
    server: 192.168.0.250
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-data-pvc
spec:
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  storageClassName: nfs
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      app: prometheus
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus
        ports:
        - containerPort: 9090
        volumeMounts:
        - name: prometheus-data
          mountPath: /prometheus
        - mountPath: "/etc/prometheus"
          name: prometheus-config
      volumes:
      - name: prometheus-data
        persistentVolumeClaim:
          claimName: prometheus-data-pvc
      - name: prometheus-config
        configMap:
          name: prometheus-cm
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-svc
spec:
  selector:
    app: prometheus
  type: NodePort
  ports:
  - name: web
    port: 9090
    targetPort: 9090
    nodePort: 30090

3. 部署grafana

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: grafana-svc
spec:
  selector:
    app: grafana
  type: NodePort
  ports:
  - name: web
    port: 3000
    targetPort: 3000
    nodePort: 30030

现成的exporter列表：
https://prometheus.io/docs/instrumenting/exporters/

grafana的视图
linux:https://grafana.com/grafana/dashboards/8919
windows:https://grafana.com/grafana/dashboards/10467
kubernetes：https://grafana.com/grafana/dashboards/13105
zookeeper：https://grafana.com/grafana/dashboards/9236
zookeeper official: https://grafana.com/grafana/dashboards/10465
kafka：https://github.com/cloudhut/kminion
elasticsearch：https://grafana.com/grafana/dashboards/2322
监控模板查找：https://grafana.com/grafana/dashboards