Kubernetes - - k8s - v1.12.3 一键部署高可用 Prometheus 并实现邮件告警

1，基本概念

• 本次部署使用的是CoreOS的prometheus-operator。
• 项目地址：https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus
• 使用Helm安装：https://github.com/helm/charts/tree/master/stable/prometheus-operator

2，安装

2.1 下载安装文件

2.2 修改相关配置

• 1 修改deploy文件中的etcd证书文件，kubeadm安装方式的无须修改
• 2 修改manifests/prometheus/prometheus-etcd.yaml的tlsConfig(kubeadm安装方式的无须修改)和addresses(etcd地址)
• 3 修改alertmanager.yaml文件的邮件告警配置和收件人配置

2.3 一键安装

[root@k8s-master01 prometheus-operator]# ./deploy
namespace/monitoring created
secret/alertmanager-main created
secret/etcd-certs created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
serviceaccount/prometheus-operator created
service/prometheus-operator created
deployment.apps/prometheus-operator created
Waiting for Operator to register custom resource definitions.............................done!
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
clusterrole.rbac.authorization.k8s.io/node-exporter created
daemonset.extensions/node-exporter created
serviceaccount/node-exporter created
service/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
deployment.extensions/kube-state-metrics created
rolebinding.rbac.authorization.k8s.io/kube-state-metrics created
role.rbac.authorization.k8s.io/kube-state-metrics-resizer created
serviceaccount/kube-state-metrics created
service/kube-state-metrics created
secret/grafana-credentials created
secret/grafana-credentials unchanged
configmap/grafana-dashboard-definitions-0 created
configmap/grafana-dashboards created
configmap/grafana-datasources created
deployment.apps/grafana created
service/grafana created
configmap/prometheus-k8s-rules created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/alertmanager created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
servicemonitor.monitoring.coreos.com/kubelet created
servicemonitor.monitoring.coreos.com/node-exporter created
servicemonitor.monitoring.coreos.com/prometheus-operator created
servicemonitor.monitoring.coreos.com/prometheus created
service/prometheus-k8s created
prometheus.monitoring.coreos.com/k8s created
service/etcd-k8s created
endpoints/etcd-k8s created
servicemonitor.monitoring.coreos.com/etcd-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
service/alertmanager-main created
alertmanager.monitoring.coreos.com/main created

2.4，验证安装

• svc

[root@k8s-master01 prometheus-operator]# kubectl get svc -n monitoring
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
alertmanager-main       NodePort    10.110.29.99     <none>        9093:30903/TCP      118s
alertmanager-operated   ClusterIP   None             <none>        9093/TCP,6783/TCP   118s
etcd-k8s                ClusterIP   None             <none>        2379/TCP            2m14s
grafana                 NodePort    10.109.144.190   <none>        3000:30902/TCP      2m28s
kube-state-metrics      ClusterIP   None             <none>        8443/TCP,9443/TCP   2m31s
node-exporter           ClusterIP   None             <none>        9100/TCP            2m35s
prometheus-k8s          NodePort    10.107.20.96     <none>        9090:30900/TCP      2m16s
prometheus-operated     ClusterIP   None             <none>        9090/TCP            2m15s
prometheus-operator     ClusterIP   10.110.197.2     <none>        8080/TCP            3m18s

• pods

[root@k8s-master01 prometheus-operator]# kubectl get pod -n monitoring
NAME                                   READY   STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2     Running   0          10m
alertmanager-main-1                    2/2     Running   0          7m13s
alertmanager-main-2                    2/2     Running   0          5m42s
grafana-644664679-gr6zh                1/1     Running   0          11m
kube-state-metrics-cf688bfc-w5tr7      4/4     Running   0          6m38s
node-exporter-5wz4v                    2/2     Running   0          11m
node-exporter-7r2jp                    2/2     Running   0          11m
node-exporter-shvzb                    2/2     Running   0          11m
prometheus-k8s-0                       2/2     Running   0          11m
prometheus-k8s-1                       2/2     Running   0          11m
prometheus-operator-69d5c87454-jqwns   1/1     Running   0          12m

• NodePortc
  • prometheus：30900
  • grafana：30902
  • alertmanager：30903

3，访问测试

3.1 prometheus

• http://k8s-master-lb:30900/graph

3.2 grafana

• http://k8s-master-lb:30902

3.3 alertmanager

• http://k8s-master-lb:30903

3.4 告警邮件

• 注：腾讯企业邮箱

• 参考：
  • https://www.cnblogs.com/dukuan/p/10177757.html
  • https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus
  • https://github.com/helm/charts/tree/master/stable/prometheus-operator