使用 Elastic 技术栈构建 K8S 全栈监控 -2: 用 Metricbeat 对 Kubernetes 集群进行监控

文章转载自：https://www.qikqiak.com/post/k8s-monitor-use-elastic-stack-2/

操作步骤

git clone https://github.com/kubernetes/kube-state-metrics.git

cd kube-state-metrics


注意：下一步会用到镜像：registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.5.0，但是拉取不了

解决办法：
方法1：(在每个集群节点主机上执行)

docker pull bitnami/kube-state-metrics:2.5.0 && docker tag bitnami/kube-state-metrics:2.5.0 registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.5.0

方法2：

修改文件使用的镜像为：bitnami/kube-state-metrics:2.5.0
文件：kube-state-metrics\examples\standard\deployment.yaml

kubectl apply -f examples/standard/

kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-state-metrics

给es创建账号时有个用户：beats_system

Metricbeat连接es使用专门的账号：beats_system = rFRUMRxrkyHxQ1amJDrZ

# 不执行：kubectl create secret generic beats-elasticsearch-pw -n elastic --from-literal=username=beats_system --from-literal=password=rFRUMRxrkyHxQ1amJDrZ

注意：使用beats_system用户，Metricbeat启动后会报如下错误 (使用elastic这个超级用户则不会报错)

error loading /usr/share/metricbeat/kibana/7/dashboard/metricbeat-nginx-overview.json: returned 403 to import file: <nil>. Response: {"statusCode":403,"error":"Forbidden","message":"Unable to bulk_create dashboard,visualization"}
error loading /usr/share/metricbeat/kibana/7/dashboard/metricbeat-vsphere-host.json: returned 403 to import file: <nil>. Response: {"statusCode":403,"error":"Forbidden","message":"Unable to bulk_create dashboard,search,visualization"}
error loading /usr/share/metricbeat/kibana/7/dashboard/metricbeat-vsphere-vm.json: returned 403 to import file: <nil>. Response: {"statusCode":403,"error":"Forbidden","message":"Unable to bulk_create dashboard,search,visualization"}
error loading /usr/share/metricbeat/kibana/7/dashboard/metricbeat-windows-service.json: returned 403 to import file: <nil>. Response: {"statusCode":403,"error":"Forbidden","message":"Unable to bulk_create dashboard,search,visualization"}

因此这里还是用elastic这个超级用户配置管理吧

kubectl create secret generic beats-elasticsearch-pw -n elastic --from-literal=username=elastic --from-literal=password=7xmKjryjpjJfAui2d90v

kubectl apply  -f metricbeat.settings.configmap.yml \
                 -f metricbeat.indice-lifecycle.configmap.yml \
                 -f metricbeat.daemonset.yml \
                 -f metricbeat.permissions.yml

kubectl get pods -n elastic -l app=metricbeat