监控，水平升宿

监控:需要先安装HeapSter组件

 

资源指标:metrics-server

自定义指标:prometheus  k8s-prometheus-adapter

自定义资源定义

开发apiserver 服务器

资源指标api

 

新一代架构:

核心指标流水线:由kubelet,metrics-server以及由API server提供的api组成；提供cpu累计使用使用率,内存实时使用率，pod资源的占用率,以及容器的磁盘占用率；

 

监控流水线:用于从系统收集各种指标数据并提供终端用户,存储,系统以及HPA，它们包含核心指标以及许多非核心指标，非核心指标不能被k8s所解析

 

metrics-server:API server

 /apis/metrics.k8s.io/v1beta1

 

部署metrics-server 获取核心指标

https://github.com/kubernetes-incubator/metrics-server/tree/master/deploy/1.8%2B

 

git clone https://github.com/kubernetes-incubator/metrics-server.git

cd /root/metrics/metrics-server-master/deploy/1.8+

将metrics-server-deployment.yaml

镜像地址改为: mirrorgooglecontainers/metrics-server-amd64:v0.3.3

vim metrics-server-deployment.yaml

---

apiVersion: v1

kind: ServiceAccount

metadata:

  name: metrics-server

  namespace: kube-system

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: metrics-server

  namespace: kube-system

  labels:

    k8s-app: metrics-server

spec:

  selector:

    matchLabels:

      k8s-app: metrics-server

  template:

    metadata:

      name: metrics-server

      labels:

        k8s-app: metrics-server

    spec:

      serviceAccountName: metrics-server

      volumes:

      # mount in tmp so we can safely use from-scratch images and/or read-only containers

      - name: tmp-dir

        emptyDir: {}

      containers:

      - name: metrics-server

        image: mirrorgooglecontainers/metrics-server-amd64:v0.3.3 修改镜像地址

        imagePullPolicy: Always

        command:      新加

        - /metrics-server  新加

        - --metric-resolution=30s  新加

        - --kubelet-insecure-tls  新加

        - --kubelet-preferred-address-types=InternalIP 新加

        volumeMounts:

        - name: tmp-dir

          mountPath: /tmp

 

vim resource-reader.yaml

rules:

- apiGroups:

  - ""

  resources:

  - pods

  - nodes

  - nodes/stats

  - namespaces

  verbs:

  - get

  - list

  - watch

 

kubectl apply -f ./

 

kubectl get pods -n kube-system  -o wide

kubectl describe pods -n kube-system metrics-server-95cc6867b-nm8g4

kubectl api-versions 会出现metrics.k8s.io/v1beta1

使用

打开反向代理

kubectl proxy --port=8080

打开另一个终端获取数据

curl http://localhost:8080/apis/metrics.k8s.io/v1beta1

获取node

 curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/nodes

获取pod

curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/pods

直接命令

kubectl top nodes node1

kubectl top pod --all-namespaces

 

prometheus监控

容器日志:/var/logs/containers

 

prometheus架构 自身就是一个数据库

node_exporte --> prometheusr <--> PromQL <-- kube-state-metrics <-- custom metrics api (k8s-prometheus-adapater)

 

部署

官方部署https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus

实际部署https://github.com/ikubernetes/k8s-prom

 

过程

mkdir prometheus && cd prometheus/

unzip k8s-prom-master.zip  && cd k8s-prom-master/

 

创建名称空间

kubectl apply -f namespace.yaml

 

创建node_exporter 客户端agent

cd node_exporter/

kubectl apply -f ./

 

创建prometheus 主体

cd prometheus/

kubectl apply -f ./

测试访问 http://192.168.81.10:30090/graph

 

创建kube-state-metrics  将数据转换成k8s能够识别的格式

cd kube-state-metrics/

修改镜像地址vim kube-state-metrics-deploy.yaml

mirrorgooglecontainers/kube-state-metrics-amd64:v1.3.1

kubectl apply -f ./

 

创建k8s-prometheus-adapter 是metric的apiserver

cd k8s-prometheus-adapter/

下载https://github.com/DirectXMan12/k8s-prometheus-adapter/tree/master/deploy/manifests 替换k8s-prometheus-adapter目录，将名称空间custom-metrics替换为prom

然后apply这个目录所有文件

kubectl exec -it   -n prom custom-metrics-apiserver-fd48dd8c-khrkn -- /bin/sh

kubectl api-versions

custom.metrics.k8s.io/v1beta1

kubectl proxy --port=8080

另一终端

curl http://localhost:8080/apis/custom.metrics.k8s.io/v1beta1/

 

整合grafana

cp grafana.yaml ../prometheus/k8s-prom-master

vim grafana.yaml

namespace: prom

service

  namespace: prom

nodePort: 32002

#- name: INFLUXDB_HOST

#  value: monitoring-influxdb 注释influxdb

 

kubectl apply -f grafana.yam

设置grafana

Name:Prometheus

Type:Prometheus

URL http://prometheus.prom.svc:9090  

就可以save test

 

模板下载

https://grafana.com/grafana/dashboards/8588 -->download_json

 

daboard主页 +号 --> Import -->upload json -->选择数据源

建议下载Kubernetes Cluster (Prometheus) 模板

 

HPA 运用规模水平自动升宿

kubectl explain hpa

kubectl explain hpa.spec

kubectl run myapp --image=ikubernetes/myapp:v1 --replicas=1 --requests='cpu=50m,memory=256Mi'  

--limits='cpu=50m,memory=256Mi' --labels='app=myapp' --expose --port=80

                                                            暴露端口

自动升宿

kubectl autoscale deployment myapp --min=1 --max=8 --cpu-percent=60

       最少pod 最多pod cpu最多使用60%

kubectl get hpa                                         

 

压测：

yum install httpd-tools -y

ab -c 100 -n 50000 http://10.96.150.45/index.html

kubectl describe hpa

kubectl delete deployments.apps myapp

kubectl delete pod pod-demo --force --grace-period=0  强制删除pod

 

使用hpa2

kubectl run myapp --image=ikubernetes/myapp:v1 --replicas=1 --requests='cpu=50m,memory=256Mi'  

--limits='cpu=50m,memory=256Mi' --labels='app=myapp' --expose --port=80

 

vim hpa-v2-demo.yaml

apiVersion: autoscaling/v2beta1

kind: HorizontalPodAutoscaler

metadata:

  name: myapp-hpa-v2-2

spec:  规格

  scaleTargetRef:  引用什么指标来定义  

    apiVersion: apps/v1 对谁来扩展

    kind: Deployment 对deploy来扩展

    name: myapp  deploy名

  minReplicas: 1 最少

  maxReplicas: 10 最多

  metrics: 依据哪些指标评估

  - type: Resource 基于资源

   resource: 资源

     name: CPU 基于cpu

     targetAverageUtilization: 55 超过60%就扩展

  - type: Resource

   resource:

     name: memory 基于内存

     targetAverageValue: 50Mi 超过50Mi就扩展

 

kubectl apply -f  hpa-v2-demo.yaml

kubectl get hpa

进行压测

ab -c 100 -n 50000 http://10.96.150.45/index.html

    -c 每个命令发送100次请求   -n 总共发送多少请求

hpa2例

kubectl run myapp-custom --image=ikubernetes/metrics-app --replicas=1 --requests='cpu=50m,memory=256Mi'  --limits='cpu=50m,memory=256Mi' --labels='app=myapp' --expose --port=80

 

vim hpa-v2-custom.yaml

apiVersion: autoscaling/v2beta1

kind: HorizontalPodAutoscaler

metadata:

  name: myapp-hpa-v2

spec:

  scaleTargetRef:

    apiVersion: apps/v1

    kind: Deployment

    name: myapp-custom

  minReplicas: 1

  maxReplicas: 10

  metrics:

  - type: Pods  对pods指标输出

    pods:

      metricName: http_requests 自定义指标 http_requests并发数

      targetAverageValue: 800m  超过800个并发就扩展

 

压测

ab -c 100 -n 50000 http://10.102.237.99/index.html