容器的监控：使用cAdvisor，weavescope监控容器
一.系统环境
| 服务器版本 | docker软件版本 | CPU架构 |
|---|---|---|
| CentOS Linux release 7.4.1708 (Core) | Docker version 20.10.12 | x86_64 |
二.前言
当生产环境中部署了多个容器，我们需要部署监控工具来监控容器的资源利用率和性能。本文介绍两种容器监控工具：cAdvisor，weavescope。
cAdvisor 是谷歌公司用来分析运行中的 Docker 容器的资源占用以及性能特性的工具。cAdvisor 是一个运行中的守护进程用来收集、聚合、处理和导出运行容器相关的信息，每个容器保持独立的参数、历史资源使用情况和完整的资源使用数据。cadvisor监控容器主要是监控物理机的目录：/var/run，/sys，/var/lib/docker。
Weave Scope是Docker和Kubernetes的可视化和监视工具。它提供了自上而下的应用程序视图以及整个基础架构视图，并允许您实时诊断将分布式容器化应用程序部署到云提供商时遇到的任何问题。github开源项目地址: https://github.com/weaveworks/scope
Weave Scope可以监控Kubernetes集群的一系列资源的状态、资源使用情况、应用拓扑、还可以直接通过UI界面进行调试，查看日志等操作。
三.容器监控的原理
Docker是基于Namespace、Cgroups和联合文件系统实现的，Cgroups不仅可以用于容器资源的限制，还可以提供容器的资源使用率。不管用什么监控方案，底层数据都来源于Cgroups，Cgroups的工作目录/sys/fs/cgroup 下包含了Cgroups的所有内容。Cgroups包含了很多子系统，可以对CPU，内存，PID，磁盘IO等资源进行限制和监控。
查看 /sys/fs/cgroup 文件夹：
每一个文件夹都代表了Cgroups的子系统，docker会在Cgroups的子系统下创建docker文件夹。
四.使用docker stats监控容器
容器的监控可以使用命令docker stats查看
[root@k8smaster ~]# docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
7b38d98d8db5 nginx 0.07% 4.355MiB / 3.32GiB 0.13% 603MB / 604MB 1.91MB / 0B 3
f52a518746ec harbor-jobservice 0.20% 31.14MiB / 3.32GiB 0.92% 16.3MB / 229MB 75.5MB / 0B 8
d2422dc8ed83 harbor-core 0.11% 48.21MiB / 3.32GiB 1.42% 608MB / 609MB 78.8MB / 0B 8
cd780fa84d5f redis 0.21% 4.336MiB / 3.32GiB 0.13% 230MB / 16.8MB 4.42MB / 2.27MB 5
7fba7cc774eb harbor-portal 0.06% 8.703MiB / 3.32GiB 0.26% 1.3MB / 5.92MB 13.4MB / 0B 3
9080c39aea05 registry 0.08% 115.7MiB / 3.32GiB 3.40% 304MB / 301MB 26.8MB / 193MB 5
fbeb3dbb7f29 harbor-db 0.09% 75.74MiB / 3.32GiB 2.23% 1.6MB / 1.54MB 18.7MB / 77.4MB 8
7142a107f480 registryctl 0.06% 14.28MiB / 3.32GiB 0.42% 914kB / 709kB 29.7MB / 0B 9
187b4a82403a harbor-log 0.06% 6.738MiB / 3.32GiB 0.20% 2.3MB / 658kB 12.8MB / 126kB 11
0797cbb90156 k8s_kube-controller-manager_kube-controller-manager-k8smaster_kube-system_0bac7daeb78f382a2a6fb01d62288896_29 0.27% 43.24MiB / 3.32GiB 1.27% 0B / 0B 156MB / 0B 5
2949d6764ab5 k8s_POD_kube-controller-manager-k8smaster_kube-system_0bac7daeb78f382a2a6fb01d62288896_20 0.00% 1.395MiB / 3.32GiB 0.04% 0B / 0B 0B / 0B 1
7288858a1ecc k8s_POD_kube-apiserver-k8smaster_kube-system_463a955551c752a0b753700879aec991_20 0.00% 1.461MiB / 3.32GiB 0.04% 0B / 0B 0B / 0B 1
e3323c5d994c k8s_POD_etcd-k8smaster_kube-system_1dcb59df47c677756cdf25f28d920325_21 0.00% 1.359MiB / 3.32GiB 0.04% 0B / 0B 0B / 0B 1
511e94439641 k8s_kube-scheduler_kube-scheduler-k8smaster_kube-system_298ac610fc4625e596fecc2821dc924e_30 1.04% 32.47MiB / 3.32GiB 0.96% 0B / 0B 84.2MB / 0B 8
45a5df586d7e k8s_POD_kube-scheduler-k8smaster_kube-system_298ac610fc4625e596fecc2821dc924e_20 0.00% 1.516MiB / 3.32GiB 0.04% 0B / 0B 668kB / 0B 1
1edbf56b01ee redhat7.2 0.00% 904KiB / 3.32GiB 0.03% 996B / 0B 4.73MB / 0B 1
c2d01513585b redhat 0.00% 860KiB / 3.32GiB 0.02% 1.09kB / 0B 10.2MB / 0B 2
五.部署cadvisor监控容器
拉取cadvisor镜像
[root@k8smaster ~]# docker pull hub.c.163.com/xbingo/cadvisor:latest
latest: Pulling from xbingo/cadvisor
09d0220f4043: Pull complete
151807d34af9: Pull complete
14cd28dce332: Pull complete
Digest: sha256:20ddac7ab8fdaff0549c5dcf046f0cb6f8ad064e20d9004dcc07132b907c4b68
Status: Downloaded newer image for hub.c.163.com/xbingo/cadvisor:latest
hub.c.163.com/xbingo/cadvisor:latest
查看cadvisor镜像的历史信息，端口为EXPOSE 8080/tcp
[root@k8smaster ~]# docker history hub.c.163.com/xbingo/cadvisor:latest
IMAGE CREATED CREATED BY SIZE COMMENT
4bc3588563b1 5 years ago /bin/sh -c #(nop) ENTRYPOINT ["/usr/bin/cadv… 0B
<missing> 5 years ago /bin/sh -c #(nop) EXPOSE 8080/tcp 0B
<missing> 5 years ago /bin/sh -c #(nop) ADD file:d8a1cb9e0c5b0d90d… 23MB
<missing> 5 years ago /bin/sh -c apk add --update ca-certificates … 20MB
<missing> 5 years ago /bin/sh -c #(nop) ENV GLIBC_VERSION=2.23-r1 0B
<missing> 5 years ago /bin/sh -c #(nop) MAINTAINER dengnan@google.… 0B
<missing> 5 years ago /bin/sh -c #(nop) ADD file:af671c91a986f389f… 5.26MB
cadvisor监控容器主要是监控物理机的目录：/var/run，/sys，/var/lib/docker，创建容器，挂载数据卷，使cadvisor容器可以监控相关目录。
[root@k8smaster ~]# docker run -dit --name=cadvisor --restart=always -p 8080:8080 -v /var/run:/var/run -v /sys:/sys:ro -v /var/lib/docker:/var/lib/docker:ro hub.c.163.com/xbingo/cadvisor:latest
db87f17745b6c3764aaa2597b94eac7e7464af333b0efcb7d87d026c9e78fde5
[root@k8smaster ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
db87f17745b6 hub.c.163.com/xbingo/cadvisor:latest "/usr/bin/cadvisor -…" 5 seconds ago Up 5 seconds 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp cadvisor
在浏览器输入物理机IP:8080访问cadvisor，浏览器打开http://192.168.110.137:8080/，即可看见cadvisor监控页面
六.部署weavescope监控容器
上传scope脚本并授予可执行权限，也可以自行下载scope脚本
[root@k8smaster ~]# curl -L git.io/scope -o /usr/local/bin/scope
[root@k8smaster ~]# ll -h scope
-rw-r--r-- 1 root root 12K 10月 7 02:54 scope
[root@k8smaster ~]# chmod +x scope
[root@k8smaster ~]# ll -h scope
-rwxr-xr-x 1 root root 12K 10月 7 02:54 scope
执行scope脚本会拉取weaveworks/scope:1.13.1镜像
[root@k8smaster ~]# ./scope
Usage:
scope launch {OPTIONS} {PEERS} - Launch Scope
scope stop - Stop Scope
scope command - Print the docker command used to start Scope
scope help - Print usage info
scope version - Print version info
PEERS are of the form HOST[:PORT]
HOST may be an ip or hostname.
PORT defaults to 4040.
Launch options:
Unable to find image 'weaveworks/scope:1.13.1' locally
1.13.1: Pulling from weaveworks/scope
c9b1b535fdd9: Pull complete
550073704c23: Pull complete
8738e5bbaf1d: Pull complete
0a8826d26027: Pull complete
387c1aa951b4: Pull complete
e72d45461bb9: Pull complete
75cc44b65e98: Pull complete
11f7584a6ade: Pull complete
a5aa3ebbe1c2: Pull complete
7cdbc028c8d2: Pull complete
Digest: sha256:4342f1c799aba244b975dcf12317eb11858f9879a3699818e2bf4c37887584dc
Status: Downloaded newer image for weaveworks/scope:1.13.1
Usage of /home/weave/scope:
-app-only
Only run the app.
-app.aws.create.tables
Create the tables in DynamoDB
......
-weave
Enable Weave Net integrations. (default true)
-weave.hostname string
Hostname to advertise/lookup in WeaveDNS (default "scope.weave.local.")
[root@k8smaster ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
weaveworks/scope 1.13.1 cd4cd8fcce1b 19 months ago 89.7MB
安装weavescope
[root@k8smaster ~]# ./scope launch
cf1b38f8dcdd5f61ae1d157800f70e59fdc11ed12bfb7b76f78c73e4048be86d
Scope probe started
Weave Scope is listening at the following URL(s):
* http://172.28.0.1:4040/
* http://172.18.0.1:4040/
* http://192.168.110.137:4040/
[root@k8smaster ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cf1b38f8dcdd weaveworks/scope:1.13.1 "/home/weave/entrypo…" 4 minutes ago Up 4 minutes weavescope
浏览器打开http://192.168.110.137:4040/，即可查看监控界面，监控项目还是多样的
致力于一条龙式的为您解决问题
