准入控制器(Admission Controller):ResourceQuota，ImagePolicyWebhook
一.系统环境
本文主要基于Kubernetes1.22.2和Linux操作系统Ubuntu 18.04。
| 服务器版本 | docker软件版本 | Kubernetes(k8s)集群版本 | CPU架构 |
|---|---|---|---|
| Ubuntu 18.04.5 LTS | Docker version 20.10.14 | v1.22.2 | x86_64 |
Kubernetes集群架构：k8scludes1作为master节点，k8scludes2，k8scludes3作为worker节点。
| 服务器 | 操作系统版本 | CPU架构 | 进程 | 功能描述 |
|---|---|---|---|---|
| k8scludes1/192.168.110.128 | Ubuntu 18.04.5 LTS | x86_64 | docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico | k8s master节点 |
| k8scludes2/192.168.110.129 | Ubuntu 18.04.5 LTS | x86_64 | docker，kubelet，kube-proxy，calico | k8s worker节点 |
| k8scludes3/192.168.110.130 | Ubuntu 18.04.5 LTS | x86_64 | docker，kubelet，kube-proxy，calico | k8s worker节点 |
二.前言
在本文中，我们将探讨Kubernetes中的准入控制器（Admission Controller）。准入控制器是Kubernetes API server的一部分，负责处理来自外部的API请求。它们确保只有满足特定条件的请求才能访问集群资源。通过使用准入控制器，我们可以实现对集群资源的细粒度控制，从而提高集群的安全性和可靠性。
使用准入控制器（Admission Controller）的前提是已经有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Ubuntu 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/17632858.html。
三.准入控制器简介
当用户账户或者service account要执行一些操作的时候，首先要进行账号认证，认证通过之后，如果账号已经被授权相关资源，就可以进行相关操作了。有时授权之后也会有Admission control准入控制器(可以理解为一系列规则，可以启用或者关闭某个功能，准入控制器也支持一系列的webhook，在webhook里可以定义一系列规则)，满足Admission control准入控制器的规则就可以创建相关资源了。OPA Gatekeeper也是利用的准入控制器功能，详情请查看博客《OPA Gatekeeper：Kubernetes的策略和管理》。
准入控制器 是一段代码，它会在请求通过认证和鉴权之后、对象被持久化之前拦截到达 API 服务器的请求。
准入控制器可以执行验证（Validating） 和/或变更（Mutating） 操作。 变更（mutating）控制器可以根据被其接受的请求更改相关对象；验证（validating）控制器则不行。
准入控制器限制创建、删除、修改对象的请求。 准入控制器也可以阻止自定义动作，例如通过 API 服务器代理连接到 Pod 的请求。 准入控制器不会 （也不能）阻止读取（get、watch 或 list）对象的请求。
Kubernetes 1.30 中的准入控制器编译进 kube-apiserver 可执行文件，并且只能由集群管理员配置。 在该列表中，有两个特殊的控制器：MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook。 它们根据 API 中的配置， 分别执行变更和验证准入控制 Webhook。
准入控制过程分为两个阶段。第一阶段，运行变更准入控制器。第二阶段，运行验证准入控制器。 再次提醒，某些控制器既是变更准入控制器又是验证准入控制器。
如果两个阶段之一的任何一个控制器拒绝了某请求，则整个请求将立即被拒绝，并向最终用户返回错误。
最后，除了对对象进行变更外，准入控制器还可能有其它副作用：将相关资源作为请求处理的一部分进行变更。 增加配额用量就是一个典型的示例，说明了这样做的必要性。 此类用法都需要相应的回收或回调过程，因为任一准入控制器都无法确定某个请求能否通过所有其它准入控制器。
四.为什么需要准入控制器
Kubernetes 的若干重要功能都要求启用一个准入控制器，以便正确地支持该特性。 因此，没有正确配置准入控制器的 Kubernetes API 服务器是不完整的，它无法支持你所期望的所有特性。
五.启用/禁用ResourceQuota资源配额
5.1 查看默认启用/禁用的准入控制器插件
本文中的kube-apiserver是以pod的方式运行的，名字为：kube-apiserver-k8scludes1。
root@k8scludes1:~# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-65898446b5-qd9q6 1/1 Running 21 (2d8h ago) 28d
calico-node-d6564 1/1 Running 58 (2d8h ago) 59d
calico-node-jgvjb 1/1 Running 67 (10h ago) 59d
calico-node-snkxp 1/1 Running 58 (2d8h ago) 59d
coredns-7f6cbbb7b8-5gpjt 1/1 Running 20 (2d8h ago) 28d
coredns-7f6cbbb7b8-xm6pl 1/1 Running 20 (2d8h ago) 28d
etcd-k8scludes1 1/1 Running 54 (2d8h ago) 58d
kube-apiserver-k8scludes1 1/1 Running 15 (10h ago) 19d
kube-controller-manager-k8scludes1 1/1 Running 249 (10h ago) 59d
kube-proxy-464tx 1/1 Running 52 (2d8h ago) 59d
kube-proxy-mkwpx 1/1 Running 52 (2d8h ago) 59d
kube-proxy-vsc9q 1/1 Running 53 (2d8h ago) 59d
kube-scheduler-k8scludes1 1/1 Running 247 (10h ago) 59d
查看kube-apiserver的帮助信息。
root@k8scludes1:~# kubectl exec -it kube-apiserver-k8scludes1 -n kube-system -- kube-apiserver -h
The Kubernetes API server validates and configures data
for the api objects which include pods, services, replicationcontrollers, and
others. The API Server services REST operations and provides the frontend to the
......
--goaway-chance float
To prevent HTTP/2 clients from getting stuck on a single apiserver, randomly close a connection (GOAWAY). The client's other in-flight requests won't be affected, and the client will
reconnect, likely landing on a different apiserver after going through the load balancer again. This argument sets the fraction of requests that will be sent a GOAWAY. Clusters with single
apiservers, or which don't use a load balancer, should NOT enable this. Min is 0 (off), Max is .02 (1/50 requests); .001 (1/1000) is a recommended starting point.
--livez-grace-period duration
This option represents the maximum amount of time it should take for apiserver to complete its startup sequence and become live. From apiserver's start time to when this amount of time has
elapsed, /livez will assume that unfinished post-start hooks will complete successfully and therefore return true.
--vmodule moduleSpec
comma-separated list of pattern=N settings for file-filtered logging
查看默认启用/禁止的准入控制器插件有哪些？可以看到默认启用的准入控制器插件有：CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, PodSecurity, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook。
root@k8scludes1:~# kubectl exec -it kube-apiserver-k8scludes1 -n kube-system -- kube-apiserver -h | grep admission-plugins
--admission-control strings Admission is divided into two phases. In the first phase, only mutating admission plugins run. In the second phase, only validating admission plugins run. The names in the below list may represent a validating plugin, a mutating plugin, or both. The order of plugins in which they are passed to this flag does not matter. Comma-delimited list of: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. (DEPRECATED: Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.)
--disable-admission-plugins strings admission plugins that should be disabled although they are in the default enabled plugins list (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
--enable-admission-plugins strings admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
5.2 ResourceQuota资源配额示例
创建目录存放yaml文件。
root@k8scludes1:~# mkdir admissioncontr
root@k8scludes1:~# cd admissioncontr/
创建命名空间。
root@k8scludes1:~/admissioncontr# kubectl create ns admissioncontr
namespace/admissioncontr created
切换命名空间到admissioncontr。
root@k8scludes1:~/admissioncontr# kubens admissioncontr
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "admissioncontr".
编辑pod配置文件，使用nginx镜像生成pod。
root@k8scludes1:~/admissioncontr# vim pod.yaml
root@k8scludes1:~/admissioncontr# cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: podtest
name: podtest
spec:
#当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
terminationGracePeriodSeconds: 0
containers:
- image: hub.c.163.com/library/nginx:latest
#imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
imagePullPolicy: IfNotPresent
name: podtest
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {}
创建pod。
root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml
pod/podtest created
使用相同的yaml文件，生成另外两个pod。
root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created
root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
pod/podtest2 created
现在没有任何限制，可以正常创建多个pod。
root@k8scludes1:~/admissioncontr# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
podtest 1/1 Running 0 61s 10.244.218.179 k8scludes2 <none> <none>
podtest1 1/1 Running 0 15s 10.244.218.133 k8scludes2 <none> <none>
podtest2 1/1 Running 0 9s 10.244.1.84 k8scludes3 <none> <none>
删除pod。
root@k8scludes1:~/admissioncontr# kubectl delete pod podtest podtest1 podtest2
pod "podtest" deleted
pod "podtest1" deleted
pod "podtest2" deleted
现在没有设置resourcequota资源配额。
root@k8scludes1:~/admissioncontr# kubectl get quota
No resources found in admissioncontr namespace.
root@k8scludes1:~/admissioncontr# kubectl get resourcequota
No resources found in admissioncontr namespace.
编辑ResourceQuota配置文件，表示创建一个resourcequota资源配额，当前命名空间只能创建2个pod。
resourcequota 资源配额用来设置一个命名空间最多能创建多少个资源对象，比如能创建多少svc，能创建多少pod或者deploy，详细信息请查看博客《Kubernetes(k8s) 资源限制：resources，LimitRange，ResourceQuota》。
root@k8scludes1:~/admissioncontr# vim resourcequota.yaml
root@k8scludes1:~/admissioncontr# cat resourcequota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: myresourcequota
spec:
#hard：指定资源的硬性限制，即最大允许使用的资源数量。
hard:
pods: "2"
创建ResourceQuota。
root@k8scludes1:~/admissioncontr# kubectl apply -f resourcequota.yaml
resourcequota/myresourcequota created
查看resourcequota资源配额，可以看到pod资源限额是2，现在有0个pod。
root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME AGE REQUEST LIMIT
myresourcequota 11s pods: 0/2
再次创建pod。
root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml
pod/podtest created
root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created
创建第三个pod的时候报错了，pod资源配额是2，超过2个pod就创建失败了。
root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
Error from server (Forbidden): error when creating "STDIN": pods "podtest2" is forbidden: exceeded quota: myresourcequota, requested: pods=1, used: pods=2, limited: pods=2
查看resourcequota，可以看到pods: 2/2，资源配额满了。
root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME AGE REQUEST LIMIT
myresourcequota 5m48s pods: 2/2
删除pod。
root@k8scludes1:~/admissioncontr# kubectl delete pod podtest podtest1
pod "podtest" deleted
pod "podtest1" deleted
5.3 禁用ResourceQuota
resourcequota 资源配额是一种准入控制器插件，默认是启用的。我们可以把resourcequota禁用了。
修改kube-apiserver.yaml文件，禁用resourcequota 资源配额。
disable-admission-plugins=ResourceQuota表示禁用准入控制器插件ResourceQuota。
root@k8scludes1:~/admissioncontr# vim /etc/kubernetes/manifests/kube-apiserver.yaml
root@k8scludes1:~/admissioncontr# grep disable-admission-plugins /etc/kubernetes/manifests/kube-apiserver.yaml
- --disable-admission-plugins=ResourceQuota
重启kubelet使配置生效。
root@k8scludes1:~/admissioncontr# systemctl restart kubelet
现在resourcequota 资源配额还在，还是要求当前命名空间只能创建2个pod。
root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME AGE REQUEST LIMIT
myresourcequota 16m pods: 0/2
一口气创建了5个pod还是没有报错。
root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml
pod/podtest created
root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created
root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
pod/podtest2 created
root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest3/' pod.yaml | kubectl apply -f -
pod/podtest3 created
root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest4/' pod.yaml | kubectl apply -f -
pod/podtest4 created
root@k8scludes1:~/admissioncontr# kubectl get pod
NAME READY STATUS RESTARTS AGE
podtest 1/1 Running 0 30s
podtest1 1/1 Running 0 21s
podtest2 1/1 Running 0 16s
podtest3 1/1 Running 0 11s
podtest4 1/1 Running 0 6s
可以发现，禁用resourcequota功能之后，就算有resourcequota对象存在，资源配额也不生效。
root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME AGE REQUEST LIMIT
myresourcequota 3m59s pods: 5/2
删除pod。
root@k8scludes1:~# kubectl delete pod podtest podtest1 podtest2 podtest3 podtest4
pod "podtest" deleted
pod "podtest1" deleted
pod "podtest2" deleted
pod "podtest3" deleted
pod "podtest4" deleted
root@k8scludes1:~# kubectl get pod
No resources found in admissioncontr namespace.
编辑kube-apiserver.yaml，去掉disable-admission-plugins=ResourceQuota，让ResourceQuota默认启用。
root@k8scludes1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml
root@k8scludes1:~# grep disable-admission-plugins /etc/kubernetes/manifests/kube-apiserver.yaml
#- --disable-admission-plugins=ResourceQuota
重启kubelet使配置生效。
root@k8scludes1:~# systemctl restart kubelet
现在resourcequota 资源配额还是要求当前命名空间只能创建2个pod。
root@k8scludes1:~# kubectl get resourcequota
NAME AGE REQUEST LIMIT
myresourcequota 5h31m pods: 0/2
现在ResourceQuota资源配额又生效了，只能创建2个pod。
root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml
pod/podtest created
root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created
root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
Error from server (Forbidden): error when creating "STDIN": pods "podtest2" is forbidden: exceeded quota: myresourcequota, requested: pods=1, used: pods=2, limited: pods=2
删除pod。
root@k8scludes1:~/admissioncontr# kubectl delete pod podtest podtest1
pod "podtest" deleted
pod "podtest1" deleted
删除ResourceQuota资源配额。
root@k8scludes1:~/admissioncontr# kubectl delete resourcequota myresourcequota
resourcequota "myresourcequota" deleted
root@k8scludes1:~/admissioncontr# kubectl get resourcequotas
No resources found in admissioncontr namespace.
六.配置ImagePolicyWebhook准入控制器禁止使用后缀为latest的镜像
6.1 搭建Webhook服务器
ImagePolicyWebhook的类别为验证。ImagePolicyWebhook 准入控制器允许使用后端 Webhook 做出准入决策。此准入控制器默认被禁用。
准入 Webhook 是一种用于接收准入请求并对其进行处理的 HTTP 回调机制。 可以定义两种类型的准入 webhook，即验证性质的准入 Webhook 和 修改性质的准入 Webhook。 修改性质的准入 Webhook 会先被调用。它们可以更改发送到 API 服务器的对象，以执行自定义的设置默认值操作。在完成了所有对象修改并且 API 服务器也验证了所传入的对象之后， 验证性质的 Webhook 会被调用，并通过拒绝请求的方式来强制实施自定义的策略。
说明： 如果准入 Webhook 需要保证它们所看到的是对象的最终状态以实施某种策略。 则应使用验证性质的准入 Webhook，因为对象被修改性质 Webhook 看到之后仍然可能被修改。
首先需要找一台机器搭建Webhook服务器，我们使用etcd2机器当Webhook服务器。
首先需要安装docker。
[root@etcd2 ~]# yum -y install docker-ce
查看docker版本。
[root@etcd2 ~]# docker -v
Docker version 20.10.12, build e91ed57
配置docker镜像加速器。
[root@etcd2 ~]# cat /etc/docker/daemon.json
{
"registry-mirrors": [
"https://frz7i079.mirror.aliyuncs.com"
]
}
拉取flavio/kube-image-bouncer镜像，flavio/kube-image-bouncer这个镜像的功能为：不允许使用后缀为latest的镜像。
[root@etcd2 ~]# docker pull flavio/kube-image-bouncer
Using default tag: latest
latest: Pulling from flavio/kube-image-bouncer
ff3a5c916c92: Pull complete
4947cf5a98cf: Pull complete
37ff781c0e4d: Pull complete
1545c1d26daf: Pull complete
dd581c9cf10b: Pull complete
Digest: sha256:01e1e0873d20cee1ffefb45421c798cb26250b7a9384978d34ffb1f57403d501
Status: Downloaded newer image for flavio/kube-image-bouncer:latest
docker.io/flavio/kube-image-bouncer:latest
查看flavio/kube-image-bouncer镜像历史信息，可以发现开放端口为1323(EXPOSE 1323)，使用的是web账号登录(USER web)。关于镜像Dockerfile的详细信息，请查看博客《构建自定义镜像并优化dockerfile文件》。
[root@etcd2 ~]# docker history flavio/kube-image-bouncer
IMAGE CREATED CREATED BY SIZE COMMENT
3974e07cc0c8 3 years ago /bin/sh -c #(nop) EXPOSE 1323 0B
<missing> 3 years ago /bin/sh -c #(nop) ENTRYPOINT ["./kube-image… 0B
<missing> 3 years ago /bin/sh -c #(nop) USER web 0B
<missing> 3 years ago /bin/sh -c chown -R web:web * 19.9MB
<missing> 3 years ago /bin/sh -c #(nop) COPY file:de99d16a3731b75b… 19.9MB
<missing> 3 years ago /bin/sh -c adduser -h /app -D web 4.79kB
<missing> 3 years ago /bin/sh -c #(nop) WORKDIR /app 0B
<missing> 4 years ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B
<missing> 4 years ago /bin/sh -c #(nop) ADD file:093f0723fa46f6cdb… 4.15MB
使用flavio/kube-image-bouncer镜像创建容器。
-vpwd/webhook-key.pem:/certs/webhook-key.pem:ro -v pwd/webhook.pem:/certs/webhook.pem:ro 表示做目录映射（-v 本地目录：容器目录），webhook-key.pem公钥和webhook.pem私钥不存在也没问题，ro表示容器只有只读权限。
-p 1323:1323表示做端口映射（物理机端口1323：容器端口1323）。关于容器更多详细信息，请查看博客《一文搞懂docker容器基础：docker镜像管理，docker容器管理》。
[root@etcd2 ~]# docker run -dit --name=c1 --restart=always -v `pwd`/webhook-key.pem:/certs/webhook-key.pem:ro -v `pwd`/webhook.pem:/certs/webhook.pem:ro -p 1323:1323 flavio/kube-image-bouncer
7c677fb67522073932617f07ad72cd1a17e60eddc6a68dd25ffe0b8c4f80aaae
容器创建成功了，这时候访问192.168.110.131:1323 就访问到该webhook服务器了。
[root@etcd2 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7c677fb67522 flavio/kube-image-bouncer "./kube-image-bouncer" 10 seconds ago Up 8 seconds 0.0.0.0:1323->1323/tcp, :::1323->1323/tcp c1
6.2 配置kubernetes连接后端webhook服务器
回到kubernetes集群。
root@k8scludes1:~/admissioncontr# cd /etc/kubernetes/
root@k8scludes1:/etc/kubernetes# ls
admin.conf admission-control-config-file controller-manager.conf kubelet.conf manifests pki scheduler.conf
root@k8scludes1:/etc/kubernetes# cd admission-control-config-file/
root@k8scludes1:/etc/kubernetes/admission-control-config-file# ls
admission_configuration.json apiserver-client-key.pem apiserver-client.pem kubeconfig.yaml webhook-key.pem webhook.pem
root@k8scludes1:/etc/kubernetes/admission-control-config-file# pwd
/etc/kubernetes/admission-control-config-file
ImagePolicyWebhook 使用配置文件来为后端行为设置选项。该文件可以是 JSON 或 YAML。
admission_configuration.json是ImagePolicyWebhook 的配置文件，参数解释如下：
- allowTTL以秒计的时长，控制批准请求的缓存时间；
- denyTTL以秒计的时长，控制拒绝请求的缓存时间；
- retryBackoff以毫秒计的时长，控制重试间隔；
- defaultAllow确定 Webhook 后端失效时的行为；
- kubeConfigFile指定kubeconfig文件的路径。
root@k8scludes1:/etc/kubernetes/admission-control-config-file# vim admission_configuration.json
root@k8scludes1:/etc/kubernetes/admission-control-config-file# cat admission_configuration.json
{
"imagePolicy": {
"kubeConfigFile": "/etc/kubernetes/admission-control-config-file/kubeconfig.yaml",
"allowTTL": 50,
"denyTTL": 50,
"retryBackoff": 500,
"defaultAllow": true
}
}
root@k8scludes1:/etc/kubernetes/admission-control-config-file# ls /etc/kubernetes/admission-control-config-file/admission_configuration.json
/etc/kubernetes/admission-control-config-file/admission_configuration.json
kubeconfig.yaml用来设置与后端Webhook服务器的连接，要求后端使用 TLS 进行通信。
kubeconfig.yaml文件的 clusters 字段需要指向远端Webhook服务，server指定webhook服务器进行连接。
users 字段需要包含已返回的授权者。关于kubeconfig文件的详细信息，请查看博客《Kubernetes(k8s)访问控制：身份认证》。
root@k8scludes1:/etc/kubernetes/admission-control-config-file# vim kubeconfig.yaml
#指定各种证书
root@k8scludes1:/etc/kubernetes/admission-control-config-file# cat kubeconfig.yaml
apiVersion: v1
kind: Config
clusters:
- cluster:
# CA 用于验证远程服务
certificate-authority: /etc/kubernetes/admission-control-config-file/webhook.pem
# 要查询的远程服务的 URL
server: http://192.168.110.131:1323/image_policy
name: bouncer_webhook
contexts:
- context:
cluster: bouncer_webhook
user: api-server
name: bouncer_validator
current-context: bouncer_validator
preferences: {}
users:
- name: api-server
user:
# Webhook 准入控制器使用的证书
client-certificate: /etc/kubernetes/admission-control-config-file/apiserver-client.pem
# 证书匹配的密钥
client-key: /etc/kubernetes/admission-control-config-file/apiserver-client-key.pem
ImagePolicyWebhook默认是没有开启的，现在启用ImagePolicyWebhook。
enable-admission-plugins=NodeRestriction,ImagePolicyWebhook表示启用ImagePolicyWebhook准入控制器。
通过 --admission-control-config-file 参数指定准入控制器ImagePolicyWebhook配置文件的位置。
root@k8scludes1:/etc/kubernetes/admission-control-config-file# vim /etc/kubernetes/manifests/kube-apiserver.yaml
root@k8scludes1:/etc/kubernetes/admission-control-config-file# grep admission /etc/kubernetes/manifests/kube-apiserver.yaml
- --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
- --admission-control-config-file=/etc/kubernetes/admission-control-config-file/admission_configuration.json
重启kubelet使配置生效。
root@k8scludes1:/etc/kubernetes/admission-control-config-file# systemctl restart kubelet
可以发现kube-apiserver.yaml添加参数之后，连接不上集群了。
root@k8scludes1:/etc/kubernetes/admission-control-config-file# kubectl get node
The connection to the server 192.168.110.128:6443 was refused - did you specify the right host or port?
下面开始排查问题，kubectl使用不了，那就只能使用docker排查问题了，使用docker查看容器，可以发现k8s_kube-apiserver容器处于退出状态。
root@k8scludes1:/etc/kubernetes/admission-control-config-file# cd
root@k8scludes1:~# docker ps -a | grep api
2aacfa2bdbc4 e64579b7d886 "kube-apiserver --ad…" 19 seconds ago Exited (1) 18 seconds ago k8s_kube-apiserver_kube-apiserver-k8scludes1_kube-system_054ae2c0df7fd5edd4ebb3b7057a0462_8
cb0e2e181dda registry.aliyuncs.com/google_containers/pause:3.5 "/pause" 4 minutes ago Up 4 minutes k8s_POD_kube-apiserver-k8scludes1_kube-system_054ae2c0df7fd5edd4ebb3b7057a0462_0
ac234a8ee92b e64579b7d886 "kube-apiserver --ad…" 4 minutes ago Exited (1) 4 minutes ago k8s_kube-apiserver_kube-apiserver-k8scludes1_kube-system_f38197908cec7cb32a42e0862e367c1c_0
49e9b9c1b389 registry.aliyuncs.com/google_containers/pause:3.5 "/pause" 4 minutes ago Up 4 minutes k8s_POD_kube-apiserver-k8scludes1_kube-system_f38197908cec7cb32a42e0862e367c1c_0
5d4985a8833c e64579b7d886 "kube-apiserver --ad…" 5 minutes ago Exited (1) 5 minutes ago k8s_kube-apiserver_kube-apiserver-k8scludes1_kube-system_3dc64835bcbf5bdf937660d07f41b90b_87
c674a6a6c794 registry.aliyuncs.com/google_containers/pause:3.5 "/pause" About an hour ago Up About an hour k8s_POD_kube-apiserver-k8scludes1_kube-system_3dc64835bcbf5bdf937660d07f41b90b_2
查看k8s_kube-apiserver容器日志，报错为：“open /etc/kubernetes/admission-control-config-file/admission_configuration.json: no such file or directory“，发现admission_configuration.json文件找不到。
root@k8scludes1:~# docker logs 2aacfa2bdbc4
I0616 01:39:59.927352 1 server.go:553] external host was not specified, using 192.168.110.128
I0616 01:39:59.927965 1 server.go:161] Version: v1.22.2
Error: failed to initialize admission: failed to read plugin config: unable to read admission control configuration from "/etc/kubernetes/admission-control-config-file/admission_configuration.json" [open /etc/kubernetes/admission-control-config-file/admission_configuration.json: no such file or directory]
/etc/kubernetes/admission-control-config-file/admission_configuration.json文件是存在的，但是报错。
原因为：/etc/kubernetes/admission-control-config-file/admission_configuration.json这个文件是在宿主机里的，并不在容器里。
解决方法：做一个数据卷，把宿主机里的文件映射到容器里。
root@k8scludes1:~# ls /etc/kubernetes/admission-control-config-file/admission_configuration.json
/etc/kubernetes/admission-control-config-file/admission_configuration.json
注意：生产环境里可以先备份下kube-apiserver.yaml 文件再修改，以免改不回来。
定义一个数据卷把宿主机的/etc/kubernetes/admission-control-config-file目录挂载到容器/etc/kubernetes/admission-control-config-file目录。
root@k8scludes1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml
root@k8scludes1:~# grep -A3 admission-control-config-file /etc/kubernetes/manifests/kube-apiserver.yaml
- --admission-control-config-file=/etc/kubernetes/admission-control-config-file/admission_configuration.json
#- --disable-admission-plugins=ResourceQuota
#- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy
- --token-auth-file=/etc/kubernetes/pki/mytok.csv
--
- mountPath: /etc/kubernetes/admission-control-config-file
name: admissionconfile
readOnly: true
hostNetwork: true
--
path: /etc/kubernetes/admission-control-config-file
type: DirectoryOrCreate
name: admissionconfile
- hostPath:
直接上截图可能更直观：
重启kubelet使配置生效。
root@k8scludes1:~# systemctl restart kubelet
现在k8s正常了。
root@k8scludes1:~# kubectl get pod
No resources found in admissioncontr namespace.
6.3 验证
编辑pod配置文件，表示使用hub.c.163.com/library/nginx:latest镜像创建pod。
root@k8scludes1:~# cd admissioncontr/
root@k8scludes1:~/admissioncontr# vim pod.yaml
root@k8scludes1:~/admissioncontr# cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: podtest
name: podtest
spec:
#当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
terminationGracePeriodSeconds: 0
containers:
- image: hub.c.163.com/library/nginx:latest
#imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
imagePullPolicy: IfNotPresent
name: podtest
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {}
创建pod，可以发现：使用tag为latest的镜像不能创建pod。
root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml
Error from server (Forbidden): error when creating "pod.yaml": pods "podtest" is forbidden: image policy webhook backend denied one or more images: Images using latest tag are not allowed
修改pod配置文件，表示使用hub.c.163.com/library/nginx:test镜像创建pod。
root@k8scludes1:~/admissioncontr# vim pod.yaml
root@k8scludes1:~/admissioncontr# cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: podtest
name: podtest
spec:
#当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
terminationGracePeriodSeconds: 0
containers:
- image: hub.c.163.com/library/nginx:test
#- image: hub.c.163.com/library/nginx:latest
#imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
imagePullPolicy: IfNotPresent
name: podtest
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {}
镜像tag为test，pod创建成功。
root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml
pod/podtest created
root@k8scludes1:~/admissioncontr# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
podtest 1/1 Running 0 6s 10.244.218.145 k8scludes2 <none> <none>
删除pod。
root@k8scludes1:~/admissioncontr# kubectl delete pod podtest
pod "podtest" deleted
七.总结
准入控制器是Kubernetes中一个非常重要的组件，它负责拦截和处理来自用户或其他应用程序的API请求。通过使用准入控制器，我们可以实现对集群资源的细粒度控制，从而提高集群的安全性和可靠性。
