admission webhook 初探（编译篇）
编译开源的 admission webhook
本文参考的代码 admission-webhook-example，k8s-admission-webhook 文档
admission webhook 的作用我简单的总结下，当用户的请求到达 k8s apiserver 后，apiserver 根据 MutatingWebhookConfiguration 和 ValidatingWebhookConfiguration 的配置，先调用 MutatingWebhookConfiguration 去修改用户请求的配置文件，最后会调用 ValidatingWebhookConfiguration 来验证这个修改后的配置文件是否合法。
我们可以利用 mutating 的机制，将一些特殊的配置自动加上，而不用用户来操心。同时也可以在 validating 中编写代码设置自己的规则，看请求是否合法。
因为已经研究过怎么编写 webhook 代码/怎么部署 webhook，要使用好 webhook 需要注意几个点：
- webhook 的本质是个 http server，因此，需要用代码实现这么一个 server，供 apiserver 调用
- mutating webhook 和 validating webhook 分别实现了不同的功能，这两者没有直接联系，本质上说，如果修改/验证都需要，我们需要编写两个 http server
- 编写好的 webhook，建议使用 k8s deployment 部署，同时使用 service 来暴露能力
- apiserver 是否调用 webhook，是由
k8s MutatingWebhookConfiguration/ValidatingWebhookConfiguration来定义的 - 需要注意 apiserver 和 webhook 通信时的身份认证和权限控制问题
编译示例 webhook
https://github.com/cnych/admission-webhook-example 项目中将 mutating、validating 的功能都放到一个 http server 中了，这当然也是可以的。但是 k8s MutatingWebhookConfiguration/ValidatingWebhookConfiguration 肯定是需要都部署的。
[root@localhost lihao04]# cd /root/lihao04
[root@localhost lihao04]# git clone https://github.com/cnych/admission-webhook-example.git
[root@localhost lihao04]# cd admission-webhook-example/
[root@localhost github.com]# export GOPATH=/root/lihao04/go/
[root@localhost github.com]# mkdir -p /root/lihao04/go/src/github.com/cnych/
[root@localhost github.com]# ln -s /root/lihao04/admission-webhook-example /root/lihao04/go/src/github.com/cnych/
[root@localhost github.com]# export http_proxy=http://jarvis:lihao-root@10.191.67.135:3128/
[root@localhost github.com]# export https_proxy=http://jarvis:lihao-root@10.191.67.135:3128/
[root@localhost github.com]# cd /root/lihao04/go/src/github.com/cnych/admission-webhook-example
[root@localhost admission-webhook-example]# dep ensure -v
...
[root@localhost admission-webhook-example]# export GO111MODULE=on
[root@localhost admission-webhook-example]# CGO_ENABLED=0 GOOS=linux go build -mod=vendor -a -installsuffix cgo -o admission-webhook-example
go: creating new go.mod: module github.com/cnych/admission-webhook-example
go: copying requirements from Gopkg.lock
go: converting Gopkg.lock: stat github.com/json-iterator/go@f2b4162afba35581b6d4a50d3b8f34e33c144682: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat golang.org/x/text@v0.3.0: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/google/gofuzz@24818f796faf91cd76ec7bddd72458fbced7a6c1: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/api@12444147eb1150aa5c80d2aae532cbc5b7be73d0: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/opencontainers/go-digest@v1.0.0-rc1: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/golang/glog@23def4e6c14b4da8ac2ed8007337bc5eb5007998: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/modern-go/reflect2@1df9eeb2bb81f327b96228865c5687bc2194af3f: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/apimachinery@e386b2658ed20923da8cc9250e552f082899a1ee: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/apiserver@88d4601515c27f180f7efc8705e4cc18dc19100d: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat golang.org/x/net@892bf7b0c6e2f93b51166bf3882e50277fa5afc6: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat gopkg.in/yaml.v2@v2.1.1: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/gogo/protobuf@v1.0.0: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/spf13/pflag@v1.0.0: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/docker/distribution@34c706e759240975178df82495f147559cc0edc1: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/modern-go/concurrent@bacd9c7ef1dd9b15be4a9909b8ac7a4e313eec94: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/apiextensions-apiserver@f584b16eb23bd2a3fd292a027d698d95db427c5d: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/ghodss/yaml@v1.0.0: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/kubernetes@efe960cdc41ee7b18d408128dfb80babb5bc746a: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat gopkg.in/inf.v0@v0.9.0: repo version lookup disabled by -mod=vendor
制作镜像
将编译后的代码打包进 image，这样就可以使用 k8s deployment 来部署
Dockerfile
注意修改 FROM 的
FROM alpine:3.10.3
ADD admission-webhook-example /admission-webhook-example
ENTRYPOINT ["./admission-webhook-example"]
制作命令
[root@localhost admission-webhook-example]# docker build --no-cache -t docker-registry.lihao04.virtual/jarvis-image/admission-webhook-example:v1 .
Sending build context to Docker daemon 43MB
Step 1/3 : FROM docker-registry.lihao04.virtual/jarvis-image/alpine:3.10.3
3.10.3: Pulling from jarvis-image/alpine
89d9c30c1d48: Already exists
Digest: sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a
Status: Downloaded newer image for docker-registry.lihao04.virtual/jarvis-image/alpine:3.10.3
---> 965ea09ff2eb
Step 2/3 : ADD admission-webhook-example /admission-webhook-example
---> 2c65bd92673c
Step 3/3 : ENTRYPOINT ["./admission-webhook-example"]
---> Running in 41ac87194f86
Removing intermediate container 41ac87194f86
---> 7c57f014ab9a
Successfully built 7c57f014ab9a
Successfully tagged docker-registry.lihao04.virtual/admission-webhook-example:v1
k8s 的配置
启用 MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook
MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook 默认是不启用的，apiserver 想调用 webhook，还得 enable 相关的能力
[root@master ~]# kubectl get pods kube-apiserver-master.lihao04.virtual -n kube-system -o yaml|grep enable
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
enableServiceLinks: true
因为 enable-admission-plugins 缺失 feature，我们要 enable
# 修改 /etc/kubernetes/manifests/kube-apiserver.yaml
- --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
修改配置文件后立刻生效
[root@master manifests]# kubectl get pods kube-apiserver-master.lihao04.virtual -n kube-system -o yaml|grep enable
- --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
- --enable-bootstrap-token-auth=true
enableServiceLinks: true
# 确实重启过
[root@master manifests]# kubectl get pod -n kube-system|grep api
kube-apiserver-master.lihao04.virtual 1/1 Running 0 54s
部署 webhook
prepare
这个 server 是供 apiserver 调用的，它的运行代码正是我们编译的 admission-webhook-example，他使用的镜像正是我们构建的 docker-registry.lihao04.virtual/admission-webhook-example:v1
部署前，需要先创建 CertificateSigningRequest、secret 等，这个是给 apiserver 与 webhook 通信做身份认证的
[root@master lihao04]# cd /root/lihao04/admission-webhook-example/deployment
[root@master lihao04]# ./webhook-create-signed-cert.sh
部署 webhook server
[root@master deployment]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: admission-webhook-example-deployment
labels:
app: admission-webhook-example
spec:
replicas: 1
selector:
matchLabels:
app: admission-webhook-example
template:
metadata:
labels:
app: admission-webhook-example
spec:
containers:
- name: admission-webhook-example
image: docker-registry.lihao04.virtual/jarvis-image/admission-webhook-example:v2
#image: cnych/admission-webhook-example:v1
args:
- -tlsCertFile=/etc/webhook/certs/cert.pem
- -tlsKeyFile=/etc/webhook/certs/key.pem
- -alsologtostderr
- -v=4
- 2>&1
volumeMounts:
- name: webhook-certs
mountPath: /etc/webhook/certs
readOnly: true
volumes:
- name: webhook-certs
secret:
secretName: admission-webhook-example-certs
[root@master deployment]# kubectl apply -f deployment.yaml
deployment.apps/admission-webhook-example-deployment create
部署 service
让该 deployment(webhook server) 能被 apiserver 访问，必须套一个 service，注意，访问 service 的 443 会访问 webhook deployment 中的 pod 的 443 端口，而 pod 中的服务正是启动在 443 端口上，这个大家可以看 admission-webhook-example 源码。
[root@master deployment]# cat service.yaml
apiVersion: v1
kind: Service
metadata:
name: admission-webhook-example-svc
labels:
app: admission-webhook-example
spec:
ports:
- port: 443
targetPort: 443
selector:
app: admission-webhook-example
[root@master deployment]# kubectl create -f service.yaml
在 k8s 中使能 webhook
当 webhook server 部署好了之后，apiserver 如何知道它要访问 webhook server？
因此需要配置 k8s 集群，让 webhook 被 k8s 感知到，并能在适当的时候调用到。
因为 webhook 有两个部分的功能
- 修改
- 验证
修改在先，验证在后，因此这两个功能都需要独立配置；
配置 ValidatingWebhookConfiguration
配置名为：ValidatingWebhookConfiguration
[root@master deployment]# cat validatingwebhook-ca-bundle.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
name: validation-webhook-example-cfg
labels:
app: admission-webhook-example
webhooks:
- name: required-labels.qikqiak.com
clientConfig:
service:
name: admission-webhook-example-svc
namespace: default
path: "/validate"
# 通过 kubectl config view --raw --flatten -o json|grep certificate-authority-data 获得
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1Ea3lOekEzTVRZd04xb1hEVEk1TURreU5EQTNNVFl3TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUlFCmZnRDJva2RpK3BiK2U5VmVMRFJaamxQRG9UTGpwS1NVc05OaHVSbW9Xbmoyek9od3J5cGRmd2V3QTJtdlYyVGIKdDFKYTRHSENWRmNZeWVveFE2REw5SkVLWGZoOGxKQkZxS0JuNzc3RFlFWXl5b1hqWjE5Wk5NUUZzUnZhL2RtRgpPVGlrR3ZIdDZqS0xiQmhqVlRhNTlPbmhXZTFiRUlGR0I3SXNlcG8vRXEwSDhPMW53UlVONEVYRnBPMXhiQktxCk5saS9IQ0FLZ2pWRlQrWjdpUjY3QXlMbzFUUk5JRkl1VkhDU2xRaERjaU9xTE1OM1FLeWN5ZnY2VXFsNllSeW8KeW42eGFkb1JQM2RJOUlnNWpJQ3c0dUNZeEJlbW14YnNURkQ5Tm90YkdqSnBNeVhKbGhjcGM1aVlUVDR3WUJMUwpRakNkcWlJYmNzODNSWkhqTW1jQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMeTN4ZWZFYmtUaUgxczVmbHlMVzZEb08wNUgKdWMrUW83RmdnSmlqZjA3eFNwVXpYVTRiVjhtSk55RlFJS1BiRzhHa3dRRFhiVzMvYXlZa2gyT3Z1Si9Bb2U5dAoxeGlFL3NMbTlGeTdlMTJRenZFcjlsanVpMzJWSjBtYkRpVm9Sd3FEMFh0R1JnZGhVeGltdU9PZzd0aXd1WUtkCkhBY0NDdUNReHRnWXpuQXdnVnJYVWpaSnRRV2RoNUpiVUwrZGp6ejYxSVdpTytiQSt2c0d3cjRoV09SVlk2K0sKcFNIMlhiL2JIaG5XOHBSREdsbEl2U2piZmlzY0d6SU1tUHN5end4ZG9sRXlqcnpCdGdLaW5pcjJvZmdJcEVHQwpuUzdZSDR3WE1kTkp4TEQ3U2JWdWpDbEwwQzhaUWFvamFkNUlkU2FZaldjSHFKelFmczRaKzQvRHN1OD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
rules:
- operations: [ "CREATE" ]
apiGroups: ["apps", ""]
apiVersions: ["v1"]
resources: ["deployments","services"]
# 对 namespace 中打了 admission-webhook-example = enabled 的进行验证，其他 namespace 不验证
namespaceSelector:
matchLabels:
admission-webhook-example: enabled
[root@master deployment]# kubectl apply -f validatingwebhook-ca-bundle.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io/validation-webhook-example-cfg created
测试
[root@master deployment]# cat sleep.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: sleep
spec:
replicas: 3
selector:
matchLabels:
app: sleep
template:
metadata:
labels:
app: sleep
spec:
containers:
- name: sleep
image: tutum/curl
command: ["/bin/sleep","infinity"]
imagePullPolicy: IfNotPresent
[root@master deployment]# kubectl apply -f sleep.yaml
Error from server (required labels are not set): error when creating "sleep.yaml": admission webhook "required-labels.qikqiak.com" denied the request: required labels are not set
可以发现，如果没有带特定的内容，会报错，达到了验证的效果
配置 MutatingWebhookConfiguration
配置名为：MutatingWebhookConfiguration
[root@master deployment]# cat mutatingwebhook-ca-bundle.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
name: mutating-webhook-example-cfg
labels:
app: admission-webhook-example
webhooks:
- name: mutating-example.qikqiak.com
clientConfig:
service:
name: admission-webhook-example-svc
namespace: default
path: "/mutate"
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1Ea3lOekEzTVRZd04xb1hEVEk1TURreU5EQTNNVFl3TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUlFCmZnRDJva2RpK3BiK2U5VmVMRFJaamxQRG9UTGpwS1NVc05OaHVSbW9Xbmoyek9od3J5cGRmd2V3QTJtdlYyVGIKdDFKYTRHSENWRmNZeWVveFE2REw5SkVLWGZoOGxKQkZxS0JuNzc3RFlFWXl5b1hqWjE5Wk5NUUZzUnZhL2RtRgpPVGlrR3ZIdDZqS0xiQmhqVlRhNTlPbmhXZTFiRUlGR0I3SXNlcG8vRXEwSDhPMW53UlVONEVYRnBPMXhiQktxCk5saS9IQ0FLZ2pWRlQrWjdpUjY3QXlMbzFUUk5JRkl1VkhDU2xRaERjaU9xTE1OM1FLeWN5ZnY2VXFsNllSeW8KeW42eGFkb1JQM2RJOUlnNWpJQ3c0dUNZeEJlbW14YnNURkQ5Tm90YkdqSnBNeVhKbGhjcGM1aVlUVDR3WUJMUwpRakNkcWlJYmNzODNSWkhqTW1jQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMeTN4ZWZFYmtUaUgxczVmbHlMVzZEb08wNUgKdWMrUW83RmdnSmlqZjA3eFNwVXpYVTRiVjhtSk55RlFJS1BiRzhHa3dRRFhiVzMvYXlZa2gyT3Z1Si9Bb2U5dAoxeGlFL3NMbTlGeTdlMTJRenZFcjlsanVpMzJWSjBtYkRpVm9Sd3FEMFh0R1JnZGhVeGltdU9PZzd0aXd1WUtkCkhBY0NDdUNReHRnWXpuQXdnVnJYVWpaSnRRV2RoNUpiVUwrZGp6ejYxSVdpTytiQSt2c0d3cjRoV09SVlk2K0sKcFNIMlhiL2JIaG5XOHBSREdsbEl2U2piZmlzY0d6SU1tUHN5end4ZG9sRXlqcnpCdGdLaW5pcjJvZmdJcEVHQwpuUzdZSDR3WE1kTkp4TEQ3U2JWdWpDbEwwQzhaUWFvamFkNUlkU2FZaldjSHFKelFmczRaKzQvRHN1OD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
rules:
- operations: [ "CREATE" ]
apiGroups: ["apps", ""]
apiVersions: ["v1"]
resources: ["deployments","services"]
namespaceSelector:
matchLabels:
admission-webhook-example: enabled
[root@master deployment]# kubectl apply -f mutatingwebhook-ca-bundle.yaml
mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating-webhook-example-cfg created
测试
同样的 sleep 任务
[root@master deployment]# kubectl apply -f sleep.yaml
deployment.apps/sleep created
可以创建出来了，并且可以看到，增加了很多附加的 label，因为信息被添加，所以 validation 也通过了
[root@master deployment]# kubectl describe deployment sleep
Name: sleep
Namespace: default
CreationTimestamp: Fri, 25 Oct 2019 19:17:51 +0800
Labels: app.kubernetes.io/component=not_available
app.kubernetes.io/instance=not_available
app.kubernetes.io/managed-by=not_available
app.kubernetes.io/name=not_available
app.kubernetes.io/part-of=not_available
app.kubernetes.io/version=not_available
Annotations: admission-webhook-example.qikqiak.com/status: mutated
deployment.kubernetes.io/revision: 1
Selector: app=sleep
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=sleep
Containers:
sleep:
Image: tutum/curl
Port: <none>
Host Port: <none>
Command:
/bin/sleep
infinity
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: sleep-674f75ff4d (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 57s deployment-controller Scaled up replica set sleep-674f75ff4d to 3
遇到的问题
- webhook server 验证不起作用，发现是 bad certificate
[root@master deployment]# kubectl log admission-webhook-example-deployment-5594df959f-6rjql
log is DEPRECATED and will be removed in a future version. Use logs instead.
I1025 09:32:21.348823 1 main.go:50] Server started
2019/10/25 09:33:01 http: TLS handshake error from 10.244.0.0:10131: remote error: tls: bad certificate
发现，参考文档中，下面步骤有问题
cat ./deployment/validatingwebhook.yaml | ./deployment/webhook-patch-ca-bundle.sh > ./deployment/validatingwebhook-ca-bundle.yaml
