Kubernetes 审计（Auditing）
目录
- 一.系统环境
- 二.前言
- 三.Kubernetes 审计简介
- 四.审计策略简介
- 五.启用审计
- 六.审计策略
- 6.1 记录审计阶段为：ResponseStarted，审计级别为Metadata，apiVersion为group: "" 的日志
- 6.2 只记录audit命名空间里的日志
- 6.3 只记录audit命名空间的pods操作日志
- 6.4 只记录audit命名空间的pods,services,deployments操作日志
- 6.5 只记录audit命名空间的pods操作,审计级别为RequestResponse
- 6.6 只记录audit命名空间下的tom用户的pods操作，其他用户操作不记录
- 6.7 rules规则是从上往下匹配的，第一条规则已经匹配了，第二条就不匹配了
- 6.8 在 Metadata 级别为所有请求生成日志
- 七.总结
一.系统环境
本文主要基于Kubernetes1.22.2和Linux操作系统Ubuntu 18.04。
| 服务器版本 | docker软件版本 | Kubernetes(k8s)集群版本 | CPU架构 |
|---|---|---|---|
| Ubuntu 18.04.5 LTS | Docker version 20.10.14 | v1.22.2 | x86_64 |
Kubernetes集群架构：k8scludes1作为master节点，k8scludes2，k8scludes3作为worker节点。
| 服务器 | 操作系统版本 | CPU架构 | 进程 | 功能描述 |
|---|---|---|---|---|
| k8scludes1/192.168.110.128 | Ubuntu 18.04.5 LTS | x86_64 | docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico | k8s master节点 |
| k8scludes2/192.168.110.129 | Ubuntu 18.04.5 LTS | x86_64 | docker，kubelet，kube-proxy，calico | k8s worker节点 |
| k8scludes3/192.168.110.130 | Ubuntu 18.04.5 LTS | x86_64 | docker，kubelet，kube-proxy，calico | k8s worker节点 |
二.前言
在当今的云计算时代，容器编排技术已成为企业部署和管理应用程序的关键组成部分。Kubernetes 作为最流行的容器编排平台，提供了强大的功能来部署、管理和扩展容器化应用程序。然而，随着应用程序的日益复杂和部署环境的扩大，确保 Kubernetes 集群的安全性变得越来越重要。在这篇博客中，我们将深入了解 Kubernetes 审计的概念，并探讨如何在 Kubernetes 环境中实施审计策略。
使用Kubernetes 审计（Auditing）的前提是已经有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Ubuntu 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/17632858.html。
三.Kubernetes 审计简介
Kubernetes 审计（Auditing） 功能提供了与安全相关的、按时间顺序排列的记录集， 记录每个用户、使用 Kubernetes API 的应用以及控制面自身引发的活动。
审计功能使得集群管理员能够回答以下问题：
- 发生了什么？
- 什么时候发生的？
- 谁触发的？
- 活动发生在哪个（些）对象上？
- 在哪观察到的？
- 它从哪触发的？
- 活动的后续处理行为是什么？
审计记录最初产生于 kube-apiserver 内部。每个请求在不同执行阶段都会生成审计事件；这些审计事件会根据特定策略被预处理并写入后端。 策略确定要记录的内容和用来存储记录的后端，当前的后端支持日志文件和 webhook。
每个请求都可被记录其相关的阶段（stage），阶段（stage）可以理解为什么时候记录。。已定义的阶段有：
RequestReceived- 此阶段对应审计处理器接收到请求后， 并且在委托给其余处理器之前生成的事件。ResponseStarted- 在响应消息的头部发送后，响应消息体发送前生成的事件。 只有长时间运行的请求（例如 watch）才会生成这个阶段。ResponseComplete- 当响应消息体完成并且没有更多数据需要传输的时候。Panic- 当 panic 发生时生成。
审计日志记录功能会增加 API server 的内存消耗，因为需要为每个请求存储审计所需的某些上下文。 内存消耗取决于审计日志记录的配置。
Kubernetes 审计是一种监控和记录 Kubernetes 集群中资源操作的方法，用于确保集群的安全性和符合性。通过审计，管理员可以跟踪对集群资源的访问和修改，以便在发生安全事件时进行调查和响应。Kubernetes 提供了审计日志记录的框架，允许管理员自定义审计策略，以确定哪些资源操作应该被记录。
四.审计策略简介
审计策略定义了关于应记录哪些事件以及应包含哪些数据的规则。 审计策略对象结构定义在 audit.k8s.io API 组。 处理事件时，将按顺序与规则列表进行比较。第一个匹配规则设置事件的审计级别（Audit Level），审计级别（Audit Level）可以理解为记录什么？ 已定义的审计级别有：
- None - 符合这条规则的日志将不会记录。
- Metadata - 记录请求的元数据（请求的用户、时间戳、资源、动词等等）， 但是不记录请求或者响应的消息体。
- Request - 记录事件的元数据和请求的消息体，但是不记录响应的消息体。 这不适用于非资源类型的请求。
- RequestResponse - 记录事件的元数据，请求和响应的消息体。这不适用于非资源类型的请求。
你可以使用 --audit-policy-file 标志将包含策略的文件传递给 kube-apiserver。 如果不设置该标志，则不记录事件。 注意 rules 字段必须在审计策略文件中提供。没有（0）规则的策略将被视为非法配置。
审计策略定义了哪些资源操作应该被审计以及审计记录的格式。在 Kubernetes 中，审计策略通过 Admission Controllers 实现，可以通过 Webhook 的方式进行集成。审计策略可以根据资源的类型、操作的类型和用户身份等信息进行过滤，以满足不同场景下的审计需求。
五.启用审计
5.1 引入审计
本次使用etcd2机器作为kubernetes集群的客户端，kctom是kubeconfig文件，etcd2机器使用kctom连接kubernetes集群，现在用户tom没有权限，需要先授权。kubeconfig文件在博客《Kubernetes(k8s)访问控制：身份认证》里已经详细介绍过了，这里不再赘述。
[root@etcd2 ~]# ls kc* -lh
-rw------- 1 root root 5.5K 5月 5 17:46 kctom
[root@etcd2 ~]# kubectl get node --kubeconfig=kctom
Error from server (Forbidden): nodes is forbidden: User "tom" cannot list resource "nodes" in API group "" at the cluster scope
去kubernetes集群给tom用户授予cluster-admin权限，也可以授予部分权限。关于授权的详细信息，请查看博客《Kubernetes(k8s)访问控制：权限管理之RBAC授权/鉴权》。
root@k8scludes1:~# kubectl create clusterrolebinding crbindtom --clusterrole=cluster-admin --user=tom
clusterrolebinding.rbac.authorization.k8s.io/crbindtom created
授予cluster-admin权限之后，客户端就具有权限了。
[root@etcd2 ~]# kubectl get node --kubeconfig=kctom
NAME STATUS ROLES AGE VERSION
k8scludes1 Ready control-plane,master 66d v1.22.2
k8scludes2 Ready <none> 65d v1.22.2
k8scludes3 Ready <none> 65d v1.22.2
编辑pod配置文件，表示使用nginx镜像创建pod。
root@k8scludes1:~# vim pod.yaml
root@k8scludes1:~# cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: podtest
name: podtest
spec:
#当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
terminationGracePeriodSeconds: 0
containers:
- image: hub.c.163.com/library/nginx:latest
#imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
imagePullPolicy: IfNotPresent
name: podtest
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {}
在admissioncontr命名空间创建一个pod。
root@k8scludes1:~# kubectl apply -f pod.yaml
pod/podtest created
在客户端etcd2机器可以看到这个pod。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n admissioncontr
NAME READY STATUS RESTARTS AGE
podtest 1/1 Running 0 34s
在客户端删除pod。
[root@etcd2 ~]# kubectl delete pod podtest --kubeconfig=kctom -n admissioncontr
pod "podtest" deleted
kubectl get ev （查看事件），可以看到pod被删除掉了，但是看不到pod被谁删除掉了！
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
LAST SEEN TYPE REASON OBJECT SUBOBJECT SOURCE MESSAGE FIRST SEEN COUNT NAME
3m37s Normal Pulled pod/podtest spec.containers{podtest} kubelet, k8scludes3 Container image "hub.c.163.com/library/nginx:latest" already present on machine 3m37s 1 podtest.16fab43e14342a3f
3m37s Normal Created pod/podtest spec.containers{podtest} kubelet, k8scludes3 Created container podtest 3m37s 1 podtest.16fab43e1db484f4
3m36s Normal Started pod/podtest spec.containers{podtest} kubelet, k8scludes3 Started container podtest 3m36s 1 podtest.16fab43e3315ada7
2m47s Normal Killing pod/podtest spec.containers{podtest} kubelet, k8scludes3 Stopping container podtest 2m47s 1 podtest.16fab449c468f9af
我们的k8s集群会被各种用户连接使用，我们想知道连接k8s集群的各个用户做了什么操作？应该怎么办？通过审计解决！
5.2 启用审计
创建audit目录存放yaml文件。
root@k8scludes1:~# mkdir audit
root@k8scludes1:~# cd audit/
root@k8scludes1:~/audit# pwd
/root/audit
创建命名空间audit。
root@k8scludes1:~/audit# kubectl create ns audit
namespace/audit created
切换到命名空间audit。
root@k8scludes1:~/audit# kubens audit
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "audit".
root@k8scludes1:~/audit# kubectl get pod
No resources found in audit namespace.
kubernetes默认并没有启用审计，需要启用审计功能。指定审计策略文件的路径：--audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml
root@k8scludes1:~# grep audit-policy-file /etc/kubernetes/manifests/kube-apiserver.yaml
- --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
创建审计策略文件，文件内容可以先为空。
root@k8scludes1:~# mkdir /etc/kubernetes/audit/
root@k8scludes1:~# cd /etc/kubernetes/audit/
root@k8scludes1:/etc/kubernetes/audit# touch audit-policy.yaml
root@k8scludes1:/etc/kubernetes/audit# ls /etc/kubernetes/audit/audit-policy.yaml
/etc/kubernetes/audit/audit-policy.yaml
指定审计日志存放的路径：--audit-log-path=/var/log/kubernetes/audit/audit.log 。审计日志格式为JSONlines 格式 。
root@k8scludes1:/etc/kubernetes/audit# vim /etc/kubernetes/manifests/kube-apiserver.yaml
root@k8scludes1:/etc/kubernetes/audit# grep audit-log-path /etc/kubernetes/manifests/kube-apiserver.yaml
- --audit-log-path=/var/log/kubernetes/audit/audit.log
创建审计日志文件。
root@k8scludes1:/etc/kubernetes/audit# mkdir -p /var/log/kubernetes/audit/
root@k8scludes1:/etc/kubernetes/audit# touch /var/log/kubernetes/audit/audit.log
root@k8scludes1:/etc/kubernetes/audit# ls /var/log/kubernetes/audit/audit.log
/var/log/kubernetes/audit/audit.log
审计日志参数如下：
- --audit-log-path 指定用来写入审计事件的日志文件路径。不指定此标志会禁用日志后端。- 意味着标准化
- --audit-log-maxage 定义保留旧审计日志文件的最大天数
- --audit-log-maxbackup 定义要保留的审计日志文件的最大数量
- --audit-log-maxsize 定义审计日志文件轮转之前的最大大小（兆字节）
配置审计日志的参数。
root@k8scludes1:/etc/kubernetes/audit# vim /etc/kubernetes/manifests/kube-apiserver.yaml
root@k8scludes1:/etc/kubernetes/audit# grep audit /etc/kubernetes/manifests/kube-apiserver.yaml
- --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
- --audit-log-path=/var/log/kubernetes/audit/audit.log
- --audit-log-maxage=365
- --audit-log-maxbackup=730
- --audit-log-maxsize=250
如果你的集群控制面以 Pod 的形式运行 kube-apiserver，需要通过 hostPath 数据卷来访问策略文件和日志文件所在的目录，这样审计记录才会持久保存下来。
现在/etc/kubernetes/audit/audit-policy.yaml和/var/log/kubernetes/audit/audit.log在宿主机里是存在的，但是kube-apiserver是以pod的方式运行的，pod里并不存在这两个文件，需要通过hostPath 数据卷进行映射。
/etc/kubernetes/manifests/kube-apiserver.yaml所有的修改如下截图：
注意：/var/log/kubernetes/audit/的readOnly为false，否则不能写入日志！
自此，成功启用审计了，下面开始配置审计策略。
六.审计策略
6.1 记录审计阶段为：ResponseStarted，审计级别为Metadata，apiVersion为group: "" 的日志
先不用重启kubelet，因为审计策略文件还没写好。
现在编写审计策略文件，该审计策略表示什么都不记录。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "ResponseStarted"
rules:
- level: None
resources:
- group: ""
重启kubelet使配置生效。
root@k8scludes1:~# systemctl restart kubelet
root@k8scludes1:~# kubectl get node
NAME STATUS ROLES AGE VERSION
k8scludes1 Ready control-plane,master 66d v1.22.2
k8scludes2 Ready <none> 66d v1.22.2
k8scludes3 Ready <none> 66d v1.22.2
在客户端etcd2机器执行命令：kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide。
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.
查看审计日志，没有任何日志记录。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
审计级别修改为Metadata，表示记录请求的元数据（请求的用户、时间戳、资源、动词等等）， 但是不记录请求或者响应的消息体。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "ResponseStarted"
rules:
- level: Metadata
resources:
- group: ""
重启kubelet使配置生效。
root@k8scludes1:~# systemctl daemon-reload ; systemctl restart kubelet
在客户端etcd2机器执行命令：kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide 。
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.
查看审计日志，发现还是没有日志产生，看来重启kubelet还不行，现在重启机器。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
现在直接重启k8scludes1机器，k8s worker节点不用重启。
root@k8scludes1:~# reboot
重启k8scludes1机器之后，现在/var/log/kubernetes/audit/audit.log里有很多数据，先删除数据，然后客户端执行命令，查看审计日志。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
3317
清空日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log
客户端执行kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide命令。
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.
现在不停的产生日志。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
32
这是最新的日志。
root@k8scludes1:~# tail -1 /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"72736d33-508f-4279-89a5-d95aea7b98c5","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader","verb":"update","user":{"username":"system:serviceaccount:ingress-nginx:ingress-nginx","uid":"70a36f2c-225f-450d-849e-2432db224f40","groups":["system:serviceaccounts","system:serviceaccounts:ingress-nginx","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["ingress-nginx-controller-684bbc4b45-jmpxk"],"authentication.kubernetes.io/pod-uid":["308fdf30-854b-49f7-b041-4e5d770c0667"]}},"sourceIPs":["192.168.110.130"],"userAgent":"nginx-ingress-controller/v1.0.0 (linux/amd64) ingress-nginx/041eb167c7bfccb1d1653f194924b0c5fd885e10","objectRef":{"resource":"configmaps","namespace":"ingress-nginx","name":"ingress-controller-leader","uid":"db625974-8e33-4bd6-a2e2-1816301af942","apiVersion":"v1","resourceVersion":"2739119"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-22T13:03:36.751232Z","stageTimestamp":"2022-06-22T13:03:36.759442Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ingress-nginx/ingress-nginx\" of Role \"ingress-nginx\" to ServiceAccount \"ingress-nginx/ingress-nginx\"","mutation.webhook.admission.k8s.io/round_0_index_1":"{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}"}}
客户端不执行命令就产生了太多审计日志了，需要修改审计策略。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
1174
6.2 只记录audit命名空间里的日志
修改审计策略，现在配置只记录某个命名空间里的审计日志，namespaces: ["audit"]表示只记录audit命名空间里的日志。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "ResponseStarted"
rules:
- level: Metadata
resources:
- group: ""
namespaces: ["audit"]
重启kubelet使其生效，但是审计策略没有生效，重启机器。
root@k8scludes1:~# systemctl restart kubelet
重启k8s master节点。
root@k8scludes1:~# reboot
root@k8scludes1:~# kubectl get node
NAME STATUS ROLES AGE VERSION
k8scludes1 Ready control-plane,master 66d v1.22.2
k8scludes2 Ready <none> 66d v1.22.2
k8scludes3 Ready <none> 66d v1.22.2
重启k8s master节点之后，清空日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log
客户端执行kubectl get pod --kubeconfig=kctom -n default -o wide。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n default -o wide
No resources found in default namespace.
客户端执行 kubectl get pod --kubeconfig=kctom -n audit -o wide。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n audit -o wide
No resources found in audit namespace.
查看审计日志，只记录了audit命名空间的操作，default命名空间的操作没有记录。
root@k8scludes1:~# tail -2 /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08b8fca7-8390-4f59-8a21-6ff3541fd2a2","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-22T13:26:08.732751Z","stageTimestamp":"2022-06-22T13:26:08.732751Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08b8fca7-8390-4f59-8a21-6ff3541fd2a2","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-22T13:26:08.732751Z","stageTimestamp":"2022-06-22T13:26:08.741665Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"crbindtom\" of ClusterRole \"cluster-admin\" to User \"tom\""}}
6.3 只记录audit命名空间的pods操作日志
修改审计策略，该审计策略表示只记录audit命名空间的pods操作。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "ResponseStarted"
rules:
- level: Metadata
resources:
- group: ""
resources: ["pods"]
namespaces: ["audit"]
重启机器使审计策略生效。
root@k8scludes1:~# reboot
重启之后，清空日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log
在客户端执行kubectl get pod --kubeconfig=kctom -n audit -o wide。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n audit -o wide
No resources found in audit namespace.
在客户端执行 kubectl get svc --kubeconfig=kctom -n audit -o wide 。
[root@etcd2 ~]# kubectl get svc --kubeconfig=kctom -n audit -o wide
No resources found in audit namespace.
查看日志，发现只记录了audit命名空间的pod操作，svc操作没有记录。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4748e76d-e9cb-46de-af28-d467171ca105","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T00:24:45.351036Z","stageTimestamp":"2022-06-23T00:24:45.351036Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4748e76d-e9cb-46de-af28-d467171ca105","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-23T00:24:45.351036Z","stageTimestamp":"2022-06-23T00:24:45.434147Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"clusterrolebinding1\" of ClusterRole \"clusterole1\" to User \"tom\""}}
6.4 只记录audit命名空间的pods,services,deployments操作日志
编辑审计策略文件，表示只记录audit命名空间的pods,services,deployments操作，因为deployments的apiVersion的父级为apps，所以需要group: "apps" 。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "ResponseStarted"
rules:
- level: Metadata
resources:
- group: ""
resources: ["pods","services"]
- group: "apps"
resources: ["deployments"]
namespaces: ["audit"]
6.5 只记录audit命名空间的pods操作,审计级别为RequestResponse
编辑审计策略文件，表示只记录audit命名空间的pods操作,审计级别为RequestResponse，记录事件的元数据，请求和响应的消息体。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "ResponseStarted"
rules:
- level: RequestResponse
resources:
- group: ""
resources: ["pods"]
namespaces: ["audit"]
重启机器使审计策略生效。
root@k8scludes1:~# reboot
创建一个pod。
root@k8scludes1:~# kubectl apply -f pod.yaml
pod/podtest created
root@k8scludes1:~# kubectl get pod
NAME READY STATUS RESTARTS AGE
podtest 1/1 Running 0 5s
清空审计日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log
客户端执行kubectl get pod --kubeconfig=kctom -n audit -o wide 。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n audit -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
podtest 1/1 Running 0 48s 10.244.1.97 k8scludes3 <none> <none>
查看审计日志，记录的内容很丰富，元数据，请求和响应的消息体都显示出来了。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"d6836f9b-4918-49f2-80dc-ad02e246949b","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T00:43:30.782241Z","stageTimestamp":"2022-06-23T00:43:30.782241Z"}
......
23T00:42:46Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{".":{},"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PodScheduled\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.97\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"subresource":"status"}]}}}]},"requestReceivedTimestamp":"2022-06-23T00:43:30.782241Z","stageTimestamp":"2022-06-23T00:43:30.785883Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"clusterrolebinding1\" of ClusterRole \"clusterole1\" to User \"tom\""}}
6.6 只记录audit命名空间下的tom用户的pods操作，其他用户操作不记录
编辑审计策略文件，设置审计策略：只记录audit命名空间下的tom用户的pods操作，其他用户的操作不记录。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "ResponseStarted"
rules:
- level: Metadata
users: ["tom"]
resources:
- group: ""
resources: ["pods"]
namespaces: ["audit"]
重启机器使审计策略生效。
root@k8scludes1:~# reboot
清空日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log
使用管理员用户查看pod。
root@k8scludes1:~# kubectl get pod -n audit
NAME READY STATUS RESTARTS AGE
podtest 1/1 Running 0 29m
使用tom用户在客户端查看pod 。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n audit -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
podtest 1/1 Running 0 29m 10.244.1.97 k8scludes3 <none> <none>
查看日志，可以发现只记录了tom用户的pod查询操作。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a70f1b39-3c54-4ed8-ab45-9cbb7b4688ef","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T01:12:24.103036Z","stageTimestamp":"2022-06-23T01:12:24.103036Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a70f1b39-3c54-4ed8-ab45-9cbb7b4688ef","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-23T01:12:24.103036Z","stageTimestamp":"2022-06-23T01:12:24.107872Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"crbindtom\" of ClusterRole \"cluster-admin\" to User \"tom\""}}
6.7 rules规则是从上往下匹配的，第一条规则已经匹配了，第二条就不匹配了
编辑审计策略文件，有两条策略，一条是：apiVersion为group: ""的操作不记录日志，另外一条是：只记录audit命名空间下tom用户的pod操作。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "ResponseStarted"
rules:
#apiVersion为group: ""的操作不记录日志
- level: None
resources:
- group: ""
#只记录audit命名空间下tom用户的pod操作
- level: Metadata
users: ["tom"]
resources:
- group: ""
resources: ["pods"]
namespaces: ["audit"]
重启机器使配置生效。
root@k8scludes1:~# reboot
清空日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log
使用管理员用户查询audit命名空间的pod。
root@k8scludes1:~# kubectl get pod -n audit
NAME READY STATUS RESTARTS AGE
podtest 1/1 Running 0 37m
使用tom用户在客户端机器etcd2上查询audit命名空间的pod。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n audit -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
podtest 1/1 Running 0 37m 10.244.1.97 k8scludes3 <none> <none>
查看审计日志，可以发现在客户端使用tom用户查询pod和使用管理员用户查看pod都没有生成审计日志，rules规则是从上往下匹配的，第一条规则已经匹配了，第二条就不匹配了。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
6.8 在 Metadata 级别为所有请求生成日志
编辑审计策略文件，在 Metadata 级别为所有请求生成日志。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
七.总结
在 Kubernetes 1.22.2 环境中实施审计策略可以帮助管理员监控和记录集群中的资源操作，确保集群的安全性和符合性。通过启用审计 Admission Controller 和配置相应的审计策略，我们可以灵活地控制审计记录的格式和范围。
致力于一条龙式的为您解决问题
