Kubernetes 中审计策略--事件处理
一、审计日志的策略
1、日志记录阶段
kube-apiserver 是负责接收及相应用户请求的一个组件,每一个请求都会有几个阶段,每个阶段都有对应的日志,当前支持的阶段有:
- RequestReceived - apiserver 在接收到请求后且在将该请求下发之前会生成对应的审计日志。
- ResponseStarted - 在响应 header 发送后并在响应 body 发送前生成日志。这个阶段仅为长时间运行的请求生成(例如 watch)。
- ResponseComplete - 当响应 body 发送完并且不再发送数据。
- Panic - 当有 panic 发生时生成。
也就是说对 apiserver 的每一个请求理论上会有三个阶段的审计日志生成。
2、日志记录级别
当前支持的日志记录级别有:
- None - 不记录日志。
- Metadata - 只记录 Request 的一些 metadata (例如 user, timestamp, resource, verb 等),但不记录 Request 或 Response 的body。
- Request - 记录 Request 的 metadata 和 body。
- RequestResponse - 最全记录方式,会记录所有的 metadata、Request 和 Response 的 body。
3、日志记录策略
在记录日志的时候尽量只记录所需要的信息,不需要的日志尽可能不记录,避免造成系统资源的浪费。
- 一个请求不要重复记录,每个请求有三个阶段,只记录其中需要的阶段
- 不要记录所有的资源,不要记录一个资源的所有子资源
- 系统的请求不需要记录,kubelet、kube-proxy、kube-scheduler、kube-controller-manager 等对 kube-apiserver 的请求不需要记录
- 对一些认证信息(secerts、configmaps、token 等)的 body 不记录
审计记录最初产生于 kube-apiserver 内部。每个请求在不同执行阶段都会生成审计事件;这些审计事件会根据特定策略 被预处理并写入后端。策略确定要记录的内容和用来存储记录的后端。 当
你可以使用 --audit-policy-file
标志将包含策略的文件传递给 kube-apiserver
。 如果不设置该标志,则不记录事件。 注意 rules
字段 必须 在审计策略文件中提供。没有(0)规则的策略将被视为非法配置。
k8s 审计日志的一个示例:
{ "kind": "EventList", "apiVersion": "audit.k8s.io/v1beta1", "Items": [ { "Level": "Request", "AuditID": "793e7ae2-5ca7-4ad3-a632-19708d2f8265", "Stage": "RequestReceived", "RequestURI": "/api/v1/namespaces/default/pods/test-pre-sf-de7cc-0", "Verb": "get", "User": { "Username": "system:unsecured", "UID": "", "Groups": [ "system:masters", "system:authenticated" ], "Extra": null }, "ImpersonatedUser": null, "SourceIPs": [ "192.168.1.11" ], "UserAgent": "kube-scheduler/v1.12.2 (linux/amd64) kubernetes/73f3294/scheduler", "ObjectRef": { "Resource": "pods", "Namespace": "default", "Name": "test-pre-sf-de7cc-0", "UID": "", "APIGroup": "", "APIVersion": "v1", "ResourceVersion": "", "Subresource": "" }, "ResponseStatus": null, "RequestObject": null, "ResponseObject": null, "RequestReceivedTimestamp": "2019-01-11T06:51:43.528703Z", "StageTimestamp": "2019-01-11T06:51:43.528703Z", "Annotations": null } ] }
二、启用审计日志
当前的审计日志支持两种收集方式:保存为日志文件和调用自定义的 webhook,在 v1.13 中还支持动态的 webhook。
1、将审计日志以 json 格式保存到本地文件
apiserver 配置文件的 KUBE_API_ARGS 中需要添加如下参数:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/kube-audit --audit-log-format=json
日志保存到本地后再通过 fluentd 等其他组件进行收集。
还有其他几个选项可以指定保留审计日志文件的最大天数、文件的最大数量、文件的大小等。
2、将审计日志打到后端指定的 webhook
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
webhook 配置文件实际上是一个 kubeconfig,apiserver 会将审计日志发送 到指定的 webhook 后,webhook 接收到日志后可以再分发到 kafka 或其他组件进行收集。
audit-webhook-kubeconfig
示例:
apiVersion: v1 clusters: - cluster: server: http://127.0.0.1:8081/audit/webhook name: metric contexts: - context: cluster: metric user: "" name: default-context current-context: default-context kind: Config preferences: {} users: []
前面提到过,apiserver 的每一个请求会记录三个阶段的审计日志,但是在实际中并不是需要所有的审计日志,官方也说明了启用审计日志会增加 apiserver 对内存的使用量。
audit-policy.yaml
配置示例:
apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "RequestReceived" rules: # Log pod changes at RequestResponse level - level: RequestResponse resources: - group: "" # Resource "pods" doesn't match requests to any subresource of pods, # which is consistent with the RBAC policy. resources: ["pods"] # Log "pods/log", "pods/status" at Metadata level - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] # Don't log requests to a configmap called "controller-leader" - level: None resources: - group: "" resources: ["configmaps"] resourceNames: ["controller-leader"] # Don't log watch requests by the "system:kube-proxy" on endpoints or services - level: None users: ["system:kube-proxy"] verbs: ["watch"] resources: - group: "" # core API group resources: ["endpoints", "services"] # Don't log authenticated requests to certain non-resource URL paths. - level: None userGroups: ["system:authenticated"] nonResourceURLs: - "/api*" # Wildcard matching. - "/version" # Log the request body of configmap changes in kube-system. - level: Request resources: - group: "" # core API group resources: ["configmaps"] # This rule only applies to resources in the "kube-system" namespace. # The empty string "" can be used to select non-namespaced resources. namespaces: ["kube-system"] # Log configmap and secret changes in all other namespaces at the Metadata level. - level: Metadata resources: - group: "" # core API group resources: ["secrets", "configmaps"] # Log all other resources in core and extensions at the Request level. - level: Request resources: - group: "" # core API group - group: "extensions" # Version of group should NOT be included. # A catch-all rule to log all other requests at the Metadata level. - level: Metadata # Long-running requests like watches that fall under this rule will not # generate an audit event in RequestReceived. omitStages: - "RequestReceived"
你可以使用最低限度的审计策略文件在 Metadata
级别记录所有请求:
# 在 Metadata 级别为所有请求生成日志 apiVersion: audit.k8s.io/v1beta1 kind: Policy rules: - level: Metadata
kubernetes 启用了审计策略
我们可以使用 Webhooks 将审核日志发送到文件或远程 Web API。
在本文中,我们将强制 kube api-server 将审核日志发送到文件。
K8sMeetup
在 Kubernetes 中启用审计策略(对于审计日志文件)
- 创建审计策略 YAML 文件:前往 Kubernetes 集群,并使用以下规则创建 audit-policy.yaml:
apiVersion: audit.k8s.io/v1 kind: Policy rules: # Log the request body of configmap changes in kube-system. - level: Request resources: - group: "" # core API group resources: ["configmaps"] namespaces: ["kube-system"] # Log configmap and secret changes in all other namespaces at the Metadata level. - level: Metadata resources: - group: "" # core API group resources: ["secrets", "configmaps"] # A catch-all rule to log all other requests at the Metadata level. - level: Metadata omitStages: - "RequestReceived"
更新 kube api-server 清单文件:(kubeadm安装方式)
/etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.150.90.242:6443
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=10.150.11.111
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=6443
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-cluster-ip-range=10.96.0.0/12
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
- --audit-policy-file=/etc/kubernetes/audit-policy.yaml
- --audit-log-path=/var/log/kubernetes/audit/audit.log
image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.20.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 10.150.90.242
path: /healthz
port: 6443
scheme: HTTPS
initialDelaySeconds: 15
timeoutSeconds: 15
name: kube-apiserver
resources:
requests:
cpu: 250m
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/pki
name: etc-pki
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
- mountPath: /etc/kubernetes/audit-policy.yaml
name: audit
readOnly: true
- mountPath: /var/log/kubernetes/audit/
name: audit-log
readOnly: false
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/pki
type: DirectoryOrCreate
name: etc-pki
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
- name: audit
hostPath:
path: /etc/kubernetes/audit-policy.yaml
type: File
- name: audit-log
hostPath:
path: /var/log/kubernetes/audit/
type: DirectoryOrCreate
status: {}
重启kube api-server 清单文件:(kubeadm安装方式)
systemctl restart kubelet
参考:
https://www.kubernetes.org.cn/1031.html
https://www.kubernetes.org.cn/1090.html
https://www.kubernetes.org.cn/1195.html
https://kubernetes.io/zh/docs/tasks/debug-application-cluster/audit/
https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-apiserver/
https://kubernetes.io/zh/docs/reference/config-api/apiserver-audit.v1/#audit-k8s-io-v1-Event
https://www.kubernetes.org.cn/2611.html?spm=a2c6h.12873639.0.0.616c3b5fuDXKll
https://developer.aliyun.com/article/686982