filebeat 篇章——Run Filebeat on k8s
Run Filebeat on k8s
https://www.elastic.co/guide/en/beats/filebeat/8.7/running-on-kubernetes.html
一、Kubernetes deploy manifests
You deploy Filebeat as a DaemonSet to ensure there’s a running instance on each node of the cluster.
为了确保集群中每个节点都有一个运行实例，您可以将 Filebeat 部署为一个守护进程集（DaemonSet）。
The container logs
host folder (/var/log/containers) is mounted on the Filebeat container. Filebeat starts
an input for the files and begins harvesting them as soon as they appear in the folder.
容器日志主机文件夹(/var/log/Container)已挂载在 Filebeat 容器上。一旦文件出现在文件夹中，Filebeat 就会启动一个 input 并开始收集它们。
Everything is
deployed under the kube-system namespace by default. To change the namespace,
modify the manifest file.
默认情况下，所有内容都部署在kube-system命名空间下。要更改命名空间，请修改清单文件。
To download the manifest file, run:
要下载清单文件，请运行:
curl -L -O https://raw.githubusercontent.com/elastic/beats/8.7/deploy/kubernetes/filebeat-kubernetes.yaml
1.1、 Running Filbeat on master nodes
Kubernetes master nodes can use taints to limit the workloads that can run on them. To run Filebeat on master nodes you may need to update the Daemonset spec to include proper tolerations:
Kubernetes的Master节点可以使用“污点”（Taints）来限制可在其上运行的工作负载。如果您需要在Master节点上运行Filebeat，则需要在Daemonset规范中包含正确的tolerations。为实现此目的，您可以在Filebeat的Daemonset的yaml清单文件中定义tolerations字段，以便Filebeat容器可以在Master节点上运行。例如：
spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule
1.2、Deploy
部署Filebeat到Kubernetes，运行以下命令：
kubectl create -f filebeat-kubernetes.yaml
检查状态，运行以下命令：
kubectl --namespace=kube-system get ds/filebeat
二、我个人的
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-script-config
namespace: ops-department
labels:
k8s-app: filebeat
data:
set-kafka-topic.js: |
function process(event) {
if (event.Get("kubernetes.namespace")) {
event.Put("kafka_topic", "log-k8s-" + event.Get("kubernetes.namespace"));
} else {
throw new Error("Kubernetes namespace is not defined.");
}
return event;
}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: ops-department
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
http.enabled: true
http.host: 0.0.0.0
http.port: 5066
filebeat.inputs:
- type: container
stream: stdout
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
in_cluster: true
default_matchers.enabled: true
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- rename:
fields:
- from: message
to: "@message"
- from: source
to: "@path"
- from: node_name
to: "@hostname"
- from: "kubernetes.namespace_name"
to: "kubernetes_namespace"
ignore_missing: true
- script:
lang: javascript
id: "set_kafka_topic"
file: "/usr/share/filebeat/scripts/set-kafka-topic.js"
ignore_imssing: true
processors:
- add_cloud_metadata:
exclude_fields: ["host"]
- add_host_metadata:
exclude_fields: ["host"]
- decode_json_fields:
fields: ["message"]
target: ""
overwrite_keys: true
add_error_key: true
- if:
contains:
message: kafka_topic
then:
- dissect:
tokenizer: "%{[@metadata][beat]} %{[@metadata][version]} [%{loglevel}] [%{module}] [%{namespace}] [%{podname}] %{[@metadata][message]} kafka_topic:%{kafka_topic}"
field: "message"
target_prefix: ""
- rename:
fields:
- { from: "kafka_topic", to: "topic" }
- drop_fields:
fields: ["beat", "input", "prospector.type", "offset", "source", "log", "ecs", "host", "container", "agent", "cloud", "tags", "kubernetes.replicaset", "kubernetes.labels", "kubernetes.namespace labels", "kubernetes.container", "kubernetes.node", "kubernetes.namespace_labels"]
ignore_missing: true
- drop_event:
when:
not:
or:
- equals:
kubernetes.namespace: "ops-department"
- equals:
kubernetes.namespace: "account"
- equals:
kubernetes.namespace: "jj-online"
output.kafka:
enabled: true
hosts:
- 10.10.10.10:9092
topic: "%{[kafka_topic]}"
protocol_version: "2.0.0"
compression: gzip
max_message_bytes: 1000000
multiline:
pattern: ^\d{4}-\d{2}-\d{2}
negate: true
match: after
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: ops-department
labels:
k8s-app: filebeat
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.17.9
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: scripts
mountPath: /usr/share/filebeat/scripts
readOnly: true
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
- name: host-time
mountPath: /etc/localtime
volumes:
- name: scripts
configMap:
name: filebeat-script-config
- name: config
configMap:
defaultMode: 0640
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
# When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
path: /var/lib/filebeat-data
type: DirectoryOrCreate
- name: host-time
hostPath:
path: /etc/localtime
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: ops-department
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: filebeat
namespace: ops-department
subjects:
- kind: ServiceAccount
name: filebeat
namespace: ops-department
roleRef:
kind: Role
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: filebeat-kubeadm-config
namespace: ops-department
subjects:
- kind: ServiceAccount
name: filebeat
namespace: ops-department
roleRef:
kind: Role
name: filebeat-kubeadm-config
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
- nodes
verbs:
- get
- watch
- list
- apiGroups: ["apps"]
resources:
- replicasets
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: filebeat
# should be the namespace where filebeat is running
namespace: ops-department
labels:
k8s-app: filebeat
rules:
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: filebeat-kubeadm-config
namespace: ops-department
labels:
k8s-app: filebeat
rules:
- apiGroups: [""]
resources:
- configmaps
resourceNames:
- kubeadm-config
verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: ops-department
labels:
k8s-app: filebeat
---
