k8s实现elk日志收集系统部署

目录

k8s实现elk日志收集系统

说明：es搭建集群，3台master，3台node，node角色分别为热数据，温数据，冷数据

kibana进行数据展示，同时连接es的工具使用es-head。

此处使用deployment+service结合搭建,logstash还未进行试验，等我继续更新。。。

master节点搭建#

Copy
######### 创建名称空间 ###########
kind: Namespace
apiVersion: v1
metadata:
  name: elastic
---
######## service类型为nodeport用以暴露es端口供外部访问 ########
kind: Service
apiVersion: v1
metadata:
  name: es-cluster-nodeport
  namespace: elastic
spec:
  type: NodePort
  selector:
    app: es-cluster
  ports:
    - port: 9200
      targetPort: 9200
      nodePort: 32700
---
####### es集群间通讯端口，使用service进行暴露，但不像nodeport可对集群外服务器提供访问 #############
kind: Service
apiVersion: v1
metadata:
  name: es-svc
  namespace: elastic
  labels:
    app: es-cluster
  annotations:
    es.kubernetes.io/es: es-cluster
spec:
  selector:
    app: es-cluster
    role: master
  ports:
    - port: 9300
      targetPort: 9300
---
###### es的master创建 #############################
kind: Deployment				       # 资源类型
apiVersion: apps/v1  				       # 版本
metadata:				              	# 元数据
  namespace: elastic   				       # 名称空间
  name: es-cluster-master  			        # deployment类型的元数据名
  labels:						# 标签
    app: es-cluster-master			        # 定义deployment的标签
    role: master		
spec:							# 说明				
  replicas: 3						# 副本数为3,表示创建3个pod
  selector:						# 选择器
    matchLabels:				        # 精确选择
      app: es-cluster				       # 选择标签为 app: es-cluster
      role: master					 # 选择标签为 role: master
  template:						# 模板
    metadata:						# 元数据信息
      labels:						# 标签
        app: es-cluster				       # 此模板的信息
        role: master
    spec:						# 模板对应的说明
      containers:					 # 容器
        - name: elasticsearch-master	# 名称
          image: registry.cn-hangzhou.aliyuncs.com/k8sos/es:v1		# 镜像
          ports:						# 端口
            - containerPort: 9200	            # 这里的端口不提供暴露，仅说明
              protocol: TCP
            - containerPort: 9300
              protocol: TCP
          lifecycle:					# 生命周期，也是回调钩子
            postStart:					# 当容器启动后执行的操作
              exec:					# 执行命令
                command:				
                  - "/bin/sh"
                  - "-c"
                  - "sysctl -w vm.max_map_count=262144; ulimit -l unlimited;"
          securityContext:				# 安全上下文，指具备对系统的操作权限
            privileged: true			
          readinessProbe:				# 就绪性探测
            httpGet:					# 探测9200端口是否就绪
              port: 9200
          volumeMounts:					# 存储性挂载
            - mountPath: /usr/share/elasticsearch/data	# 容器中需要挂载的路径
              name: es-data				 # 挂载到的地方，这个名称对应下方的挂载
          env:					      # 此处定义的环境变量相当于在命令行执行命令，替代了配置文件的操作
            - name: "cluster.name"
              value: "es-cluster"
            - name: "bootstrap.memory_lock"
              value: "true"
            - name: "discovery.zen.ping.unicast.hosts"
              value: "es-svc"
            - name: "discovery.zen.minimum_master_nodes"
              value: "2"
            - name: "discovery.zen.ping_timeout"
              value: "5s"
            - name: "node.master"
              value: "true"
            - name: "node.data"
              value: "false"
            - name: "ES_JAVA_OPTS"
              value: "-Xms256m -Xmx256m"
            - name: "http.cors.enabled"
              value: "true"
            - name: "http.cors.allow-origin"
              value: "*"
      volumes:					# 对应上方的挂载名
        - name: es-data
          emptyDir: {}

head插件#

Copy
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: elastic
  name: es-head
spec:
  selector:
    matchLabels:				# deployment精确匹配选择对应的模板
      app: es-head
  template:
    metadata:
      labels:
        app: es-head			      # 模板标签信息，供deployment选择
    spec:
      containers:
        - name: head
          image: alvinos/elasticsearch-head		
          ports:
            - containerPort: 9100
              protocol: TCP
---
##### 使用nodeport型的service，将pod中的端口进行暴露 ##########
kind: Service
apiVersion: v1
metadata:
  name: es-head-svc
  namespace: elastic
spec:
  type: NodePort
  selector:
    app: es-head
  ports:
    - port: 9100
      targetPort: 9100
      nodePort: 32701

node节点--hot#

Copy
kind: Deployment
apiVersion: apps/v1
metadata:
  name: es-cluster-data-hot
  namespace: elastic
  labels:						# deployment的标签
    app: es-cluster-data-hot
    role: data
spec:
  selector:
    matchLabels:					# deployment选择器精确匹配模板中的信息
      app: es-cluster
      role: data
      datatype: hot
  template:
    metadata:
      labels:						# 供上方deployment选择器选择
        app: es-cluster
        role: data
        datatype: hot
    spec:
      containers:
        - name: es-cluster-hot
          image: registry.cn-hangzhou.aliyuncs.com/k8sos/es:v1
          ports:
            - containerPort: 9200
              protocol: TCP
            - containerPort: 9300
              protocol: TCP
          env:								# 环境变量
            - name: "cluster.name"
              value: "es-cluster"
            - name: "bootstrap.memory_lock"
              value: "true"
            - name: "discovery.zen.ping.unicast.hosts"
              value: "es-svc"
            - name: "node.master"
              value: "false"
            - name: "node.data"
              value: "true"
            - name: "ES_JAVA_OPTS"
              value: "-Xms256m -Xmx256m"
            - name: "node.attr.box_type"
              value: "hot"					# 指定为hot数据类型
          lifecycle:
            postStart:						# 容器启动后的操作
              exec:
                command:
                  - "/bin/sh"
                  - "-c"
                  - "sysctl -w vm.max_map_count=262144; ulimit -l unlimited;"
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /usr/share/elasticsearch/data
              name: es-data
      volumes:
        - name: es-data
          emptyDir: {}

node节点--warm#

Copy
kind: Deployment
apiVersion: apps/v1
metadata:
  name: es-cluster-data-warm
  namespace: elastic
  labels:
    app: es-cluster-data-warm
    role: data
spec:
  selector:
    matchLabels:
      app: es-cluster
      role: data
      datatype: warm
  template:
    metadata:
      labels:
        app: es-cluster
        role: data
        datatype: warm
    spec:
      containers:
        - name: es-cluster-warm
          image: registry.cn-hangzhou.aliyuncs.com/k8sos/es:v1
          ports:
            - containerPort: 9200
              protocol: TCP
            - containerPort: 9300
              protocol: TCP
          env:
            - name: "cluster.name"
              value: "es-cluster"
            - name: "bootstrap.memory_lock"
              value: "true"
            - name: "discovery.zen.ping.unicast.hosts"
              value: "es-svc"
            - name: "node.master"
              value: "false"
            - name: "node.data"
              value: "true"
            - name: "ES_JAVA_OPTS"
              value: "-Xms256m -Xmx256m"
            - name: "node.attr.box_type"
              value: "warm"
          lifecycle:
            postStart:
              exec:
                command:
                  - "/bin/sh"
                  - "-c"
                  - "sysctl -w vm.max_map_count=262144; ulimit -l unlimited;"
          securityContext:
            privileged: true
          readinessProbe:
            httpGet:
              port: 9200
          volumeMounts:
            - mountPath: /usr/share/elasticsearch/data
              name: es-data
      volumes:
        - name: es-data
          emptyDir: {}

node节点--cold#

Copy
kind: Deployment
apiVersion: apps/v1
metadata:
  name: es-cluster-data-cold
  namespace: elastic
  labels:
    app: es-cluster-data-cold
    role: data
spec:
  selector:
    matchLabels:
      app: es-cluster
      role: data
      datatype: cold
  template:
    metadata:
      labels:
        app: es-cluster
        role: data
        datatype: cold
    spec:
      containers:
        - name: es-cluster-cold
          image: registry.cn-hangzhou.aliyuncs.com/k8sos/es:v1
          ports:
            - containerPort: 9200
              protocol: TCP
            - containerPort: 9300
              protocol: TCP
          env:
            - name: "cluster.name"
              value: "es-cluster"
            - name: "bootstrap.memory_lock"
              value: "true"
            - name: "discovery.zen.ping.unicast.hosts"
              value: "es-svc"
            - name: "node.master"
              value: "false"
            - name: "node.data"
              value: "true"
            - name: "ES_JAVA_OPTS"
              value: "-Xms256m -Xmx256m"
            - name: "node.attr.box_type"
              value: "cold"
          lifecycle:
            postStart:
              exec:
                command:
                  - "/bin/sh"
                  - "-c"
                  - "sysctl -w vm.max_map_count=262144; ulimit -l unlimited;"
          securityContext:
            privileged: true
          readinessProbe:
            httpGet:
              port: 9200
          volumeMounts:
            - mountPath: /usr/share/elasticsearch/data
              name: es-data
      volumes:
        - name: es-data
          emptyDir: {}
posted @   元气少女郭德纲！！  阅读(1182)  评论(0)  编辑  收藏  举报
编辑推荐：
· Linux系列：如何用heaptrack跟踪.NET程序的非托管内存泄露
· 开发者必知的日志记录最佳实践
· SQL Server 2025 AI相关能力初探
· Linux系列：如何用 C#调用 C方法造成内存泄露
· AI与.NET技术实操系列（二）：开始使用ML.NET
阅读排行：
· 被坑几百块钱后，我竟然真的恢复了删除的微信聊天记录！
· 没有Manus邀请码？试试免邀请码的MGX或者开源的OpenManus吧
· 【自荐】一款简洁、开源的在线白板工具 Drawnix
· 园子的第一款AI主题卫衣上架——"HELLO! HOW CAN I ASSIST YOU TODAY
· Docker 太简单，K8s 太复杂？w7panel 让容器管理更轻松！
点击右上角即可分享
微信分享提示
CONTENTS
✕