手把手系列 - 使用docker容器搭建efk8

前言

使用 docker 容器部署 EFK（elasticsearch+filebeat+kibana）日志系统。

root@efk-node(192.168.1.101)/opt/efk> tree -L 2
.
├── elasticsearch	 # elasticsearch:8.2.2
│   ├── data	# es数据存储目录
│   ├── logs	# es日志目录
│   ├── plugins	# 插件目录
│   └── start.sh # 容器启动脚本
├── filebeat	# filebeat:8.2.2
│   ├── filebeat.docker.yml	# 收集 docker 容器日志
│   └── start.sh	# 容器启动脚本
├── images	# 整体打包的基础镜像（elasticsearch:8.2.2 | filebeat:8.2.2 | kibana:8.2.2）
│   └── efk-images-8.2.2.tar.gz
└── kibana	# kibana:8.2.2
    └── start.sh	# 容器启动脚本

7 directories, 5 files

下载链接：

链接：https://pan.baidu.com/s/1HfT6_S_52fxXXBjXRp2Faw?pwd=hkey
提取码：hkey
获取文件目录：docker / efk8.2.2 / efk-8.2.2.zip

系统版本

System: CentOS Linux release 7.9.2009 (Core)
Kernel: 3.10.0-1160.el7.x86_64

docker 版本

Docker-CE
  * Server Version: 20.10.7
  * Storage Driver: overlay2

实现过程

主机名	ip地址
efk-node	192.168.1.101

系统初始化

系统初始化分为以下几步：

修改主机名
关闭selinux 和 firewalld
配置国内yum源
校对时间

修改主机名

>hostnamectl set-hostname efk-node
>hostname efk-node
>echo "192.168.1.101  efk-node" >> /etc/hosts

#断开会话重新连接
root@efk-node(192.168.1.101)/root>hostname
efk-node

关闭selinux 和 firewalld

### 关闭 selinux
>sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
>systemctl disable firewalld
>reboot

配置国内yum源

>cd /etc/yum.repos.d/
#centos-7源
>curl http://mirrors.aliyun.com/repo/Centos-7.repo -o ./Centos-7.repo
>sed -i '/aliyuncs/d' Centos-7.repo

#docker-ce源
>curl http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -o ./docker-ce.repo

#epel-7源
>curl http://mirrors.aliyun.com/repo/epel-7.repo -o ./epel-7.repo

#efk源
>cat << EOF > elasticstack.repo
[elasticstack]
name = elasticstack
gpgcheck = 0
baseurl = https://mirrors.tuna.tsinghua.edu.cn/elasticstack/yum/elastic-8.x/
EOF

校对时间

>yum install -y ntpdate
>ntpdate tiger.sina.com.cn

Docker-ce

安装docker-ce

>yum install -y docker-ce

添加docker-ce 配置

>mkdir /etc/docker/
>cat << 'EOF' > /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
	"max-file": "3"
  },
  "exec-opts": ["native.cgroupdriver=systemd"],
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ],
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn",
    "https://hub-mirror.c.163.com"
  ]
}
EOF

启动 docker

systemctl enable docker; systemctl start docker

查看docker 信息

>cat << 'EOF' >> /etc/sysctl.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
>sysctl --system

>docker info

Elasticsearch

将下载好的压缩包上传到服务器解压

> unzip efk-8.2.2.zip -d /opt/

赋予权限（否则后面容器启动时，会报权限问题。）

> chown -R 1000:1000 /opt/efk/

导入镜像

> cd /opt/efk/images/
> docker load < efk-images-8.2.2.tar.gz

启动 elasticsearch

> cd /opt/efk/elasticsearch/
> ./start.sh

第一次启动 elasticsearch 容器时，控制台会打印很多日志信息，主要获取了下面信息，就可以多次使用 Ctrl+C 停止容器。

图里打印的信息就是 elasticsearch的口令及 kibana token信息。需要记录下来。

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-> Elasticsearch security features have been automatically configured!
-> Authentication is enabled and cluster connections are encrypted.

->  Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):
  R+80iMZAShBNrOfFW2*k

->  HTTP CA certificate SHA-256 fingerprint:
  56bb845e700c9a57161c707f90c946491a46c42bfdae218ccee7c71a17e13ff7

->  Configure Kibana to use this cluster:
* Run Kibana and click the configuration link in the terminal when Kibana starts.
* Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
  eyJ2ZXIiOiI4LjIuMiIsImFkciI6WyIxNzIuMTguMC4yOjkyMDAiXSwiZmdyIjoiNTZiYjg0NWU3MDBjOWE1NzE2MWM3MDdmOTBjOTQ2NDkxYTQ2YzQyYmZkYWUyMThjY2VlN2M3MWExN2UxM2ZmNyIsImtleSI6IkhBXzc4SUVCZ0tWT0R6X081cHM5OnlJU0I0MlBKUlVXd3JOb3AyNWFWelEifQ==

-> Configure other nodes to join this cluster:
* Copy the following enrollment token and start new Elasticsearch nodes with `bin/elasticsearch --enrollment-token <token>` (valid for the next 30 minutes):
  eyJ2ZXIiOiI4LjIuMiIsImFkciI6WyIxNzIuMTguMC4yOjkyMDAiXSwiZmdyIjoiNTZiYjg0NWU3MDBjOWE1NzE2MWM3MDdmOTBjOTQ2NDkxYTQ2YzQyYmZkYWUyMThjY2VlN2M3MWExN2UxM2ZmNyIsImtleSI6IkhnXzc4SUVCZ0tWT0R6X081cHRLOlZKR0l3VC1lUkE2N1VWYVlyU1dCaGcifQ==

  If you're running in Docker, copy the enrollment token and run:
  `docker run -e "ENROLLMENT_TOKEN=<token>" docker.elastic.co/elasticsearch/elasticsearch:8.2.2`
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

如果这里没有记录，后面也可通过指令来修改。上面时效时间为 30分钟。

当获取到上面信息后，再次启动 elasticsearch 容器。

> docker start elasticsearch

通过浏览器访问 https://192.168.1.101:9200

用户名：elastic
密码：R+80iMZAShBNrOfFW2*k

Kibana

> cd /opt/efk/kibana
> ./start.sh

浏览器访问 http://192.168.1.101:5601

kibana token:
eyJ2ZXIiOiI4LjIuMiIsImFkciI6WyIxNzIuMTguMC4yOjkyMDAiXSwiZmdyIjoiNTZiYjg0NWU3MDBjOWE1NzE2MWM3MDdmOTBjOTQ2NDkxYTQ2YzQyYmZkYWUyMThjY2VlN2M3MWExN2UxM2ZmNyIsImtleSI6IkhBXzc4SUVCZ0tWT0R6X081cHM5OnlJU0I0MlBKUlVXd3JOb3AyNWFWelEifQ==

查看验证码：

> docker logs -f kibana
...
Your verification code is:  257 354

用户名：elastic
密码：R+80iMZAShBNrOfFW2*k

Filebeat

这里采用了一个收集容器日志的示例来做演示，具体更多使用，参考官方文档及目录中其他文章。

> cd /opt/efk/filebeat/


### 这里采用了收集容器一个实例来做演示
> cat filebeat.docker.yml
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
- add_cloud_metadata: ~
- drop_fields:
    fields: ["log","docker","agent","ecs","host","log.offset","agent.hostname","container.id","agent.id"]

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  ssl.verification_mode: "none"
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

修改启动脚本中 elasticsearch 密码：