Docker安装部署ELK教程 (Elasticsearch+Kibana+Logstash+spring boot 集成logstash) 基于7.16.3版本
Docker安装部署ELK教程 (Elasticsearch+Kibana+Logstash+spring boot 集成logstash) 基于7.16.3版本
环境说明:
- 容器版本 Docker version 20.10.2, build 2291f61
- 操作系统版本 Linux debian 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux
- 机器内存64G
Elasticsearch安装
-
下载组件
docker pull 10.11.105.11:8181/elasticsearch/elasticsearch:7.16.3 -
创建自定义的网络
docker network create somenetwork
-
运行 elasticsearch
docker run -d --name elasticsearch --net somenetwork -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" 10.11.105.11:8181/elasticsearch/elasticsearch:7.16.3 -
检测 elasticsearch 是否启动成功
curl 127.0.0.1:9200 或者访问
## Kibana安装
1. **下载组件**
```shell
docker pull 10.11.105.11:8181/kibana/kibana:7.16.3
-
运行Kibana
docker run -d --name kibana --net somenetwork -p 5601:5601 10.11.105.11:8181/kibana/kibana:7.16.3
-
检测 Kibana 是否启动成功
访问 http://10.11.74.126:5601 ##注意点: - 10.11.74.126 修改为自己机器的地址 - 如果加载慢需要等待一会(主要看机器性能 内存)
-
修改中文模式
##进入容器 0bd65e421183 为容器id docker exec -it 0bd65e421183 /bin/bash ##修改 config 文件 cd config vi kibana.yml - i18n.locale: "en" 改为 i18n.locale: "zh-CN" ##退出容器 exit ##重启kibana容器 docker restart 0bd65e421183
重新访问:
-
注意点:
kibana 默认和 elasticsearch 9200端口连接
Logstash安装
-
下载组件
docker pull 10.11.105.11:8181/logstash/logstash:7.16.3 -
创建logstash 的配置文件
##创建目录 /data/logstash/config 和 配置文件 logstash.yml /data/logstash/config/logstash.yml ##创建目录 /data/logstash/conf.d/ 和配置文件 logstash.conf /data/logstash/conf.d/logstash.conflogstash.yml文件内容如下:
##路径都为容器内文件路径 ##合理配置文件内容 http.host: "0.0.0.0" xpack.monitoring.elasticsearch.hosts: [ "http://10.11.74.126:9200" ] xpack.monitoring.enabled: true path.config: /usr/share/logstash/conf.d/*.conf path.logs: /var/log/logstashlogstash.conf文件内容如下:
input {
tcp {
host => "0.0.0.0"
port => 9603
codec => json_lines
}
}
output {
elasticsearch {
hosts => "10.11.74.126:9200"
}
}
3. **运行logstash **
```shell
##建议第一次使用使用此运行命令1 如果运行不成功可以使用3查看运行错误信息
##1.后台运行 初次使用可以使用此命令运行
docker run -it -d -p 5044:5044 -p 9603:9603 --name logstash --privileged=true --net somenetwork -v /data/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml -v /data/logstash/conf.d/:/usr/share/logstash/conf.d/ 10.11.105.11:8181/logstash/logstash:7.16.3
##2.后台运行 配置模板
docker run -it -d -p 5044:5044 -p 9603:9603 --name logstash --privileged=true --net somenetwork -v /data/logstash/ls_templates:/opt/lsConfig/ls_templates/ -v /data/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml -v /data/logstash/conf.d/:/usr/share/logstash/conf.d/ 10.11.105.11:8181/logstash/logstash:7.16.3
##3.前台打印运行 可以清楚看到运行的错误信息
docker run -h logstash -p 5044:5044 -p 9602:9602 --name logstash --privileged=true --network somenetwork -it --rm -v /app/software/logstash/config-dir:/config-dir 10.11.105.11:8181/logstash/logstash:7.16.3 -f /config-dir/logstash.conf
##4.前台运行 使用模板并且配置config
docker run -h logstash -p 5045:5045 -p 5044:5044 -p 9603:9603 --name logstash --privileged=true --network somenetwork -it --rm -v /data/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml -v /data/logstash/conf.d/:/usr/share/logstash/conf.d/ -v /data/logstash/ls_templates:/opt/lsConfig/ls_templates/ 10.11.105.11:8181/logstash/logstash:7.16.3
-
检测 logstash 是否启动成功
docker ps
遇到的问题:
- logstash 端口冲突导致移动报错 更换端口9602启动成功
spring boot 集成logstash
-
引入依赖
<!--logstash 依赖 start--> <dependency> <groupId>net.logstash.log4j</groupId> <artifactId>jsonevent-layout</artifactId> <version>1.7</version> </dependency> <dependency> <groupId>net.logstash.logback</groupId> <artifactId>logstash-logback-encoder</artifactId> <version>7.0.1</version> </dependency> -
配置logback日志
<?xml version="1.0" encoding="UTF-8"?> <configuration debug="true"> <!-- 获取spring配置 --> <springProperty scope="context" name="logPath" source="log.path" defaultValue="/app/logdir"/> <springProperty scope="context" name="appName" source="spring.application.name"/> <!-- 定义变量值的标签 --> <property name="LOG_HOME" value="${logPath}" /> <property name="SPRING_NAME" value="${appName}" /> <!-- 彩色日志依赖的渲染类 --> <conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter"/> <conversionRule conversionWord="wex" converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter"/> <conversionRule conversionWord="wEx" converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter"/> <!-- 链路追踪sleuth 格式化输出 以及 控制台颜色设置变量 --> <property name="CONSOLE_LOG_PATTERN" value="%d{yyyy-MM-dd HH:mm:ss.SSS} %highlight(%-5level) [${appName},%yellow(%X{X-B3-TraceId}),%green(%X{X-B3-SpanId}),%blue(%X{X-B3-ParentSpanId})] [%yellow(%thread)] %green(%logger:%L) :%msg%n"/> <!-- #############################################定义日志输出格式以及输出位置########################################## --> <!--控制台输出设置--> <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>${CONSOLE_LOG_PATTERN}</pattern> <!-- <charset>GBK</charset> --> </encoder> </appender> <!--普通文件输出设置--> <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <FileNamePattern>${LOG_HOME}/log_${SPRING_NAME}_%d{yyyy-MM-dd}_%i.log</FileNamePattern> <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"> <maxFileSize>200MB</maxFileSize> </timeBasedFileNamingAndTriggeringPolicy> </rollingPolicy> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern> </encoder> </appender> <!--aop接口日志拦截文件输出--> <appender name="bizAppender" class="ch.qos.logback.core.rolling.RollingFileAppender"> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <FileNamePattern>/app/log/biz/log_%d{yyyy-MM-dd}_%i.log</FileNamePattern> <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"> <maxFileSize>200MB</maxFileSize> </timeBasedFileNamingAndTriggeringPolicy> </rollingPolicy> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern> </encoder> </appender> <!--开启tcp格式的logstash传输，通过TCP协议连接Logstash--> <!-- <appender name="STASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">--> <!-- <destination>10.11.74.123:9600</destination>--> <!-- <encoder class="net.logstash.logback.encoder.LogstashEncoder" />--> <!-- </appender>--> <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender"> <!--可以访问的logstash日志收集端口--> <destination>10.11.74.126:9600</destination> <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder"/> <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"> <providers> <timestamp> <timeZone>Asia/Shanghai</timeZone> </timestamp> <pattern> <pattern> { "app_name":"${SPRING_NAME}", "traceid":"%X{traceid}", "ip": "%X{ip}", "server_name": "%X{server_name}", "level": "%level", "trace": "%X{X-B3-TraceId:-}", "span": "%X{X-B3-SpanId:-}", "parent": "%X{X-B3-ParentSpanId:-}", "thread": "%thread", "class": "%logger{40} - %M:%L", "message": "%message", "stack_trace": "%exception{10}" } </pattern> </pattern> </providers> </encoder> </appender> <!-- #############################################设置输出日志输出等级########################################## --> <!-- mybatis log configure--> <!-- logger设置某一个包或者具体的某一个类的日志打印级别 --> <logger name="com.apache.ibatis" level="TRACE"/> <logger name="java.sql.Connection" level="DEBUG"/> <logger name="java.sql.Statement" level="DEBUG"/> <logger name="java.sql.PreparedStatement" level="DEBUG"/> <logger name="org.apache.ibatis.logging.stdout.StdOutImpl" level="DEBUG"/> <!-- SaveLogAspect log configure外部接口调用--> <!-- logger设置某一个包或者具体的某一个类的日志打印级别 --> <logger name="com.springweb.baseweb.log.aop.SaveLogAspect" additivity="false" level="INFO"> <!-- 同时输出到两个文件 --> <appender-ref ref="bizAppender"/> <appender-ref ref="FILE"/> </logger> <root level="INFO"> <!-- 默认日志文件输出 --> <appender-ref ref="FILE"/> <appender-ref ref="STDOUT"/> <!-- 默认日志文件输出logstash --> <appender-ref ref="LOGSTASH"/> </root> </configuration> -
启动项目验证日志输出
查看索引管理日志文件已经输出到es
创建索引模式查看日志输出内容:
