离线日志数据导入elk

上一篇: docker搭建elk日志收集系统

非系统数据或者离线日志进行分析,可以采用端口导入elk方式来实现

测试数据

2020-06-27 06:19:04.856 [] [INFO] [main] [com.lyf.BaseApplication:50] - Starting BaseApplication on NVX1F8W7MJSQCJI with PID 11448 (E:\xianghaizing\SpringbootBase\target\classes started by Administrator in E:\xianghaizing\SpringbootBase)
2020-06-27 06:19:04.863 [] [INFO] [main] [com.lyf.BaseApplication:675] - No active profile set, falling back to default profiles: default
2020-06-27 06:19:06.141 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:244] - Multiple Spring Data modules found, entering strict repository configuration mode!
2020-06-27 06:19:06.144 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:126] - Bootstrapping Spring Data repositories in DEFAULT mode.
2020-06-27 06:19:06.236 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:182] - Finished Spring Data repository scanning in 84ms. Found 1 repository interfaces.
2020-06-27 06:19:06.251 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:244] - Multiple Spring Data modules found, entering strict repository configuration mode!
2020-06-27 06:19:06.252 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:126] - Bootstrapping Spring Data repositories in DEFAULT mode.
2020-06-27 06:19:06.279 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationExtensionSupport:363] - Spring Data Redis - Could not safely identify store assignment for repository candidate interface com.lyf.dao.ElasticRepository.
2020-06-27 06:19:06.279 [] [INFO] [main] [org.springframework.data.repository.config.RepositoryConfigurationDelegate:182] - Finished Spring Data repository scanning in 13ms. Found 0 repository interfaces.
2020-06-27 06:19:06.416 [] [WARN] [main] [org.mybatis.spring.mapper.ClassPathMapperScanner:44] - Skipping MapperFactoryBean with name 'elasticRepository' and 'com.lyf.dao.ElasticRepository' mapperInterface. Bean already defined with the same name!
2020-06-27 06:19:06.847 [] [INFO] [main] [org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker:330] - Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$81d28fde] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-06-27 06:19:06.992 [] [INFO] [main] [org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker:330] - Bean 'redisConfig' of type [com.lyf.redis.RedisConfig$$EnhancerBySpringCGLIB$$aac139eb] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-06-27 06:19:07.651 [] [INFO] [main] [org.springframework.boot.web.embedded.tomcat.TomcatWebServer:90] - Tomcat initialized with port(s): 8765 (http)
2020-06-27 06:19:07.666 [] [INFO] [main] [org.apache.coyote.http11.Http11NioProtocol:173] - Initializing ProtocolHandler ["http-nio-8765"]
2020-06-27 06:19:07.676 [] [INFO] [main] [org.apache.catalina.core.StandardService:173] - Starting service [Tomcat]
2020-06-27 06:19:07.676 [] [INFO] [main] [org.apache.catalina.core.StandardEngine:173] - Starting Servlet engine: [Apache Tomcat/9.0.17]
2020-06-27 06:19:07.843 [] [INFO] [main] [org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]:173] - Initializing Spring embedded WebApplicationContext
2020-06-27 06:19:07.843 [] [INFO] [main] [org.springframework.web.context.ContextLoader:296] - Root WebApplicationContext: initialization completed in 2851 ms
2020-06-27 06:19:09.878 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:190] - no modules loaded
2020-06-27 06:19:09.880 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:193] - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
2020-06-27 06:19:09.881 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:193] - loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
2020-06-27 06:19:09.882 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:193] - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2020-06-27 06:19:09.883 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:193] - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
2020-06-27 06:19:09.884 [] [INFO] [main] [org.elasticsearch.plugins.PluginsService:193] - loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2020-06-27 06:19:13.168 [] [INFO] [main] [org.springframework.data.elasticsearch.client.TransportClientFactoryBean:88] - Adding transport node : 192.168.37.190:9300
2020-06-27 06:19:25.891 [] [ERROR] [main] [org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository:91] - failed to load elasticsearch nodes : org.elasticsearch.index.mapper.MapperParsingException: analyzer [ik_max_word] not found for field [title]
2020-06-27 06:19:26.912 [] [INFO] [main] [org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor:171] - Initializing ExecutorService 'applicationTaskExecutor'
2020-06-27 06:19:27.100 [] [INFO] [main] [org.springframework.boot.autoconfigure.web.servlet.WelcomePageHandlerMapping:56] - Adding welcome page: class path resource [static/index.html]
2020-06-27 06:19:28.079 [] [INFO] [main] [org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver:59] - Exposing 2 endpoint(s) beneath base path '/actuator'
2020-06-27 06:19:28.240 [] [INFO] [main] [org.springframework.amqp.rabbit.connection.CachingConnectionFactory:460] - Attempting to connect to: [192.168.37.190:5672]
2020-06-27 06:19:28.651 [] [INFO] [main] [org.springframework.amqp.rabbit.connection.CachingConnectionFactory:496] - Created new connection: rabbitConnectionFactory#e784320:0/SimpleConnection@1be4a7e3 [delegate=amqp://guest@192.168.37.190:5672/, localPort= 57385]
2020-06-27 06:19:28.938 [] [INFO] [main] [org.apache.coyote.http11.Http11NioProtocol:173] - Starting ProtocolHandler ["http-nio-8765"]
2020-06-27 06:19:28.954 [] [INFO] [main] [org.springframework.boot.web.embedded.tomcat.TomcatWebServer:204] - Tomcat started on port(s): 8765 (http) with context path ''
2020-06-27 06:19:28.957 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [main] [com.lyf.BaseApplication:59] - Started BaseApplication in 24.844 seconds (JVM running for 28.197)
2020-06-27 06:19:29.234 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [RMI TCP Connection(6)-127.0.0.1] [com.zaxxer.hikari.HikariDataSource:110] - UserHikariCP - Starting...
2020-06-27 06:19:29.236 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [RMI TCP Connection(8)-127.0.0.1] [org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]:173] - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-06-27 06:19:29.236 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [RMI TCP Connection(8)-127.0.0.1] [org.springframework.web.servlet.DispatcherServlet:524] - Initializing Servlet 'dispatcherServlet'
2020-06-27 06:19:29.254 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [RMI TCP Connection(8)-127.0.0.1] [org.springframework.web.servlet.DispatcherServlet:546] - Completed initialization in 17 ms
2020-06-27 06:19:29.424 [670dfbbf-982e-4e10-981a-e1e11fd12262] [INFO] [RMI TCP Connection(6)-127.0.0.1] [com.zaxxer.hikari.HikariDataSource:123] - UserHikariCP - Start completed.
2020-06-27 06:19:29.546 [670dfbbf-982e-4e10-981a-e1e11fd12262] [WARN] [RMI TCP Connection(6)-127.0.0.1] [org.springframework.boot.actuate.elasticsearch.ElasticsearchHealthIndicator:89] - Elasticsearch health check failed
org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.
	at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:72)
	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:54)
	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:44)
	at org.springframework.boot.actuate.elasticsearch.ElasticsearchHealthIndicator.doHealthCheck(ElasticsearchHealthIndicator.java:80)
	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:84)
	at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:98)
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
	at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76)
	at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:61)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:126)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:99)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
	at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
	at sun.rmi.transport.Transport$1.run(Transport.java:200)
	at sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:235)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:69)
	at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:70)
	... 38 common frames omitted
2020-06-27 06:19:29.960 [] [INFO] [RMI TCP Connection(6)-127.0.0.1] [io.lettuce.core.EpollProvider:104] - Starting without optional epoll library
2020-06-27 06:19:29.961 [] [INFO] [RMI TCP Connection(6)-127.0.0.1] [io.lettuce.core.KqueueProvider:104] - Starting without optional kqueue library
2020-06-27 07:25:25.480 [] [INFO] [main] [org.apache.coyote.http11.Http11NioProtocol:173] - Starting ProtocolHandler ["http-nio-8765"]
2020-06-27 07:25:25.498 [] [INFO] [main] [org.springframework.boot.web.embedded.tomcat.TomcatWebServer:204] - Tomcat started on port(s): 8765 (http) with context path ''
2020-06-27 07:25:25.501 [] [INFO] [main] [com.lyf.BaseApplication:59] - Started BaseApplication in 18.721 seconds (JVM running for 21.761)
2020-06-27 07:25:25.937 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(2)-127.0.0.1] [org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]:173] - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-06-27 07:25:25.937 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(3)-127.0.0.1] [com.zaxxer.hikari.HikariDataSource:110] - UserHikariCP - Starting...
2020-06-27 07:25:25.938 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(2)-127.0.0.1] [org.springframework.web.servlet.DispatcherServlet:524] - Initializing Servlet 'dispatcherServlet'
2020-06-27 07:25:25.963 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(2)-127.0.0.1] [org.springframework.web.servlet.DispatcherServlet:546] - Completed initialization in 25 ms
2020-06-27 07:25:26.209 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(3)-127.0.0.1] [com.zaxxer.hikari.HikariDataSource:123] - UserHikariCP - Start completed.
2020-06-27 07:25:26.397 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(3)-127.0.0.1] [io.lettuce.core.EpollProvider:104] - Starting without optional epoll library
2020-06-27 07:25:26.400 [dc1ad0e3-9dba-4484-9969-4a0051567096] [INFO] [RMI TCP Connection(3)-127.0.0.1] [io.lettuce.core.KqueueProvider:104] - Starting without optional kqueue library

logstash配置

input {
  tcp {
    mode => "server"
    host => "0.0.0.0"
    port => 4560
    codec => json_lines
    type => "business"
  }
  tcp {
    mode => "server"
    host => "0.0.0.0"
    port => 4561
    codec => multiline {
      pattern => "^2020"
      negate => true
      what => "previous"
    }
    type => "debug"
  }
}
filter {
  if [type] == "debug" {
    grok {
      match => ["message", "%{TIMESTAMP_ISO8601:logdate}"]
    }
    date {
      match => ["logdate", "yyyy-MM-dd HH:mm:ss.SSS"]
    }
  }
}
output {
  elasticsearch {
    hosts => ["es:9200"]
    action => "index"
    codec => json
    index => "%{type}-%{+YYYY.MM.dd}"
  }
  stdout {
    codec => rubydebug {
    }
  }
}

配置简要说明:

codec => multiline {
		pattern => "^2"
		negate => true
		what => "previous"
	}

使用4561端口来接收离线数据
multiline设置多行合并,否则异常信息会被分成多行存入
pattern设置合并规则,测试数据起始行都是2020,所以设置^20202020开头的为新的一行,否则合并显示!!!

filter {
  if [type] == "debug" {
    grok {
      match => ["message", "%{TIMESTAMP_ISO8601:logdate}"]
    }
    date {
      match => ["logdate", "yyyy-MM-dd HH:mm:ss.SSS"]
    }
  }
}