Go语言之高级篇beego框架之日志收集系统
一、日志收集系统架构设计
图1
图2
二、开发环境
1、安装jdk
jdk-8u51-windows-x64.exe
安装目录:C:\Program Files\jdk8
2、安装zookeeper
zookeeper-3.3.6.tar.gz
安装目录: C:\Program Files\zookeeper-3.3.6
配置:
C:\Program Files\zookeeper-3.3.6\conf
把配置文件:zoo_sample.cfg,改名成:zoo.cfg
修改部署路径为你安装的路径。
配置环境变量:
C:\Program Files\zookeeper-3.3.6\bin
#启动服务:
进行目录,直接启动服务:C:\Program Files\zookeeper-3.3.6\bin
直接运行服务名称: zkServer.cmd
h had sessionid 0x16908b8a4920000 2019-02-20 10:52:39,153 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53568 2019-02-20 10:52:39,154 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53568 2019-02-20 10:52:39,155 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53568 2019-02-20 10:52:39,158 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:39,161 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:39,163 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53568 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:41,214 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53570 2019-02-20 10:52:41,216 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53570 2019-02-20 10:52:41,217 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53570 2019-02-20 10:52:41,220 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:41,224 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:41,226 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53570 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:42,438 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53571 2019-02-20 10:52:42,439 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53571 2019-02-20 10:52:42,441 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53571 2019-02-20 10:52:42,444 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:42,448 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:42,449 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53571 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:44,219 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53572 2019-02-20 10:52:44,220 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53572 2019-02-20 10:52:44,221 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53572 2019-02-20 10:52:44,223 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:44,227 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:44,229 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53572 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:45,462 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53573 2019-02-20 10:52:45,464 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53573 2019-02-20 10:52:45,465 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53573 2019-02-20 10:52:45,468 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:45,473 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:45,475 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53573 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:52:46,769 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53574 2019-02-20 10:52:46,770 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53574 2019-02-20 10:52:46,772 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53574 2019-02-20 10:52:46,775 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:46,779 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:46,781 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53574 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:48,284 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53575 2019-02-20 10:52:48,285 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53575 2019-02-20 10:52:48,286 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53575 2019-02-20 10:52:48,289 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:48,293 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:48,295 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53575 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:50,069 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53576 2019-02-20 10:52:50,071 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53576 2019-02-20 10:52:50,072 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53576 2019-02-20 10:52:50,075 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:50,080 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:50,081 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53576 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:51,547 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53577 2019-02-20 10:52:51,548 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53577 2019-02-20 10:52:51,550 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53577 2019-02-20 10:52:51,552 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:51,557 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:51,559 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53577 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:53,315 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53579 2019-02-20 10:52:53,316 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53579 2019-02-20 10:52:53,318 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53579 2019-02-20 10:52:53,321 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:53,325 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:53,326 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53579 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:52:54,537 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53580 2019-02-20 10:52:54,539 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53580 2019-02-20 10:52:54,540 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53580 2019-02-20 10:52:54,545 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:54,551 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:52:54,552 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53580 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:52:56,153 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53581 2019-02-20 10:52:56,154 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53581 2019-02-20 10:52:56,156 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53581 2019-02-20 10:52:56,159 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:56,163 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:52:56,164 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53581 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:58,023 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53582 2019-02-20 10:52:58,024 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53582 2019-02-20 10:52:58,026 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53582 2019-02-20 10:52:58,028 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:58,032 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:52:58,034 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53582 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:52:59,424 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53584 2019-02-20 10:52:59,425 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53584 2019-02-20 10:52:59,425 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53584 2019-02-20 10:52:59,428 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:59,431 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:52:59,431 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53584 which had sessionid 0x16908b8a4920000 2019-02-20 10:53:01,246 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53585 2019-02-20 10:53:01,247 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53585 2019-02-20 10:53:01,249 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53585 2019-02-20 10:53:01,252 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:53:01,259 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:53:01,260 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53585 which had sessionid 0x16908b8a4920000 2019-02-20 10:53:02,677 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53587 2019-02-20 10:53:02,678 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53587 2019-02-20 10:53:02,679 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53587 2019-02-20 10:53:02,682 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:53:02,685 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:53:02,686 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53587 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:53:04,384 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53588 2019-02-20 10:53:04,385 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53588 2019-02-20 10:53:04,387 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53588 2019-02-20 10:53:04,390 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:53:04,399 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:53:04,400 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53588 which had sessionid 0x16908b8a4920000 2019-02-20 10:53:06,078 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53589 2019-02-20 10:53:06,079 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53589 2019-02-20 10:53:06,081 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53589 2019-02-20 10:53:06,084 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:53:06,089 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:53:06,090 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53589 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:53:07,834 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53590 2019-02-20 10:53:07,835 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53590 2019-02-20 10:53:07,836 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53590 2019-02-20 10:53:07,838 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14
3、kafka
kafka_2.12-2.1.1.tgz
安装目录:D:\run_file\kafka_2.12-2.1.1 (不能这在C:\Program Files\ 下面启动服务会报错。)
#修改配置文件
D:\run_file\kafka_2.12-2.1.1\config
修改为本机IP地址:
advertised.host.name=172.28.5.25
#配置日志地址:
########### Log Basics ############# # A comma separated list of directories under which to store log files #log.dirs=/tmp/kafka-logs log.dirs=D:/run_file/kafka_2.12-2.1.1/kafka-logs #增加这行代码
启动服务:
进入安装目录中:
在命令行下: cd D:\run_file\kafka_2.12-2.1.1 ./bin/windows/kafka-server-start.bat ./config/server.preperties
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSock NIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145 [2019-02-20 10:53:06,087] INFO [ZooKeeperClient] Waiting until connected. (kaf .zookeeper.ZooKeeperClient) [2019-02-20 10:53:06,089] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.Z KeeperClient) [2019-02-20 10:53:06,189] INFO [ZooKeeperClient] Waiting until connected. (kaf .zookeeper.ZooKeeperClient) [2019-02-20 10:53:06,190] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.Z KeeperClient) [2019-02-20 10:53:06,190] INFO [ZooKeeperClient] Waiting until connected. (kaf .zookeeper.ZooKeeperClient) [2019-02-20 10:53:06,191] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.Z KeeperClient) [2019-02-20 10:53:07,832] INFO Opening socket connection to server localhost/1 .0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org. ache.zookeeper.ClientCnxn) [2019-02-20 10:53:07,834] INFO Socket connection established to localhost/127. 0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2019-02-20 10:53:07,836] WARN Connected to an old server; r-o mode will be un ailable (org.apache.zookeeper.ClientCnxnSocket) [2019-02-20 10:53:07,837] INFO Session establishment complete on server localh t/127.0.0.1:2181, sessionid = 0x16908b8a4920000, negotiated timeout = 6000 (or apache.zookeeper.ClientCnxn) [2019-02-20 10:53:07,844] INFO [SocketServer brokerId=0] Started processors fo 1 acceptors (kafka.network.SocketServer) [2019-02-20 10:53:07,853] INFO Kafka version : 2.1.1 (org.apache.kafka.common. ils.AppInfoParser) [2019-02-20 10:53:07,853] INFO Kafka commitId : 21234bee31165527 (org.apache.k ka.common.utils.AppInfoParser) [2019-02-20 10:53:07,856] INFO [KafkaServer id=0] started (kafka.server.KafkaS ver) [2019-02-20 10:53:12,993] INFO [GroupMetadataManager brokerId=0] Removed 0 exp ed offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
二、tail
在cmd中安装:
go get github.com/hpcloud/tail
#代码:
创建一个my.log的文件
路径:
D:\GoFiles\my.log
示例:
vi my.log
############################# Producer Basics ############################# # list of brokers used for bootstrapping knowledge about the rest of the cluster # format: host1:port1,host2:port2 ... bootstrap.servers=localhost:9092 # specify the compression codec for all data generated: none, gzip, snappy, lz4, zstd compression.type=none # name of the partitioner class for partitioning events; default partition spreads data randomly #partitioner.class= # the maximum amount of time the client will wait for the response of a request #request.timeout.ms= # how long `KafkaProducer.send` and `KafkaProducer.partitionsFor` will block for #max.block.ms= # the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together #linger.ms= # the maximum size of a request in bytes #max.request.size= # the default batch size in bytes when batching multiple records sent to a partition #batch.size= # the total bytes of memory the producer can use to buffer records waiting to be sent to the server #buffer.memory=
示例:把文件中内容读出来
vi main.go
package main import ( "fmt" "time" "github.com/hpcloud/tail" ) func main() { filename := "D:\\GoFiles\\my.log" tails, err := tail.TailFile(filename, tail.Config{ ReOpen: true, Follow: true, //Location: &tail.SeekInfo{Offset: 0, Whence: 2}, MustExist: false, Poll: true, }) if err != nil { fmt.Println("tail file err:", err) return } var msg *tail.Line var ok bool for true { msg, ok = <-tails.Lines if !ok { fmt.Printf("tail file close reopen,filename:%s\n", tails, filename) time.Sleep(100 * time.Millisecond) continue } fmt.Println("msg:", msg.Text) } }
执行结果:
#编译
#在终端中输入: D:\GoFiles\src\log_collect\tail>go build D:\GoFiles\src\log_collect\tail #运行 D:\GoFiles\src\log_collect\tail>tail.exe #返回结果: ############################# Producer Basics ############################# msg: msg: # list of brokers used for bootstrapping knowledge about the rest of the cl uster msg: # format: host1:port1,host2:port2 ... msg: bootstrap.servers=localhost:9092 msg: msg: # specify the compression codec for all data generated: none, gzip, snappy, lz4, zstd msg: compression.type=none msg: msg: # name of the partitioner class for partitioning events; default partition spreads data randomly msg: #partitioner.class= msg: msg: # the maximum amount of time the client will wait for the response of a req uest msg: #request.timeout.ms= msg: msg: # how long `KafkaProducer.send` and `KafkaProducer.partitionsFor` will bloc k for msg: #max.block.ms= msg: msg: # the producer will wait for up to the given delay to allow other records t o be sent so that the sends can be batched together msg: #linger.ms= msg: msg: # the maximum size of a request in bytes msg: #max.request.size= msg: msg: # the default batch size in bytes when batching multiple records sent to a partition msg: #batch.size= msg: msg: # the total bytes of memory the producer can use to buffer records waiting to be sent to the server msg: #buffer.memory=
#
二、els安装
1、安装Elasticsearch
elasticsearch-5.5.0.msi
安装目录
C:\Program Files\Elastic\Elasticsearch
#启动目录
C:\Program Files\Elastic\Elasticsearch\bin
运行:
elasticsearch.exe
#访问: http://127.0.0.1:9200/
2、kafka(windows版本安装)
软件部署:
kibana-5.5.1-windows-x86.zip
解压到下面目录中:
C:\Program Files (x86)\kibana
#修改配置文件
elasticsearch.url: "http://127.0.0.1:9200" #指向es的启动页面
#启动kibana
C:\Program Files (x86)\kibana\bin
运行:
kibana.bat
示例:
运行程序的前提条件:
先安装elastic.v2
安装包:
go get gopkg.in/olivere/elastic.v2
使用方法:
import "gopkg.in/olivere/elastic.v2"
目录:
![](https://img2018.cnblogs.com/blog/1053682/201902/1053682-20190222151957033-2093230296.png)
代码:
vi main.go
package main import ( "fmt" elastic "gopkg.in/olivere/elastic.v2" ) type Tweet struct { User string Message string } func main() { client, err := elastic.NewClient(elastic.SetSniff(false), elastic.SetURL("http://127.0.0.1:9200/")) if err != nil { fmt.Println("connect es error", err) return } fmt.Println("conn es succ") for i := 0; i < 10000; i++ { tweet := Tweet{User: "olivere", Message: "Take Five"} _, err = client.Index(). Index("twitter"). Type("tweet"). Id(fmt.Sprintf("%d", i)). BodyJson(tweet). Do() if err != nil { // Handle error panic(err) return } } fmt.Println("insert success") }
#打开页面:
http://localhost:5601
输入: twitter --->再点Discover就可以看到下面页面。
3、