Go语言之高级篇beego框架之日志收集系统
一、日志收集系统架构设计
图1
图2
二、开发环境
1、安装jdk
jdk-8u51-windows-x64.exe
安装目录：C:\Program Files\jdk8
2、安装zookeeper
zookeeper-3.3.6.tar.gz
安装目录： C:\Program Files\zookeeper-3.3.6
配置：
C:\Program Files\zookeeper-3.3.6\conf
把配置文件：zoo_sample.cfg，改名成：zoo.cfg
修改部署路径为你安装的路径。
配置环境变量：
C:\Program Files\zookeeper-3.3.6\bin
#启动服务：
进行目录，直接启动服务：C:\Program Files\zookeeper-3.3.6\bin
直接运行服务名称: zkServer.cmd
h had sessionid 0x16908b8a4920000 2019-02-20 10:52:39,153 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53568 2019-02-20 10:52:39,154 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53568 2019-02-20 10:52:39,155 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53568 2019-02-20 10:52:39,158 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:39,161 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:39,163 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53568 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:41,214 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53570 2019-02-20 10:52:41,216 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53570 2019-02-20 10:52:41,217 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53570 2019-02-20 10:52:41,220 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:41,224 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:41,226 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53570 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:42,438 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53571 2019-02-20 10:52:42,439 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53571 2019-02-20 10:52:42,441 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53571 2019-02-20 10:52:42,444 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:42,448 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:42,449 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53571 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:44,219 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53572 2019-02-20 10:52:44,220 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53572 2019-02-20 10:52:44,221 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53572 2019-02-20 10:52:44,223 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:44,227 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:44,229 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53572 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:45,462 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53573 2019-02-20 10:52:45,464 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53573 2019-02-20 10:52:45,465 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53573 2019-02-20 10:52:45,468 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:45,473 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:45,475 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53573 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:52:46,769 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53574 2019-02-20 10:52:46,770 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53574 2019-02-20 10:52:46,772 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53574 2019-02-20 10:52:46,775 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:46,779 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:46,781 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53574 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:48,284 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53575 2019-02-20 10:52:48,285 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53575 2019-02-20 10:52:48,286 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53575 2019-02-20 10:52:48,289 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:48,293 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:48,295 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53575 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:50,069 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53576 2019-02-20 10:52:50,071 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53576 2019-02-20 10:52:50,072 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53576 2019-02-20 10:52:50,075 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:50,080 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:50,081 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53576 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:51,547 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53577 2019-02-20 10:52:51,548 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53577 2019-02-20 10:52:51,550 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53577 2019-02-20 10:52:51,552 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:51,557 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:51,559 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53577 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:53,315 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53579 2019-02-20 10:52:53,316 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53579 2019-02-20 10:52:53,318 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53579 2019-02-20 10:52:53,321 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:53,325 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java .io.IOException: 远程主机强迫关闭了一个现有的连接。 2019-02-20 10:52:53,326 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53579 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:52:54,537 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53580 2019-02-20 10:52:54,539 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53580 2019-02-20 10:52:54,540 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53580 2019-02-20 10:52:54,545 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:54,551 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:52:54,552 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53580 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:52:56,153 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53581 2019-02-20 10:52:56,154 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53581 2019-02-20 10:52:56,156 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53581 2019-02-20 10:52:56,159 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:56,163 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:52:56,164 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53581 which had sessionid 0x16908b8a4920000 2019-02-20 10:52:58,023 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53582 2019-02-20 10:52:58,024 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53582 2019-02-20 10:52:58,026 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53582 2019-02-20 10:52:58,028 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:58,032 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:52:58,034 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53582 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:52:59,424 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53584 2019-02-20 10:52:59,425 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53584 2019-02-20 10:52:59,425 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53584 2019-02-20 10:52:59,428 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:52:59,431 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:52:59,431 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53584 which had sessionid 0x16908b8a4920000 2019-02-20 10:53:01,246 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53585 2019-02-20 10:53:01,247 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53585 2019-02-20 10:53:01,249 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53585 2019-02-20 10:53:01,252 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:53:01,259 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:53:01,260 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53585 which had sessionid 0x16908b8a4920000 2019-02-20 10:53:02,677 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53587 2019-02-20 10:53:02,678 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53587 2019-02-20 10:53:02,679 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53587 2019-02-20 10:53:02,682 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:53:02,685 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:53:02,686 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53587 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:53:04,384 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53588 2019-02-20 10:53:04,385 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53588 2019-02-20 10:53:04,387 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53588 2019-02-20 10:53:04,390 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:53:04,399 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:53:04,400 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53588 which had sessionid 0x16908b8a4920000 2019-02-20 10:53:06,078 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53589 2019-02-20 10:53:06,079 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0 :0:0:0:1:53589 2019-02-20 10:53:06,081 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /0:0:0:0:0:0:0:1:53589 2019-02-20 10:53:06,084 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14 2019-02-20 10:53:06,089 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0x16908b8a4920000, likely client has closed socket 2019-02-20 10:53:06,090 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53589 whic h had sessionid 0x16908b8a4920000 2019-02-20 10:53:07,834 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53590 2019-02-20 10:53:07,835 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0 .1:53590 2019-02-20 10:53:07,836 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6 000 for client /127.0.0.1:53590 2019-02-20 10:53:07,838 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe eperServer@593] - Dropping packet at server of type 14
3、kafka
kafka_2.12-2.1.1.tgz
安装目录：D:\run_file\kafka_2.12-2.1.1 （不能这在C:\Program Files\ 下面启动服务会报错。）
#修改配置文件
D:\run_file\kafka_2.12-2.1.1\config
修改为本机IP地址：
advertised.host.name=172.28.5.25
#配置日志地址：
########### Log Basics ############# # A comma separated list of directories under which to store log files #log.dirs=/tmp/kafka-logs log.dirs=D:/run_file/kafka_2.12-2.1.1/kafka-logs #增加这行代码
启动服务：
进入安装目录中：
在命令行下： cd D:\run_file\kafka_2.12-2.1.1 ./bin/windows/kafka-server-start.bat ./config/server.preperties
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSock
NIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145
[2019-02-20 10:53:06,087] INFO [ZooKeeperClient] Waiting until connected. (kaf
.zookeeper.ZooKeeperClient)
[2019-02-20 10:53:06,089] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.Z
KeeperClient)
[2019-02-20 10:53:06,189] INFO [ZooKeeperClient] Waiting until connected. (kaf
.zookeeper.ZooKeeperClient)
[2019-02-20 10:53:06,190] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.Z
KeeperClient)
[2019-02-20 10:53:06,190] INFO [ZooKeeperClient] Waiting until connected. (kaf
.zookeeper.ZooKeeperClient)
[2019-02-20 10:53:06,191] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.Z
KeeperClient)
[2019-02-20 10:53:07,832] INFO Opening socket connection to server localhost/1
.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.
ache.zookeeper.ClientCnxn)
[2019-02-20 10:53:07,834] INFO Socket connection established to localhost/127.
0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2019-02-20 10:53:07,836] WARN Connected to an old server; r-o mode will be un
ailable (org.apache.zookeeper.ClientCnxnSocket)
[2019-02-20 10:53:07,837] INFO Session establishment complete on server localh
t/127.0.0.1:2181, sessionid = 0x16908b8a4920000, negotiated timeout = 6000 (or
apache.zookeeper.ClientCnxn)
[2019-02-20 10:53:07,844] INFO [SocketServer brokerId=0] Started processors fo
1 acceptors (kafka.network.SocketServer)
[2019-02-20 10:53:07,853] INFO Kafka version : 2.1.1 (org.apache.kafka.common.
ils.AppInfoParser)
[2019-02-20 10:53:07,853] INFO Kafka commitId : 21234bee31165527 (org.apache.k
ka.common.utils.AppInfoParser)
[2019-02-20 10:53:07,856] INFO [KafkaServer id=0] started (kafka.server.KafkaS
ver)
[2019-02-20 10:53:12,993] INFO [GroupMetadataManager brokerId=0] Removed 0 exp
ed offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
二、tail
在cmd中安装：
go get github.com/hpcloud/tail
#代码：
创建一个my.log的文件
路径：
D:\GoFiles\my.log
示例：
vi my.log
############################# Producer Basics ############################# # list of brokers used for bootstrapping knowledge about the rest of the cluster # format: host1:port1,host2:port2 ... bootstrap.servers=localhost:9092 # specify the compression codec for all data generated: none, gzip, snappy, lz4, zstd compression.type=none # name of the partitioner class for partitioning events; default partition spreads data randomly #partitioner.class= # the maximum amount of time the client will wait for the response of a request #request.timeout.ms= # how long `KafkaProducer.send` and `KafkaProducer.partitionsFor` will block for #max.block.ms= # the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together #linger.ms= # the maximum size of a request in bytes #max.request.size= # the default batch size in bytes when batching multiple records sent to a partition #batch.size= # the total bytes of memory the producer can use to buffer records waiting to be sent to the server #buffer.memory=
示例：把文件中内容读出来
vi main.go
package main
import (
"fmt"
"time"
"github.com/hpcloud/tail"
)
func main() {
filename := "D:\\GoFiles\\my.log"
tails, err := tail.TailFile(filename, tail.Config{
ReOpen: true,
Follow: true,
//Location: &tail.SeekInfo{Offset: 0, Whence: 2},
MustExist: false,
Poll: true,
})
if err != nil {
fmt.Println("tail file err:", err)
return
}
var msg *tail.Line
var ok bool
for true {
msg, ok = <-tails.Lines
if !ok {
fmt.Printf("tail file close reopen,filename:%s\n", tails, filename)
time.Sleep(100 * time.Millisecond)
continue
}
fmt.Println("msg:", msg.Text)
}
}
执行结果：
#编译
#在终端中输入： D:\GoFiles\src\log_collect\tail>go build D:\GoFiles\src\log_collect\tail #运行 D:\GoFiles\src\log_collect\tail>tail.exe #返回结果： ############################# Producer Basics ############################# msg: msg: # list of brokers used for bootstrapping knowledge about the rest of the cl uster msg: # format: host1:port1,host2:port2 ... msg: bootstrap.servers=localhost:9092 msg: msg: # specify the compression codec for all data generated: none, gzip, snappy, lz4, zstd msg: compression.type=none msg: msg: # name of the partitioner class for partitioning events; default partition spreads data randomly msg: #partitioner.class= msg: msg: # the maximum amount of time the client will wait for the response of a req uest msg: #request.timeout.ms= msg: msg: # how long `KafkaProducer.send` and `KafkaProducer.partitionsFor` will bloc k for msg: #max.block.ms= msg: msg: # the producer will wait for up to the given delay to allow other records t o be sent so that the sends can be batched together msg: #linger.ms= msg: msg: # the maximum size of a request in bytes msg: #max.request.size= msg: msg: # the default batch size in bytes when batching multiple records sent to a partition msg: #batch.size= msg: msg: # the total bytes of memory the producer can use to buffer records waiting to be sent to the server msg: #buffer.memory=
#
二、els安装
1、安装Elasticsearch
elasticsearch-5.5.0.msi
安装目录
C:\Program Files\Elastic\Elasticsearch
#启动目录
C:\Program Files\Elastic\Elasticsearch\bin
运行：
elasticsearch.exe
#访问： http://127.0.0.1:9200/
2、kafka（windows版本安装）
软件部署：
kibana-5.5.1-windows-x86.zip
解压到下面目录中：
C:\Program Files (x86)\kibana
#修改配置文件
elasticsearch.url: "http://127.0.0.1:9200" #指向es的启动页面
#启动kibana
C:\Program Files (x86)\kibana\bin
运行：
kibana.bat
示例：
运行程序的前提条件：
先安装elastic.v2
安装包：
go get gopkg.in/olivere/elastic.v2
使用方法：
import "gopkg.in/olivere/elastic.v2"
目录：
代码：
vi main.go
package main
import (
"fmt"
elastic "gopkg.in/olivere/elastic.v2"
)
type Tweet struct {
User string
Message string
}
func main() {
client, err := elastic.NewClient(elastic.SetSniff(false), elastic.SetURL("http://127.0.0.1:9200/"))
if err != nil {
fmt.Println("connect es error", err)
return
}
fmt.Println("conn es succ")
for i := 0; i < 10000; i++ {
tweet := Tweet{User: "olivere", Message: "Take Five"}
_, err = client.Index().
Index("twitter").
Type("tweet").
Id(fmt.Sprintf("%d", i)).
BodyJson(tweet).
Do()
if err != nil {
// Handle error
panic(err)
return
}
}
fmt.Println("insert success")
}
#打开页面：
http://localhost:5601
输入： twitter --->再点Discover就可以看到下面页面。
3、
