kafka 集群的部署安装
这里我们罗列一下我们的环境 10.19.18.88 zk1 10.19.16.84 zk2 10.19.11.44 zk3
这里公司需要接入kafka用于zipkin来定位调用链
kafka 的地址是http://kafka.apache.org/
zipkin 的地址是https://github.com/openzipkin/zipkin/tree/master/zipkin-server#environment-variables
kafka-manager 地址是https://github.com/yahoo/kafka-manager
这里首先我们现在kafka的下载包
kafka download http://kafka.apache.org/downloads https://archive.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz
下载包之后我们直接解压使用，因为这里我们环境已经配置过zookeeper
所以这里我就不配置kafka/config/zookeeper.properties
我们直接修改kafka的配置文件：
# The id of the broker. This must be set to a unique integer for each broker. broker.id=1 ############################# Socket Server Settings ############################# listeners=PLAINTEXT://10.19.18.88:1092 port=1092 host.name=10.19.18.88 # The number of threads handling network requests num.network.threads=8 # The number of threads doing disk I/O num.io.threads=8 # The send buffer (SO_SNDBUF) used by the socket server socket.send.buffer.bytes=1048576 # The receive buffer (SO_RCVBUF) used by the socket server socket.receive.buffer.bytes=1048576 # The maximum size of a request that the socket server will accept (protection against OOM) socket.request.max.bytes=104857600 # The number of queued requests allowed before blocking the network threads queued.max.requests=100 # The purge interval (in number of requests) of the fetch request purgatory fetch.purgatory.purge.interval.requests=200 # The purge interval (in number of requests) of the producer request purgatory producer.purgatory.purge.interval.requests=200 ############################# Log Basics ############################# # A comma seperated list of directories under which to store log files log.dirs=/data/package/kafka/kafka-logs # The default number of log partitions per topic. num.partitions=24 # The number of threads per data directory to be used for log recovery at startup and flushing at shutdown. num.recovery.threads.per.data.dir=2 # The maximum size of message that the server can receive message.max.bytes=1000000 # Enable auto creation of topic on the server auto.create.topics.enable=true # The interval with which we add an entry to the offset index log.index.interval.bytes=4096 # The maximum size in bytes of the offset index log.index.size.max.bytes=10485760 # Allow to delete topics delete.topic.enable=true ############################# Log Flush Policy ############################# # The number of messages to accept before forcing a flush of data to disk log.flush.interval.messages=20000 # The maximum amount of time a message can sit in a log before we force a flush log.flush.interval.ms=10000 # The frequency in ms that the log flusher checks whether any log needs to be flushed to disk log.flush.scheduler.interval.ms=2000 ############################# Log Retention Policy ############################# # The minimum age of a log file to be eligible for deletion log.retention.hours=168 # A size-based retention policy for logs. log.retention.bytes=1073741824 # The maximum size of a log segment file. When this size is reached a new log segment will be created. log.segment.bytes=1073741824 # The interval at which log segments are checked to see if they can be deleted according # to the retention policies log.retention.check.interval.ms=300000 # The maximum time before a new log segment is rolled out (in hours) log.roll.hours=168 ############################# Zookeeper ############################# # Zookeeper connection string (see zookeeper docs for details). zookeeper.connect=10.19.18.88:12081,10.19.16.84:12081,10.19.11.44:12081 # Timeout in ms for connecting to zookeeper zookeeper.connection.timeout.ms=6000 # How far a ZK follower can be behind a ZK leader zookeeper.sync.time.ms=2000 ############################# Replication configurations ################ # default replication factors for automatically created topics default.replication.factor=3 # Number of fetcher threads used to replicate messages from a source broker. num.replica.fetchers=4 # The number of bytes of messages to attempt to fetch for each partition. replica.fetch.max.bytes=1048576 # max wait time for each fetcher request issued by follower replicas. replica.fetch.wait.max.ms=500 # The frequency with which the high watermark is saved out to disk replica.high.watermark.checkpoint.interval.ms=5000 # The socket timeout for network requests. replica.socket.timeout.ms=30000 # The socket receive buffer for network requests replica.socket.receive.buffer.bytes=65536 # If a follower hasn't sent any fetch requests or hasn't consumed up to the leaders log end offset for at least this time, the leader will remove the follower from isr replica.lag.time.max.ms=10000 # The socket timeout for controller-to-broker channels controller.socket.timeout.ms=30000 controller.message.queue.size=10
这里不同的就是上面红色标注的，我们这里有三台机器组成的kafka集群
vim bin/kafka-server-start.sh 修改最后一行 #exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@" exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "/data/package/kafka/config/server.properties"
同样的方法，我们修改一下其他的两台server的配置文件
然后我们就可以一次启动kafka了。
最后我们安装kafka-manager
功能： 为了简化开发者和服务工程师维护Kafka集群的工作，yahoo构建了一个叫做Kafka管理器的基于Web工具，叫做 Kafka Manager。这个管理工具可以很容易地发现分布在集群中的哪些topic分布不均匀，或者是分区在整个集群分布不均匀的的情况。它支持管理多个集群、选择副本、副本重新分配以及创建Topic。同时，这个管理工具也是一个非常好的可以快速浏览这个集群的工具，有如下功能： 1.管理多个kafka集群 2.便捷的检查kafka集群状态(topics,brokers,备份分布情况,分区分布情况) 3.选择你要运行的副本 4.基于当前分区状况进行 5.可以选择topic配置并创建topic(0.8.1.1和0.8.2的配置不同) 6.删除topic(只支持0.8.2以上的版本并且要在broker配置中设置delete.topic.enable=true) 7.Topic list会指明哪些topic被删除（在0.8.2以上版本适用） 8.为已存在的topic增加分区 9.为已存在的topic更新配置 10.在多个topic上批量重分区 11.在多个topic上批量重分区(可选partition broker位置) 安装步骤 1、获取kafka-manager源码，并编译打包 # cd /usr/local # git clone https://github.com/yahoo/kafka-manager # cd kafka-manager # ./sbt clean dist 注: 执行sbt编译打包可能花费很长时间，如果你hang在如下情况 将project/plugins.sbt 中的logLevel参数修改为logLevel := Level.Debug(默认为Warn) 2、安装配置 编译成功后，会在target/universal下生成一个zip包 # cd /usr/local/kafka-manager/target/universal # unzip kafka-manager-1.3.3.7.zip 将application.conf中的kafka-manager.zkhosts的值设置为你的zk地址 如：kafka-manager.zkhosts="172.16.218.201:2181,172.16.218.202:2181,172.16.218.203:2181" 3、启动,指定配置文件位置和启动端口号，默认为9000 直接启动： # cd kafka-manager-1.3.3.7/bin # ./kafka-manager -Dconfig.file=../conf/application.conf 后台运行： # ./kafka-manager -h # nohup ./kafka-manager -Dconfig.file=../conf/application.conf & 指定端口，例如： # nohup bin/kafka-manager -Dconfig.file=conf/application.conf -Dhttp.port=9001 & 第一次进入web UI要进行kafka cluster的相关配置，根据自己的信息进行配置。
参考文章链接：
kafka https://www.cnblogs.com/shiyiwen/p/6150213.html http://blog.51cto.com/xiangcun168/1933509 http://orchome.com/41 http://www.360doc.com/content/16/1117/16/37253246_607304757.shtml http://jayveehe.github.io/2017/02/01/elk-stack/ http://blog.51cto.com/wuyebamboo/1963786 https://facingissuesonit.com/2017/05/29/integrate-filebeat-kafka-logstash-elasticsearch-and-kibana/ https://www.yuanmas.com/info/GlypPG18y2.html https://www.cnblogs.com/yinchengzhe/p/5111635.html kafka 参数 https://www.cnblogs.com/weixiuli/p/6413109.html kafka 配置文件参数详解
