linux上kafka搭建小结
1、安装java
yum -y list java
可以查看java的版本，然后选1.8的64位那个安装
yum install java_1.8.0_openjdk.x86_64
或者从网站下载安装
tar -zxvf jre-8u271-linux-x64.tar.gz
mkdir /usr/lib/jvm
mv jre-8u271-linux-x64 /usr/lib/jvm
chown root:root -R /usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.272.b10-1.el7_9.x86_64
java安装之后就开始设置环境变量
vi /etc/profile
#set java environment JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.272.b10-1.el7_9.x86_64 PATH=$PATH:$JAVA_HOME/bin CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOME CLASSPATH PATH
使之生效命令
source /etc/profile
或者
. /etc/profile
2、 zookeeper
装kafka的时候才知道，必须要先安装zookeeper
从官网下载了3.5.5的包
http://archive.apache.org/dist/zookeeper/zookeeper-3.5.5/
tar zxvf apache-zookeeper-3.5.5-bin.tar.gz cd apache-zookeeper-3.5.5-bin cd conf cp zoo_sample.cfg zoo.cfg vi zoo.cfg
修改dataDir=后面的路径，就改成程序路径下，然后mkdir data | cd data | mkdir zookeeper | mkdir log
修改环境变量/etc/profile
ZOOKEEPER_HOME=/root/hzt/apache-zookeeper-3.5.5-bin
PATH=$PATH:$ZOOKEEPER_HOME/bin
并生效
source /etc/profile
启动程序
sh zkServer.sh start
停止命令把start改成stop就行。
测试下可以用windows的cmd
telnet 替换实际的机器ip 2181
或者linux下看状态
sh zkServer.sh status
查看状态发现了一个状况
ZooKeeper JMX enabled by default Using config: /home/opt/hz/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Error contacting service. It is probably not running.
查看logs文件夹下的.out文件
2020-12-14 16:59:13,825 [myid:] - INFO [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0 .0:2181 2020-12-14 16:59:13,825 [myid:] - INFO [main:QuorumPeerConfig@389] - secureClientPort is not set 2020-12-14 16:59:13,830 [myid:] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 2020-12-14 16:59:13,832 [myid:] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0 2020-12-14 16:59:13,832 [myid:] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled. 2020-12-14 16:59:13,832 [myid:] - WARN [main:QuorumPeerMain@125] - Either no config or no quorum define d
zoo.cfg中datadir也配置过了，修改了下也没效果。然后把系统中开启的tomcat关掉后再启动就行了。不知道是不是跟8080端口被占用有关系
[opt@test bin]$ zkServer.sh status ZooKeeper JMX enabled by default Using config: /home/opt/hz/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Mode: standalone
3、安装kafka
从官网下载非源码版本的程序，带有src安装了会有问题
tar xzvf kafka_2.12-2.3.0.tgz
cd kafka_2.12-2.3.0
启动程序
bin/kafka-server-start.sh config/server.properties &
检测2181与9092端口
netstat -tunlp|egrep '(2181|9092)'
在用普通用户启动时候会报
Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient) [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper) ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
换成root用户启动就好了，怀疑普通用户没有默认配置中跟路径权限。后期把配置修改下再试试。
功能测试，新开一个窗口作为生产者
运行producer
bin/kafka-console-producer.sh --broker-list 192.168.1.1:9092 --topic test
再开一个窗口作为消费者
sh kafka_2.13-2.5.0/bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.1:9092 --topic test --from-beginning
然后生产者界面输入的内容会在消费者界面出现
4、客户端开发
从github上download相应的src，使用librdkafka
unzip librdkafka-master.zip
cd librdkafka-master
sh ./configure
make
sudo make install
https://github.com/edenhill/librdkafka
完了之后可以进examples文件夹中运行demo
生产者：
./producer kafka机器的ip:9092 topic
然后就可以丢数据进去了
消费者：
./consumer kafka机器的ip:9092 groupid(Consumer group id) topics
可以订阅一个或者多个主题，组id的用途还没搞懂
如果遇到连接被kafka拒绝的情况，则需要修改下kafka的conf/server.properties中的listeners这项
本机运行的话可以改成
listeners=PLAINTEXT://localhost:9092
参考文档：
1、http://www.xwood.net/_site_domain_/_root/5870/5930/5932/t_c261426.html
2、https://www.cnblogs.com/wjup/p/11041274.html
3、https://blog.csdn.net/qq_41094332/article/details/104366315
4、https://www.cnblogs.com/justuntil/p/8033792.html
5、https://www.cnblogs.com/wangyangliuping/p/5546506.html
