Kafka 的安装及启动
Linux 服务器CentOS 为例。 Kafka 安装获取下载地址，下载后解压。 $ wget https://mirror.bit.edu.cn/apache/kafka/2.5.0/kafka_2.12-2.5.0.tgz
$ tar -zxvf kafka_2.12-2.5.0.tgz && cd kafka_2.12-2.5.0
具体参考官网 Kafka - Quickstart。 启动 KafkaKafka 依赖 ZooKeeper，所以需要先运行后者。Kafak 安装目录下自带了 ZooKeeper，可直接运行无须单独安装。 $ bin/zookeeper-server-start.sh config/zookeeper.properties
运行 Zookeeper 时发现机器上没有 Java，报错信息如下： $ bin/zookeeper-server-start.sh config/zookeeper.properties
/root/dev/kafka/bin/kafka-run-class.sh: line 315: exec: java: not found
所以需要先安装 Java。 安装 Java 过程中如果 yum 报 https 错误，提示 404 repo 地址无效等， yum install java 时的报错信息$ yum install java-1.6.0-openjdk
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
base | 3.6 kB 00:00:00
epel | 4.7 kB 00:00:00
extras | 3.4 kB 00:00:00
https://repo.mongodb.org/yum/redhat/2.2/mongodb-org/4.2/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article
https://access.redhat.com/articles/1320623
If above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/
One of the configured repositories failed (MongoDB Repository),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Disable the repository, so yum won't use it by default. Yum will then
just ignore the repository until you permanently enable it again or use
--enablerepo for temporary usage:
yum-config-manager --disable mongodb-org-4.2
4. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=mongodb-org-4.2.skip_if_unavailable=true
failure: repodata/repomd.xml from mongodb-org-4.2: [Errno 256] No more mirrors to try.
https://repo.mongodb.org/yum/redhat/2.2/mongodb-org/4.2/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
根据提示，执行 检查 Java 的安装： $ java -version
java version "1.6.0_38"
OpenJDK Runtime Environment (IcedTea6 1.13.10) (rhel-1.13.10.0.el7_2-x86_64)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)
另，如果需要卸载，执行 再次运行 $ bin/zookeeper-server-start.sh config/zookeeper.properties
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/zookeeper/server/quorum/QuorumPeerMain : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.zookeeper.server.quorum.QuorumPeerMain. Program will exit.
根据提示最小应该为 52 的版本，根据 Unsupported major.minor version 52.0 [duplicate]可知 52 对应的语义化版本。 同时，可通过 再次检查安装： $ java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
启动 Kafka 服务$ bin/kafka-server-start.sh config/server.properties
创建 Topic$ bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
创建生产者创建生产者发送消息 $ bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test
创建消费者$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
此时发消息的地方有新增数据时，消费者处会实时获取到。 mac 本机修改 Kafka 配置$ vi /usr/local/etc/kafka/server.properties
将 # The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
- listeners=PLAINTEXT://:9092
+ listeners=PLAINTEXT://localhost:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
启动 zookeeper$ brew services start zookeeper
==> Successfully started `zookeeper` (label: homebrew.mxcl.zookeeper)
注： 通过 brew 和 zookeeper 自己的命令启动时，停止服务也需要对应的命令，即， 启动 kafka$ brew services start kafka
==> Successfully started `kafka` (label: homebrew.mxcl.kafka)
上面是以服务形式常驻启动，如果临时启动，可使用下面的命令： $ zkServer start
$ kafka-server-start /usr/local/etc/kafka/server.properties
创建 Topic$ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
如果出现如下错误： Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:262)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:258)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1863)
at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:341)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:55)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
$ zkServer status 22:22:32
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.
如果是像上面这样，zookeeper 根本没启动成功。虽然 然后不要相信网上列的那一系列可能原因：
此时需要做的是先停掉服务， $ brew services start zookeeper
# 或
$ zkServer stop
然后打开 $ cat /usr/local/etc/zookeeper/log4j.properties
log4j.rootCategory=WARN, zklog
log4j.appender.zklog = org.apache.log4j.RollingFileAppender
log4j.appender.zklog.File = /usr/local/var/log/zookeeper/zookeeper.log
log4j.appender.zklog.Append = true
log4j.appender.zklog.layout = org.apache.log4j.PatternLayout
log4j.appender.zklog.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n
可以看到，日志文件在 新开窗口使用 $ tail -n 50 -f /usr/local/var/log/zookeeper/zookeeper.log
2020-05-13 09:58:43 QuorumPeerMain [WARN] Either no config or no quorum defined in config, running in standalone mode
2020-05-13 09:58:43 ContextHandler [WARN] o.e.j.s.ServletContextHandler@e45f292{/,null,UNAVAILABLE} contextPath ends with /*
2020-05-13 09:58:43 ContextHandler [WARN] Empty contextPath
2020-05-13 09:58:43 NIOServerCnxnFactory [ERROR] Thread Thread[main,5,main] died
java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
at org.apache.jute.BinaryOutputArchive.stringToByteBuffer(BinaryOutputArchive.java:77)
at org.apache.jute.BinaryOutputArchive.writeString(BinaryOutputArchive.java:107)
at org.apache.zookeeper.data.Id.serialize(Id.java:50)
at org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
at org.apache.zookeeper.data.ACL.serialize(ACL.java:51)
at org.apache.zookeeper.server.ReferenceCountedACLCache.serialize(ReferenceCountedACLCache.java:136)
at org.apache.zookeeper.server.DataTree.serialize(DataTree.java:1218)
at org.apache.zookeeper.server.util.SerializeUtils.serializeSnapshot(SerializeUtils.java:152)
at org.apache.zookeeper.server.persistence.FileSnap.serialize(FileSnap.java:210)
at org.apache.zookeeper.server.persistence.FileSnap.serialize(FileSnap.java:227)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.save(FileTxnSnapLog.java:406)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:248)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
原因就很明朗了， java 中报错找不到对应方法。根据 Kafka with Zookeeper 3.5.7 Crash NoSuchMethodError: java.nio.ByteBuffer.flip() 这个类似问题下的回答，解决办法有两个：
看了下使用 brew 安装后的版本： $ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b09, mixed mode)
$ brew info zookeeper
zookeeper: stable 3.5.7 (bottled), HEAD
Centralized server for distributed coordination of services
...
因为是本机，升级 java 会比较容易操作些，直接： $ brew cask install java
...
完成后查看 java 版本，已经来到了最新的 14.0： $ java -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7)
OpenJDK 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)
再次启动 zookeeper， $ zkServer start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Starting zookeeper ... STARTED
同时查看日志输出： 可以看到，仍然报错，但错误已经变了。看来前面的步骤起作用了。再来看新的报错， 错误信息里关键信息是 看来是端口占用的问题，查看 zookeeper 配置并添加 $ vi /usr/local/etc/zookeeper/zoo.cfg
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
+ admin.serverPort=8089
再次尝试启动并查看日志： 错误已经更新成了 $ brew services start zookeeper
# 或
$ zkServer stop
再次启动： $ zkServer start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Starting zookeeper ... STARTED
查看日志已经没有异样了： 再通过 $ zkServer status 11:57:01
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: standalone
最后，可以成功创建 Topic 了: $ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic test.
创建生产者创建生产者并生成消息： $ kafka-console-producer --broker-list localhost:9092 --topic test
> hello
以上命令执行后会进入交互命令行，每输入一行代表一条消息。 创建消费者消费消息： $ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning
> hello
启动后会从队列起始处开始消费消息并输出。 相关资源 |
|
The text was updated successfully, but these errors were encountered: |
CC BY-NC-SA 署名-非商业性使用-相同方式共享
