Kafka - 消费接口分析
转自: http://www.cnblogs.com/smartloli/p/5241067.html#top
1.概述
在 Kafka 中,官方对外提供了两种消费 API,一种是高等级消费 API,另一种是低等级的消费 API。在 《高级消费 API》一文中,介绍了其高级消费的 API 实现。今天给大家介绍另一种消费 API。
2.内容
在使用过 Kafka 的高级消费 API 后,我们知道它是一种高度抽象的消费 API,使用起来简单,方便,但是对于某些特殊的需求我们可能要用到第二种更加底层的 API。那么,我们首先需要知道低级消费 API 的作用。它能帮助我们去做那些事情:
- 一个消息进行多次读取
- 在处理过程中只消费 Partition 其中的某一部分消息
- 添加事物管理机制以保证消息仅被处理一次
当然,在使用的过程当中也是有些弊端的,其内容如下:
- 必须在程序中跟踪 Offset 的值
- 必须找出指定的 Topic Partition 中的 Lead Broker
- 必须处理 Broker 的变动
使用其 API 的思路步骤如下所示:
- 从所有处于 Active 状态的 Broker 中找出哪个是指定 Topic Partition 中的 Lead Broker
- 找出指定 Topic Partition 中的所有备份 Broker
- 构造请求
- 发送请求并查询数据
- 处理 Leader Broker 的变动
3.代码实现
3.1 Java Project
若是使用 Java Project 工程去实现该部分代码,需要添加相关以来 JAR 文件,其内容包含如下:
- scala-xml_${version}-${version}.jar
- scala-library-${version}.jar
- metrics-core-${version}.jar
- kafka-client-${version}.jar
- kafka_${version}-${version}.jar
针对 Java Project 工程,需要自己筛选 JAR 去添加。保证代码的顺利执行。
3.2 Maven Project
对 Maven 工程,在 pom.xml 文件中添加相应的依赖信息即可,简单方便。让 Maven 去管理相应的依赖 JAR 文件。内容如下所示:
1 <dependency> 2 <groupId>org.apache.kafka</groupId> 3 <artifactId>kafka_2.11</artifactId> 4 <version>0.8.2.1</version> 5 <exclusions> 6 <exclusion> 7 <groupId>org.apache.zookeeper</groupId> 8 <artifactId>zookeeper</artifactId> 9 </exclusion> 10 <exclusion> 11 <groupId>log4j</groupId> 12 <artifactId>log4j</artifactId> 13 </exclusion> 14 </exclusions> 15 </dependency>
这样在 Maven 工程中相应的依赖 JAR 文件就添加完成了。
3.3 代码实现
在低级消费 API 中,实现代码如下所示:
1 public class SimpleKafkaConsumer { 2 private static Logger log = LoggerFactory.getLogger(SimpleKafkaConsumer.class); 3 private List<String> m_replicaBrokers = new ArrayList<String>(); 4 5 public SimpleKafkaConsumer() { 6 m_replicaBrokers = new ArrayList<String>(); 7 } 8 9 public static void main(String[] args) { 10 SimpleKafkaConsumer example = new SimpleKafkaConsumer(); 11 // Max read number 12 long maxReads = SystemConfig.getIntProperty("kafka.read.max"); 13 // To subscribe to the topic 14 String topic = SystemConfig.getProperty("kafka.topic"); 15 // Find partition 16 int partition = SystemConfig.getIntProperty("kafka.partition"); 17 // Broker node's ip 18 List<String> seeds = new ArrayList<String>(); 19 String[] hosts = SystemConfig.getPropertyArray("kafka.server.host", ","); 20 for (String host : hosts) { 21 seeds.add(host); 22 } 23 int port = SystemConfig.getIntProperty("kafka.server.port"); 24 try { 25 example.run(maxReads, topic, partition, seeds, port); 26 } catch (Exception e) { 27 log.error("Oops:" + e); 28 e.printStackTrace(); 29 } 30 } 31 32 public void run(long a_maxReads, String a_topic, int a_partition, List<String> a_seedBrokers, int a_port) 33 throws Exception { 34 // Get point topic partition's meta 35 PartitionMetadata metadata = findLeader(a_seedBrokers, a_port, a_topic, a_partition); 36 if (metadata == null) { 37 log.info("[SimpleKafkaConsumer.run()] - Can't find metadata for Topic and Partition. Exiting"); 38 return; 39 } 40 if (metadata.leader() == null) { 41 log.info("[SimpleKafkaConsumer.run()] - Can't find Leader for Topic and Partition. Exiting"); 42 return; 43 } 44 String leadBroker = metadata.leader().host(); 45 String clientName = "Client_" + a_topic + "_" + a_partition; 46 47 SimpleConsumer consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName); 48 long readOffset = getLastOffset(consumer, a_topic, a_partition, kafka.api.OffsetRequest.EarliestTime(), 49 clientName); 50 int numErrors = 0; 51 while (a_maxReads > 0) { 52 if (consumer == null) { 53 consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName); 54 } 55 FetchRequest req = new FetchRequestBuilder().clientId(clientName) 56 .addFetch(a_topic, a_partition, readOffset, 100000).build(); 57 FetchResponse fetchResponse = consumer.fetch(req); 58 59 if (fetchResponse.hasError()) { 60 numErrors++; 61 // Something went wrong! 62 short code = fetchResponse.errorCode(a_topic, a_partition); 63 log.info("[SimpleKafkaConsumer.run()] - Error fetching data from the Broker:" + leadBroker 64 + " Reason: " + code); 65 if (numErrors > 5) 66 break; 67 if (code == ErrorMapping.OffsetOutOfRangeCode()) { 68 // We asked for an invalid offset. For simple case ask for 69 // the last element to reset 70 readOffset = getLastOffset(consumer, a_topic, a_partition, kafka.api.OffsetRequest.LatestTime(), 71 clientName); 72 continue; 73 } 74 consumer.close(); 75 consumer = null; 76 leadBroker = findNewLeader(leadBroker, a_topic, a_partition, a_port); 77 continue; 78 } 79 numErrors = 0; 80 81 long numRead = 0; 82 for (MessageAndOffset messageAndOffset : fetchResponse.messageSet(a_topic, a_partition)) { 83 long currentOffset = messageAndOffset.offset(); 84 if (currentOffset < readOffset) { 85 log.info("[SimpleKafkaConsumer.run()] - Found an old offset: " + currentOffset + " Expecting: " 86 + readOffset); 87 continue; 88 } 89 90 readOffset = messageAndOffset.nextOffset(); 91 ByteBuffer payload = messageAndOffset.message().payload(); 92 93 byte[] bytes = new byte[payload.limit()]; 94 payload.get(bytes); 95 System.out.println(String.valueOf(messageAndOffset.offset()) + ": " + new String(bytes, "UTF-8")); // Message deal enter 96 numRead++; 97 a_maxReads--; 98 } 99 100 if (numRead == 0) { 101 try { 102 Thread.sleep(1000); 103 } catch (InterruptedException ie) { 104 } 105 } 106 } 107 if (consumer != null) 108 consumer.close(); 109 } 110 111 public static long getLastOffset(SimpleConsumer consumer, String topic, int partition, long whichTime, 112 String clientName) { 113 TopicAndPartition topicAndPartition = new TopicAndPartition(topic, partition); 114 Map<TopicAndPartition, PartitionOffsetRequestInfo> requestInfo = new HashMap<TopicAndPartition, PartitionOffsetRequestInfo>(); 115 requestInfo.put(topicAndPartition, new PartitionOffsetRequestInfo(whichTime, 1)); 116 kafka.javaapi.OffsetRequest request = new kafka.javaapi.OffsetRequest(requestInfo, 117 kafka.api.OffsetRequest.CurrentVersion(), clientName); 118 OffsetResponse response = consumer.getOffsetsBefore(request); 119 120 if (response.hasError()) { 121 log.info("[SimpleKafkaConsumer.getLastOffset()] - Error fetching data Offset Data the Broker. Reason: " 122 + response.errorCode(topic, partition)); 123 return 0; 124 } 125 long[] offsets = response.offsets(topic, partition); 126 return offsets[0]; 127 } 128 129 /** 130 * @param a_oldLeader 131 * @param a_topic 132 * @param a_partition 133 * @param a_port 134 * @return String 135 * @throws Exception 136 * find next leader broker 137 */ 138 private String findNewLeader(String a_oldLeader, String a_topic, int a_partition, int a_port) throws Exception { 139 for (int i = 0; i < 3; i++) { 140 boolean goToSleep = false; 141 PartitionMetadata metadata = findLeader(m_replicaBrokers, a_port, a_topic, a_partition); 142 if (metadata == null) { 143 goToSleep = true; 144 } else if (metadata.leader() == null) { 145 goToSleep = true; 146 } else if (a_oldLeader.equalsIgnoreCase(metadata.leader().host()) && i == 0) { 147 // first time through if the leader hasn't changed give 148 // ZooKeeper a second to recover 149 // second time, assume the broker did recover before failover, 150 // or it was a non-Broker issue 151 // 152 goToSleep = true; 153 } else { 154 return metadata.leader().host(); 155 } 156 if (goToSleep) { 157 try { 158 Thread.sleep(1000); 159 } catch (InterruptedException ie) { 160 } 161 } 162 } 163 throw new Exception("Unable to find new leader after Broker failure. Exiting"); 164 } 165 166 private PartitionMetadata findLeader(List<String> a_seedBrokers, int a_port, String a_topic, int a_partition) { 167 PartitionMetadata returnMetaData = null; 168 loop: for (String seed : a_seedBrokers) { 169 SimpleConsumer consumer = null; 170 try { 171 consumer = new SimpleConsumer(seed, a_port, 100000, 64 * 1024, "leaderLookup"); 172 List<String> topics = Collections.singletonList(a_topic); 173 TopicMetadataRequest req = new TopicMetadataRequest(topics); 174 kafka.javaapi.TopicMetadataResponse resp = consumer.send(req); 175 176 List<TopicMetadata> metaData = resp.topicsMetadata(); 177 for (TopicMetadata item : metaData) { 178 for (PartitionMetadata part : item.partitionsMetadata()) { 179 if (part.partitionId() == a_partition) { 180 returnMetaData = part; 181 break loop; 182 } 183 } 184 } 185 } catch (Exception e) { 186 log.error("Error communicating with Broker [" + seed + "] to find Leader for [" + a_topic + ", " 187 + a_partition + "] Reason: " + e); 188 } finally { 189 if (consumer != null) 190 consumer.close(); 191 } 192 } 193 if (returnMetaData != null) { 194 m_replicaBrokers.clear(); 195 for (kafka.cluster.Broker replica : returnMetaData.replicas()) { 196 m_replicaBrokers.add(replica.host()); 197 } 198 } 199 return returnMetaData; 200 } 201 }
4.总结
在使用 Kafka 低级消费 API 时,要明确我们所使用的业务场景,一般建议还是使用高级消费 API,除非遇到特殊需要。另外,在使用过程中,注意 Leader Broker 的处理,和 Offset 的管理。