|NO.Z.00050|——————————|BigDataEnd|——|Hadoop&Flink.V05|——|Flink.v05|Flink Connector|kafka|源码理解|源码说明.V3|

一、源码提取说明

### --- kafkaFetcher的runFetchLoop方法
~~~     此方法为FlinkKafkaConsumer获取数据的主入口，通过一个循环来不断获取kafka broker的数据。

~~~     # 源码提取说明：KafkaFetcher.java：方法实现
~~~     # 第124~144行
    @Override
    public void runFetchLoop() throws Exception {
        try {
            // kick off the actual Kafka consumer
            consumerThread.start(); # consumerThread线程启动，需观察consumerThread线程的run方法

            while (running) {
                // this blocks until we get the next records
                // it automatically re-throws exceptions encountered in the consumer thread
                final ConsumerRecords<byte[], byte[]> records = handover.pollNext();

                // get the records for each topic partition
                for (KafkaTopicPartitionState<T, TopicPartition> partition : subscribedPartitionStates()) {

                    List<ConsumerRecord<byte[], byte[]>> partitionRecords =
                        records.records(partition.getKafkaPartitionHandle());

                    partitionConsumerRecordsHandler(partitionRecords, partition);
                }
            }
        }

### --- KafkaConsumerThread线程的run方法实例化handover

~~~     # 源码提取方法：KafkaConsumerThread.java
~~~     # 第153~161行
    @Override
    public void run() {
        // early exit check
        if (!running) {
            return;
        }

        // this is the means to talk to FlinkKafkaConsumer's main thread
        final Handover handover = this.handover; # 实例化handover，作用是和FlinkKafkaConsumer的main方法保持会话

### --- 回到KafkaFecher类中的runFetchLoop方法

~~~     # 源码提取说明：KafkaFetcher.java：方法实现
~~~     # 第124~148行
    @Override
    public void runFetchLoop() throws Exception {
        try {
            // kick off the actual Kafka consumer
            consumerThread.start(); # 消费线程启动，定期将消费到的数据转交给handover

            while (running) {
                // this blocks until we get the next records
                // it automatically re-throws exceptions encountered in the consumer thread
                final ConsumerRecords<byte[], byte[]> records = handover.pollNext();
                                                            # 获取handover中的数据，在consumerThread线程没有
                // get the records for each topic partition # 将数据转发给handover之前，这个方法会阻塞
                for (KafkaTopicPartitionState<T, TopicPartition> partition : subscribedPartitionStates()) {
                                                            # 所有订阅区
                    List<ConsumerRecord<byte[], byte[]>> partitionRecords =
                        records.records(partition.getKafkaPartitionHandle()); # 获取属于该分区的record

                    partitionConsumerRecordsHandler(partitionRecords, partition);
                }
            }
        }
        finally {
            // this signals the consumer thread that no more work is to be done
            consumerThread.shutdown();
        }

### --- partitionConsumerRecordsHandler方法

~~~     # 源码提取说明：KafkaFetcher.java
~~~     # 第176~197行
    protected void partitionConsumerRecordsHandler(
            List<ConsumerRecord<byte[], byte[]>> partitionRecords,
            KafkaTopicPartitionState<T, TopicPartition> partition) throws Exception {

        for (ConsumerRecord<byte[], byte[]> record : partitionRecords) {
            deserializer.deserialize(record, kafkaCollector);
            # 反序列化record，并将数据交给kafkaCollector，以备将数据向下游发送
            // emit the actual records. this also updates offset state atomically and emits
            // watermarks
            emitRecordsWithTimestamps( # 发送数据，更新offset，生成timestamp和watermarkets
                kafkaCollector.getRecords(),
                partition,
                record.offset(),
                record.timestamp());
                # 如果数据源已到末尾（收到流结束信号），停止fetcher循环
            if (kafkaCollector.isEndOfStreamSignalled()) {
                // end of stream signaled
                running = false;
                break;
            }
        }
    }

---

 

 

 

 

 

 

 

 

 

---

Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart

                                                                                                                                                   ——W.S.Landor

---