Kafka学习笔记——存储结构

1，由cdh安装的kafka的默认存储路径如图所示在/var/local/kafka/data，一般会进行修改

kafka配置参考：apache kafka系列之server.properties配置文件参数说明

路径下文件如下

如果是多个路径的话，使用,进行分隔，比如/data01/kafka/data,/data02/kafka/data，注意data的权限需要是kafka用户和kafka组

对应kafka manager上的topic

具体的topic的目录下的文件

lintong@master:/var/local/kafka/data/test_topic-0$ ls
00000000000000000187.index  00000000000000000187.snapshot   leader-epoch-checkpoint
00000000000000000187.log    00000000000000000187.timeindex

其中

187就是这个日志数据文件开始的offset

00000000000000000187.log是日志数据文件

可以使用解码命令查看日志片段中的内容

/opt/cloudera/parcels/KAFKA/bin/kafka-run-class kafka.tools.DumpLogSegments --files ./00000000000000000187.log
20/07/19 17:28:55 INFO utils.Log4jControllerRegistration$: Registered kafka:type=kafka.Log4jController MBean
Dumping ./00000000000000000187.log
Starting offset: 187
baseOffset: 187 lastOffset: 187 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 0 CreateTime: 1595149737820 size: 71 magic: 2 compresscodec: NONE crc: 1738564593 isvalid: true
baseOffset: 188 lastOffset: 188 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 71 CreateTime: 1595149744449 size: 74 magic: 2 compresscodec: NONE crc: 3794338178 isvalid: true
baseOffset: 189 lastOffset: 189 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 145 CreateTime: 1595149757514 size: 75 magic: 2 compresscodec: NONE crc: 3084259622 isvalid: true

log.segment.bytes设置是1G，如果log文件的大小达到1G之后会生成另外一个log文件

该参数在1.0.1及以下的kafka有bug，可能会影响消费者消费topic的数据，但是不影响生产者，参考：https://issues.apache.org/jira/browse/KAFKA-6292

当kafka的broker读取segment文件的时候，会判断当前当前读取的segment的偏移量position在继续读取一段HEADER_SIZE_UP_TO_MAGIC之后和该segment文件最大可读取的偏移量end之间的大小

当调高了log.segment.bytes=2G，注意此处2G=有符号INT的最大值=2147483647，有可能导致position+HEADER_SIZE_UP_TO_MAGIC的大小超过int最大值，从而成为负数，小于end，返回null，并导致从log文件的末尾开始读取数据

参考kafka 1.0.1版本源码：https://github.com/apache/kafka/blob/1.0.1/clients/src/main/java/org/apache/kafka/common/record/FileLogInputStream.java

 

flume消费者报错：

24 Mar 2020 01:55:05,055 WARN  [PollableSourceRunner-KafkaSource-source1] (org.apache.kafka.clients.consumer.internals.Fetcher.handleFetchResponse:600)  - Unknown error fetching data for topic-partition xxxx-17

kafka端报错：

2020-03-24 15:57:60,265 ERROR kafka.server.ReplicaManager: [ReplicaManager broker=xxx] Error processing fetch operatio
n on partition xxx-17, offset 4095495430
org.apache.kafka.common.KafkaException: java.io.EOFException: Failed to read `log header` from file channel `sun.nio.ch
.FileChannelImpl@73e125b5`. Expected to read 18 bytes, but reached end of file after reading 0 bytes. Started read from
 position 2147454346.
    at org.apache.kafka.common.record.RecordBatchIterator.makeNext(RecordBatchIterator.java:40)
    at org.apache.kafka.common.record.RecordBatchIterator.makeNext(RecordBatchIterator.java:24)
    at org.apache.kafka.common.utils.AbstractIterator.maybeComputeNext(AbstractIterator.java:79)
    at org.apache.kafka.common.utils.AbstractIterator.hasNext(AbstractIterator.java:45)
    at org.apache.kafka.common.record.FileRecords.searchForOffsetWithSize(FileRecords.java:287)
    at kafka.log.LogSegment.translateOffset(LogSegment.scala:174)
    at kafka.log.LogSegment.read(LogSegment.scala:226)
    at kafka.log.Log$$anonfun$read$2.apply(Log.scala:1002)
    at kafka.log.Log$$anonfun$read$2.apply(Log.scala:958)
    at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
    at kafka.log.Log.read(Log.scala:958)
    at kafka.server.ReplicaManager.kafka$server$ReplicaManager$$read$1(ReplicaManager.scala:900)
    at kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:962)
    at kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:961)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at kafka.server.ReplicaManager.readFromLocalLog(ReplicaManager.scala:961)
    at kafka.server.ReplicaManager.readFromLog$1(ReplicaManager.scala:790)
    at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:803)
    at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:601)
    at kafka.server.KafkaApis.handle(KafkaApis.scala:99)
    at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:65)
    at java.lang.Thread.run(Thread.java:748)

 

00000000000000000187.index是偏移量索引文件

因为index文件中key存储是的offset，而value存储的是对应log文件中的偏移量，所以index文件的作用是通过二分查找帮助client更快地找到所需offset的message在log文件中的位置

参考：https://mrbird.cc/Kafka%E5%AD%98%E5%82%A8%E6%9C%BA%E5%88%B6.html

/opt/cloudera/parcels/KAFKA/bin/kafka-run-class kafka.tools.DumpLogSegments --files ./00000000000000000187.index
20/07/19 19:05:08 INFO utils.Log4jControllerRegistration$: Registered kafka:type=kafka.Log4jController MBean
Dumping ./00000000000000000187.index
offset: 187 position: 0

00000000000000000187.timeindex是时间戳索引文件

/opt/cloudera/parcels/KAFKA/bin/kafka-run-class kafka.tools.DumpLogSegments --files ./00000000000000000187.timeindex
20/07/19 18:47:31 INFO utils.Log4jControllerRegistration$: Registered kafka:type=kafka.Log4jController MBean
Dumping ./00000000000000000187.timeindex
timestamp: 0 offset: 187
Found timestamp mismatch in :/var/local/kafka/data/test_topic-0/./00000000000000000187.timeindex
  Index timestamp: 0, log timestamp: 1595149737820
  Index timestamp: 0, log timestamp: 1595149737820
Found out of order timestamp in :/var/local/kafka/data/test_topic-0/./00000000000000000187.timeindex
  Index timestamp: 0, Previously indexed timestamp: 0

1595149737820毫秒转换成unix时间是2020/7/19 17:8:57，即这个日志文件开始的时间戳

index文件和log文件的大小还可以在创建topic的时候设置，如下

segment.index.bytes参数决定了index文件大小达到多大之后进行切分，默认大小是10M，如下

 

segment.bytes参数决定了log文件大小达到多大之后进行切分

 

 

可能还会有其他文件，比如：

.snapshot(快照文件)、.deleted()、.cleaned(日志清理临时文件)、.swap(Log Compaction之后的临时文件)、leader-epoch-checkpoint（保存了每一任leader开始写入消息时的offset，会定时更新。 follower被选为leader时会根据这个确定哪些消息可用）

 

kafka在写入和读取上都做了优化

参考：https://juejin.im/post/5cd2db8951882530b11ee976

写入：顺序写入、MMFile（Memory Mapped Files，内存映射文件）

读取：基于sendfile实现Zero Copy、批量压缩

 

Kafka中的索引机制

其中介绍了跳表