kafka运维填坑
转载自：https://www.jianshu.com/p/d2cbaae38014
- 前提: 只针对Kafka 0.9.0.1版本;
- 说是运维,其实偏重于问题解决;
- 大部分解决方案都是google而来, 我只是作了次搬运工;
- 有些问题的解决方案未必一定是通用的, 若应用到线上请慎重;
- 如有疏漏之处, 欢迎大家批评指正;
- 列表:
- Replica无法从leader同步消息
- Broker到zk集群的连接不时会断开重断
- Broker重启耗时很久
- 不允许脏主选举导致Broker被强制关闭
- Replica从错误的Partition leader上去同步数据
- __consumer_offsets日志无法被清除
- GC问题
- zk和kafka部署
- 监控很重要
- 大量异常:
Attempted to decrease connection count for address with no connections - 新版sdk访问较旧版的kafka, 发送kafka不支持的request
- 频繁FullGC
- 机器Swap使用
Replica无法从leader同步消息
- 现象: 集群上某topic原来只有单复本, 增加双复本后,发现有些partition没有从leader同步数据,导致isr列表中一直没有新增的replica;
- 日志分析:
[2017-09-20 19:37:05,265] ERROR Found invalid messages during fetch for partition [xxxx,87] offset 1503297 error Message is corrupt (stored crc = 286782282, computed crc = 400317671) (kafka.server.ReplicaFetcherThread)
[2017-09-20 19:37:05,458] ERROR Found invalid messages during fetch for partition [xxxx,75] offset 1501373 error Message found with corrupt size (0) in shallow iterator (kafka.server.ReplicaFetcherThread)
[2017-09-20 19:37:07,455] ERROR [ReplicaFetcherThread-0-5], Error due to (kafka.server.ReplicaFetcherThread)
kafka.common.KafkaException: error processing data for partition [xxxx,87] offset 1503346
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:147)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:122)
at scala.Option.foreach(Option.scala:257)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:122)
at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:120)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach