golang kafka clinet 内存泄露问题处理

go 内存泄露

新版本服务跑上一天内存占用20g，显然是内存泄露

内存泄露的问题难在定位

技术上的定位 主要靠 pprof 生成统计文件

之前写web项目 基于net/http/pprof 可以看到运行态信息，协助排查

https://github.com/cclient/gowebframework/blob/f691321b898484327ceb0dcc4897315d5b8b0637/src/server/api/profiler.go#L4

现服务是个后台服务，预先并没加pprof模块

先从数据流程和代码变更上排查

不限于 go 语言，内存泄露的原因最主要是三种情况，1资源类（文件，io流）未释放，2 ”全局“对象，附加了大量额外引用，运行时只加不减，3两者结合，资源类对象又附加了大量额外引用。

20g的量，基本是每日的数据处理量，引起泄露的版本主要变更是引入了kafka处理模块，猜测定位到kafka，数据流处理在写入kafka 部分出现泄露

kafka 处理部分引用 github.com/Shopify/sarama

查看代码 初步定位在

dataCollector,err := newDataCollector([]string{KFKHost})
if err == nil {
    err = dataCollector.SendMessages(messages)
}

 

查看dataCollector 发现有 Close 方法

type SyncProducer interface {

// SendMessage produces a given message, and returns only when it either has
// succeeded or failed to produce. It will return the partition and the offset
// of the produced message, or an error if the message failed to produce.
 SendMessage(msg *ProducerMessage) (partition int32, offset int64, err error)

// SendMessages produces a given set of messages, and returns only when all
// messages in the set have either succeeded or failed. Note that messages
// can succeed and fail individually; if some succeed and some fail,
// SendMessages will return an error.
 SendMessages(msgs []*ProducerMessage) error

// Close shuts down the producer and waits for any buffered messages to be
// flushed. You must call this function before a producer object passes out of
// scope, as it may otherwise leak memory. You must call this before calling
// Close on the underlying client.
 Close() error
}

 

添加

dataCollector,err := newDataCollector([]string{KFKHost})
if err == nil {
    err = dataCollector.SendMessages(messages)
}
defer dataCollector.Close()

上线，问题解决

如果代码梳理没定位到问题，再去依赖技术手段

之前只通过net/http/pprof 应用过

看了些资料
https://github.com/hyper0x/go_command_tutorial/blob/master/0.12.md
引入net/http/pprof是更简介的选择