graylog 的InputBuffer 简单说明
InputBuffer 是graylog 实际处理日志的部分,内部处理基于了Disruptor,同时还依赖了Journalling 能力
内部实际处理是依赖LocalKafka的(而且是推荐的)
InputBuffer 处理的几个模式
目前主要包含了3大类 DirectMessageHandler,RawMessageEncoderHandler,JournallingMessageHandler
构造(InputBufferImpl 构造函数)
final Disruptor<RawMessageEvent> disruptor = new Disruptor<>(
RawMessageEvent.FACTORY,
configuration.getInputBufferRingSize(),
threadFactory(metricRegistry),
ProducerType.MULTI,
configuration.getInputBufferWaitStrategy());
disruptor.setDefaultExceptionHandler(new LoggingExceptionHandler(LOG));
// 处理器核心是disruptor 的workpool 实现负载均衡能力
final int numberOfHandlers = configuration.getInputbufferProcessors();
if (configuration.isMessageJournalEnabled()) {
LOG.info("Message journal is enabled.");
final RawMessageEncoderHandler[] handlers = new RawMessageEncoderHandler[numberOfHandlers];
for (int i = 0; i < numberOfHandlers; i++) {
handlers[i] = rawMessageEncoderHandlerProvider.get();
}
disruptor.handleEventsWithWorkerPool(handlers).then(spoolingMessageHandlerProvider.get());
} else {
LOG.info("Message journal is disabled.");
final DirectMessageHandler[] handlers = new DirectMessageHandler[numberOfHandlers];
for (int i = 0; i < numberOfHandlers; i++) {
handlers[i] = directMessageHandlerProvider.get();
}
disruptor.handleEventsWithWorkerPool(handlers);
}
ringBuffer = disruptor.start();
JournallingMessageHandler 依赖MessageQueueWriter 能力,核心实现是onEvent 方法,调用MessageQueueWriter 的write
同时也会依赖Filter ,进行过滤处理
DirectMessageHandler 依赖ProcessBuffer ,具体处理后边会介绍 ProcessBuffer 也依赖Disruptor 进行处理
insert 处理
通过MessageInput 我们可以知道MessageInput核心是依赖的是InputBuffer 的insert 方法,此处依赖了Disruptor
RingBuffer 的publishEvent 方法,具体消息处理就需要依赖上边的MessageHandler 了
说明
graylog 的InputBuffer 比较依赖Disruptor 进行数据处理,毕竟Disruptor 性能是很不错的,好多高性能框架都依赖这个
进行消息处理
参考资料
https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/shared/buffers/JournallingMessageHandler.java
https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/shared/buffers/InputBufferImpl.java
https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/shared/buffers/ProcessBuffer.java