disruptor

 disruptor提供了线程间消息通信机制，通常会拿它和jdk中的blockingQueue作比较，blockingQueue使用了ReentrantLock。

 

下例的逻辑是，生产500个ValueEvent，先后由toDbHandler和businessHandler消费事件。

normalTest使用的是低级接口，DSLWizardTest使用的是高级接口，后者明显要简洁一些。

class ValueEvent {
  private long value;

  public long getValue() {
    return value;
  }

  public void setValue(final long value) {
    this.value = value;
  }

  @Override
  public String toString() {
    return "ValueEvent{" +
            "value=" + value +
            '}';
  }
}
// 生产者的线程工厂
class SimpleThreadFactory implements java.util.concurrent.ThreadFactory {
  @Override
  public Thread newThread(Runnable r) {
    return new Thread(r, "simpleThread");
  }
};

// RingBuffer生产工厂,初始化RingBuffer的时候使用
class SimpleEventFactory implements EventFactory<ValueEvent> {
  @Override
  public ValueEvent newInstance() {
    return new ValueEvent();
  }
};

class JournalEventHandler implements EventHandler<ValueEvent> {
  public void onEvent(final ValueEvent event, final long sequence, final boolean endOfBatch) throws Exception {
    // process a new event.
    System.out.println("to db: " + event);
  }
};
class ReplicationEventHandler implements EventHandler<ValueEvent> {
  public void onEvent(final ValueEvent event, final long sequence, final boolean endOfBatch) throws Exception {
    // process a new event.
    System.out.println("replication: " + event);
  }
};
class ApplicationEventHandler implements EventHandler<ValueEvent> {
  public void onEvent(final ValueEvent event, final long sequence, final boolean endOfBatch) throws Exception {
      // process a new event.
      System.out.println(event);
            // System.out.println("sequence: " + sequence);
            // System.out.println("end of batch: " + endOfBatch);
  }
};

 

public class TestDisruptor {

  // 指定RingBuffer的大小
  private int bufferSize = 16;
  // 阻塞策略
  BlockingWaitStrategy strategy = new BlockingWaitStrategy();

  @Test
  public void normalTest() throws InterruptedException {
    RingBuffer<ValueEvent> ringBuffer =
            RingBuffer.createSingleProducer(new SimpleEventFactory(), bufferSize, strategy);

    SequenceBarrier barrier = ringBuffer.newBarrier();
    BatchEventProcessor<ValueEvent> journalEventProcessor = new BatchEventProcessor<>(ringBuffer, barrier, new JournalEventHandler());
    BatchEventProcessor<ValueEvent> replicationEventProcessor = new BatchEventProcessor<>(ringBuffer, barrier, new ReplicationEventHandler());

    barrier = ringBuffer.newBarrier(journalEventProcessor.getSequence(), replicationEventProcessor.getSequence());
    BatchEventProcessor<ValueEvent> businessEventProcessor = new BatchEventProcessor<>(ringBuffer, barrier, new ApplicationEventHandler());


    ringBuffer.addGatingSequences(businessEventProcessor.getSequence());

    ExecutorService executor = Executors.newCachedThreadPool(new SimpleThreadFactory());
    // Each EventProcessor can run on a separate thread
    executor.submit(businessEventProcessor);
    executor.submit(journalEventProcessor);
    executor.submit(replicationEventProcessor);

    for (int i = 0; i < 500; i++) {
      // Publishers claim events in sequence
      long sequence = ringBuffer.next();
      try {
        ValueEvent event = ringBuffer.get(sequence);
        event.setValue(i); // this could be more complex with multiple fields
      } finally {
        ringBuffer.publish(sequence);
      }
      // make the event available to EventProcessors
      Thread.sleep(10);
    }
    Thread.sleep(100);
  }

  /**
   * DSLWizard
   *
   * @throws Exception
   */
  @Test
  public void dslWizardTest() throws Exception {
    // 创建disruptor，采用单生产者模式
    Disruptor<ValueEvent> disruptor =
            new Disruptor<>(new SimpleEventFactory(), bufferSize, new SimpleThreadFactory(), ProducerType.SINGLE, strategy);
    // 设置EventHandler
    disruptor.handleEventsWith(new JournalEventHandler(), new ReplicationEventHandler())
            .then(new ApplicationEventHandler());
    // 启动disruptor的线程
    disruptor.start();

    EventTranslator<ValueEvent> eventTranslator = new EventTranslator<ValueEvent>() {
      private AtomicInteger value = new AtomicInteger(0);

      @Override
      public void translateTo(ValueEvent event, long sequence) {
        event.setValue(value.getAndIncrement());
      }
    };
    for (int l = 0; l < 500; l++) {
      disruptor.publishEvent(eventTranslator);
      Thread.sleep(10);
    }
  }
}

上例实现有一个问题，ApplicationConsumer（ApplicationEventHandler）非常耗时时会批量阻塞Producer的生产，比如JournalConsumer、ReplicationConsumer可以很快处理16个事件（因为此例中RingBuffer大小设为16），ApplicationConsumer慢速处理导致生产者迟迟不能生产，直至ApplicationConsumer将RingBuffer全部消费完。

while (true)
{
  try
  {
      //获取（最大）可消费序号
    final long availableSequence = sequenceBarrier.waitFor(nextSequence);
    if (batchStartAware != null)
    {
      batchStartAware.onBatchStart(availableSequence - nextSequence + 1);
    }

    while (nextSequence <= availableSequence)
    {
      event = dataProvider.get(nextSequence);
      eventHandler.onEvent(event, nextSequence, nextSequence == availableSequence);
      nextSequence++;
    }
        //只有批量处理完成后才设置sequence，而producer在生产时需要通过RingBuffer的Sequencer.next获得可分配序号，
        //Sequencer.next又依赖于最低级的Consumer（EventProcesser）的Sequence，
        //即只有最低级的Consumer的Sequence向前移动了，producer才能继续生产。
    sequence.set(availableSequence);
  }
  catch (final TimeoutException e)
  {
      //......
  }
}

 

上述实现的功能可以使用下图表示，ApplicationConsumer（ApplicationEventHandler）依赖于JournalConsumer（JournalEventHandler）和ReplicationConsumer（ReplicationEventHandler），这种依赖关系体现在ApplicationConsumer的SequenceBarrier依赖ReplicationConsumer和JournalConsumer的Sequence（三者的SequenceBarrier都依赖于RingBuffer的Sequence）。

我们看到Sequencer依赖于ApplicationConsumer的Sequence，ApplicationConsumer作为最后一级消费者，其消费速度如果过慢，就会阻塞Sequencer产生下一个序号的速度（RingBuffer.next()调用了Sequencer.next()）。

Sequencer的next使用CAS分配序号，避免了锁开销，当RingBuffer满时，不断循环判断是否可以分配序号，如果不可分配则LockSupport.parkNanos(1)。

每个Consumer都有一个独立的Sequence，Consumers和Sequencer共享一个waitStrategy，Consumer在取事件时，会通过waitStrategy的waitFor获得可用序号。

waitStrategy有不同的策略，比如BlockingWaitStrategy、YieldingWaitStrategy等，这些策略决定了在无可用序号时线程的等待策略，比如BlockingWaitStrategy会阻塞（涉及锁），YieldingWaitStrategy会不断循环并yield。

disruptor中的几个重要组件：

RingBuffer：存放元素的环形容器；

Sequence：序号，类似AtomicLong，考虑了缓存行带来的伪共享（false sharing），对value进行了填充，用于并发获取RingBuffer中可存入元素的序号；

Sequencer：有两个实现类，SingleProducerSequencer和MultiProducerSequencer，这是一个非常关键的组件，next()会得到RingBuffer的下一个可分配序号，然后判断该序号是否可以被分配，判断依据就是最低级的Consumer的消费速度是否跟上了生产者的生产速度（通过最低级的Sequence与待分配序号比较即可判断）；

SequenceBarrier：可以理解为Sequence的屏障，Sequence不能跨过该屏障取序号；

EventHandler：消费者的消费逻辑；

EventProccessor：消费者，循环通过SequenceBarrier.waitFor得到可用（可消费）序号，回调EventHandler的onEvent消费；

WaitStrategy：Consumer（EventProcessor）的等待策略；

Producer：用户逻辑；

 

 

参考：https://github.com/LMAX-Exchange/disruptor/wiki/Introduction

　　http://lmax-exchange.github.io/disruptor/