基于Disruptor并发框架的分类任务并发
并发的场景
最近在编码中遇到的场景,我的程序需要处理不同类型的任务,场景要求如下:
1.同类任务串行、不同类任务并发。
2.高吞吐量。
3.任务类型动态增减。
思路
思路一:
最直接的想法,每有一个任务种类被新建,就创建对应的处理线程。
这样的思路问题在于线程数量不可控、创建、销毁线程开销大。不可取。
思路二:
比较常规的想法,所有任务共享线程池每有一个任务种类被创建,就新建一个队列,以保证同类任务串行。
这样的思路问题在于数据结构开销不可控,如果是任务种类繁多,但每种任务数量并不多的情况,那么建如此多的队列显得可笑。
于是我期望能够使用一个线程池、一个队列搞定这些事。
设计
代码实现
引入disruptor依赖:
<dependency>
<groupId>com.lmax</groupId>
<artifactId>disruptor</artifactId>
<version>3.4.2</version>
</dependency>
Task接口
public interface Task<T> {
public T exec();
}
TaskEvent类
import com.lmax.disruptor.EventFactory;
public class TaskEvent<T> {
private Task<T> input;
//用于标记一个并发分组
private int partitionId;
//disruptor的任务事件工场
public static final EventFactory<TaskEvent> FACTORY = TaskEvent::new;
public Task<T> getInput() {
return input;
}
public void setInput(Task<T> input) {
this.input = input;
}
public int getPartitionId() {
return partitionId;
}
public void setPartitionId(int partitionId) {
this.partitionId = partitionId;
}
}
TaskHandler类
import com.lmax.disruptor.EventHandler;
import com.lmax.disruptor.LifecycleAware;
public class EventAdaptor<T> implements EventHandler<TaskEvent<T>>, LifecycleAware {
//决定我处理那些任务
private final int partitionId;
public EventAdaptor(int partitionId) {
super();
this.partitionId = partitionId;
}
public void onEvent(TaskEvent<T> taskEvent, long arg1, boolean arg2) throws Exception {
if(partitionId == taskEvent.getPartitionId()) {
taskEvent.getInput().exec();
}
}
@Override
public void onShutdown() {
Thread.currentThread().setName("handler-" + partitionId);
}
@Override
public void onStart() {
Thread.currentThread().setName("handler-" + partitionId);
}
}
TaskService类
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import javax.annotation.PostConstruct;
import com.lmax.disruptor.BlockingWaitStrategy;
import com.lmax.disruptor.dsl.Disruptor;
import com.lmax.disruptor.dsl.ProducerType;
@Service
@Singleton
public class TaskService<V extends Task<T>, T> {
private static final int BUFFER_SIZE = 1024;
private static final int DEFAULT_POOL_SIZE = 5;
private ThreadPoolExecutor executor;
private Disruptor<TaskEvent> disruptor;
private Map<String,Integer> taskClassMapperPartition = new ConcurrentHashMap<>();
private static final int PARALLEL_NUM = 5;
private List<String> taskTypes = new ArrayList<>();
@PostConstruct
public void init() {
//初始化处理器和线程池
this.executor = new ThreadPoolExecutor(DEFAULT_POOL_SIZE, DEFAULT_POOL_SIZE, 15L, TimeUnit.SECONDS, new LinkedBlockingQueue<>());
this.executor.prestartAllCoreThreads();
this.disruptor = new Disruptor<>(TaskEvent.FACTORY, BUFFER_SIZE, executor, ProducerType.SINGLE, new BlockingWaitStrategy());
EventAdaptor[] handlers = new EventAdaptor[PARALLEL_NUM];
for(int i = 0; i < PARALLEL_NUM; i++) {
handlers[i] = new EventAdaptor(i);
}
this.disruptor.handleEventsWith(handlers);
this.disruptor.start();
rePartition();
}
public void addTaskType(String type) {
taskClassMapperPartition.put(type, taskTypes.size() % PARALLEL_NUM);
taskTypes.add(type);
}
public void deleteTaskType(String type) {
taskTypes.remove(type);
taskClassMapperPartition.remove(type);
rePartition();
}
private void rePartition() {
for(int i = 0, length = taskTypes.size(); i < length; i++) {
//给各任务处理器均衡发放任务
taskClassMapperPartition.put(taskTypes.get(i), i % PARALLEL_NUM);
}
}
}
这里给出的代码是一套物业无得简单框架,你只要实现Task接口就可以了。
许多年前 你有一双清澈的双眼
想看遍这世界 去最遥远的远方
感觉有双翅膀 能飞越高山和海洋