线程池问题记录以及处理

现象

每天到业务高峰期就会出现提交线程被拒绝。

疑问点

什么原因会导致active threads远远小于poolsize的情况下，提交任务失败

关键信息

线程池配置

ExecutorService aService = new ThreadPoolExecutor(
                130
                , 300
                , 60L
                , TimeUnit.SECONDS
                , new LinkedBlockingQueue<>(10)
                , new ThreadFactoryBuilder().setNameFormat("aService-pool-%d").build()
                , new ThreadPoolExecutor.AbortPolicy()
        );

拒绝任务的日志信息

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.CompletableFuture$AsyncSupply@4e4dd145 
rejected from java.util.concurrent.ThreadPoolExecutor@5c39c043[Running, pool size = 300, active threads = 4, queued tasks = 8, completed tasks = 3851427]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) ~[?:?]
	at java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1782) ~[?:?]
	at java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:2005)

其他信息

1. 这个线程池在使用时存在父子线程都使用的情况，这个用法本身有问题。
2. 从监控来看，java.util.concurrent.ThreadPoolExecutor#getActiveCount 指标最高到40；而 java.util.concurrent.ThreadPoolExecutor#getPoolSize 指标已到高峰期就保持在300

复现

代码示例

@Test
    public void testReject() throws InterruptedException {
        /* 线上拒绝任务日志
        java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.CompletableFuture$AsyncSupply@5b5b5892 rejected from java.util.concurrent.ThreadPoolExecutor@1c3a066b[Running, pool size = 300, active threads = 20, queued tasks = 10, completed tasks = 3517903]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) ~[?:?]
	at java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1782) ~[?:?]
	at java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:2005) ~[?:?]
         */
        ThreadPoolExecutor executorService = new ThreadPoolExecutor(
                13
                , 30
                , 60L
                , TimeUnit.SECONDS
                , new LinkedBlockingQueue<>(5)
                , new ThreadFactoryBuilder().setNameFormat("pool-%d").build()
                , new ThreadPoolExecutor.AbortPolicy()
        );
        // 打印线程池状态
        statPrint(executorService);
        Supplier<Double> supplier = () -> {
            try {
                TimeUnit.MILLISECONDS.sleep(10L);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
            return RandomUtils.nextLong()/Math.PI;
        };
        CountDownLatch latch = new CountDownLatch(1);
        for (int i = 0; i < 10; i++) {
            new Thread(() -> {
                while (true) {
                    try {
                        TimeUnit.MILLISECONDS.sleep(8L);
                    } catch (InterruptedException e) {
                        throw new RuntimeException(e);
                    }
                    CompletableFuture<Double> future = supplyAsync(supplier, executorService, -1D);
                    Double result = getResult(future, 12L, 0.0, "get result error");
                }
            }).start();
        }
        // hold
        latch.await();
    }
     /**
     * 提交一步任务,如果提交失败返回带有默认值的CompletableFuture
     */
    public static <T> CompletableFuture<T> supplyAsync(Supplier<T> supplier, ExecutorService executor, T defaultValue) {
        try {
            return CompletableFuture.supplyAsync(supplier, executor);
        } catch (Exception e) {
            log.error("提交任务失败使用默认带有结果的future");
        }
        return CompletableFuture.completedFuture(defaultValue);
    }
    /**
    * Future get 结果，超时或者其他异常时返回默认值，并取消任务
    **/
    public static  <T> T getResult(Future<T> future, long timeout, T defaultValue, String errorMessage) {
        boolean needCancel = true;
        try {
            T result = future.get(timeout, TimeUnit.MILLISECONDS);
            needCancel = false;
            return result;
        } catch (ExecutionException | TimeoutException e) {
            log.error("获取结果异常:{}", errorMessage);
        } catch (InterruptedException e1) {
            log.error("InterruptedException :{}", errorMessage);
            Thread.currentThread().interrupt();
        } finally {
            if (needCancel) {
                log.warn("task cancel");
                future.cancel(true);
            }
        }
        return defaultValue;
    }

测试用例部分日志

poolSize=30, activeThreads=5, numberOfTasksCompleted=3569, numberOfTasksInQueue=0, time: 7405
poolSize=30, activeThreads=8, numberOfTasksCompleted=3615, numberOfTasksInQueue=0, time: 7507
poolSize=30, activeThreads=2, numberOfTasksCompleted=3665, numberOfTasksInQueue=0, time: 7605
poolSize=30, activeThreads=1, numberOfTasksCompleted=3714, numberOfTasksInQueue=1, time: 7705
提交任务失败使用默认带有结果的future
提交任务失败使用默认带有结果的future
poolSize=30, activeThreads=6, numberOfTasksCompleted=3759, numberOfTasksInQueue=0, time: 7805
poolSize=30, activeThreads=8, numberOfTasksCompleted=3807, numberOfTasksInQueue=0, time: 7905

解决方案

增加动态修改线程池配置的功能，在线上调整，已达到更好的效果