Netty中NioEventLoopGroup的创建源码分析

NioEventLoopGroup的无参构造：

public NioEventLoopGroup() {
    this(0);
}

调用了单参的构造：

public NioEventLoopGroup(int nThreads) {
    this(nThreads, (Executor)null);
}

继续看到双参构造：

public NioEventLoopGroup(int nThreads, Executor executor) {
    this(nThreads, executor, SelectorProvider.provider());
}

在这里是使用JDK中NIO的原生API：SelectorProvider的provider，产生了一个SelectorProvider对象调用，继续调用三参构造。
关于SelectorProvider在我前面的博客中有介绍过：【Java】NIO中Selector的创建源码分析，在Windows下默认创建了WindowsSelectorProvider对象。

继续看三参构造：

public NioEventLoopGroup(int nThreads, ThreadFactory threadFactory, SelectorProvider selectorProvider) {
    this(nThreads, threadFactory, selectorProvider, DefaultSelectStrategyFactory.INSTANCE);
}

在这里创建了一个单例的DefaultSelectStrategyFactory 对象：

public final class DefaultSelectStrategyFactory implements SelectStrategyFactory {
    public static final SelectStrategyFactory INSTANCE = new DefaultSelectStrategyFactory();

    private DefaultSelectStrategyFactory() {
    }

    public SelectStrategy newSelectStrategy() {
        return DefaultSelectStrategy.INSTANCE;
    }
}

DefaultSelectStrategyFactory实现的是SelectStrategyFactory 接口：

public interface SelectStrategyFactory {
    SelectStrategy newSelectStrategy();
}

该接口提供一个用来产生Select策略的方法，SelectStrategy接口如下：

public interface SelectStrategy {
    int SELECT = -1;
    int CONTINUE = -2;

    int calculateStrategy(IntSupplier var1, boolean var2) throws Exception;
}

根据IntSupplier 和一个boolean值为Select策略提供了一个计算策略的方法。
在Netty中只提供了DefaultSelectStrategy这一种默认实现：

final class DefaultSelectStrategy implements SelectStrategy {
    static final SelectStrategy INSTANCE = new DefaultSelectStrategy();

    private DefaultSelectStrategy() {
    }

    public int calculateStrategy(IntSupplier selectSupplier, boolean hasTasks) throws Exception {
        return hasTasks ? selectSupplier.get() : -1;
    }
}

其中IntSupplier :

public interface IntSupplier {
    int get() throws Exception;
}

结合上面的来看，最终的选择策略主要是根据IntSupplier的get值来得到的。

再回到构造：

public NioEventLoopGroup(int nThreads, ThreadFactory threadFactory, SelectorProvider selectorProvider, SelectStrategyFactory selectStrategyFactory) {
    super(nThreads, threadFactory, new Object[]{selectorProvider, selectStrategyFactory, RejectedExecutionHandlers.reject()});
}

这里产生了一个拒绝策略：

public static RejectedExecutionHandler reject() {
    return REJECT;
}

private static final RejectedExecutionHandler REJECT = new RejectedExecutionHandler() {
    public void rejected(Runnable task, SingleThreadEventExecutor executor) {
        throw new RejectedExecutionException();
    }
};

public interface RejectedExecutionHandler {
    void rejected(Runnable var1, SingleThreadEventExecutor var2);
}

将selectorProvider、selectStrategyFactory以及这个拒绝策略封装在一个Object数组里，再调用了父类MultithreadEventLoopGroup的构造：

protected MultithreadEventLoopGroup(int nThreads, ThreadFactory threadFactory, Object... args) {
    super(nThreads == 0 ? DEFAULT_EVENT_LOOP_THREADS : nThreads, threadFactory, args);
}

在这里对nThreads的大小进行了调整：

private static final int DEFAULT_EVENT_LOOP_THREADS = Math.max(1, SystemPropertyUtil.getInt("io.netty.eventLoopThreads", NettyRuntime.availableProcessors() * 2));

SystemPropertyUtil.getInt是根据key值"io.netty.eventLoopThreads"，获取系统配置值，在没用设置时使用NettyRuntime.availableProcessors() * 2的值
NettyRuntime的availableProcessors实现如下：

synchronized int availableProcessors() {
    if (this.availableProcessors == 0) {
        int availableProcessors = SystemPropertyUtil.getInt("io.netty.availableProcessors", Runtime.getRuntime().availableProcessors());
        this.setAvailableProcessors(availableProcessors);
    }

    return this.availableProcessors;
}

还是一样，根据key值"io.netty.availableProcessors"，获取系统配置值，在没用设置时使用Runtime.getRuntime().availableProcessors()，是用来获取处理器的个数。

这样保证了在默认情况下nThreads的大小是总是cpu个数的2倍。

继续回到构造，MultithreadEventLoopGroup继续调用父类MultithreadEventExecutorGroup的构造：

protected MultithreadEventExecutorGroup(int nThreads, Executor executor, Object... args) {
    this(nThreads, executor, DefaultEventExecutorChooserFactory.INSTANCE, args);
}

在这里又初始化了一个单例的DefaultEventExecutorChooserFactory对象：

public static final DefaultEventExecutorChooserFactory INSTANCE = new DefaultEventExecutorChooserFactory();

DefaultEventExecutorChooserFactory 实现的是EventExecutorChooserFactory接口：

public interface EventExecutorChooserFactory {
    EventExecutorChooserFactory.EventExecutorChooser newChooser(EventExecutor[] var1);

    public interface EventExecutorChooser {
        EventExecutor next();
    }
}

DefaultEventExecutorChooserFactory 的具体实现：

public EventExecutorChooser newChooser(EventExecutor[] executors) {
    return (EventExecutorChooser)(isPowerOfTwo(executors.length) ? new DefaultEventExecutorChooserFactory.PowerOfTwoEventExecutorChooser(executors) : new DefaultEventExecutorChooserFactory.GenericEventExecutorChooser(executors));
}

private static boolean isPowerOfTwo(int val) {
    return (val & -val) == val;
}

isPowerOfTwo是用来检查executors的大小是否是二的整数次方，若是二的整数次方，产生PowerOfTwoEventExecutorChooser，反之产生GenericEventExecutorChooser：

private static final class GenericEventExecutorChooser implements EventExecutorChooser {
    private final AtomicInteger idx = new AtomicInteger();
    private final EventExecutor[] executors;

    GenericEventExecutorChooser(EventExecutor[] executors) {
        this.executors = executors;
    }

    public EventExecutor next() {
        return this.executors[Math.abs(this.idx.getAndIncrement() % this.executors.length)];
    }
}

private static final class PowerOfTwoEventExecutorChooser implements EventExecutorChooser {
    private final AtomicInteger idx = new AtomicInteger();
    private final EventExecutor[] executors;

    PowerOfTwoEventExecutorChooser(EventExecutor[] executors) {
        this.executors = executors;
    }

    public EventExecutor next() {
        return this.executors[this.idx.getAndIncrement() & this.executors.length - 1];
    }
}

这两种其实都是用了取模运算，只不过因为二的整数次方的特殊性而使用位运算。

回到构造，MultithreadEventExecutorGroup继续调用本省的构造：

private final EventExecutor[] children;
private final Set<EventExecutor> readonlyChildren;
private final AtomicInteger terminatedChildren;
private final Promise<?> terminationFuture;
private final EventExecutorChooser chooser;

protected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {
    this.terminatedChildren = new AtomicInteger();
    this.terminationFuture = new DefaultPromise(GlobalEventExecutor.INSTANCE);
    if (nThreads <= 0) {
        throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));
    } else {
        if (executor == null) {
            executor = new ThreadPerTaskExecutor(this.newDefaultThreadFactory());
        }

        this.children = new EventExecutor[nThreads];

        int j;
        for(int i = 0; i < nThreads; ++i) {
            boolean success = false;
            boolean var18 = false;

            try {
                var18 = true;
                this.children[i] = this.newChild((Executor)executor, args);
                success = true;
                var18 = false;
            } catch (Exception var19) {
                throw new IllegalStateException("failed to create a child event loop", var19);
            } finally {
                if (var18) {
                    if (!success) {
                        int j;
                        for(j = 0; j < i; ++j) {
                            this.children[j].shutdownGracefully();
                        }

                        for(j = 0; j < i; ++j) {
                            EventExecutor e = this.children[j];

                            try {
                                while(!e.isTerminated()) {
                                    e.awaitTermination(2147483647L, TimeUnit.SECONDS);
                                }
                            } catch (InterruptedException var20) {
                                Thread.currentThread().interrupt();
                                break;
                            }
                        }
                    }

                }
            }

            if (!success) {
                for(j = 0; j < i; ++j) {
                    this.children[j].shutdownGracefully();
                }

                for(j = 0; j < i; ++j) {
                    EventExecutor e = this.children[j];

                    try {
                        while(!e.isTerminated()) {
                            e.awaitTermination(2147483647L, TimeUnit.SECONDS);
                        }
                    } catch (InterruptedException var22) {
                        Thread.currentThread().interrupt();
                        break;
                    }
                }
            }
        }

        this.chooser = chooserFactory.newChooser(this.children);
        FutureListener<Object> terminationListener = new FutureListener<Object>() {
            public void operationComplete(Future<Object> future) throws Exception {
                if (MultithreadEventExecutorGroup.this.terminatedChildren.incrementAndGet() == MultithreadEventExecutorGroup.this.children.length) {
                    MultithreadEventExecutorGroup.this.terminationFuture.setSuccess((Object)null);
                }

            }
        };
        EventExecutor[] var24 = this.children;
        j = var24.length;

        for(int var26 = 0; var26 < j; ++var26) {
            EventExecutor e = var24[var26];
            e.terminationFuture().addListener(terminationListener);
        }

        Set<EventExecutor> childrenSet = new LinkedHashSet(this.children.length);
        Collections.addAll(childrenSet, this.children);
        this.readonlyChildren = Collections.unmodifiableSet(childrenSet);
    }
}

首先是对terminatedChildren的初始化，没什么好说的，对terminationFuture的初始化使用DefaultPromise，用来异步处理终止事件。executor初始化产生一个线程池。

接下来就是对children的操作，根据nThreads的大小，产生一个EventExecutor数组，然后遍历这个数组，调用newChild给每一个元素初始化。

newChild是在NioEventLoopGroup中实现的：

protected EventLoop newChild(Executor executor, Object... args) throws Exception {
    return new NioEventLoop(this, executor, (SelectorProvider)args[0], ((SelectStrategyFactory)args[1]).newSelectStrategy(), (RejectedExecutionHandler)args[2]);
}

在这里直接使用executor，和之前放在args数组中的SelectorProvider、SelectStrategyFactory（newSelectStrategy方法产生DefaultSelectStrategy）和RejectedExecutionHandler产生了一个NioEventLoop对象：

private Selector selector;
private Selector unwrappedSelector;
private SelectedSelectionKeySet selectedKeys;
private final SelectorProvider provider;
private final AtomicBoolean wakenUp = new AtomicBoolean();
private final SelectStrategy selectStrategy;

NioEventLoop(NioEventLoopGroup parent, Executor executor, SelectorProvider selectorProvider, SelectStrategy strategy, RejectedExecutionHandler rejectedExecutionHandler) {
   super(parent, executor, false, DEFAULT_MAX_PENDING_TASKS, rejectedExecutionHandler);
    if (selectorProvider == null) {
        throw new NullPointerException("selectorProvider");
    } else if (strategy == null) {
        throw new NullPointerException("selectStrategy");
    } else {
        this.provider = selectorProvider;
        NioEventLoop.SelectorTuple selectorTuple = this.openSelector();
        this.selector = selectorTuple.selector;
        this.unwrappedSelector = selectorTuple.unwrappedSelector;
        this.selectStrategy = strategy;
    }
}

NioEventLoop首先在继承链上调用父类的构造，都是一些成员的赋值操作，简单看一看：

protected SingleThreadEventLoop(EventLoopGroup parent, Executor executor, boolean addTaskWakesUp, int maxPendingTasks, RejectedExecutionHandler rejectedExecutionHandler) {
    super(parent, executor, addTaskWakesUp, maxPendingTasks, rejectedExecutionHandler);
    this.tailTasks = this.newTaskQueue(maxPendingTasks);
}

protected SingleThreadEventExecutor(EventExecutorGroup parent, Executor executor, boolean addTaskWakesUp, int maxPendingTasks, RejectedExecutionHandler rejectedHandler) {
    super(parent);
    this.threadLock = new Semaphore(0);
    this.shutdownHooks = new LinkedHashSet();
    this.state = 1;
    this.terminationFuture = new DefaultPromise(GlobalEventExecutor.INSTANCE);
    this.addTaskWakesUp = addTaskWakesUp;
    this.maxPendingTasks = Math.max(16, maxPendingTasks);
    this.executor = (Executor)ObjectUtil.checkNotNull(executor, "executor");
    this.taskQueue = this.newTaskQueue(this.maxPendingTasks);
    this.rejectedExecutionHandler = (RejectedExecutionHandler)ObjectUtil.checkNotNull(rejectedHandler, "rejectedHandler");
}

protected AbstractScheduledEventExecutor(EventExecutorGroup parent) {
    super(parent);
}

protected AbstractEventExecutor(EventExecutorGroup parent) {
    this.selfCollection = Collections.singleton(this);
    this.parent = parent;
}

在经过这继承链上的一系列调用后，给provider成员赋值selectorProvider，就是之前创建好的WindowsSelectorProvider，然后使用openSelector方法，最终创建JDK原生的Selector：

private NioEventLoop.SelectorTuple openSelector() {
    final AbstractSelector unwrappedSelector;
    try {
        unwrappedSelector = this.provider.openSelector();
    } catch (IOException var7) {
        throw new ChannelException("failed to open a new selector", var7);
    }

    if (DISABLE_KEYSET_OPTIMIZATION) {
        return new NioEventLoop.SelectorTuple(unwrappedSelector);
    } else {
        final SelectedSelectionKeySet selectedKeySet = new SelectedSelectionKeySet();
        Object maybeSelectorImplClass = AccessController.doPrivileged(new PrivilegedAction<Object>() {
            public Object run() {
                try {
                    return Class.forName("sun.nio.ch.SelectorImpl", false, PlatformDependent.getSystemClassLoader());
                } catch (Throwable var2) {
                    return var2;
                }
            }
        });
        if (maybeSelectorImplClass instanceof Class && ((Class)maybeSelectorImplClass).isAssignableFrom(unwrappedSelector.getClass())) {
            final Class<?> selectorImplClass = (Class)maybeSelectorImplClass;
            Object maybeException = AccessController.doPrivileged(new PrivilegedAction<Object>() {
                public Object run() {
                    try {
                        Field selectedKeysField = selectorImplClass.getDeclaredField("selectedKeys");
                        Field publicSelectedKeysField = selectorImplClass.getDeclaredField("publicSelectedKeys");
                        Throwable cause = ReflectionUtil.trySetAccessible(selectedKeysField, true);
                        if (cause != null) {
                            return cause;
                        } else {
                            cause = ReflectionUtil.trySetAccessible(publicSelectedKeysField, true);
                            if (cause != null) {
                                return cause;
                            } else {
                                selectedKeysField.set(unwrappedSelector, selectedKeySet);
                                publicSelectedKeysField.set(unwrappedSelector, selectedKeySet);
                                return null;
                            }
                        }
                    } catch (NoSuchFieldException var4) {
                        return var4;
                    } catch (IllegalAccessException var5) {
                        return var5;
                    }
                }
            });
            if (maybeException instanceof Exception) {
                this.selectedKeys = null;
                Exception e = (Exception)maybeException;
                logger.trace("failed to instrument a special java.util.Set into: {}", unwrappedSelector, e);
                return new NioEventLoop.SelectorTuple(unwrappedSelector);
            } else {
                this.selectedKeys = selectedKeySet;
                logger.trace("instrumented a special java.util.Set into: {}", unwrappedSelector);
                return new NioEventLoop.SelectorTuple(unwrappedSelector, new SelectedSelectionKeySetSelector(unwrappedSelector, selectedKeySet));
            }
        } else {
            if (maybeSelectorImplClass instanceof Throwable) {
                Throwable t = (Throwable)maybeSelectorImplClass;
                logger.trace("failed to instrument a special java.util.Set into: {}", unwrappedSelector, t);
            }

            return new NioEventLoop.SelectorTuple(unwrappedSelector);
        }
    }
}

可以看到在一开始就使用provider的openSelector方法，即WindowsSelectorProvider的openSelector方法，创建了WindowsSelectorImpl对象（【Java】NIO中Selector的创建源码分析 ）

然后根据DISABLE_KEYSET_OPTIMIZATION判断：

private static final boolean DISABLE_KEYSET_OPTIMIZATION = SystemPropertyUtil.getBoolean("io.netty.noKeySetOptimization", false);

可以看到这个系统配置在没有设置默认是false，如果设置了则直接创建一个SelectorTuple对象返回：

private static final class SelectorTuple {
    final Selector unwrappedSelector;
    final Selector selector;

    SelectorTuple(Selector unwrappedSelector) {
        this.unwrappedSelector = unwrappedSelector;
        this.selector = unwrappedSelector;
    }

    SelectorTuple(Selector unwrappedSelector, Selector selector) {
        this.unwrappedSelector = unwrappedSelector;
        this.selector = selector;
    }
}

可以看到仅仅是将unwrappedSelector和selector封装了，unwrappedSelector对应的是JDK原生Selector没有经过更改的，而selector对应的是经过更改修饰操作的。

在没有系统配置下，就对Selector进行更改修饰操作：
首先创建SelectedSelectionKeySet对象，这个SelectedSelectionKeySet继承自AbstractSet：

final class SelectedSelectionKeySet extends AbstractSet<SelectionKey> {
    SelectionKey[] keys = new SelectionKey[1024];
    int size;
    
    SelectedSelectionKeySet() {
    }
    ......
}

后面是通过反射机制，使得WindowsSelectorImpl的selectedKeys和publicSelectedKeys成员直接赋值为SelectedSelectionKeySet对象。

WindowsSelectorImpl的这两个成员是在SelectorImpl中定义的：

protected Set<SelectionKey> selectedKeys = new HashSet();
private Set<SelectionKey> publicSelectedKeys;

从这里就可以明白，在JDK原生的Selector中，selectedKeys和publicSelectedKeys这两个Set的初始化大小都为0，而在这里仅仅就是使其初始化大小变为1024。
后面就是对一些异常的处理，没什么好说的。

openSelector结束后，就可以分别对包装过的Selector和未包装过的Selector，即selector和unwrappedSelector成员赋值，再由selectStrategy保存刚才产生的选择策略，用于Selector的轮询。

回到MultithreadEventExecutorGroup的构造，在调用newChild方法时即NioEventLoop创建的过程中可能出现异常情况，就需要遍历children数组，将之前创建好的NioEventLoop使用shutdownGracefully优雅地关闭掉：
shutdownGracefully在AbstractEventExecutor中实现：

public Future<?> shutdownGracefully() {
    return this.shutdownGracefully(2L, 15L, TimeUnit.SECONDS);
}

这里设置了超时时间，继续调用SingleThreadEventExecutor的shutdownGracefully方法：

public Future<?> shutdownGracefully(long quietPeriod, long timeout, TimeUnit unit) {
    if (quietPeriod < 0L) {
        throw new IllegalArgumentException("quietPeriod: " + quietPeriod + " (expected >= 0)");
    } else if (timeout < quietPeriod) {
        throw new IllegalArgumentException("timeout: " + timeout + " (expected >= quietPeriod (" + quietPeriod + "))");
    } else if (unit == null) {
        throw new NullPointerException("unit");
    } else if (this.isShuttingDown()) {
        return this.terminationFuture();
    } else {
        boolean inEventLoop = this.inEventLoop();

        while(!this.isShuttingDown()) {
            boolean wakeup = true;
            int oldState = this.state;
            int newState;
            if (inEventLoop) {
                newState = 3;
            } else {
                switch(oldState) {
                case 1:
                case 2:
                    newState = 3;
                    break;
                default:
                    newState = oldState;
                    wakeup = false;
                }
            }

            if (STATE_UPDATER.compareAndSet(this, oldState, newState)) {
                this.gracefulShutdownQuietPeriod = unit.toNanos(quietPeriod);
                this.gracefulShutdownTimeout = unit.toNanos(timeout);
                if (oldState == 1) {
                    try {
                        this.doStartThread();
                    } catch (Throwable var10) {
                        STATE_UPDATER.set(this, 5);
                        this.terminationFuture.tryFailure(var10);
                        if (!(var10 instanceof Exception)) {
                            PlatformDependent.throwException(var10);
                        }

                        return this.terminationFuture;
                    }
                }

                if (wakeup) {
                    this.wakeup(inEventLoop);
                }

                return this.terminationFuture();
            }
        }

        return this.terminationFuture();
    }
}

前三个判断没什么好说的，isShuttingDown判断：

public boolean isShuttingDown() {
    return this.state >= 3;
}

在之前NioEventLoop创建的时候，调用了一系列的继承链，其中state是在SingleThreadEventExecutor的构造方法中实现的，初始值是1，state有如下几种状态：

private static final int ST_NOT_STARTED = 1;
private static final int ST_STARTED = 2;
private static final int ST_SHUTTING_DOWN = 3;
private static final int ST_SHUTDOWN = 4;
private static final int ST_TERMINATED = 5;

可见在NioEventLoop初始化后处于尚未启动状态，并没有Channel的注册，也就不需要轮询。

isShuttingDown就必然是false，就进入了else块：
首先得到inEventLoop的返回值，该方法在AbstractEventExecutor中实现：

public boolean inEventLoop() {
    return this.inEventLoop(Thread.currentThread());
}

他传入了一个当前线程，接着调用inEventLoop的重载，这个方法是在SingleThreadEventExecutor中实现：

public boolean inEventLoop(Thread thread) {
    return thread == this.thread;
}

通过观察之前的SingleThreadEventExecutor构造方法，发现并没有对thread成员初始化，此时的this.thread就是null，那么返回值就是false，即inEventLoop为false。

在while循环中又对isShuttingDown进行了判断，shutdownGracefully当让不仅仅使用在创建NioEventLoop对象失败时才调用的，无论是在EventLoopGroup的关闭，还是Bootstrap的关闭，都会关闭绑定的NioEventLoop，所以在多线程环境中，有可能会被其他线程关闭。

在while循环中，结合上面可知满足进入switch块，在switch块中令newState为3；
然后调用STATE_UPDATER的compareAndSet方法，STATE_UPDATER是用来原子化更新state成员的：

private static final AtomicIntegerFieldUpdater<SingleThreadEventExecutor> STATE_UPDATER = AtomicIntegerFieldUpdater.newUpdater(SingleThreadEventExecutor.class, "state");

所以这里就是使用CAS操作，原子化更新state成员为3，也就是使当前状态由ST_NOT_STARTED 变为了ST_SHUTTING_DOWN 状态。

gracefulShutdownQuietPeriod和gracefulShutdownTimeout分别保存quietPeriod和timeout的纳秒级颗粒度。

前面可知oldState使1，调用doStartThread方法：

private void doStartThread() {
    assert this.thread == null;

    this.executor.execute(new Runnable() {
        public void run() {
            SingleThreadEventExecutor.this.thread = Thread.currentThread();
            if (SingleThreadEventExecutor.this.interrupted) {
                SingleThreadEventExecutor.this.thread.interrupt();
            }

            boolean success = false;
            SingleThreadEventExecutor.this.updateLastExecutionTime();
            boolean var112 = false;

            int oldState;
            label1685: {
                try {
                    var112 = true;
                    SingleThreadEventExecutor.this.run();
                    success = true;
                    var112 = false;
                    break label1685;
                } catch (Throwable var119) {
                    SingleThreadEventExecutor.logger.warn("Unexpected exception from an event executor: ", var119);
                    var112 = false;
                } finally {
                    if (var112) {
                        int oldStatex;
                        do {
                            oldStatex = SingleThreadEventExecutor.this.state;
                        } while(oldStatex < 3 && !SingleThreadEventExecutor.STATE_UPDATER.compareAndSet(SingleThreadEventExecutor.this, oldStatex, 3));

                        if (success && SingleThreadEventExecutor.this.gracefulShutdownStartTime == 0L) {
                            SingleThreadEventExecutor.logger.error("Buggy " + EventExecutor.class.getSimpleName() + " implementation; " + SingleThreadEventExecutor.class.getSimpleName() + ".confirmShutdown() must be called before run() implementation terminates.");
                        }

                        try {
                            while(!SingleThreadEventExecutor.this.confirmShutdown()) {
                                ;
                            }
                        } finally {
                            try {
                                SingleThreadEventExecutor.this.cleanup();
                            } finally {
                                SingleThreadEventExecutor.STATE_UPDATER.set(SingleThreadEventExecutor.this, 5);
                                SingleThreadEventExecutor.this.threadLock.release();
                                if (!SingleThreadEventExecutor.this.taskQueue.isEmpty()) {
                                    SingleThreadEventExecutor.logger.warn("An event executor terminated with non-empty task queue (" + SingleThreadEventExecutor.this.taskQueue.size() + ')');
                                }

                                SingleThreadEventExecutor.this.terminationFuture.setSuccess((Object)null);
                            }
                        }

                    }
                }

                do {
                    oldState = SingleThreadEventExecutor.this.state;
                } while(oldState < 3 && !SingleThreadEventExecutor.STATE_UPDATER.compareAndSet(SingleThreadEventExecutor.this, oldState, 3));

                if (success && SingleThreadEventExecutor.this.gracefulShutdownStartTime == 0L) {
                    SingleThreadEventExecutor.logger.error("Buggy " + EventExecutor.class.getSimpleName() + " implementation; " + SingleThreadEventExecutor.class.getSimpleName() + ".confirmShutdown() must be called before run() implementation terminates.");
                }

                try {
                    while(!SingleThreadEventExecutor.this.confirmShutdown()) {
                        ;
                    }

                    return;
                } finally {
                    try {
                        SingleThreadEventExecutor.this.cleanup();
                    } finally {
                        SingleThreadEventExecutor.STATE_UPDATER.set(SingleThreadEventExecutor.this, 5);
                        SingleThreadEventExecutor.this.threadLock.release();
                        if (!SingleThreadEventExecutor.this.taskQueue.isEmpty()) {
                            SingleThreadEventExecutor.logger.warn("An event executor terminated with non-empty task queue (" + SingleThreadEventExecutor.this.taskQueue.size() + ')');
                        }

                        SingleThreadEventExecutor.this.terminationFuture.setSuccess((Object)null);
                    }
                }
            }

            do {
                oldState = SingleThreadEventExecutor.this.state;
            } while(oldState < 3 && !SingleThreadEventExecutor.STATE_UPDATER.compareAndSet(SingleThreadEventExecutor.this, oldState, 3));

            if (success && SingleThreadEventExecutor.this.gracefulShutdownStartTime == 0L) {
                SingleThreadEventExecutor.logger.error("Buggy " + EventExecutor.class.getSimpleName() + " implementation; " + SingleThreadEventExecutor.class.getSimpleName() + ".confirmShutdown() must be called before run() implementation terminates.");
            }

            try {
                while(!SingleThreadEventExecutor.this.confirmShutdown()) {
                    ;
                }
            } finally {
                try {
                    SingleThreadEventExecutor.this.cleanup();
                } finally {
                    SingleThreadEventExecutor.STATE_UPDATER.set(SingleThreadEventExecutor.this, 5);
                    SingleThreadEventExecutor.this.threadLock.release();
                    if (!SingleThreadEventExecutor.this.taskQueue.isEmpty()) {
                        SingleThreadEventExecutor.logger.warn("An event executor terminated with non-empty task queue (" + SingleThreadEventExecutor.this.taskQueue.size() + ')');
                    }

                    SingleThreadEventExecutor.this.terminationFuture.setSuccess((Object)null);
                }
            }

        }
    });
}

刚才说过this.thread并没有初始化，所以等于null成立，断言可以继续。

然后直接使executor运行了一个线程，这个executor其实就是在刚才的MultithreadEventExecutorGroup中产生的ThreadPerTaskExecutor对象。

在线程中，首先将SingleThreadEventExecutor的thread成员初始化为当前线程。

在这里可能就有疑问了，为什么会在关闭时会调用名为doStartThread的方法，这个方法不因该在启动时调用吗？
其实doStartThread在启动时是会被调用的，当在启动时被调用的话，每一个NioEventLoop都会被绑定一个线程用来处理真正的Selector操作，根据之前的说法就可以知道，每个EventLoopGroup在创建后都会被绑定cpu个数的二倍个NioEventLoop，而每个NioEventLoop都会绑定一个Selector对象，上面又说了在启动时SingleThreadEventExecutor绑定了一个线程，即NioEventLoop绑定了一个线程来处理其绑定的Selector的轮询。
至于关闭时还会启动Selector的轮询，就是为了解决注册了的Channel没有被处理的情况。

回到doStartThread方法，其实这个doStartThread方法的核心是SingleThreadEventExecutor.this.run()，这个方法就是正真的Selector的轮询操作，在NioEventLoop中实现：

protected void run() {
    while(true) {
        while(true) {
            try {
                switch(this.selectStrategy.calculateStrategy(this.selectNowSupplier, this.hasTasks())) {
                case -2:
                    continue;
                case -1:
                    this.select(this.wakenUp.getAndSet(false));
                    if (this.wakenUp.get()) {
                        this.selector.wakeup();
                    }
                default:
                    this.cancelledKeys = 0;
                    this.needsToSelectAgain = false;
                    int ioRatio = this.ioRatio;
                    if (ioRatio == 100) {
                        try {
                            this.processSelectedKeys();
                        } finally {
                            this.runAllTasks();
                        }
                    } else {
                        long ioStartTime = System.nanoTime();
                        boolean var13 = false;

                        try {
                            var13 = true;
                            this.processSelectedKeys();
                            var13 = false;
                        } finally {
                            if (var13) {
                                long ioTime = System.nanoTime() - ioStartTime;
                                this.runAllTasks(ioTime * (long)(100 - ioRatio) / (long)ioRatio);
                            }
                        }

                        long ioTime = System.nanoTime() - ioStartTime;
                        this.runAllTasks(ioTime * (long)(100 - ioRatio) / (long)ioRatio);
                    }
                }
            } catch (Throwable var21) {
                handleLoopException(var21);
            }

            try {
                if (this.isShuttingDown()) {
                    this.closeAll();
                    if (this.confirmShutdown()) {
                        return;
                    }
                }
            } catch (Throwable var18) {
                handleLoopException(var18);
            }
        }
    }
}

进入switch块，首先调用之前准备好的选择策略，其中this.selectNowSupplier在NioEventLoop创建的时候就被创建了：

private final IntSupplier selectNowSupplier = new IntSupplier() {
    public int get() throws Exception {
        return NioEventLoop.this.selectNow();
    }
};

实际上调用了selectNow方法：

int selectNow() throws IOException {
    int var1;
    try {
        var1 = this.selector.selectNow();
    } finally {
        if (this.wakenUp.get()) {
            this.selector.wakeup();
        }

    }

    return var1;
}

这里就直接调用了JDK原生的selectNow方法。
之前说过的选择策略：

public int calculateStrategy(IntSupplier selectSupplier, boolean hasTasks) throws Exception {
    return hasTasks ? selectSupplier.get() : -1;
}

其中hasTasks是根据hasTasks方法来判断，而hasTasks方法就是判断任务队列是否为空，那么在一开始初始化，必然是空的，所以这里calculateStrategy的返回值就是-1；

那么case为-1条件成立，执行this.select(this.wakenUp.getAndSet(false))，其中wakenUp是一个原子化的Boolean，用来表示是需要唤醒Selector的轮询阻塞，初始化是为true，这里通过CAS操作设置为false代表不需要唤醒，后面在select执行完后，又判断wakenUp是否需要唤醒，说明在select中对Selector的阻塞进行了检查，若是需要唤醒，就通过Selector的原生API完成唤醒【Java】NIO中Selector的select方法源码分析

来看看这里的select实现：

private void select(boolean oldWakenUp) throws IOException {
    Selector selector = this.selector;

    try {
        int selectCnt = 0;
        long currentTimeNanos = System.nanoTime();
        long selectDeadLineNanos = currentTimeNanos + this.delayNanos(currentTimeNanos);

        while(true) {
            long timeoutMillis = (selectDeadLineNanos - currentTimeNanos + 500000L) / 1000000L;
            if (timeoutMillis <= 0L) {
                if (selectCnt == 0) {
                    selector.selectNow();
                    selectCnt = 1;
                }
                break;
            }

            if (this.hasTasks() && this.wakenUp.compareAndSet(false, true)) {
                selector.selectNow();
                selectCnt = 1;
                break;
            }

            int selectedKeys = selector.select(timeoutMillis);
            ++selectCnt;
            if (selectedKeys != 0 || oldWakenUp || this.wakenUp.get() || this.hasTasks() || this.hasScheduledTasks()) {
                break;
            }

            if (Thread.interrupted()) {
                if (logger.isDebugEnabled()) {
                    logger.debug("Selector.select() returned prematurely because Thread.currentThread().interrupt() was called. Use NioEventLoop.shutdownGracefully() to shutdown the NioEventLoop.");
                }

                selectCnt = 1;
                break;
            }

            long time = System.nanoTime();
            if (time - TimeUnit.MILLISECONDS.toNanos(timeoutMillis) >= currentTimeNanos) {
                selectCnt = 1;
            } else if (SELECTOR_AUTO_REBUILD_THRESHOLD > 0 && selectCnt >= SELECTOR_AUTO_REBUILD_THRESHOLD) {
                logger.warn("Selector.select() returned prematurely {} times in a row; rebuilding Selector {}.", selectCnt, selector);
                this.rebuildSelector();
                selector = this.selector;
                selector.selectNow();
                selectCnt = 1;
                break;
            }

            currentTimeNanos = time;
        }

        if (selectCnt > 3 && logger.isDebugEnabled()) {
            logger.debug("Selector.select() returned prematurely {} times in a row for Selector {}.", selectCnt - 1, selector);
        }
    } catch (CancelledKeyException var13) {
        if (logger.isDebugEnabled()) {
            logger.debug(CancelledKeyException.class.getSimpleName() + " raised by a Selector {} - JDK bug?", selector, var13);
        }
    }

}

这个方法虽然看着很长，但核心就是判断这个存放任务的阻塞队列是否还有任务，若是有，就直接调用Selector的selectNow方法获取就绪的文件描述符，若是没有就绪的文件描述符该方法也会立即返回；若是阻塞队列中没有任务，就调用Selector的select(timeout)方法，尝试在超时时间内取获取就绪的文件描述符。

因为现在是在执行NioEventLoopGroup的创建，并没有Channel的注册，也就没有轮询到任何文件描述符就绪。
在轮询结束后，回到run方法，进入default块：
其中ioRatio是执行IO操作和执行任务队列的任务用时比率，默认是50。若是ioRatio设置为100，就必须等到tasks阻塞队列中的所有任务执行完毕才再次进行轮询；若是小于100，那么就根据(100 - ioRatio) / ioRatio的比值乘以ioTime计算出的超时时间让所有任务尝试在超时时间内执行完毕，若是到达超时时间还没执行完毕，就在下一轮的轮询中执行。

processSelectedKeys方法就是获取Selector轮询的SelectedKeys结果：

private void processSelectedKeys() {
    if (this.selectedKeys != null) {
        this.processSelectedKeysOptimized();
    } else {
        this.processSelectedKeysPlain(this.selector.selectedKeys());
    }

}

selectedKeys 在openSelector时被初始化过了，若是在openSelector中出现异常selectedKeys才会为null。

processSelectedKeysOptimized方法：

private void processSelectedKeysOptimized() {
    for(int i = 0; i < this.selectedKeys.size; ++i) {
        SelectionKey k = this.selectedKeys.keys[i];
        this.selectedKeys.keys[i] = null;
        Object a = k.attachment();
        if (a instanceof AbstractNioChannel) {
            this.processSelectedKey(k, (AbstractNioChannel)a);
        } else {
            NioTask<SelectableChannel> task = (NioTask)a;
            processSelectedKey(k, task);
        }

        if (this.needsToSelectAgain) {
            this.selectedKeys.reset(i + 1);
            this.selectAgain();
            i = -1;
        }
    }

}

这里就通过遍历在openSelector中注入进Selector的SelectedKeys，得到SelectionKey 对象。
在这里可以看到Netty很巧妙地通过SelectionKey的attachment附件，将JDK中的Channel和Netty中的Channel联系了起来。
根据得到的附件Channel的类型，执行不同的processSelectedKey方法，去处理IO操作。

processSelectedKey(SelectionKey k, AbstractNioChannel ch)方法：

private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
    NioUnsafe unsafe = ch.unsafe();
    if (!k.isValid()) {
        NioEventLoop eventLoop;
        try {
            eventLoop = ch.eventLoop();
        } catch (Throwable var6) {
            return;
        }

        if (eventLoop == this && eventLoop != null) {
            unsafe.close(unsafe.voidPromise());
        }
    } else {
        try {
            int readyOps = k.readyOps();
            if ((readyOps & 8) != 0) {
                int ops = k.interestOps();
                ops &= -9;
                k.interestOps(ops);
                unsafe.finishConnect();
            }

            if ((readyOps & 4) != 0) {
                ch.unsafe().forceFlush();
            }

            if ((readyOps & 17) != 0 || readyOps == 0) {
                unsafe.read();
            }
        } catch (CancelledKeyException var7) {
            unsafe.close(unsafe.voidPromise());
        }

    }
}

这里的主要核心就是根据SelectedKey的readyOps值来判断，处理不同的就绪事件，有如下几种事件：

public static final int OP_READ = 1 << 0;
public static final int OP_WRITE = 1 << 2;
public static final int OP_CONNECT = 1 << 3;
public static final int OP_ACCEPT = 1 << 4;

结合来看上面的判断就对应：连接就绪、写就绪、侦听或者读就绪（或者缺省状态0，该状态是未来注册时的默认状态，后续博客会介绍），交由Netty的AbstractNioChannel的NioUnsafe去处理不同事件的byte数据，NioUnsafe会将数据再交由ChannelPipeline双向链表去处理。
关于ChannelPipeline会在后续的博客中详细介绍。

processSelectedKey(SelectionKey k, NioTask<SelectableChannel> task)这个方法的实现细节需要由使用者实现NioTask<SelectableChannel>接口，就不说了。

回到processSelectedKeys方法，在this.selectedKeys等于null的情况下：

private void processSelectedKeysPlain(Set<SelectionKey> selectedKeys) {
    if (!selectedKeys.isEmpty()) {
        Iterator i = selectedKeys.iterator();

        while(true) {
            SelectionKey k = (SelectionKey)i.next();
            Object a = k.attachment();
            i.remove();
            if (a instanceof AbstractNioChannel) {
                this.processSelectedKey(k, (AbstractNioChannel)a);
            } else {
                NioTask<SelectableChannel> task = (NioTask)a;
                processSelectedKey(k, task);
            }

            if (!i.hasNext()) {
                break;
            }

            if (this.needsToSelectAgain) {
                this.selectAgain();
                selectedKeys = this.selector.selectedKeys();
                if (selectedKeys.isEmpty()) {
                    break;
                }

                i = selectedKeys.iterator();
            }
        }

    }
}

这是在openSelector中注入进Selector的SelectedKeys失败的情况下，直接遍历Selector本身的SelectedKeys，和processSelectedKeysOptimized没有差别。

继续回到run方法，在调用完processSelectedKeys方法后，就需要调用runAllTasks处理任务队列中的任务：
runAllTasks（）方法：

protected boolean runAllTasks() {
    assert this.inEventLoop();

    boolean ranAtLeastOne = false;

    boolean fetchedAll;
    do {
        fetchedAll = this.fetchFromScheduledTaskQueue();
        if (this.runAllTasksFrom(this.taskQueue)) {
            ranAtLeastOne = true;
        }
    } while(!fetchedAll);

    if (ranAtLeastOne) {
        this.lastExecutionTime = ScheduledFutureTask.nanoTime();
    }

    this.afterRunningAllTasks();
    return ranAtLeastOne;
}

首先调用fetchFromScheduledTaskQueue方法：

private boolean fetchFromScheduledTaskQueue() {
    long nanoTime = AbstractScheduledEventExecutor.nanoTime();

    for(Runnable scheduledTask = this.pollScheduledTask(nanoTime); scheduledTask != null; scheduledTask = this.pollScheduledTask(nanoTime)) {
        if (!this.taskQueue.offer(scheduledTask)) {
            this.scheduledTaskQueue().add((ScheduledFutureTask)scheduledTask);
            return false;
        }
    }

    return true;
}

这里就是通过pollScheduledTask不断地从延时任务队列获取到期的任务，将到期的任务添加到taskQueue任务队列中，为上面的runAllTasksFrom执行做准备；若是添加失败，再把它放进延时任务队列。

pollScheduledTask方法：

protected final Runnable pollScheduledTask(long nanoTime) {
    assert this.inEventLoop();

    Queue<ScheduledFutureTask<?>> scheduledTaskQueue = this.scheduledTaskQueue;
    ScheduledFutureTask<?> scheduledTask = scheduledTaskQueue == null ? null : (ScheduledFutureTask)scheduledTaskQueue.peek();
    if (scheduledTask == null) {
        return null;
    } else if (scheduledTask.deadlineNanos() <= nanoTime) {
        scheduledTaskQueue.remove();
        return scheduledTask;
    } else {
        return null;
    }
}

从延时任务队列中获取队首的任务scheduledTask，若是scheduledTask的deadlineNanos小于等于nanoTime，说明该任务到期。

回到runAllTasks，将到期了的延时任务放在了任务队列，由runAllTasksFrom执行这些任务：

protected final boolean runAllTasksFrom(Queue<Runnable> taskQueue) {
    Runnable task = pollTaskFrom(taskQueue);
    if (task == null) {
        return false;
    } else {
        do {
            safeExecute(task);
            task = pollTaskFrom(taskQueue);
        } while(task != null);

        return true;
    }
}

不断地从任务队列队首获取任务，然后执行，直到没有任务。

pollTaskFrom是获取队首任务：

protected static Runnable pollTaskFrom(Queue<Runnable> taskQueue) {
    Runnable task;
    do {
        task = (Runnable)taskQueue.poll();
    } while(task == WAKEUP_TASK);

    return task;
}

其中WAKEUP_TASK，是用来巧妙地控制循环：

private static final Runnable WAKEUP_TASK = new Runnable() {
    public void run() {
    }
};

safeExecute是执行任务：

protected static void safeExecute(Runnable task) {
    try {
        task.run();
    } catch (Throwable var2) {
        logger.warn("A task raised an exception. Task: {}", task, var2);
    }

}

实际上就是执行Runnable 的run方法。

继续回到runAllTasks方法，当所有到期任务执行完毕后，根据ranAtLeastOne判断是否需要修改最后一次执行时间lastExecutionTime，最后调用afterRunningAllTasks方法，该方法是在SingleThreadEventLoop中实现的：

protected void afterRunningAllTasks() {
    this.runAllTasksFrom(this.tailTasks);
}

这里就仅仅执行了tailTasks队列中的任务。runAllTasks到这里执行完毕。

再来看看runAllTasks(timeoutNanos)方法：

protected boolean runAllTasks(long timeoutNanos) {
    this.fetchFromScheduledTaskQueue();
    Runnable task = this.pollTask();
    if (task == null) {
        this.afterRunningAllTasks();
        return false;
    } else {
        long deadline = ScheduledFutureTask.nanoTime() + timeoutNanos;
        long runTasks = 0L;

        long lastExecutionTime;
        while(true) {
            safeExecute(task);
            ++runTasks;
            if ((runTasks & 63L) == 0L) {
                lastExecutionTime = ScheduledFutureTask.nanoTime();
                if (lastExecutionTime >= deadline) {
                    break;
                }
            }

            task = this.pollTask();
            if (task == null) {
                lastExecutionTime = ScheduledFutureTask.nanoTime();
                break;
            }
        }

        this.afterRunningAllTasks();
        this.lastExecutionTime = lastExecutionTime;
        return true;
    }
}

这个方法前面的runAllTasks方法类似，先通过fetchFromScheduledTaskQueue将所有到期了的延时任务放在taskQueue中，然后不断从taskQueue队首获取任务，但是，若是执行到了到超过了63个任务，判断是否达到了超时时间deadline，若是达到结束循环，留着下次执行，反之继续循环执行任务。

 

回到run方法，在轮询完毕，并且执行完任务后，通过isShuttingDown判断当前状态，在之前的CAS操作中，state已经变为了3，所以isShuttingDown成立，就需要调用closeAll方法

private void closeAll() {
    this.selectAgain();
    Set<SelectionKey> keys = this.selector.keys();
    Collection<AbstractNioChannel> channels = new ArrayList(keys.size());
    Iterator var3 = keys.iterator();

    while(var3.hasNext()) {
        SelectionKey k = (SelectionKey)var3.next();
        Object a = k.attachment();
        if (a instanceof AbstractNioChannel) {
            channels.add((AbstractNioChannel)a);
        } else {
            k.cancel();
            NioTask<SelectableChannel> task = (NioTask)a;
            invokeChannelUnregistered(task, k, (Throwable)null);
        }
    }

    var3 = channels.iterator();

    while(var3.hasNext()) {
        AbstractNioChannel ch = (AbstractNioChannel)var3.next();
        ch.unsafe().close(ch.unsafe().voidPromise());
    }

}

在这里首先调用selectAgain进行一次轮询：

private void selectAgain() {
    this.needsToSelectAgain = false;

    try {
        this.selector.selectNow();
    } catch (Throwable var2) {
        logger.warn("Failed to update SelectionKeys.", var2);
    }

}

通过这次的轮询，将当前仍有事件就绪的JDK的SelectionKey中绑定的Netty的Channel添加到channels集合中，遍历这个集合，通过unsafe的close方法关闭Netty的Channel。

之后调用confirmShutdown方法：

protected boolean confirmShutdown() {
    if (!this.isShuttingDown()) {
        return false;
    } else if (!this.inEventLoop()) {
        throw new IllegalStateException("must be invoked from an event loop");
    } else {
        this.cancelScheduledTasks();
        if (this.gracefulShutdownStartTime == 0L) {
            this.gracefulShutdownStartTime = ScheduledFutureTask.nanoTime();
        }

        if (!this.runAllTasks() && !this.runShutdownHooks()) {
            long nanoTime = ScheduledFutureTask.nanoTime();
            if (!this.isShutdown() && nanoTime - this.gracefulShutdownStartTime <= this.gracefulShutdownTimeout) {
                if (nanoTime - this.lastExecutionTime <= this.gracefulShutdownQuietPeriod) {
                    this.wakeup(true);

                    try {
                        Thread.sleep(100L);
                    } catch (InterruptedException var4) {
                        ;
                    }

                    return false;
                } else {
                    return true;
                }
            } else {
                return true;
            }
        } else if (this.isShutdown()) {
            return true;
        } else if (this.gracefulShutdownQuietPeriod == 0L) {
            return true;
        } else {
            this.wakeup(true);
            return false;
        }
    }
}

首先调用cancelScheduledTasks，取消所有的延时任务：

protected void cancelScheduledTasks() {
    assert this.inEventLoop();

    PriorityQueue<ScheduledFutureTask<?>> scheduledTaskQueue = this.scheduledTaskQueue;
    if (!isNullOrEmpty(scheduledTaskQueue)) {
        ScheduledFutureTask<?>[] scheduledTasks = (ScheduledFutureTask[])scheduledTaskQueue.toArray(new ScheduledFutureTask[scheduledTaskQueue.size()]);
        ScheduledFutureTask[] var3 = scheduledTasks;
        int var4 = scheduledTasks.length;

        for(int var5 = 0; var5 < var4; ++var5) {
            ScheduledFutureTask<?> task = var3[var5];
            task.cancelWithoutRemove(false);
        }

        scheduledTaskQueue.clearIgnoringIndexes();
    }
}

遍历scheduledTasks这个延时任务对立中所有的任务，通过cancelWithoutRemove将该任务取消。

至此轮询的整个生命周期完成。

回到SingleThreadEventExecutor的doStartThread方法，在run方法执行完毕后，说明Selector轮询结束，调用SingleThreadEventExecutor.this.cleanup()方法关闭Selector：

protected void cleanup() {
    try {
        this.selector.close();
    } catch (IOException var2) {
        logger.warn("Failed to close a selector.", var2);
    }

}

这次终于可以回到MultithreadEventExecutorGroup的构造，在children创建完毕后，用chooserFactory根据children的大小创建chooser，前面说过。

然后产生terminationListener异步中断监听对象，给每个NioEventLoop设置中断监听，然后对children进行了备份处理，通过readonlyChildren保存。

至此NioEventLoopGroup的创建全部结束。