mongodb"failed to create thread after accepting new connection, closing connection"问题定位
事件
lxc宿主机10.11.164.28上所有mongodb数据节点，在同一时刻报错：failed to create thread after accepting new connection, closing connection
宿主机版本：oracle linux 6.5，lxc版本:1.0.6
数据库版本:mongodb 3.2.11
报错信息：
2017-07-26T10:23:18.734+0800 I NETWORK [initandlisten] failed to create thread after accepting new connection, closing connection 2017-07-26T10:23:23.781+0800 I NETWORK [initandlisten] connection accepted from 127.0.0.1:44334 #5874 (14 connections now open) 2017-07-26T10:23:23.781+0800 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable 2017-07-26T10:23:23.781+0800 I NETWORK [initandlisten] failed to create thread after accepting new connection, closing connection 2017-07-26T10:23:26.670+0800 I NETWORK [initandlisten] connection accepted from 192.168.4.206:54601 #5875 (14 connections now open) 2017-07-26T10:23:26.670+0800 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
问题确认
报错信息所在文件：
./mongodb-src-r3.2.16/src/mongo/util/net//message_server_port.cpp: log() << "failed to create thread after accepting new connection, closing connection";
./mongodb-src-r3.2.16/src/mongo/util/net//message_server_port.cpp: log() << "pthread_create failed: " << errnoWithDescription(failed) << endl;
virtual void accepted(std::shared_ptr<Socket> psocket, long long connectionId) { ScopeGuard sleepAfterClosingPort = MakeGuard(sleepmillis, 2); std::unique_ptr<MessagingPortWithHandler> portWithHandler( new MessagingPortWithHandler(psocket, _handler, connectionId)); if (!Listener::globalTicketHolder.tryAcquire()) { log() << "connection refused because too many open connections: " << Listener::globalTicketHolder.used() << endl; return; } try { #ifndef __linux__ // TODO: consider making this ifdef _WIN32 { stdx::thread thr(stdx::bind(&handleIncomingMsg, portWithHandler.get())); thr.detach(); } #else pthread_attr_t attrs; //声明pthread_attr_t对象attrs pthread_attr_init(&attrs); //初始化attrs pthread_attr_setdetachstate(&attrs, PTHREAD_CREATE_DETACHED); //设置线程attrs状态为PTHREAD_CREATE_DETACHED，退出时自行释放所占用的资源 static const size_t STACK_SIZE = //设置静态常量stack_size，数据类型为size_t,正整数 1024 * 1024; // if we change this we need to update the warning struct rlimit limits; //声明rlimit类型的结构体limits，详细内容在下文解释 verify(getrlimit(RLIMIT_STACK, &limits) == 0); //验证,RLIMIT_STACK(最大的进程堆栈)和limits比较，如果 if (limits.rlim_cur > STACK_SIZE) { //如果需要的stack大小大于建议设置的stack大小比较，则分配建议的stack_size(1M) size_t stackSizeToSet = STACK_SIZE; #if !__has_feature(address_sanitizer) if (kDebugBuild) // stackSizeToSet /= 2; #endif pthread_attr_setstacksize(&attrs, stackSizeToSet); //为线程attrs分配堆栈大小，大小为stackSizeToSet } else if (limits.rlim_cur < 1024 * 1024) { //如果需要的limit值小于1M，则warning warning() << "Stack size set to " << (limits.rlim_cur / 1024) << "KB. We suggest 1MB" << endl; } pthread_t thread; //声明进程 int failed = pthread_create(&thread, &attrs, &handleIncomingMsg, portWithHandler.get()); //创建进程(进程号，属性，其实函数地址等，启动变量等)，fail值，成功为0，失败-1 pthread_attr_destroy(&attrs); //释放占用的sttrs资源 if (failed) { //创建失败，日志打印， log() << "pthread_create failed: " << errnoWithDescription(failed) << endl; //errnoWithDescription(failed)在这里为Resource temporarily unavailable throw std::system_error( std::make_error_code(std::errc::resource_unavailable_try_again)); } #endif // __linux__ portWithHandler.release(); //释放定制的函数指针 sleepAfterClosingPort.Dismiss(); // } catch (...) { //抛出异常，释放监听进程等 Listener::globalTicketHolder.release(); log() << "failed to create thread after accepting new connection, closing connection"; } }
其中limits是rlimit类型的结构体，定义如下，rlimit是linux系统的结构体，定义一个进程在运行过程中能得到的最大进程，针对soft limit(软限制)或者hard limit(硬限制)
struct rlimit { rlim_t rlim_cur; //soft limit rlim_t rlim_max; //hard limit };
有两种函数控制：
int getrlimit(int resource, struct rlimit *rlim); //查询进程是否满足一个进程的rlimit
int setrlimit(int resource, const struct rlimit *rlim);
报错含义是：不能创建新的进程，来映射新链接
确认原因
lxc虚机与stack相关的参数信息如下：
ulimit -s stack size The maximum stack size cat /etc/security/limits.conf mongodb soft nproc 4096 max number of processes mongodb hard nproc 16384 max number of processes mongodb soft nofile 131072 max number of open file descriptors mongodb hard nofile 131072 max number of open file descriptors mongodb soft stack 1024 max stack size (KB) mongodb hard stack 1024 max stack size (KB)
宿主机与stack相关的参数表
stack size (kbytes, -s) 8192 # End of file * soft nproc 65536 * hard nproc 65536 * soft nofile 131072 * hard nofile 131072
尝试修改虚机的nproc限制，并不起作用。
除此，在linux 6x版本中，还引入了一个配置cat /etc/security/limits.d/90-nproc.conf
宿主机的配置：
# cat /etc/security/limits.d/90-nproc.conf # Default limit for number of user's processes to prevent # accidental fork bombs. # See rhbz #432903 for reasoning. * soft nproc 65536 root soft nproc unlimited
虚机的配置：
#/etc/security/limits.d/90-nproc.conf # Default limit for number of user's processes to prevent # accidental fork bombs. # See rhbz #432903 for reasoning. * soft nproc 1024 root soft nproc unlimited
尝试扩大虚机对*用户的nproc软限制，改为10240，mongodb创建链接恢复正常
结论，创建thread受限两个配置，/etc/security/limit.conf和/etc/security/limits.d/90-nproc.conf，当然也受限与宿主机的配置。
静水流深
