记一次 “Too many open files“引发的问题

这里我们用虚拟机进行模拟问题。

有一天突然发现网站页面打不开了，报500错误，赶紧查看日志，发现报错如下

[Fri Jul 24 11:02:20.686132 2020] [:error] [pid 2322] [remote 10.0.0.253:8] IOError: [Errno 24] Too many open files: '/usr/share/openstack-dashboard/openstack_dashboard/themes/default/templates/500.html'

看到"Too many open files"这个报错，大家一般都能很快的反应到是哪里的问题，很明显是最大打开文件数量限制了。

我们首先进行确认：1. 检查系统可以打开的最大文件  2. httpd进程可以打开的最大文件数据 

[root@controller ~]# ulimit -n
1024
[root@controller ~]# 
[root@controller ~]# 
[root@controller ~]# ps -ef|grep -v grep|grep httpd
root       1319      1  0 02:55 ?        00:00:07 /usr/sbin/httpd -DFOREGROUND
nova       1646   1319  0 02:55 ?        00:01:03 /usr/sbin/httpd -DFOREGROUND
nova       1647   1319  0 02:55 ?        00:01:01 /usr/sbin/httpd -DFOREGROUND
nova       1648   1319  0 02:55 ?        00:00:59 /usr/sbin/httpd -DFOREGROUND
apache     1655   1319  0 02:55 ?        00:00:01 /usr/sbin/httpd -DFOREGROUND
apache     2866   1319  0 03:01 ?        00:00:01 /usr/sbin/httpd -DFOREGROUND
apache     8363   1319  0 05:19 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    14653   1319  0 07:56 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    17966   1319  0 09:14 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    17989   1319  0 09:15 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    18618   1319  0 09:30 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    22737   1319  0 11:08 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    27765   1319  0 13:14 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    28522   1319  0 13:34 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
[root@controller ~]# 
[root@controller ~]# cat /proc/1319/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             19032                19032                processes 
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       19032                19032                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

　由输出可以看出系统默认最大只能打开文件数量为1024，httpd进程Max open files也为1024，难怪会报错了。

 

赶紧修改配置文件/etc/security/limits.conf，在修改这个文件之前首先查看系统支持最大打开多少文件数量，注意不同的系统下面的这个值可能会不一致，我的系统为CentOS7的虚拟机。

[root@controller ~]# cat /proc/sys/fs/file-max 
480850　　

 修改好的配置如下

[root@controller ~]# egrep -v '^#|^$' /etc/security/limits.conf 
*		 -	 nofile          480850　　

修改完了之后不需要重启系统，只需要退出当前session，重新登录修改即可生效。现在我们进行验证

[root@controller ~]# ulimit -n
480850
[root@controller ~]# 
[root@controller ~]# systemctl restart httpd
[root@controller ~]# 
[root@controller ~]# ss -tlnp|grep 80
LISTEN     0      128          *:4369                     *:*                   users:(("epmd",pid=1806,fd=3),("systemd",pid=1,fd=42))
LISTEN     0      100          *:6080                     *:*                   users:(("nova-novncproxy",pid=1300,fd=4))
LISTEN     0      128       [::]:80                    [::]:*                   users:(("httpd",pid=31526,fd=4),("httpd",pid=31455,fd=4),("httpd",pid=31428,fd=4),("httpd",pid=31427,fd=4),("httpd",pid=31426,fd=4),("httpd",pid=31425,fd=4),("httpd",pid=31424,fd=4),("httpd",pid=31415,fd=4))
[root@controller ~]# ps -ef|grep httpd
root      31415      1  0 14:36 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
nova      31416  31415  4 14:36 ?        00:00:04 /usr/sbin/httpd -DFOREGROUND
nova      31417  31415  2 14:36 ?        00:00:02 /usr/sbin/httpd -DFOREGROUND
nova      31418  31415  2 14:36 ?        00:00:02 /usr/sbin/httpd -DFOREGROUND
apache    31424  31415  0 14:36 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    31425  31415  0 14:36 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    31426  31415  0 14:36 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    31427  31415  0 14:36 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    31428  31415  0 14:36 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    31455  31415  0 14:36 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    31526  31415  0 14:37 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
root      31603  31322  0 14:38 pts/1    00:00:00 grep --color=auto httpd
[root@controller ~]# 
[root@controller ~]# cat /proc/31415/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             19032                19032                processes 
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       19032                19032                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us  

系统文件数量修改生效了，但是我们发现重启完httpd服务之后，httpd服务进程所能打开的最大文件数量依然是1024，怎么回事儿？

费了九牛二虎之力终于找到了原因，因为我们的httpd服务使用systemd来启动的，而systemd我们并没有配置最大可以打开的文件数量，默认为1024，所以必须要修改它的配置文件

vim /etc/systemd/system.conf，增加一行

DefaultLimitNOFILE=480850

这里这一步完事之后只需要systemctl daemon-reexec，即可使配置文件生效，不需要重新操作系统。

daemon-reexec
     Reexecute the systemd manager. This will serialize the manager state, reexecute the process and deserialize the state
     again. This command is of little use except for debugging and package upgrades. Sometimes, it might be helpful as a
     heavy-weight daemon-reload. While the daemon is being reexecuted, all sockets systemd listening on behalf of user
     configuration will stay accessible.

　　

然后我们验证

[root@controller ~]# systemctl daemon-reexec 
[root@controller ~]# 
[root@controller ~]# 
[root@controller ~]# 
[root@controller ~]# systemctl restart httpd
[root@controller ~]# 
[root@controller ~]# 
[root@controller ~]# ps -ef|grep httpd
root      34213      1  1 14:57 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
nova      34214  34213  0 14:57 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
nova      34215  34213  0 14:57 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
nova      34216  34213  1 14:57 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    34222  34213  0 14:57 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    34223  34213  0 14:57 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    34224  34213  0 14:57 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    34225  34213  0 14:57 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    34226  34213  0 14:57 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
root      34260  31322  0 14:57 pts/1    00:00:00 grep --color=auto httpd
[root@controller ~]# 
[root@controller ~]# 
[root@controller ~]# cat /proc/34213/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             19032                19032                processes 
Max open files            480850               480850               files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       19032                19032                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us      

我们可以看到httpd进程所能打开的最大文件数据量已经为480850了，已经设置成功了。