“Too many open files” 小记

pycharm远程链接centos7开发过程中突然遇到“Too many open files”。

几点记录：

1. 命令：ulimit -n

查看系统配置，其中的

open files                      (-n) 8192

表示每个进程最多能打开8192个文件句柄（是句柄数，socket连接也算在里面）。如果单个进程打开的文件句柄数量超过了系统定义的值，就会提到“too many files open”的错误提示。

2.  cat /proc/sys/fs/file-max

查看系统允许打开的文件句柄的最大数量。

3. cat /proc/sys/fs/file-nr

查看已经打开的句柄数，转载如下：

[root@hostname ~]# cat /proc/sys/fs/file-nr
53536 0     64000
|     |       |
|     |       |
|     |       maximum open file descriptors
|     total free allocated file descriptors
total allocated file descriptors
(the number of file descriptors allocated since boot)

不知道总的句柄数超过了系统限制会不会报同一个错。

4. lsof -n|awk '{print $2}'|sort|uniq -c |sort -nr

使用lsof查看打开的文件，这条命令统计了每个进程打开了多少个文件。

但是，lsof的统计的范围过于广泛。

What is an open file?

Is an open file a file that is being used, or is it an open file descriptor? A file descriptor is a data structure used by a program to get a handle on a file, the most well know being 0,1,2 for standard in, standard out, and standard error. The file-max kernel parameter refers to open file descriptors, and file-nr gives us the current number of open file descriptors. But lsof lists all open files, including files which are not using file descriptors – such as current working directories, memory mapped library files, and executable text files. To illustrate, let’s examine the difference between the output of lsof for a given pid and the file descriptors listed for that pid in /proc.
--------------------------------
Some of the open files which are not using file descriptors: library files, the program itself (executable text), and so on as listed above. These files are accounted for elsewhere in the kernel data structures (cat /proc/PID/maps to see the libraries, for instance), but they are not using file descriptors and therefore do not exhaust the kernel’s file descriptor maximum.

5. ls -l /proc/175072/fd/

使用/proc/下的信息是最准确的，写了一个脚本统计<username>用户的进程打开的文件句柄数的排序：

#!/usr/bin/perl

use strict;
use Data::Dumper;

my @pids = `ps aux | grep ^username | awk '{print \$2}'`;
my $res = [];

foreach my $pid (@pids) {
  $pid =~ s/\n//;

  my $fds = `ls /proc/$pid/fd/ | wc -l`;
  $fds =~ s/\n//;

  push(@$res, {pid => $pid, fd => $fds});
}

$res = [ sort { $b->{fd} <=> $a->{fd} } @$res];
foreach my $val (@$res) {
  print "$val->{pid}:$val->{fd}\n";
}

 

ref.

http://www.jb51.net/article/97706.htm

https://stackoverflow.com/questions/21999820/what-are-the-number-of-open-files-for-a-user-on-linux-and-system-wide-for-linux

https://unix.stackexchange.com/questions/66235/how-to-display-open-file-descriptors-but-not-using-lsof-command

https://www.netadmintools.com/art295.html