文件描述符即fd个数其实分为两种,
一种是系统本身的总的限制个数,另一种是进程能够打开的具体的限制的个数。
1. 系统最大打开文件描述符数:/proc/sys/fs/file-max
2. 进程最大打开文件描述符数:ulimit -n open files (-n) 65536
3.查看当前系统使用的打开文件描述符数
[root@cl-node01 ~]# cat /proc/sys/fs/file-max
6534666
[root@cl-node01 ~]# cat /proc/sys/fs/file-nr
1664 0 6534666
第一个值表示当前系统已分配使用的打开文件描述符数,第二个数为分配后已释放的(目前已不再使用),第三个数等于file-max。
1.当前进程分配过的文件描述符的近似的最高值
cat /proc/进程号/status
在这个文件中会有一个FDSize字段,该字段是表示当前进程分配过的文件描述符的近似的最高值。
为什么是近似,因为这里FDSize本身是不会减少的,如果刚开始打开了18个文件,则这里的FDSize等于32,若大于32,则以32为单位递增,例如33则是64;
这个32的单位是依赖于系统位数的,如果是32位则是32,若是64位系统则是以64倍增。
2.实际的已经打开的fd
ls -l /proc/进程号/fd | wc
文件描述符
ulimit命令: 针对登录用户的当前shell
/etc/security/limits.conf: the resource limits for the users logged in via PAM. not affect resource limits of the system services
/proc/sys/fs/file-nr, /proc/sys/fs/file-max
[root@cl-server ~]# cat /proc/sys/fs/file-max
174192
[root@cl-server ~]# cat /proc/sys/fs/file-nr
1120 0 174192
已分配文件句柄的数目
已使用文件句柄的数目
文件句柄的最大数目
[soupman@myos ~]$ ulimit -n 6 [root@myos ~]# cd /proc/2693549/fd [root@myos fd]# ll total 0 lrwx------ 1 soupman soupman 64 Apr 2 09:45 0 -> /dev/pts/0 lrwx------ 1 soupman soupman 64 Apr 2 09:45 1 -> /dev/pts/0 lrwx------ 1 soupman soupman 64 Apr 2 09:45 2 -> /dev/pts/0 lrwx------ 1 soupman soupman 64 Apr 2 09:46 255 -> /dev/pts/0 lr-x------ 1 soupman soupman 64 Apr 2 09:45 3 -> /var/lib/sss/mc/passwd [soupman@myos ~]$ vi aa.txt E138: Can't write viminfo file /home/soupman/.viminfo.tmp! Press ENTER or type command to continue [soupman@myos ~]$ ulimit -n 6 ### 普通用户只允许往小的改,不允许往大的改, 1024->512成功, 1024->2048失败;root用户不受限制 [soupman@myos ~]$ ulimit -n 9 -bash: ulimit: open files: cannot modify limit: Operation not permitted [soupman@myos ~]$ lsof -u soupman | grep ulimit [soupman@myos ~]$ ulimit -n 8 [soupman@myos ~]$ lsof -u soupman | grep ulimit lsof: can't open pipes: Too many open files [soupman@myos ~]$ ulimit -a open files (-n) 8 [soupman@myos ~]$ lsof -R -u soupman lsof: can't open pipes: Too many open files [soupman@myos ~]$ sudo -i sudo: PAM account management error: Permission denied [soupman@myos ~]$ ulimit -n 9 [soupman@myos ~]$ lsof -R -u soupman COMMAND PID PPID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 2695529 2695527 soupman cwd unknown /proc/2695529/cwd (readlink: Permission denied) sshd 2695529 2695527 soupman rtd unknown /proc/2695529/root (readlink: Permission denied) sshd 2695529 2695527 soupman txt unknown /proc/2695529/exe (readlink: Permission denied) sshd 2695529 2695527 soupman NOFD /proc/2695529/fd (opendir: Permission denied) ----------- bash 2695530 2695529 soupman cwd DIR 253,1 4096 1839302 /home/soupman bash 2695530 2695529 soupman rtd DIR 253,1 4096 2 / bash 2695530 2695529 soupman txt REG 253,1 1150704 1706258 /usr/bin/bash bash 2695530 2695529 soupman mem REG 253,1 2586930 1705916 /usr/lib/locale/en_US.utf8/LC_COLLATE bash 2695530 2695529 soupman mem REG 253,1 9253600 2494054 /var/lib/sss/mc/passwd bash 2695530 2695529 soupman mem REG 253,1 46424 1711970 /usr/lib64/libnss_sss.so.2 bash 2695530 2695529 soupman mem REG 253,1 3154704 1705958 /usr/lib64/libc-2.28.so bash 2695530 2695529 soupman mem REG 253,1 28968 1705960 /usr/lib64/libdl-2.28.so bash 2695530 2695529 soupman mem REG 253,1 208616 1705909 /usr/lib64/libtinfo.so.6.1 bash 2695530 2695529 soupman mem REG 253,1 252280 1705858 /usr/lib64/ld-2.28.so bash 2695530 2695529 soupman mem REG 253,1 337024 1713187 /usr/lib/locale/en_US.utf8/LC_CTYPE bash 2695530 2695529 soupman mem REG 253,1 54 1713190 /usr/lib/locale/en_US.utf8/LC_NUMERIC bash 2695530 2695529 soupman mem REG 253,1 3316 1970006 /usr/lib/locale/en_US.utf8/LC_TIME bash 2695530 2695529 soupman mem REG 253,1 286 1970004 /usr/lib/locale/en_US.utf8/LC_MONETARY bash 2695530 2695529 soupman mem REG 253,1 26998 1706278 /usr/lib64/gconv/gconv-modules.cache bash 2695530 2695529 soupman mem REG 253,1 57 1705919 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES bash 2695530 2695529 soupman mem REG 253,1 34 1835693 /usr/lib/locale/en_US.utf8/LC_PAPER bash 2695530 2695529 soupman mem REG 253,1 77 1713189 /usr/lib/locale/en_US.utf8/LC_NAME bash 2695530 2695529 soupman mem REG 253,1 167 1966622 /usr/lib/locale/en_US.utf8/LC_ADDRESS bash 2695530 2695529 soupman mem REG 253,1 59 1970005 /usr/lib/locale/en_US.utf8/LC_TELEPHONE bash 2695530 2695529 soupman mem REG 253,1 23 1970003 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT bash 2695530 2695529 soupman mem REG 253,1 368 1970002 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION bash 2695530 2695529 soupman 0u CHR 136,2 0t0 5 /dev/pts/2 bash 2695530 2695529 soupman 1u CHR 136,2 0t0 5 /dev/pts/2 bash 2695530 2695529 soupman 2u CHR 136,2 0t0 5 /dev/pts/2 bash 2695530 2695529 soupman 3r REG 253,1 9253600 2494054 /var/lib/sss/mc/passwd bash 2695530 2695529 soupman 255u CHR 136,2 0t0 5 /dev/pts/2 ---------- lsof 2695575 2695530 soupman cwd DIR 253,1 4096 1839302 /home/soupman lsof 2695575 2695530 soupman rtd DIR 253,1 4096 2 / lsof 2695575 2695530 soupman txt REG 253,1 179608 1707457 /usr/bin/lsof lsof 2695575 2695530 soupman mem REG 253,1 9253600 2494054 /var/lib/sss/mc/passwd lsof 2695575 2695530 soupman mem REG 253,1 46424 1711970 /usr/lib64/libnss_sss.so.2 lsof 2695575 2695530 soupman mem REG 253,1 95400 1706374 /usr/lib64/libz.so.1.2.11 lsof 2695575 2695530 soupman mem REG 253,1 123320 1705974 /usr/lib64/libresolv-2.28.so lsof 2695575 2695530 soupman mem REG 253,1 3079664 1710148 /usr/lib64/libcrypto.so.1.1.1k lsof 2695575 2695530 soupman mem REG 253,1 16320 1706661 /usr/lib64/libkeyutils.so.1.6 lsof 2695575 2695530 soupman mem REG 253,1 71640 1708516 /usr/lib64/libkrb5support.so.0.1 lsof 2695575 2695530 soupman mem REG 253,1 320504 1705972 /usr/lib64/libpthread-2.28.so lsof 2695575 2695530 soupman mem REG 253,1 17336 1706388 /usr/lib64/libcom_err.so.2.1 lsof 2695575 2695530 soupman mem REG 253,1 96648 1708508 /usr/lib64/libk5crypto.so.3.1 lsof 2695575 2695530 soupman mem REG 253,1 967880 1708514 /usr/lib64/libkrb5.so.3.3 lsof 2695575 2695530 soupman mem REG 253,1 356136 1708504 /usr/lib64/libgssapi_krb5.so.2.2 lsof 2695575 2695530 soupman mem REG 253,1 28968 1705960 /usr/lib64/libdl-2.28.so lsof 2695575 2695530 soupman mem REG 253,1 543160 1705848 /usr/lib64/libpcre2-8.so.0.7.1 lsof 2695575 2695530 soupman mem REG 253,1 3154704 1705958 /usr/lib64/libc-2.28.so lsof 2695575 2695530 soupman mem REG 253,1 371032 1708527 /usr/lib64/libtirpc.so.3.0.0 lsof 2695575 2695530 soupman mem REG 253,1 168568 1705208 /usr/lib64/libselinux.so.1 lsof 2695575 2695530 soupman mem REG 253,1 252280 1705858 /usr/lib64/ld-2.28.so lsof 2695575 2695530 soupman mem REG 253,1 337024 1713187 /usr/lib/locale/en_US.utf8/LC_CTYPE lsof 2695575 2695530 soupman mem REG 253,1 26998 1706278 /usr/lib64/gconv/gconv-modules.cache lsof 2695575 2695530 soupman 0u CHR 136,2 0t0 5 /dev/pts/2 lsof 2695575 2695530 soupman 1u CHR 136,2 0t0 5 /dev/pts/2 lsof 2695575 2695530 soupman 2u CHR 136,2 0t0 5 /dev/pts/2 lsof 2695575 2695530 soupman 3r REG 253,1 9253600 2494054 /var/lib/sss/mc/passwd lsof 2695575 2695530 soupman 4r DIR 0,4 0 1 /proc lsof 2695575 2695530 soupman 5r DIR 0,4 0 39302662 /proc/2695575/fd lsof 2695575 2695530 soupman 6w FIFO 0,13 0t0 39302667 pipe lsof 2695575 2695530 soupman 7r FIFO 0,13 0t0 39302668 pipe ------- lsof 2695576 2695575 soupman cwd DIR 253,1 4096 1839302 /home/soupman lsof 2695576 2695575 soupman rtd DIR 253,1 4096 2 / lsof 2695576 2695575 soupman txt REG 253,1 179608 1707457 /usr/bin/lsof ...... 同上面的mem 条目 lsof 2695576 2695575 soupman 5r FIFO 0,13 0t0 39302667 pipe lsof 2695576 2695575 soupman 8w FIFO 0,13 0t0 39302668 pipe
[root@myos ~]# cat /etc/security/limits.conf # /etc/security/limits.conf # #This file sets the resource limits for the users logged in via PAM. #It does not affect resource limits of the system services. # #Also note that configuration files in /etc/security/limits.d directory, #which are read in alphabetical order, override the settings in this #file in case the domain is the same or more specific. #That means for example that setting a limit for wildcard domain here #can be overriden with a wildcard setting in a config file in the #subdirectory, but a user specific setting here can be overriden only #with a user specific setting in the subdirectory. # #Each line describes a limit for a user in the form: # #<domain> <type> <item> <value> # #Where: #<domain> can be: # - a user name # - a group name, with @group syntax # - the wildcard *, for default entry # - the wildcard %, can be also used with %group syntax, # for maxlogin limit # #<type> can have the two values: # - "soft" for enforcing the soft limits # - "hard" for enforcing hard limits # #<item> can be one of the following: # - core - limits the core file size (KB) # - data - max data size (KB) # - fsize - maximum filesize (KB) # - memlock - max locked-in-memory address space (KB) # - nofile - max number of open file descriptors # - rss - max resident set size (KB) # - stack - max stack size (KB) # - cpu - max CPU time (MIN) # - nproc - max number of processes # - as - address space limit (KB) # - maxlogins - max number of logins for this user # - maxsyslogins - max number of logins on the system # - priority - the priority to run user process with # - locks - max number of file locks the user can hold # - sigpending - max number of pending signals # - msgqueue - max memory used by POSIX message queues (bytes) # - nice - max nice priority allowed to raise to values: [-20, 19] # - rtprio - max realtime priority # #<domain> <type> <item> <value> # #* soft core 0 #* hard rss 10000 #@student hard nproc 20 #@faculty soft nproc 20 #@faculty hard nproc 50 #ftp hard nproc 0 #@student - maxlogins 4 # End of file [root@myos ~]# cat /proc/sys/fs/file-max 1000000 [root@myos ~]# cat /proc/sys/fs/file-nr 2112 0 1000000 [root@myos ~]# cat /proc/sys/fs/nr_open 1048576
/proc/sys/fs/inode-max (only present until Linux 2.2) This file contains the maximum number of in-memory inodes. This value should be 3–4 times larger than the value in file-max, since stdin, stdout and network sockets also need an inode to handle them. When you regularly run out of inodes, you need to increase this value. Starting with Linux 2.4, there is no longer a static limit on the number of inodes, and this file is removed. /proc/sys/fs/nr_open (since Linux 2.6.25) This file imposes a ceiling on the value to which the RLIMIT_NOFILE resource limit can be raised (see getrlimit(2)). This ceiling is enforced for both unprivileged and privileged process. The default value in this file is 1048576. (Before Linux 2.6.25, the ceiling for RLIMIT_NOFILE was hard-coded to the same value.) /proc/sys/fs/inode-nr This file contains the first two values from inode-state. /proc/sys/fs/inode-state This file contains seven numbers: nr_inodes, nr_free_inodes, preshrink, and four dummy values (always zero). nr_inodes is the number of inodes the system has allocated. nr_free_inodes represents the number of free inodes. preshrink is nonzero when the nr_inodes > inode-max and the system needs to prune the inode list instead of allocating more; since Linux 2.4, this field is a dummy value (always zero). /proc/sys/fs/file-max This file defines a system-wide limit on the number of open files for all processes. System calls that fail when encountering this limit fail with the error ENFILE. (See also setrlimit(2), which can be used by a process to set the per-process limit, RLIMIT_NOFILE, on the number of files it may open.) If you get lots of error messages in the kernel log about running out of file handles (open file descriptions) (look for "VFS: file-max limit <number> reached"), try increasing this value: echo 100000 > /proc/sys/fs/file-max Privileged processes (CAP_SYS_ADMIN) can override the file-max limit. /proc/sys/fs/file-nr This (read-only) file contains three numbers: the number of allocated file handles (i.e., the number of open file descriptions; see open(2)); the number of free file handles; and the maximum number of file handles (i.e., the same value as /proc/sys/fs/file-max). If the number of allocated file handles is close to the maximum, you should consider increasing the maximum. Before Linux 2.6, the kernel allocated file handles dynamically, but it didn't free them again. Instead the free file handles were kept in a list for reallocation; the "free file handles" value indicates the size of that list. A large number of free file handles indicates that there was a past peak in the usage of open file handles. Since Linux 2.6, the kernel does deallocate freed file handles, and the "free file handles" value is always zero.
[root@myos kernel]# cat threads-max 14293 [root@myos kernel]# cat pid_max 4194304 /proc/sys/kernel/threads-max (since Linux 2.3.11) This file specifies the system-wide limit on the number of threads (tasks) that can be created on the system. Since Linux 4.1, the value that can be written to threads-max is bounded. The minimum value that can be written is 20. The maximum value that can be written is given by the constant FUTEX_TID_MASK (0x3fffffff). If a value outside of this range is written to threads-max, the error EINVAL occurs. The value written is checked against the available RAM pages. If the thread structures would occupy too much (more than 1/8th) of the available RAM pages, threads-max is reduced accordingly. /proc/sys/kernel/pid_max (since Linux 2.5.34) This file specifies the value at which PIDs wrap around (i.e., the value in this file is one greater than the maximum PID). PIDs greater than this value are not allocated; thus, the value in this file also acts as a system-wide limit on the total number of processes and threads. The default value for this file, 32768, results in the same range of PIDs as on earlier kernels. On 32-bit platforms, 32768 is the maximum value for pid_max. On 64-bit systems, pid_max can be set to any value up to 2^22 (PID_MAX_LIMIT, approximately 4 million).
查看所有进程的文件打开数
[root@cl-server ~]# lsof | wc -l
7526
查看某个进程打开的文件数
[root@cl-server ~]# lsof -p 983 | wc -l
19
系统设置-- ulimit
语 法:ulimit [-aHS][-c <core文件上 限>][-d <数据节区大小>][-f <文件大小>][-m <内存大小>][-n <文件数 目>][-p <缓冲区大小>][-s <堆叠大小>][-t <CPU时间>][-u <程序数 目>][-v <虚拟内存大小>]
补充说明:ulimit为shell内建指令,可用来控制shell执行程序的资源。
参 数:
-a 显示目前资源限制的设定。
-c <core文件上限> 设定core文件的最大值,单位为区块。
-d <数据节区大小> 程序数据节区的最大值,单位为KB。
-f <文件大小> shell所能建立的最大文件,单位为区块。
-H 设定资源的硬性限制,也就是管理员所设下的限制。
-m <内存大小> 指定可使用内存的上限,单位为KB。
-n <文件数目> 指定同一时间最多可开启的文件数。
-p <缓冲区大小> 指定管道缓冲区的大小,单位512字节。
-s <堆叠大小> 指定堆叠的上限,单位为KB。
-S 设定资源的弹性限制。
-t <CPU时间> 指定CPU使用时间的上限,单位为秒。
-u <程序数目> 用户最多可开启的程序数目。
-v <虚拟内存大小> 指定可使用的虚拟内存上限,单位为KB。
ulimit -n:
1、这个限制是针对单个程序的限制
2、这个限制不会改变之前已经在运行的程序的限制
3、对这个值的修改,退出了当前的shell就会消失
[root@cl-server ~]# vi /etc/security/limits.conf
root soft nofile 102400
root hard nofile 102400
adminx soft nofile 10240
adminx hard nofile 10240
[root@cl-server ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 7184
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 102400
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 7184
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
1、修改用户进程可打开文件数限制
在Linux平台上,无论编写客户端程序还是服务端程序,在进行高并发TCP连接处理时,最高的并发数量都要受到系统对用户单一进程同时可打开文件数量的限制(这是因为系统为每个TCP连接都要创建一个socket句柄,每个socket句柄同时也是一个文件句柄)。可使用ulimit命令查看系统允许当前用户进程打开的文件数限制:
[speng@as4 ~]$ ulimit -n
1024
这表示当前用户的每个进程最多允许同时打开1024个文件,这1024个文件中还得除去每个进程必然打开的标准输入,标准输出,标准错误,服务器监听 socket,进程间通讯的unix域socket等文件,那么剩下的可用于客户端socket连接的文件数就只有大概1024-10=1014个左右。也就是说缺省情况下,基于Linux的通讯程序最多允许同时1014个TCP并发连接。
对于想支持更高数量的TCP并发连接的通讯处理程序,就必须修改Linux对当前用户的进程同时打开的文件数量的软限制(soft limit)和硬限制(hardlimit)。其中软限制是指Linux在当前系统能够承受的范围内进一步限制用户同时打开的文件数;硬限制则是根据系统硬件资源状况(主要是系统内存)计算出来的系统最多可同时打开的文件数量。通常软限制小于或等于硬限制。
修改上述限制的最简单的办法就是使用ulimit命令:
[speng@as4 ~]$ ulimit -n <file_num>
上述命令中,在<file_num>中指定要设置的单一进程允许打开的最大文件数。如果系统回显类似于“Operation notpermitted”之类的话,说明上述限制修改失败,实际上是因为在<file_num>中指定的数值超过了Linux系统对该用户打开文件数的软限制或硬限制。因此,就需要修改Linux系统对用户的关于打开文件数的软限制和硬限制。
第一步,修改/etc/security/limits.conf文件,在文件中添加如下行:
speng soft nofile 10240
speng hard nofile 10240
其中speng指定了要修改哪个用户的打开文件数限制,可用'*'号表示修改所有用户的限制;soft或hard指定要修改软限制还是硬限制;10240则指定了想要修改的新的限制值,即最大打开文件数(请注意软限制值要小于或等于硬限制)。修改完后保存文件。
第二步,修改/etc/pam.d/login文件,在文件中添加如下行:
session required /lib/security/pam_limits.so
这是告诉Linux在用户完成系统登录后,应该调用pam_limits.so模块来设置系统对该用户可使用的各种资源数量的最大限制(包括用户可打开的最大文件数限制),而pam_limits.so模块就会从/etc/security/limits.conf文件中读取配置来设置这些限制值。修改完后保存此文件。
第三步,查看Linux系统级的最大打开文件数限制,使用如下命令:
[speng@as4 ~]$ cat /proc/sys/fs/file-max
12158
这表明这台Linux系统最多允许同时打开(即包含所有用户打开文件数总和)12158个文件,是Linux系统级硬限制,所有用户级的打开文件数限制都不应超过这个数值。通常这个系统级硬限制是Linux系统在启动时根据系统硬件资源状况计算出来的最佳的最大同时打开文件数限制,如果没有特殊需要,不应该修改此限制,除非想为用户级打开文件数限制设置超过此限制的值。修改此硬限制的方法是修改/etc/rc.local脚本,在脚本中添加如下行:
echo 22158 > /proc/sys/fs/file-max
这是让Linux在启动完成后强行将系统级打开文件数硬限制设置为22158。修改完后保存此文件。
完成上述步骤后重启系统,一般情况下就可以将Linux系统对指定用户的单一进程允许同时打开的最大文件数限制设为指定的数值。如果重启后用 ulimit-n命令查看用户可打开文件数限制仍然低于上述步骤中设置的最大值,这可能是因为在用户登录脚本/etc/profile中使用 ulimit-n命令已经将用户可同时打开的文件数做了限制。由于通过ulimit-n修改系统对用户可同时打开文件的最大数限制时,新修改的值只能小于或等于上次ulimit-n设置的值,因此想用此命令增大这个限制值是不可能的。所以,如果有上述问题存在,就只能去打开/etc/profile脚本文件,在文件中查找是否使用了ulimit-n限制了用户可同时打开的最大文件数量,如果找到,则删除这行命令,或者将其设置的值改为合适的值,然后保存文件,用户退出并重新登录系统即可。
通过上述步骤,就为支持高并发TCP连接处理的通讯处理程序解除关于打开文件数量方面的系统限制。
2、修改网络内核对TCP连接的有关限制
在Linux上编写支持高并发TCP连接的客户端通讯处理程序时,有时会发现尽管已经解除了系统对用户同时打开文件数的限制,但仍会出现并发TCP连接数增加到一定数量时,再也无法成功建立新的TCP连接的现象。出现这种现在的原因有多种。
第一种原因可能是因为Linux网络内核对本地端口号范围有限制。此时,进一步分析为什么无法建立TCP连接,会发现问题出在connect()调用返回失败,查看系统错误提示消息是“Can't assign requestedaddress”。同时,如果在此时用tcpdump工具监视网络,会发现根本没有TCP连接时客户端发SYN包的网络流量。这些情况说明问题在于本地Linux系统内核中有限制。其实,问题的根本原因在于Linux内核的TCP/IP协议实现模块对系统中所有的客户端TCP连接对应的本地端口号的范围进行了限制(例如,内核限制本地端口号的范围为1024~32768之间)。当系统中某一时刻同时存在太多的TCP客户端连接时,由于每个TCP客户端连接都要占用一个唯一的本地端口号(此端口号在系统的本地端口号范围限制中),如果现有的TCP客户端连接已将所有的本地端口号占满,则此时就无法为新的TCP客户端连接分配一个本地端口号了,因此系统会在这种情况下在connect()调用中返回失败,并将错误提示消息设为“Can't assignrequested address”。有关这些控制逻辑可以查看Linux内核源代码,以linux2.6内核为例,可以查看tcp_ipv4.c文件中如下函数:
static int tcp_v4_hash_connect(struct sock *sk)
请注意上述函数中对变量sysctl_local_port_range的访问控制。变量sysctl_local_port_range的初始化则是在tcp.c文件中的如下函数中设置:
void __init tcp_init(void)
内核编译时默认设置的本地端口号范围可能太小,因此需要修改此本地端口范围限制。
第一步,修改/etc/sysctl.conf文件,在文件中添加如下行:
net.ipv4.ip_local_port_range = 1024 65000
这表明将系统对本地端口范围限制设置为1024~65000之间。请注意,本地端口范围的最小值必须大于或等于1024;而端口范围的最大值则应小于或等于65535。修改完后保存此文件。
第二步,执行sysctl命令:
[speng@as4 ~]$ sysctl -p
如果系统没有错误提示,就表明新的本地端口范围设置成功。如果按上述端口范围进行设置,则理论上单独一个进程最多可以同时建立60000多个TCP客户端连接。
第二种无法建立TCP连接的原因可能是因为Linux网络内核的IP_TABLE防火墙对最大跟踪的TCP连接数有限制。此时程序会表现为在 connect()调用中阻塞,如同死机,如果用tcpdump工具监视网络,也会发现根本没有TCP连接时客户端发SYN包的网络流量。由于 IP_TABLE防火墙在内核中会对每个TCP连接的状态进行跟踪,跟踪信息将会放在位于内核内存中的conntrackdatabase中,这个数据库的大小有限,当系统中存在过多的TCP连接时,数据库容量不足,IP_TABLE无法为新的TCP连接建立跟踪信息,于是表现为在connect()调用中阻塞。此时就必须修改内核对最大跟踪的TCP连接数的限制,方法同修改内核对本地端口号范围的限制是类似的:
第一步,修改/etc/sysctl.conf文件,在文件中添加如下行:
net.ipv4.ip_conntrack_max = 10240
这表明将系统对最大跟踪的TCP连接数限制设置为10240。请注意,此限制值要尽量小,以节省对内核内存的占用。
第二步,执行sysctl命令:
[speng@as4 ~]$ sysctl -p
如果系统没有错误提示,就表明系统对新的最大跟踪的TCP连接数限制修改成功。如果按上述参数进行设置,则理论上单独一个进程最多可以同时建立10000多个TCP客户端连接。
3、使用支持高并发网络I/O的编程技术
在Linux上编写高并发TCP连接应用程序时,必须使用合适的网络I/O技术和I/O事件分派机制。
可用的I/O技术有同步I/O,非阻塞式同步I/O(也称反应式I/O),以及异步I/O。在高TCP并发的情形下,如果使用同步I/O,这会严重阻塞程序的运转,除非为每个TCP连接的I/O创建一个线程。但是,过多的线程又会因系统对线程的调度造成巨大开销。因此,在高TCP并发的情形下使用同步I /O是不可取的,这时可以考虑使用非阻塞式同步I/O或异步I/O。非阻塞式同步I/O的技术包括使用select(),poll(),epoll等机制。异步I/O的技术就是使用AIO。
从I/O事件分派机制来看,使用select()是不合适的,因为它所支持的并发连接数有限(通常在1024个以内)。如果考虑性能,poll()也是不合适的,尽管它可以支持的较高的TCP并发数,但是由于其采用“轮询”机制,当并发数较高时,其运行效率相当低,并可能存在I/O事件分派不均,导致部分TCP连接上的I/O出现“饥饿”现象。而如果使用epoll或AIO,则没有上述问题(早期Linux内核的AIO技术实现是通过在内核中为每个I /O请求创建一个线程来实现的,这种实现机制在高并发TCP连接的情形下使用其实也有严重的性能问题。但在最新的Linux内核中,AIO的实现已经得到改进)。
综上所述,在开发支持高并发TCP连接的Linux应用程序时,应尽量使用epoll或AIO技术来实现并发的TCP连接上的I/O控制,这将为提升程序对高并发TCP连接的支持提供有效的I/O保证。
[adminx@cl-server ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 7184
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
使用root配置
[root@cl-server ~]# vi /etc/security/limits.conf
adminx soft nofile 10240
adminx hard nofile 10240
adminx soft nproc 65535
adminx hard nproc 65535
无变化,退出会话,重新登录
[adminx@cl-server ~]$ exit
登出
WARNING! The remote SSH server rejected X11 forwarding request.
Last login: Thu Oct 21 10:40:42 2021 from 192.168.234.1
[adminx@cl-server ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 7184
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 10240
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
# /etc/security/limits.conf
#
#This file sets the resource limits for the users logged in via PAM.
#It does not affect resource limits of the system services.
#
#Also note that configuration files in /etc/security/limits.d directory,
#which are read in alphabetical order, override the settings in this
#file in case the domain is the same or more specific.
#That means for example that setting a limit for wildcard domain here
#can be overriden with a wildcard setting in a config file in the
#subdirectory, but a user specific setting here can be overriden only
#with a user specific setting in the subdirectory.
#
#Each line describes a limit for a user in the form:
#
#<domain> <type> <item> <value>
#
#Where:
#<domain> can be:
# - a user name
# - a group name, with @group syntax
# - the wildcard *, for default entry
# - the wildcard %, can be also used with %group syntax,
# for maxlogin limit
#
#<type> can have the two values:
# - "soft" for enforcing the soft limits
# - "hard" for enforcing hard limits
#
#<item> can be one of the following:
# - core - limits the core file size (KB)
# - data - max data size (KB)
# - fsize - maximum filesize (KB)
# - memlock - max locked-in-memory address space (KB)
# - nofile - max number of open file descriptors
# - rss - max resident set size (KB)
# - stack - max stack size (KB)
# - cpu - max CPU time (MIN)
# - nproc - max number of processes
# - as - address space limit (KB)
# - maxlogins - max number of logins for this user
# - maxsyslogins - max number of logins on the system
# - priority - the priority to run user process with
# - locks - max number of file locks the user can hold
# - sigpending - max number of pending signals
# - msgqueue - max memory used by POSIX message queues (bytes)
# - nice - max nice priority allowed to raise to values: [-20, 19]
# - rtprio - max realtime priority
#
#<domain> <type> <item> <value>
#
#* soft core 0
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#@student - maxlogins 4
# End of file