100万连接测试资料整理

设置相关脚本：

默认：

$cat /proc/sys/net/ipv4/ip_local_port_range

32768 61000

$cat setting.sh

#/bin/bash

# modify backlog, meanwhile change listen function to 1024, default 128
# echo “1024" > /proc/sys/net/core/somaxconn

sysctl -w net.core.somaxconn = 1024

# more ports for testing
sysctl -w net.ipv4.ip_local_port_range="1025 65535" // 1025 - 6535, about 50000 port avaiable

# tcp read buffer, min, default, maximum
sysctl -w net.ipv4.tcp_rmem="2048 2048 16777216"      // adjust read buffer -> 2k

# tcp write buffer, min, default, maximum
sysctl -w net.ipv4.tcp_wmem="2048 2048 16777216"      // djust write buffer -> 2k, so every socket take about 4k bytes, tested 3.75k.

# tcp buffer, min, default, maximum, uint = page,page = 4096 byte, we can change the value based on mem
sysctl -w net.ipv4.tcp_mem="1250000 1500000 1750000" // min 5G, default 6G max 7G, 7G/4K about 175w connections

# tcp
sysctl -w net.ipv4.tcp_max_orphans="200000"          // as large as possible, consider physical mem.

echo 9999999 | tee /proc/sys/fs/nr_open
echo 9999999 | tee /proc/sys/fs/file-max

ulimit -n 9999999

#ulimit -HSn 9999999, setting soft link and hard link

# edit /etc/security/limits.conf, add line, for all user

#* - nofile 9999999

这样修改的结果是临时有效，如果系统重启将失效，如果永久生效：

echo “net.core.somaxconn = 1024" >> /etc/sysctl.conf

echo "net.ipv4.ip_local_port_range = 1024 65535" >> /etc/sysctl.conf
echo "net.ipv4.tcp_rmem = 2048 4096 16777216" >> /etc/sysctl.conf
echo "net.ipv4.tcp_wmem = 2048 4096 16777216" >> /etc/sysctl.conf
echo "net.ipv4.tcp_mem = 1250000 1500000 1750000" >> /etc/sysctl.conf
echo "net.ipv4.tcp_max_orphans=200000" >> /etc/sysctl.conf

echo "fs.nr_open = 9999999" >> /etc/sysctl.conf
echo "fs.file-max = 9999999" >> /etc/sysctl.conf

echo "* - nofile 9999999" >> /etc/security/limits.conf

测试方式：

A 服务端 8G 内存，服务端启动端口 5050 地址：192.168.1.2

B client 8G 内存，客户端启动，设置20个ip 地址别名 192.168.1.100 - 192.168.1.120, 批量设置： for i in `seq 100 120`; do sudo ifconfig eth0:$i 192.168.1.$i up ; done

客户端发送每个ip地址绑定发送 50000，共 20个 20 * 50000 = 100万

发送命令：

1. ./epoll_cli -s 192.168.1.2 -p 5050 -m -t 192.168.1.99 -n 10 -c 50000 // 绑定 100 - 110，建立50w连接

2. ./epoll_cli -s 192.168.1.2 -p 5050 -m -t 192.168.1.109 -n 10 -c 50000 // 绑定 110 - 120，建立50w连接

C 使用ACE中的性能测试工具对，多连接情况下进行测试

$ACE_ROOT/performance-tests/TCP/tcp_test -c localhost -i 500000 -b 64 -p 5050

代码下载：

https://github.com/DavadDi/blogs/blob/master/epoll_server.tgz