OVERVIEW
Issue
Packet drops on this system for connections using ip_conntrack
or nf_conntrack
. Following messages seen in /var/log/kern
on the centos nodes when one of the instances drops packets:
$ tail -f /var/log/kern
Jul 4 03:47:16 centos kernel: : nf_conntrack: table full, dropping packet
Jul 4 03:47:16 centos kernel: : nf_conntrack: table full, dropping packet
This can happen when you are being attacked, or is also very likely to happen on a busy server even if there is no malicious activity.
NOTE: By default, CentOS will set this maximum to 65,536 connections. This is enough for lightly loaded servers, but can easily be exhausted on heavy traffic servers.
How to Fix
View the current maximum configured connections
$ cat /proc/sys/net/netfilter/nf_conntrack_max
To see the current used connections
$ cat /proc/sys/net/netfilter/nf_conntrack_count
Increase maximum configured connections limit
# Temporarily Solution
echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max
# Permanent Solution
# Add following line on /etc/rc.d/rc.local
$ vim /etc/rc.d/rc.local
echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max
$ chmod a+x /etc/rc.d/rc.local