Linux - ERROR: nf_conntrack: table full, dropping packet

Sometime on a busy server, you might see the following message repeatly in /var/log/messages:

Dec 22 15:48:02 host kernel: nf_conntrack: table full, dropping packet.
Dec 22 15:48:02 host kernel: nf_conntrack: table full, dropping packet.
Dec 22 15:48:02 host kernel: nf_conntrack: table full, dropping packet.
Dec 22 15:48:02 host kernel: nf_conntrack: table full, dropping packet.
Dec 22 15:48:02 host kernel: nf_conntrack: table full, dropping packet.

In my case, I saw these messages on "Red Hat Enterprise Linux Server release 6.4".

This error message means the nf_conntrack kernel maximum number assigned value gets reached. Normally this means the server is having a heavy traffic passing by or there is a DDos attack.

What should you do?

If you see this error, first thing you should do is check the current nf_conntrack max value:

# cat /proc/sys/net/netfilter/nf_conntrack_max
65536
or
# sysctl -a | grep net.nf_conntrack_max
net.nf_conntrack_max = 65536

Now check the current nf_conntrack active connections (I captured the value after the problem, so you are seeing a much lower value here):

# sysctl -a | grep net.netfilter.nf_conntrack_count
net.netfilter.nf_conntrack_count = 31517
Note the shown connections "31517" is assigned dynamicly on each new successful TCP/IP connection.
or
# lsmod | grep 'nf_conntrack'
nf_conntrack           29645  2 nf_conntrack_ipv4,xt_state

Fixing the error：
One temporary fix is to increase the nf_conntrack_max:

# sysctl -w net.nf_conntrack_max=1000000

Remeber to increase the size of nf_conntrack hash-table as well, this hashsize value stores lists of conntrack entries, should be increased propertionally whenever net.nf_conntrack_max is increased.

# echo 250000 > /sys/module/nf_conntrack/parameters/hashsize

Normally the rule of thumb is:
hashsize = nf_conntrack_max / 4

Note "sysctl -w" is just temporarily changes the value, to make it permanently you need to edit the /etc/sysctl.conf:

# echo "net.nf_conntrack_max=1000000" >> /etc/sysctl.conf

Decreasing nf_conntrack timeout value:
You can also decrease nf_conntrack's timeout value to help the dropping connections. Because the default timeout value is too high (600 seconds), so even if you increased the max value, you might still see error messages. To check the timeout:

# sudo sysctl -a | grep conntrack | grep timeout
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_events_retry_timeout = 15

See the "net.netfilter.nf_conntrack_generic_timeout" is 600 seconds, which is pretty high. This means and NAT-ted connection not responding can stay hanging for 10 minutes. Also "net.netfilter.nf_conntrack_tcp_timeout_established" is set to 432000 (5 days) is too high. With this value, the server will become an easy target for anyone who would like to flood it.

Decrease "net.netfilter.nf_conntrack_generic_timeout" and "net.netfilter.nf_conntrack_tcp_timeout_established"

# sysctl -w net.netfilter.nf_conntrack_generic_timeout=120
# sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established = 54000

Make sure to update the sysctl.conf file if you want to change it permanently.