Dec 22 15:48:02 host kernel: nf_conntrack: table full, dropping packet. Dec 22 15:48:02 host kernel: nf_conntrack: table full, dropping packet. Dec 22 15:48:02 host kernel: nf_conntrack: table full, dropping packet. Dec 22 15:48:02 host kernel: nf_conntrack: table full, dropping packet. Dec 22 15:48:02 host kernel: nf_conntrack: table full, dropping packet.
In my case, I saw these messages on "Red Hat Enterprise Linux Server release 6.4".
This error message means the nf_conntrack kernel maximum number assigned value gets reached. Normally this means the server is having a heavy traffic passing by or there is a DDos attack.
What should you do?
If you see this error, first thing you should do is check the current nf_conntrack max value:
# cat /proc/sys/net/netfilter/nf_conntrack_max 65536 or # sysctl -a | grep net.nf_conntrack_max net.nf_conntrack_max = 65536
Now check the current nf_conntrack active connections (I captured the value after the problem, so you are seeing a much lower value here):
# sysctl -a | grep net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_count = 31517 Note the shown connections "31517" is assigned dynamicly on each new successful TCP/IP connection. or # lsmod | grep 'nf_conntrack' nf_conntrack 29645 2 nf_conntrack_ipv4,xt_state
Fixing the error:
One temporary fix is to increase the nf_conntrack_max:
# sysctl -w net.nf_conntrack_max=1000000
Remeber to increase the size of nf_conntrack hash-table as well, this hashsize value stores lists of conntrack entries, should be increased propertionally whenever net.nf_conntrack_max is increased.
# echo 250000 > /sys/module/nf_conntrack/parameters/hashsize
Normally the rule of thumb is:
hashsize = nf_conntrack_max / 4
Note "sysctl -w" is just temporarily changes the value, to make it permanently you need to edit the /etc/sysctl.conf:
# echo "net.nf_conntrack_max=1000000" >> /etc/sysctl.conf
Decreasing nf_conntrack timeout value:
You can also decrease nf_conntrack's timeout value to help the dropping connections. Because the default timeout value is too high (600 seconds), so even if you increased the max value, you might still see error messages. To check the timeout:
# sudo sysctl -a | grep conntrack | grep timeout net.netfilter.nf_conntrack_generic_timeout = 600 net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120 net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60 net.netfilter.nf_conntrack_tcp_timeout_established = 432000 net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120 net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60 net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30 net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120 net.netfilter.nf_conntrack_tcp_timeout_close = 10 net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300 net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300 net.netfilter.nf_conntrack_udp_timeout = 30 net.netfilter.nf_conntrack_udp_timeout_stream = 180 net.netfilter.nf_conntrack_icmp_timeout = 30 net.netfilter.nf_conntrack_events_retry_timeout = 15
See the "net.netfilter.nf_conntrack_generic_timeout" is 600 seconds, which is pretty high. This means and NAT-ted connection not responding can stay hanging for 10 minutes. Also "net.netfilter.nf_conntrack_tcp_timeout_established" is set to 432000 (5 days) is too high. With this value, the server will become an easy target for anyone who would like to flood it.
Decrease "net.netfilter.nf_conntrack_generic_timeout" and "net.netfilter.nf_conntrack_tcp_timeout_established"
# sysctl -w net.netfilter.nf_conntrack_generic_timeout=120 # sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established = 54000
Make sure to update the sysctl.conf file if you want to change it permanently.
No comments:
Post a Comment