Cloudera Manager - How to change ulimit for CDH process

If you are using Cloudera Manager to supervise your hadoop process, you will find the ulimit value that you change in /etc/security/limits.conf does not take effect. For example, on my server, I set hdfs user to be unlimited in my /etc/security/limits.conf, but the process still uses "32768" as max value:

# jps | grep NameNode
9994 NameNode

# cat /proc/9994/limits | grep files
Max open files            32768                32768                files

Why? Because Cloudera Manager doesn't use /etc/security/limits.conf file, or has it own file to control the limit. Check "/usr/sbin/cmf-agent":

# cat /usr/sbin/cmf-agent

and you will find

# Max number of open files
ulimit -n 32768

That's where the 32768 is from.

So, if you choose to use Cloudera Manager to supervise your hadoop process, /usr/sbin/cmf-agent is the file you should use to change your ulimit.