Cloudera Manager - Datanode denied communication with namenode

If you ran into the following issue:

2013-04-24 23:14:46,929 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-761923600-10.2.1.227-1366870456764 (storage id DS-1593183589-10.2.0.61-50010-1366870463382) service to cshdpm3/10.2.1.227:8020
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode: DatanodeRegistration(10.2.1.61, storageID=DS-1593183589-10.2.0.61-50010-1366870463382, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=cluster44;nsid=663570625;c=0)
        at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:582)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3481)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:862)

The Namenode has a configuration file called "dfs_hosts_allow".  CM prepares that based on the IP and DNS names of the managed hosts.  Tyipcally, a "denied" message means that the datanode is reporting itself not as the IP it's reporting itself to CM. To debug this, first compare the IP address here:

DatanodeRegistration(10.2.1.61, storageID=DS-1593183589-10.2.0.61-50010-1366870463382, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=cluster44;nsid=663570625;c=0)

with what you think should be the IP in the "Hosts" tab. Also, navigate to the Namenode process, "show configuration files", and check "dfs_hosts_allow.txt".  (Or run the following on the namenode host).

$sudo cat /proc/$(pgrep -f namenode.NameNode)/cwd/dfs_hosts_allow.txt
172.29.122.20
172.29.122.24
172.29.122.25
172.29.122.21
172.29.122.23
172.29.122.22
172.29.122.19

Make sure both IP are matching.