Hadoop - Datanode denied communication with namenode

If you ran into this error when you try to add hdfs1 service using Cloudera Manager:

2013-04-24 23:14:46,929 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-761923600-10.2.1.227-1366870456764 (storage id DS-1593183589-10.2.0.61-50010-1366870463382) service to cshdpm3/10.2.1.227:8020

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode: DatanodeRegistration(10.2.1.61, storageID=DS-1593183589-10.2.0.61-50010-1366870463382, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=cluster44;nsid=663570625;c=0)

        at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:582)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3481)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:862)

        at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:91)

Typically a "denied" message means that the datanode is reporting itself not as the IP it's reporting itself to CM. To debug, first compare the IP address in "DatanodeRegistration (10.2.1.61)" above, then from command line, do

$ sudo cat /proc/$(pgrep -f namenode.NameNode)/cwd/dfs_hosts_allow.txt
10.6.70.33
10.6.70.34
10.6.70.35
10.6.70.30
10.6.70.31
10.6.70.32

Make sure IP address are consistent.