Thursday, October 24, 2013

Impala Duplicate registration of subscriber: subscriber-id" Error


Impala version: Impala 1.1.1-1.p0.17

One of the datanode went down this morning and after a reboot, the Impala instance on that datanode won't start properly. It gave me this error in the log file:

I1016 08:39:49.260819 25392 thread.cc:235] Started thread 25494 - statestore-subscriber:recovery-mode-thread
I1016 08:39:49.262377 25392 status.cc:70] State Store Subscriber did not start up
E1016 08:39:49.262532 25392 impalad-main.cc:90] Impalad services did not start correctly, exiting.  Error: Duplicate registration of subscriber: ld:22002
State Store Subscriber did not start up.

This is due to a bug in Impala 1.1 (https://issues.cloudera.org/browse/IMPALA-474). It happends when a subscriber takes a long time to processes/respond to a statestore heartbeat. Additionally, it appears that if there is a duplicate registration error when Impala is starting up the daemon will fail to start.

Suppose to be fixed in Impala 1.1, but it did happened in my case. I tried restart Impala daemon several times on that specific datanode several times but still won't start. Had to restart the whole Impala service to get it re-registered.

No comments: