Wednesday, August 26, 2015

Cassandra - Upgrade from 1.x to 2.x

Important notes:

  1. Do one node first, have some sort of graphs to monitor the upgrade. For example, client request errors (Read and write), the average column family reads, the average of read latency 99th percentitle. Investigate any outstanding changes.
  2. Do the rest of nodes from the same Rack. Do not do one node from each Rack, assuming you are not using "SimpleSnitch". That way if the update screw up, worst case you only lose one Rack, and have two unaffected Racks. You will not lose data and it will be eaiser to re-build one rack.

Before you doing the upgrade, you should read the Cassandra "NEWS.txt" and "CHANGES.txt." for any upgrade instructions or caveats. Also, make sure all the current nodes in the cluster are on the same latest version using:
# nodetool version

Also check cassandra log files, cassandra infomation (nodetool info), make sure there are no warnings/errors/exceptions.

On one node:
Make sure all the SSTables are up to date:
# nodetool upgradesstables

Make a backup so the upgrade can be reverted:
# nodetool snapshot -t preupgradebkp

Shutdown Cassandra service:
# service cassandra stop
Or drain you node first, if it is a busy cluster
# nodetool -h localhost disablethrift
# nodetool -h localhost disablegossip
# nodetool -h localhost drain
# service cassandra stop

Make a copy of the backups:
Out of paranoia, we want to make sure the removal of old package will not remove any data
# rsync -av --link-dest=$CASSANDRA_DATA_DIR $CASSANDRA_DATA_DIR $CASSANDRA_DATA_BACKUP_DIR
# rsync -av --link-dest=/var/lib/cassandra/ /var/lib/cassandra/ /opt/cassandrabkp/
# rsync -av --link-dest=/etc/cassandra/ /etc/cassandra/ /opt/cassandra_conf/

Upgrade Cassandra package (rpm):
# yum remove cassandra12
# yum install cassandra20

Update cassandra configurations:
Using the backups you made of your configuration files, merge any modifications you have previously made into the new configuration files for the new version. Configuration options change often, so be sure to double check the version restrictions for additional steps and changes regarding configuration.

Start cassandra service:
# service cassandra start

Remove backups:
# Only cassandra service is running okay and there is no errors
# rm -r $CASSANDRA_DATA_BACKUP_DIR /opt/cassandrabkp /opt/cassandra_conf/
# nodetool clearsnapshot
# nodetool upgradesstables
# Notice:
# While "scrub" does rebuild your sstables, it will also discard data it deems broken and create a snapshot, which
# you have to remove manually.

Check the logs for warnings, errors, and exceptions. Repeat the same process on other nodes. It is always a good idea to let the first node run at least 24 hours before you proceed to the next node upgrade.

No comments: