Cassandra - Tips for upgrading SSTables

Recently we upgraded our Cassandra cluster from 1.2.x to 2.0.x. Here are some tips for running "nodetool upgradesstables" (Reads only SSTables created by old major versions of Cassandra and re-writes them to the current version one at a time) command I would like to share.

Normally you need to run this command after upgrading your Cassandra cluster. The purpose of this command is to convert sstables to the latest format. Alwyas read NEWS.txt (https://github.com/apache/cassandra/blob/cassandra-2.2/NEWS.txt) and CHANGES.txt (https://github.com/apache/cassandra/blob/cassandra-2.2/CHANGES.txt) before upgrade your cluster.

Always run upgrade off peak time

Avoid upgrade during the peak time, because every sstable will be rewritten (old version), you do not want to upgrade during the cluster peak time. Off peak time can give you the full throughput to the upgrade process which will speed up the process, and also avoid resource competition and cluster overloading.

How long will it take?

After issue this command, your node will be re-writing every sstable at the speed of "compaction_throughput_mb_per_sec". You can monitor the progress by using "nodetool compactionstats" and "nodetool netstats". During my upgrade process, I use "ll /var/lib/cassandra/data00/keyspace1/ | egrep -v "UPDATE_DATE" | wc -l" to calculate the remaining number of files that haven't been upgraded. This way you can have a rough estimate about how long the upgrade process will take. You can also control the speed by "nodetool setcompactionthrouighput"

Run in parallel if possible

You can run "nodetool upgradesstables" in parallel. This process is a "within the node" operation that pounds I/O. In my case, we have a 15 nodes Cassandra cluster, we ran "nodetool upgradesstables" on all nodes in parallel at the least busy time, with the "compaction_throughput_mb_per_sec" of 0 (No throttling).

Do not do streaming

DO NOT bootstrap any new nodes or repair any exiting nodes during the "nodetool upgradesstables"! Because sstable structure has changed after upgrade, streaming during the upgrade process is a bad idea. Always wait until upgrade process complete.

In case

In case of cluster overloading or whatever reason you need to stop the upgrade process, you can just kill the upgrade process by issusing the "kill -9 processID" command.