Cassandra 202: Snitches

Introduction:

Snitch is a component which determines the network topology of the whole cassandra cluster. It provides the translation from the node’s IP address to the datacenter & rack it belongs to. This ensures that the data is placed in such a way that the cluster can handle rack/datacenter level outages.

To improve the resiliency of our cassandra cluster, we decided to move from SimpleSnitch which is the default snitch to GossipingPropertyFileSnitch, a recommended snitch for production grade cluster. While the latter is rack and datacenter aware snitch, the former doesn’t recognize any of this information.

Facts:

• Every node in the cluster must be configured to use same snitch type.
• Since SimpleSnitch assigns every node to rack1in datacenter1 , you can only migrate from SimpleSnitch to GossipingPropertyFileSnitch first. None of the other snitches like Ec2Snitch or GoogleCloudSnitch are compatible to SimpleSnitch. Migrating to any incompatible snitch directly can cause data loss.
• Cassandra doesn’t allow changing rack or datacenter of a node which already has data in it. Hence the only option in such case is to first decommission the node and bootstrap it.

SimpleSnitch to GPFS Migration:

I. Change the Snitch configuration of the current cluster

Let’s say you have five nodes in your cluster configured with SimpleSnitch. You can visualize your cluster like this:

1. Stop all nodes of the current cluster.

2. Since GPFS refers to cassandra-rackdc.properties to infer the rack and datacenter of a node, update them in each node as follows
   

dc=datacenter1
rack=rack1

1. Update the snitch type in each node in cassandra.yaml

endpoint\_snitch: GossipingPropertyFileSnitch

1. Start all nodes of the current cluster.

II. Update all cassandra clients to be DCAware

Before adding the new datacenter, we need to fulfill these prerequisites:

1. Make sure that clients query the existing datacenter datacenter1 cassandra nodes only:

• Ensure clients are configured to useDCAwareRoundRobinPolicy
• Ensure clients are pointing to datacenter1
• Change consistency level from QUORUM to LOCAL_QUORUM

1. Update all cassandra keyspaces to use NetworkTopologyStrategy from SimpleStrategy

ALTER KEYSPACE sample\_keyspace WITH REPLICATION =
{ 'class' : 'NetworkTopologyStrategy', 'datacenter1' : 3 };

You need to do it for all system related keyspaces and user created keyspaces except few system keyspaces whose configuration you cant change.

Altering the keyspace replication settings doesn’t actually move any existing data. It only affects new reads/writes.

III. Add the new datacenter

Now start nodes of the new datacenter making sure that they all have the same cluster_name configuration as current cluster.

To make sure that cassandra nodes in one datacenter can see the nodes of the other datacenter, add the seed nodes of the new datacenter in all of the old datacenter’s nodes configuration and restart them. Similarly, add the seed nodes of the old datacenter in all of the new datacenter’s nodes configuration and restart them. It is always recommended to use seeds from all datacenters.

One this is done, you will notice that nodetool status will show both the datacenters in the output.

Although the new datacenter nodes have joined the cluster, they still don’t have any data. To ensure that every keyspace from old datacenter nodes is replicated to new datacenter nodes.

1. Update every keyspace adding count of expected replicas in newer datacenter.

ALTER KEYSPACE sample\_keyspace WITH REPLICATION =
{ 'class' : 'NetworkTopologyStrategy', 'datacenter1' : 3, 'datacenter2' : 3 };

You can specify replica count as 1 for the new datacenter to ensure rebuild faster.

If you specify replica count as 1 for the new datacenter and change it to lets say 3 later, you’ll need to run nodetool repair -full with the -dc option to repair nodes only in the new datacenter. This may increase the overall time.

1. Rebuild each node in the new datacenter.

nodetool rebuild -- _datacenter1_

You can rebuild on one or more nodes at the same time, but we will suggest you to run it only on once at a time. Running on one node at a time reduces the impact on the older datacenter. In our case, running it concurrently on multiple nodes caused out of memory issues for cassandra.

IV. Verify the newer datacenter is in sync with older datacenter

V. Remove references of older datacenter

Before starting the decommissioning process for the older datacenter, we need to fulfill these prerequisites:

1. Make sure that clients are pointing to the newer datacenter datacenter2 cassandra nodes only.
2. Run a full repair with nodetool repair -full to ensure that all data is propagated from the datacenter being decommissioned. You need to run it on each of the nodes of the older datacenter (and on one node at a time)
3. Update every keyspace removing the older datacenter datacenter1 from replication configuration.

ALTER KEYSPACE sample\_keyspace WITH REPLICATION =
{ 'class' : 'NetworkTopologyStrategy', 'datacenter2' : 3 };

VI. Decommission the older datacenter

1. Run nodetool decommission on every node of the older datacenter.

References:

• Cassandra: Updating the replication factor