Using Aerospike policies (correctly!)

Developers familiar with Aerospike will know that the vast majority of the API calls take a “policy” as a parameter. This policy is optional but can have a significant impact on the application. There are a number of parameters in policies that can be confusing to developers, and even creating the policies is frequently done incorrectly. This blog will attempt to clear up some of this confusion.

What are policies?

Each API call in Aerospike typically takes a policy as a parameter. For example, consider a simple get in Java:

client.put(null, new Key("test", "testSet", 1), 
           new Bin("name", "Tim"), new Bin("age", 312));

The first parameter here is the policy; in this case, it is an instance of the WritePolicy class. If null is specified, then the default policy is used. Policies typically contain fields that fall into one of three categories:

• Networking control: How should the API call behave if something goes wrong? For example, if the server cannot be reached, should the call be retried on a different server?

• Application behavior: Parameters that control how the API call should behave in various application-driven scenarios. For example, Aerospike does an “upsert” operation by default on a put, selecting to either update or insert the record depending on whether it already exists or not. However, the application might require the record to be inserted only, failing if the record already exists in the database.

• Filtering: Aerospike supports filters on operations, allowing almost all operations to become conditional. For example, an operation can be specified to retrieve a record if the amount bin in that record is greater than a thousand.

These three categories are all in the same policy object, with the networking control defined on the Policy class, which is a superclass of other policies, such as WritePolicy. Note that the Policy class is used for read operations without a subclass; there is no ReadPolicy class.

Networking control policies

The policy superclass defines a number of parameters related to networking. These are critical to optimizing application behavior but are often poorly understood. Let’s take a look at these and the interaction between them.

These networking fields are:

• connectTimeout
• maxRetries
• replica
• sleepBetweenRetries
• socketTimeout
• timeoutDelay
• totalTimeout

A diagram is helpful in understanding the interaction between these fields:

The diagram shows a timeline of what happens when an API call is submitted to the Aerospike Client Library (ACL) and can be read from left to right.

The first thing that happens is the ACL needs a connection to the server which holds the data. If this is a simple get or put, this will be a connection to a single server, but if it’s a complex operation such as a batch or query, multiple servers might be involved. The ACL has connection pools to each server node, so acquiring this connection is typically a matter of borrowing one from the pool, which is almost instantaneous. However, if the pool has no available connections, a new connection must be established, which can take some time. This is especially true if the connection is over TLS as the handshake between the client and server on a TLS connection can be slow.

This connection establishment process has its own timeout field, connectTimeout. This field is in milliseconds and includes both the connection establishment as well as user authentication (if any). By default, this is set to zero, which is usually a reasonable default – this uses the lesser of the socketTimeout and totalTimeout, or 2,000 milliseconds if both are zero. If the application can tolerate extra time when a connection is established, especially when using TLS, then this can be set to a non-zero value. For example, if you want a large connectTimeout and a small socketTimeout to allow for protracted SSL handshakes but fast application responses, setting this value to something like 5,000 milliseconds works well.

Once a connection has been retrieved, the API call is sent to the server. There are two timeouts compared at this point, the socketTimeout and the totalTimeout. Both are in milliseconds, and the socketTimeout reflects how long it takes to wait for data from this API call before retrying or failing. The totalTimeout specifies the maximum time the call can take, including retries. If the time remaining in the totalTimeout is less than the socketTimeout, then the totalTimeout is used as the socketTimeout for the call.

In Figure 1, the socketTimeout is less than the totalTimeout, so the ACL waits until one of the following occurs: socketTimeout milliseconds have elapsed, the connection returns a timeout, or the call succeeds. If the call was unsuccessful in this period, the ACL checks the maxRetries setting to see if any retries remain. For reads, this value defaults to two, giving it a total of three tries – the initial attempt plus the two retries. For writes, this value defaults to zero, so no retries are attempted.

Figure 1 depicts a read with the default two retries, so once the original socketTimeout expires, the API call does not return to the application. Instead, the ACL waits for sleepBetweenRetries milliseconds and then submits the call again.

This second call may or may not go to the same server. This is controlled by the replica policy setting, which defaults to SEQUENCE. Sequence means that the first attempt is made to the node containing the master record, but on failure, the retry is made to a replica. This can be very useful in a read-sensitive use case, for example. If the socketTimeout is set to a short interval (say 20 milliseconds) and there are retries with a replica policy of SEQUENCE if the master node is busy and cannot respond in time or has fallen over. Still, if the cluster has not detected this yet, the retry will go to the replica and may return data within the application’s SLA.

If this second call again times out after socketTimeout milliseconds, a second delay of sleepBetweenRetry milliseconds is performed then the second retry (3rd total attempt) is performed.

Let’s take a look at where this leaves us in Figure 1:

The ACL is about to start the second retry, but in this case, the remaining time in the totalTimeout is less than the socketTimeout. Hence the API call will timeout before the socketTimeout elapses when the remaining time in the totalTimeout has elapsed.

Note that once the totalTimeout has elapsed, the client API call fails, even if further retries are available.

Timeout delay

The only timeout setting not covered in the above is the timeoutDelay setting. This is not strictly related to the API behavior but rather the system's efficiency. As the above shows, when the ACL submits the call to the server node, and no response is received within socketTimeout milliseconds, either the call is abandoned, or a retry is performed. But what happens to the connection on which that original request was placed?

By default, that connection is closed, destroying the connection and not placing it back in the connection pool. This is because it is possible that the server simply did not respond in time, and it might respond some time later. If the connection was reused instead of being closed, this server response might confuse the communication of the reusing call.

However, as discussed above, connection establishment can be expensive. Closing connections typically means that this connection must be re-established later, which can impact the application.

The timeoutDelay parameter specifies a time interval in milliseconds for those connections to be kept alive. If the server responds within that timeout, the socket is drained and then returned to the connection pool. If the timeout is exceeded, then the connection is closed.

Note that there is a cost of using the timeoutDelay as the ACL must keep track of which connections have timed out. It is useful in situations where timeouts may occur frequently (for example, aggressive read settings) and connection establishment is expensive.

Default policies

Now that the timeout settings are understood, we can discuss default policies. The ACL defines default values of the various policy attributes, which users can customize to suit their application instead of repeatedly modifying the defaults in the application code. Default policies are set up when the ACL is created and are used when null is passed to the policy parameter in the API call. Most often, the default policies are customized with the network settings discussed above.

Here is an example which creates a default writePolicy with a socketTimeout of 2,000 milliseconds:

WritePolicy writePolicyDefault = new WritePolicy();
writePolicyDefault.socketTimeout = 2000;

ClientPolicy clientPolicy = new ClientPolicy();
clientPolicy.writePolicyDefault = writePolicyDefault;
IAerospikeClient client = new AerospikeClient(clientPolicy, "127.0.0.1", 3000);

Note that the writePolicyDefault is set on the ClientPolicy instance before the ACL is created. When passed to the ACL constructor, a copy of the default policies is made so no changes are possible to the default policies.

After executing this code, passing null to a write API call will result in the ACL retrieving and using the defaultWritePolicy. Hence, the following two calls are identical:

client.put(null, key, new Bin("name", "joe"));
client.put(client.getWritePolicyDefault(), key, new Bin("name", "joe"));

Creating policies

Timeout settings are typically set on default policies and reused multiple times across the application. However, application-level settings are done as needed by the application on a per-call basis. Consider a check-and-set scenario where a record has been read from the database, changes made, and then those changes pushed back to the database, but only if the record has not been changed since it was read:

Record record = client.get(null, key);
...
WritePolicy writePolicy = client.getWritePolicyDefault();
writePolicy.generation = record.generation;
writePolicy.generationPolicy = GenerationPolicy.EXPECT_GEN_EQUAL;
client.put(writePolicy, key, new Bin("name", "joe"));

This code attempts to solve the requirements, getting the defaultWritePolicy from the ACL and then setting the generation and generationPolicy correctly. However, this code is very dangerous and will create issues in the application. There is just one defaultWritePolicy on the ACL, and the getWritePolicyDefault()call will return a reference to that object, not a copy of that object. Hence, setting the generation values on that policy will affect all calls that use that default policy on all threads, almost certainly causing many calls to fail with an exception.

Another common pattern to solve this problem is:

WritePolicy writePolicy = new WritePolicy();
writePolicy.generation = record.generation;
writePolicy.generationPolicy = GenerationPolicy.EXPECT_GEN_EQUAL;
client.put(writePolicy, key, new Bin("name", "joe"));

While this code is an improvement on the previous snippet, it too, has a problem. In this case, a unique WritePolicy instance has been created, preventing issues with this call affecting other calls. However, using new WritePolicy() does not pick up the settings of the default write policy, so settings changed on this default policy, such as the socketTimeout will not apply to this new policy.

The correct way to create a new policy to override fields is as follows:

WritePolicy writePolicy = new WritePolicy(client.getWritePolicyDefault());
writePolicy.generation = record.generation;
writePolicy.generationPolicy = GenerationPolicy.EXPECT_GEN_EQUAL;
client.put(writePolicy, key, new Bin("name", "joe"));

In this case, a new WritePolicy is created, but it gets a copy of all the settings off the default write policy. This object is local to this API call, so there is no contention with other calls.

Summary

Aerospike’s Policy classes are both powerful and flexible, allowing fine-grained, per-call control over both network and application-level settings. However, it is important to understand what the settings do to correctly control the API call. Network settings are often overlooked during the application development phase and are frequently poorly understood. A thorough understanding of them is critical to developing applications that perform well not just in the happy-day scenario but also in the face of network issues or server failures.

Developers who are new to Aerospike should understand this simple pattern to create new policies. It will ensure the correct execution of the API without affecting other concurrent calls and unintended surprises.