Neel Phadnis for Aerospike

Posted on Jul 20, 2022 • Edited on Aug 1, 2022 • Originally published at developer.aerospike.com

A Quick Orientation to Aerospike API

#aerospike #api #orientation #smartclient

Source: Photo by Jametlene Reskp on Unsplash

Aerospike Database and the client API provide a rich set of capabilities that have evolved over more than a decade through an increasing number of mission critical deployments. This post provides a high level view of the Aerospike architecture and API to give developers a broader understanding of its architecture and capabilities, and help them become more productive and effective. This post also points to resources for further exploration of specific areas.

The post is organized in the following sections:

Core Concepts: Describes the core architecture and data distribution concepts.
Functional Elements: Describes major elements of functionality in the API.
Key Specifics: Describes a few things that are useful to know up front.
Performance Features: Summarizes the common performance enhancing features.
Useful Libraries: Mentions libraries you should be aware of.

A caveat: Aerospike has client libraries for many languages. Not every aspect described here may apply to all client libraries precisely. While the discussion is broadly applicable to all client libraries, some details may be specific to the Java client library as it is most widely used.

Core Concepts

The key architecture concepts to understand include:

data organization in a cluster,
workings of the client library,
transaction support and replica consistency,
server-side execution of complex data type operations,
various processing modes, and
primary and secondary index queries.

Data Organization in Cluster

Aerospike is a distributed record-oriented database with support for the document-oriented data model. An Aerospike Database holds multiple namespaces, which are equivalent to databases in the relational model. A namespace holds records (rows), organized in sets (tables) and are accessed using a unique key that serves as the record id. A record can contain one or more bins (columns), and a bin can hold a value of different data types. Aerospike supports type-specific operations on Integer, String, List, Map, Geospatial, HyperLogLog, and Blob types. Sets and records do not conform to any schema. The primary index provides fast access to a record by key, and secondary indexes are supported for predicate based access.

Records are hashed by key across 4096 data partitions which are uniformly distributed across cluster nodes. Each data partition is replicated with a Replication Factor (RF) number of copies for fault tolerance.

Please find more details in the architecture overview section of the documentation.

Smart Client

Aerospike has client libraries or clients that implement the API in multiple languages including Java, C#, Python, Go, REST, Node.js, C, Ruby, and more. A client library simplifies application development by taking care of many complex aspects. It has the smarts to actively track and adapt to the latest cluster state and data distribution, almost working as an extension of the cluster. As such, it is referred to as the Smart Client. The Smart Client implements a common wire protocol for server interactions, directly connects to all nodes in the cluster, determines the specific server nodes for a data request, sends the request to them, and coordinates a response back to the application. It also handles timeouts, automatic retries, connection pooling, request throttling, replica selection, among other things.

Please refer to the Smart Client section and the supported clients in the documentation.

Data-Type Server

The API supports many complex data types including List, Map, Blob (or Binary), GeoJSON, and HyperLogLog (HLL). Since many complex data elements can get large in size, Aerospike eliminates expensive client-server data transfer by executing the operations entirely on the server.

Developers can leverage the following features to minimize client-server data transfers:

Complex operations supported in the API. To minimize data transfer, the API provides server-side execution of operations, finer control of the returned data, as well as the ability to customize multi-element processing logic for Collection Data Type (CDT) operations. Please refer to the CDT documentation and tutorials.
Expressions allow flexible logic to be computed on the server for filtering, retrieval, and updates. Please see the tutorial on Expressions.
Lua UDFs: Allow general logic to be computed on the server for retrieval and updates (described further below).

Transactions and Consistency

Aerospike guarantees all single-record requests to be atomic, that is, they either succeed or fail. Multi-op requests (described below) on a single record are transactional. However, multi-record batch and query operations (also described below) are not transactional and there is no rollback available for partially successful requests. The transactional boundary assured is for individual record operations within a multi-record request.

Aerospike replicates data for resiliency and performance in multiple replicas. Replicas are kept in sync by applying a synchronous write operation to all replicas. Read replicas are automatically selected based on the consistency requirements that are specified in the policy.

Please view the blog post Developers: Understanding Aerospike Transactions for details.

Processing Modes

Aerospike supports multiple modes to process a request, each with its trade-offs, and the application should choose the appropriate mode.

Synchronous: In the synchronous mode, the client waits for all responses to arrive from the server nodes before handing the cumulative response to the application. The application can spawn multiple threads and process multiple synchronous requests in parallel, one per thread at a time. From the application’s standpoint, synchronous requests can be easier to implement, but may not offer best resource utilization.
Asynchronous: In the asynchronous mode, the application can submit a request without waiting for the results. The results are processed when they are available in a different callback thread. Depending on the application’s choice, the client library can call back once for each record response, or just once with all responses. The asynchronous mode has superior efficiency and performance, but may be more complex to implement. Check out the tutorial on Asynchronous Operations for details.
Background: Common updates to a large number of records that are selected by a query can be performed in a background mode, where no results are returned from the server. The application can query whether a background request is in progress, completed successfully, or failed; and if necessary must determine the details of any failure separately.

Primary and Secondary Index Queries

Queries use either the Primary Index or a Secondary Index.

A primary-index query is simply a scan, performed on either a set or the entire namespace. A set index can optionally be created to boost performance, and is automatically used in the scan of that set. If a set index is not available, the entire namespace is scanned to determine the set records. A namespace scan uses the primary index.
A secondary-index query returns records meeting a predicate or condition supported by the corresponding secondary index: equality and range for Integer, equality for String, contains and is-contained-by for GeoJSON data type. An appropriate secondary index must have been created in order to execute a secondary-index query.

Note that filter expressions (described below) provide a powerful mechanism to select records for operations, and are broadly used with queries.

You can view query examples in the tutorial on Implementing SQL: Select.

Functional Elements

This section describes major functional elements including:

single- and multi-record function variants,
multi-op requests,
Collection Data Types (CDTs),
expressions, and
User Defined Functions (UDFs).

Note, these categories are not exclusive. For example, a multi-op request can involve CDTs.

Single and Multi-Record Function Variants

In addition to the execution modes, there are function variants based on the number of records involved.

Aerospike defines distinct API functions for single-record, batch, and query operations for simplicity instead of having a common API function with a generic operand that can take a variable number of records or a query predicate. Thus, there are separate functions for single-record and batch variations of operations like exists, get, put, append, and operate as well as their sync and async invocations.

A single-record request operates on a specified key or record.
Batch operations operate over multiple keys or records, where each key is specified. Please see the blog post on Batch Operations for details.
A query operates on multiple records that are identified by:
- a primary-index query (a scan of a set or the entire namespace) or a secondary-index query (a condition or predicate that uses an existing secondary index),
- a filter expression (which does not use or require a secondary index but is calculated on each record),
- both, or
- neither (in which case all records in the specified namespace and set are selected for the operation).

Query requests can be used for retrieval or update. The latter are executed in the background mode as mentioned earlier.

You can find examples of the function variants in the tutorials on SQL Operations.

Multi-Op Requests

Aerospike allows multiple bin operations to be performed on a record with a single multi-op Operate request that takes a list of Operation objects. The operations are performed in the sequence specified. The results are returned by the bins involved - either their final state or the individual operation results in the specified order. CRUD functions including CDT operations can be specified as Operations in the Operate request. Read/Write Expressions that allow server-side bin computations can also be used. Operate can be used for single-record, batch, and query requests in sync, async, and background (query updates only) modes.

You can find examples of multi-op requests in the documentation and tutorials, such as this tutorial on Introduction to Transactions.

Collection Data Types (CDTs)

Applications commonly use Collection Data Types (CDTs), namely List and Map, to store objects. CDTs are useful to model data for efficient storage, access, and transactional updates. Please see the blog post on Data Modeling for Speed At Scale (Part 2) for details.

Expressions

Aerospike Expressions are defined using bins and metadata, API functions, and various operators, and are evaluated on the server to filter records (filter expressions), return computed values (read expressions), and write to bins (write expressions).

A filter expression is a general mechanism usable in most operations. An operation is applied to a record only if the filter expression evaluates to true. A filter expression is specified in the policy parameter of the operation.

As a selection mechanism, a filter expression allows general conditions, whereas a secondary-index query offers superior performance. For best performance, use the most selective condition in a secondary-index query when possible.

Please find details in the workshop on Unleashing the Power of Expressions and the tutorial on Expressions in Aerospike.

User Defined Functions (UDFs)

User Defined Functions (UDFs) allow custom code to be executed on the server. A UDF is written in Lua, registered with the server, and invoked through a request from the client.
There are two distinct types of UDF: Record and Stream. A Record UDF performs a read and/or write operation on a single record, and can be invoked in sync, async, or background mode. A Stream UDF performs an aggregate computation over multiple records selected by a query. A Stream UDF typically also has a client execution phase that is handled by the client library.

Please be aware that UDFs may not be appropriate for performance sensitive use cases. For record-oriented functions, expressions should be the first choice whenever possible for best performance.

Please find details in UDF documentation and the tutorials on SQL Operations.

Key Specifics

Some API specifics are useful to know to avoid potential confusion:

the policy parameter,
data persistence and expiration,
write semantics, and
metadata operations.

Policy

Most API calls take a policy parameter which includes significant info that affects both the how and what of the request. Many specifics that define how an operation is performed (such as the timeout and retries) and what data is involved (such as the filter expression) are specified in the policy object. Many operation semantics details may not be obvious by just looking at the call parameters because they are specified in the policy. Examples include:

filter-expression: condition to select records for the operation,
send-key: store the user-key at record creation,
record-exists-action: different behavior of writes depending on whether the record exists or not,
generation-policy: used to isolate concurrent read-write transactions,
expiration: assign time-to-live duration to a record, and
durable-delete: used to prevent deleted records from reappearing after node failure.

Read more about policies in the Policies section of the documentation.

Data Persistence and Expiration

The server applies updates by default in memory, to be flushed to persistent storage at regular intervals. Updates can also be made immediately durable or persistent by using the commit-to-device option in the namespace configuration.

A record by default is created never to expire, but a different time-to-live (ttl) can be specified. An expired record is automatically removed and its space reclaimed, thus relieving the lifecycle management burden in applications for temporary objects.

Write Semantics

In Aerospike, a put or write combines Create (Insert), Update, and sometimes Delete. The default create-or-replace semantics of a write can be modified for alternative behaviors.

The record-exists-action specified within the write-policy defines the operation semantics when the record already exists, with create/update/update-only/replace/replace-only variants.
Map updates have a write-mode of “update”, “create-only”, and “update-only”` to control insertion behavior based on the existence of a key.
A bin is removed from a record by writing.a NULL value to it. When the last bin is removed, the record is automatically deleted.

Metadata Operations

Not all metadata operations are available through the API in the client libraries.

Namespace: A namespace is added through the config and requires a server restart. The truncate API removes all records in a set or the entire namespace.
Set: A set is automatically created when the first record is inserted in the set. Records in a set can be truncated using the truncate API.
Index: The API supports creation and deletion of a set index and secondary index. A secondary index is defined on a bin or an element in List or Map for a specific value type.

Please view the tutorial SQL: Updates for examples.

Performance Features

Aerospike is purpose-built to deliver high performance for large data with small cluster size. All features are designed and tradeoffs are made with this overarching goal. Data modeling is key to achieving best performance. Please see the blog post Data Modeling for Speed At Scale.

Each application is different, and Aerospike provides flexibility to accommodate custom performance considerations. At the same time, a developer should be aware of the following performance features (described earlier) that are commonly used:

Collection Data Types (CDTs)
Multi-op requests
Batch requests
Expressions
Secondary indexes
Set indexes
Complex Data Types - Binary, HLL, GeoJSON
User Defined Functions (UDFs)

Useful Libraries

Here are some useful libraries.

Document API

The Aerospike Document API provides CRUD operations at arbitrary points within a JSON document. It allows a JSONPath argument to specify parts of the document simply and expressively to apply these methods. Check out the tutorial for the Document API.

Java Object Mapper

The object mapper library uses Java annotations to define the Aerospike semantics for the saving and loading behavior. Annotations are specified next to the definitions of a class, methods, and fields. The object mapper makes managing persistent data easier to implement, easier to understand, and less error prone. Check out the Java Object Mapper workshop and tutorial.

Summary

Aerospike client API provides a rich set of capabilities. This post provides a high level view of the Aerospike architecture and API to give developers a broader understanding of its architecture and capabilities, and help them become more productive and effective. It provides an orientation of the client API by describing the core concepts, functional elements, key specifics, common performance features, and useful libraries; and points out resources for further exploration of specific topics.

DEV Community