GraphQL pagination with DynamoDB - DynamoDB pagination

Just like the aforementioned GraphQL pagination, DynamoDB also uses cursor based pagination.

That being said, there are distinct differences that need to be taken into account 🔔.

🏎️ Traversal

For any given query to DynamoDB, the following takes place:

1. In-order traversal takes place on the target table (order specified by the sort key)
2. The provided key condition expression is evaluated to find matching documents
3. Optionally provided filter expressions are used to additionally constrain the matching documents

Inverting traversal order

Traversal order can be inverted using the ScanIndexForward attribute.

This (logically) inverted collection is traversed in reverse order, and the resulting items are also returned in reverse order.

📚 Paging

As matches are found, DynamoDB adds them to a result set - a "page". If a Limit is provided, DynamoDB will suspend traversal when the number of matches for the given key condition expression reaches the limit.

Unfortunately, the limit is applied prior to the filter expression evaluation; meaning a result set will never exceed the limit value, but can have a size smaller than the limit, while also having subsequent pages to follow.

🔑 Cursors

Upon the return of a page, assuming the collection hasn't been exhausted, DynamoDB provides a cursor in the form of a LastEvaluatedKey.

Providing this value in subsequent queries via the ExclusiveStartKey allows DynamoDB to continue where it left off.

Exhausting a query

In a similar fashion, this pattern can be used to retrieve all items in a query.

const exhaustQuery = async <T>(
  query: AWS.DynamoDB.DocumentClient.QueryInput,
  agg: T[] = [],
): Promise<T[]> => {
  const res = await dynamodb.query(query).promise();
  const newAgg = [...agg, ...(res.Items || [])];

  if (!res.LastEvaluatedKey || !res.Items) {
    return newAgg;
  }

  return exhaustQuery(
    {
      ...query,
      ExclusiveStartKey: res.LastEvaluatedKey,
    },
    newAgg,
  );
};

Cursor construction

DynamoDB responses only provide a cursor for the position of the last evaluated element in the collection. There doesn't look to be official documentation on how cursors are constructed, but I've found the following thanks to some experimentation.

It looks like DynamoDB needs two things to be able to continue where it left off:

• The unique identity of the last visited item (i.e. primary key)
• The position in the index where said element exists (i.e. primary key of index/table)

In the case of querying a table, the primary key typically consists of a partition (hash) and optionally a sort (range) key.

{
  productId: 1,  // partition key (pk)
  type: 'Book'   // sort key (sk)
}

For querying an index, the same rule applies. We'll still need the attributes required to uniquely identify the element (primary key), but we also need the partition and (optional) sort attributes to get back to the previous position in the query.

{
  productId: 1,          // pk
  type: 'Book',          // sk, index pk
  createdAt: 1618496921, // index sk
}

Thanks for reading!

If you enjoyed this post, be sure to react 🦄 or drop a comment below with any thoughts 🤔.

You can also hit me up on twitter - @andyrichardsonn

Disclaimer: All thoughts and opinions expressed in this article are my own.