Assuming you have a good understanding of relay pagination and DynamoDB pagination, here's a rundown on how to get the two working in harmony π₯.
π Creating a resolver
For the majority of this section, it's fair to assume that we're working inside of a resolver such as the following.
const usersResolver = () => async (
root,
{ first, after, last, before },
) => {
// ...
};
Determining direction
Before querying the database, we first need to know which direction is being requested by the user.
const isForward = Boolean(first);
The easiest way to do this is to see if the provided first
argument has a value. If so, we're working with forward pagination.
Note: Bi-directional pagination within a single query is allowed but is not advised.
Querying the Database
For the query, most of the arguments passed through are going to be your bog-standard query; but there will be some additional attributes we need to pass through.
ScanIndexForward
needs to be passed a boolean dependent on the direction of the query (i.e. isForward
from the previous example).
ExclusiveStartKey
is going to be the client provided cursor (i.e. before
or after
arguments). The current SDK doesn't support the value null
so make sure to fall back to undefined
for cases where a cursor isn't present.
await documentClient
.query({
ScanIndexForward: isForward,
ExclusiveStartKey: before || after || undefined,
// The rest of your query
})
.promise();
Page sizing
A single query won't be enough to guarantee that the client provided page size is satisfied. In order to work around this, we're going to need to create a utility to iterate through one or more DynamoDB pages to populate our collection.
export const paginateQuery = <R>(client: DynamoDB.DocumentClient) => async <T = R>({
params,
pageSize,
acc = [],
}: {
params: DynamoDB.DocumentClient.QueryInput;
pageSize: number;
acc?: T[];
}): Promise<{ page: T[]; hasNextPage: boolean }> => {
const remaining = pageSize - acc.length;
const result = await client.query(params).promise();
const newItems = result.Items || [];
const newAcc = [...acc, ...(newItems.slice(0, remaining) as T[])];
// Query exhausted
if (!result.LastEvaluatedKey) {
return {
page: newAcc,
hasNextPage: newItems.length > remaining,
};
}
if (
// Page needs to be filled more
newAcc.length < pageSize ||
// page full but hasNextPage unknown
newItems.length <= remaining
) {
return paginateQuery(client)({
params: {
...params,
ExclusiveStartKey: result.LastEvaluatedKey,
},
pageSize,
acc: newAcc,
});
}
return {
page: newAcc,
hasNextPage: true,
};
};
Having pieced this together, the previous DynamoDB query can now instead call this utility and the requested page size is passed along.
const { page, hasNextPage } = await paginateQuery(documentClient)({
pageSize: first || last,
params: {
ScanIndexForward: isForward,
ExclusiveStartKey: before || after || undefined,
// The rest of your query
},
});
Constructing Edges
The response from DynamoDB equates to the nodes in our response Edges. A cursor is also required to be colocated with these nodes.
In this example, the query is on a table (rather than an index) so the keys required correspond to the partition key and sort key of the table.
For index queries, see the Cursor Construction section of the DynamoDB Pagination post.
const cursorKeys = ['id', 'dateOfBirth'] as const;
const edges = page.map((node) => ({
node,
cursor: cursorKeys.reduce((agg, key) => ({ ...agg, [key]: node[key] }), {}),
}));
Correcting edge order
While DynamoDB inverts sort-order when paginating backward, Relay does not. For this reason, the order of edges needs to be reversed if backward pagination is being used.
if (!isForward) {
edges.reverse();
}
Constructing PageInfo
The task is almost complete! The final part to this pagination saga is to put together the PageInfo response.
Cursors
Assuming the edges have already been ordered correctly (see above) the start and end cursors can easily be set by getting the cursor values of the first and last edges.
const pageInfo = {
startCursor: edges[0]?.cursor,
endCursor: edges[edges.length - 1]?.cursor,
// ...
};
Next pages
Assuming the client is stateful, there's little-to-no need to tell the client whether there is a page available in the opposite direction. For this purpose, we can default to false
for hasPreviousPage
and hasNextPage
for forward and backward pagination, respectively.
const pageInfo = {
// ...
...(isForward
? { hasNextPage, hasPreviousPage: false }
: { hasNextPage: false, hasPreviousPage: hasNextPage }),
};
Final result
Here's what our resolver looks like after putting all these parts together.
const usersResolver = () => async (root, { first, after, last, before }) => {
const isForward = Boolean(first);
const { page, hasNextPage } = await paginateQuery(documentClient)({
pageSize: first || last,
params: {
ScanIndexForward: isForward,
ExclusiveStartKey: before || after || undefined,
// ...
},
});
const cursorKeys = ["id", "dateOfBirth"] as const;
const edges = page.map((node) => ({
node,
cursor: cursorKeys.reduce((agg, key) => ({ ...agg, [key]: node[key] }), {}),
}));
if (!isForward) {
edges.reverse();
}
const pageInfo = {
startCursor: edges[0]?.cursor,
endCursor: edges[edges.length - 1]?.cursor,
...(isForward
? { hasNextPage, hasPreviousPage: false }
: { hasNextPage: false, hasPreviousPage: hasNextPage }),
};
return { edges, pageInfo };
};
π Conclusion
If you've gotten this far - congratulations! You are now a pagination expertβ’ and are ready to take this on in the real world π!
For the sake of keeping this concise, I've left out a few additional steps (namely optimising cursors and input validation). If you would like to see that follow up post, be sure to let me know π¬.
Thanks for reading!
If you enjoyed this post, be sure to react π¦ or drop a comment below with any thoughts π€.
You can also hit me up on twitterβ-β@andyrichardsonn
Disclaimer: All thoughts and opinions expressed in this article are my own.
Top comments (2)
Hey @andyrichardsonn ! First, congratulations, I really appreciate the series, the way you wrote it, and the GIFs :)
Cursor-based pagination with DynamoDB is a topic I have an interest in because I work with it every day. I'd like to know if you have plans to create an NPM package for this method. I've created the dynamodb-cursor-based-pagination and I'm going to use some ideas of your code, like the
exhaustQuery
and passingdocumentClient
as a parameter.Tell me if you have plans to transform this into an NPM package, maybe we can work together.
Hey Pedro, thanks for the comment!
Feel free to use all/any of the code examples π The project you have looks interesting and I'll definitely keep an eye on it!
I think one of the larger challenges with encapsulating pagination into a library is how much variance needs to be considered.
For example, there is some required domain/schema knowledge to be able to construct cursors, users might require (or want to avoid) calculating the
hasPreviousPage
value, and I'm sure there are a large number of additional edge cases (post-fetch filtering, etc).It looks like you've already made a tonne of headway so maybe this is/could be a solved problem for those who have less exotic needs and are happy working within the constraints of a library π