At Theodo, we are big fans of Jeremy Dalyβs DynamoDB-Toolbox. We started using it as early as 2019 and grew fond of it... but were also well aware of its flaws π
One of them was that it had originally been coded in JavaScript. Although Jeremy rewrote the source code in TypeScript in 2020, it didn't handle type inference, a feature that I eventually came to implement myself in the v0.4.
However, there were still some features that we felt lacked: From declaring enums
on primitives, to supporting recursive schemas and types (lists and maps sub-attributes) and polymorphism.
I was also wary of the object-oriented approach: I donβt have anything against classes, but they are not tree-shakable. Meaning that they should be kept relatively light in a serverless context. Thatβs what AWS went for with the v3 of their SDK, and for good reasons: Keep bundles tight!
That just wasn't the case for DynamoDB-Toolbox: I remember working on an .update
method that was more than 1000 lines long... But why bundle it when you don't even need it?
So last year, I decided to throw myself into a complete overhaul of the code, with three main objectives:
- Support the v3 of the AWS SDK (although support has been added in the v0.8)
- Get the API and type inference on par with those of more "modern" tools like zod and electrodb
- Use a more functional and tree-shakable approach
Today, I am happy to announce the v1 beta of dynamodb-toolbox is out πΒ It includes reworked Table
and Entity
classes, as well as complete support for PutItem
, GetItem
and DeleteItem
commands (including conditions and projections), with UpdateItem
, Query
and Scan
commands soon to follow.
This article details how the new API works and the main breaking changes from previous versions - which, by the way, only concern the API: No data migration needed π₯³
Let's dive in!
Table of content
- Installation
- Tables
- Entities
- Designing Entity schemas
- Computed defaults
- Commands
- Utility helpers and types
- Errors
- Conclusion
Installation
### npm
npm i dynamodb-toolbox@1.0.0-beta.0
## yarn
yarn add dynamodb-toolbox@1.0.0-beta.0
## ...and so on
βοΈ Stay up to date with the patches by following the project GitHub releases
The v1
is built on top the v3
of the AWS SDK. It has @aws-sdk/client-dynamodb
and @aws-sdk/lib-dynamodb
as peer dependencies so youβll have to install them as well:
## npm
npm i @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb
## yarn
yarn add @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb
## ...and so on
Tables
Tables are defined pretty much the same way as in previous versions, but the key
attributes now have a type
along with their name
:
import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient } from '@aws-sdk/lib-dynamodb';
// Will be renamed Table in the official release π
import { TableV2 } from 'dynamodb-toolbox';
const dynamoDBClient = new DynamoDBClient({});
const documentClient = DynamoDBDocumentClient.from(dynamoDBClient);
const myTable = new TableV2({
name: 'MySuperTable',
partitionKey: {
name: 'PK',
type: 'string', // 'string' | 'number' | 'binary'
},
sortKey: {
name: 'SK',
type: 'string',
},
documentClient,
});
βοΈ The v1 does not support indexes yet as queries are not yet available.
As in previous versions, the v1
classes tag your data with an entity identifier through an internal entity
string attribute, saved as "_et"
by default. This can be renamed at the Table
level through the entityAttributeSavedAs
argument:
const myTable = new TableV2({
...
// π defaults to "_et"
entityAttributeSavedAs: '__entity__',
});
Entities
For Entities, the main change is that the attributes
argument becomes schema
:
// Will be renamed Entity in the official release π
import { EntityV2, schema } from 'dynamodb-toolbox';
const myEntity = new EntityV2({
name: 'MyEntity',
table: myTable,
// Attributes definition
schema: schema({ ... }),
});
Timestamps
The internal timestamp attributes are also there and behave similarly as in the previous versions. You can set the timestamps
to false
to disable them (default value is true
), or fine-tune the created
and modified
attributes names:
const myEntity = new EntityV2({
...
// π de-activate timestamps altogether
timestamps: false,
});
const myEntity = new EntityV2({
...
timestamps: {
// π de-activate only `created` attribute
created: false,
modified: true,
},
});
const myEntity = new EntityV2({
...
timestamps: {
created: {
// π defaults to "created"
name: 'creationDate',
// π defaults to "_ct"
savedAs: '__createdAt__',
},
modified: {
// π defaults to "modified"
name: 'lastModificationDate',
// π defaults to "_md"
savedAs: '__lastMod__',
},
},
});
Matching the Table schema
An important change from previous versions is that the EntityV2
key attributes are validated against the TableV2
schema, both through types and at runtime. There are two ways to match the table schema:
- The simplest one is to have an entity schema that already matches the table schema (see "Designing Entity schemas"). The Entity is then considered valid and no other argument is required:
import { string } from 'dynamodb-toolbox';
const pokemonEntity = new EntityV2({
name: 'Pokemon',
table: myTable, // <= { PK: string, SK: string } primary key
schema: schema({
// Provide a schema that matches the primary key
PK: string().key(),
// π using "savedAs" will also work
pokemonId: string().key().savedAs('SK'),
...
}),
});
- If the entity key attributes don't match the table schema, the
Entity
class will require you to add acomputeKey
property which must derive the primary key from them:
const pokemonEntity = new EntityV2({
...
table: myTable, // <= { PK: string, SK: string } primary key
schema: schema({
pokemonClass: string().key(),
pokemonId: string().key(),
...
}),
// π `computeKey` is correctly typed
computeKey: ({ pokemonClass, pokemonId }) => ({
PK: pokemonClass,
SK: pokemonId,
}),
});
SavedItem and FormattedItem
If you feel lost, you can always use the SavedItem
and FormattedItem
utility types to infer the type of your entity items:
import type { FormattedItem, SavedItem } from 'dynamodb-toolbox';
const pokemonEntity = new EntityV2({
name: 'Pokemon',
timestamps: true,
table: myTable,
schema: schema({
pokemonClass: string().key().savedAs('PK'),
pokemonId: string().key().savedAs('SK'),
level: number().default(1),
customName: string().optional(),
internalField: string().hidden(),
}),
});
// What Pokemons will look like in DynamoDB
type SavedPokemon = SavedItem<typeof pokemonEntity>;
// π Equivalent to:
// {
// _et: "Pokemon",
// _ct: string,
// _md: string,
// PK: string,
// SK: string,
// level: number,
// customName?: string | undefined,
// internalField: string | undefined,
// }
// What fetched Pokemons will look like in your code
type FormattedPokemon = FormattedItem<typeof pokemonEntity>;
// π Equivalent to:
// {
// created: string,
// modified: string,
// pokemonClass: string,
// pokemonId: string,
// level: number,
// customName?: string | undefined,
// }
Designing Entity schemas
Now letβs dive into the part that received the most significant overhaul: Schema definition.
Schema definition
Similarly to zod or yup, attributes are now defined through function builders. For TS users, this removes the need for the as const
statement previously needed for type inference (so don't forget to remove it when you migrate π).
You can either import the attribute builders through their dedicated imports, or through the attribute
or attr
shorthands. For instance, those declarations will output the same attribute schema:
import { string, attribute, attr } from 'dynamodb-toolbox';
// π More tree-shakable
const pokemonName = string();
// π Not tree-shakable, but single import
const pokemonName = attribute.string();
const pokemonName = attr.string();
Prior to being wrapped in a schema
declaration, attributes are called warm: They are not validated (at run-time) and can be used to build other schemas. By inspecting their types, you will see that they are prefixed with $
. Once frozen, validation is applied and building methods are stripped:
The main takeaway is that warm schemas can be composed while frozen schemas cannot:
import { schema } from 'dynamodb-toolbox';
const pokemonName = string();
const pokemonSchema = schema({
// π No problem
pokemonName,
...
});
const pokedexSchema = schema({
// β Not possible
pokemon: pokemonSchema,
...
});
You can create/update warm attributes by using dedicated methods or by providing option objects. The former provides a slick devX with autocomplete and shorthands, while the latter theoretically requires less compute time and memory usage, although it should be very minor (validation being only applied on freeze):
// Using methods
const pokemonName = string().required('always');
// Using options
const pokemonName = string({ required: 'always' });
All attributes share the following options:
-
required
(string?="atLeastOnce") Tag a root attribute or Map sub-attribute as required. Possible values are:-
"atLeastOnce"
Required inPutItem
commands -
"never"
: Optional in all commands -
"always"
: Required inPutItem
,GetItem
andDeleteItem
commands
-
// Equivalent
const pokemonName = string().required();
const pokemonName = string({ required: 'atLeastOnce' });
// `.optional()` is a shorthand for `.required(βneverβ)`
const pokemonName = string().optional();
const pokemonName = string({ required: 'never' });
A very important breaking change from previous versions is that root attributes and Map sub-attributes are now required by default. This was made so composition and validation work better together.
π‘ Outside of root attributes and Map sub-attributes, such as in a list of strings, it doesnβt make sense for sub-schemas to be optional. So, should I force users to write list(string().required())
every time OR make string validation and type inference aware of their context (ignore required
in lists but not in maps)? It felt more elegant to enforce string()
as required by default and prevent schemas such as list(string().optional())
.
-
hidden
(boolean?=true) Skip attribute when formatting the returned item of a command:
const pokemonName = string().hidden();
const pokemonName = string({ hidden: true });
-
key
(boolean?=true) Tag attribute as needed to compute the primary key:
// Note: The method will also modify the `required` property to "always"
// (it is often the case in practice, you can still use `.optional()` if needed)
const pokemonName = string().key();
const pokemonName = string({ key: true });
-
savedAs
(string) Previously known asmap
. Rename a root or Map sub-attribute before sending commands:
const pokemonName = string().savedAs('_n');
const pokemonName = string({ savedAs: '_n' });
-
default
: (ComputedDefault) See Computed defaults
Attributes types
Hereβs the exhaustive list of available attribute types:
Any
Define an attribute of any value. No validation will be applied at runtime, and its type will be resolved as unknown
:
import { any } from 'dynamodb-toolbox';
const pokemonSchema = schema({
...
metadata: any(),
});
type FormattedPokemon = FormattedItem<typeof pokemonEntity>;
// => {
// ...
// metadata: unknown
// }
You can provide default values through the default
option or method:
const metadata = any().default({ any: 'value' });
const metadata = any({
default: () => 'Getters also work!',
});
Primitives
Defines a string
, number
, boolean
or binary
attribute:
import { string, number, boolean, binary } from 'dynamodb-toolbox';
const pokemonSchema = schema({
...
pokemonType: string(),
level: number(),
isLegendary: boolean(),
binEncoded: binary(),
});
type FormattedPokemon = FormattedItem<typeof pokemonEntity>;
// => {
// ...
// pokemonType: string
// level: number
// isLegendary: boolean
// binEncoded: Buffer
// }
You can provide default values through the default
option or method:
// π Correctly typed!
const level = number().default(42);
const date = string().default(() => new Date().toISOString());
const level = number({ default: 42 });
const date = string({
default: () => new Date().toISOString(),
});
Primitive types have an additional enum
option. For instance, you could provide a finite list of pokemon types:
const pokemonTypeAttribute = string().enum('fire', 'grass', 'water');
// Shorthand for `.enum("POKEMON").default("POKEMON")`
const pokemonPartitionKey = string().const('POKEMON');
π‘ For type inference reasons, the enum
option is only available as a method, not as an object option
Set
Defines a set of strings, numbers or binaries. Unlike in previous versions, sets are kept as Set
classes. Let me know if you would prefer using arrays (or being able to choose from both):
import { set } from 'dynamodb-toolbox';
const pokemonSchema = schema({
...
skills: set(string()),
});
type FormattedPokemon = FormattedItem<typeof pokemonEntity>;
// => {
// ...
// skills: Set<string>
// }
Options can be provided as a 2nd argument:
const setAttr = set(string()).hidden();
const setAttr = set(string(), { hidden: true });
List
Defines a list of sub-schemas of any type:
import { list } from 'dynamodb-toolbox';
const pokemonSchema = schema({
...
skills: list(string()),
});
type FormattedPokemon = FormattedItem<typeof pokemonEntity>;
// => {
// ...
// skills: string[]
// }
As in sets, options can be povided as a 2nd argument.
Map
Defines a finite list of key-value pairs. Keys must follow a string schema, while values can be sub-schema of any type:
import { map } from 'dynamodb-toolbox';
const pokemonSchema = schema({
...
nestedMagic: map({
will: map({
work: string().const('!'),
}),
}),
});
type FormattedPokemon = FormattedItem<typeof pokemonEntity>;
// => {
// ...
// nestedMagic: {
// will: {
// work: "!"
// }
// }
// }
As in sets and lists, options can be povided as a 2nd argument.
Record
A new attribute type that translates to Partial<Record<KeyType, ValueType>>
in TypeScript. Records differ from maps as they can accept an infinite range of keys:
import { record } from 'dynamodb-toolbox';
const pokemonType = string().enum(...);
const pokemonSchema = schema({
...
weaknessesByPokemonType: record(pokemonType, number()),
});
type FormattedPokemon = FormattedItem<typeof pokemonEntity>;
// => {
// ...
// weaknessesByPokemonType: {
// [key in PokemonType]?: number
// }
// }
Options can be provided as a 3rd argument:
const recordAttr = record(string(), number()).hidden();
const recordAttr = record(string(), number(), { hidden: true });
AnyOf
A new meta-attribute type that represents a union of types, i.e. a range of possible types:
import { anyOf } from 'dynamodb-toolbox';
const pokemonSchema = schema({
...
pokemonType: anyOf([
string().const('fire'),
string().const('grass'),
string().const('water'),
]),
});
In this particular case, an enum
would have done the trick. However, anyOf
becomes particularly powerful when used in conjunction with a map
and the enum
or const
directives of a primitive attribute, to implement polymorphism:
const pokemonSchema = schema({
...
captureState: anyOf([
map({
status: string().const('caught'),
// π captureState.trainerId exists if status is "caught"...
trainerId: string(),
}),
// ...but not otherwise! π
map({ status: string().const('wild') }),
]),
});
type CaptureState = FormattedItem<typeof pokemonEntity>['captureState'];
// π Equivalent to:
// | { status: "wild" }
// | { status: "caught", trainerId: string }
As in sets, lists and maps, options can be povided as a 2nd argument.
Looking forward
Thatβs all for now! Iβm planning on including new tuple
and allOf
attributes at some point.
If there are other types youβd like to see, feel free to leave a comment on this article and/or open a discussion on the official repo with the v1
label π
Computed defaults
In previous versions, default
was used to compute attribute from other attributes values. This feature was very handy for "technical" attributes such as composite indexes.
However, it was just impossible to type correctly in TypeScript:
const pokemonSchema = schema({
...
level: number(),
levelPlusOne: number().default(
// β No way to retrieve the caller context
input => input.level + 1,
),
});
It means the input
was typed as any and it fell to the developper to type it correctly, which just didnβt cut it for me.
The solution I committed to was to split computed defaults declaration into 2 steps:
- First, declare that an attribute default should be derived from other attributes:
import { ComputedDefault } from 'dynamodb-toolbox';
const pokemonSchema = schema({
...
level: number(),
levelPlusOne: number().default(ComputedDefault),
});
π‘ ComputedDefault
is a JavaScript Symbol (TLDR: A sort of unique and custom null
), so it cannot possibly conflict with an actual desired default value.
- Then, declare a way to compute this attribute at the entity level, through the
computeDefaults
property:
const pokemonEntity = new EntityV2({
...
schema: pokemonSchema,
computeDefaults: {
// π Correctly typed!
levelPlusOne: ({ level }) => level + 1,
},
});
In the tricky case of nested attributes, computeDefaults
becomes an object with an _attributes
or _elements
property to emphasize that the computing is local:
const pokemonSchema = schema({
...
defaultLevel: number(),
// π Defaulted Map attribute
levelHistory: map({
currentLevel: number(),
// π Defaulted sub-attribute
nextLevel: number().default(ComputedDefault),
}).default(ComputedDefault),
});
const pokemonEntity = new EntityV2({
...
schema: pokemonSchema,
computeDefaults: {
levelHistory: {
// Defaulted value of Map attribute
_map: item => ({
currentLevel: item.defaultLevel,
nextLevel: item.defaultLevel,
}),
_attributes: {
// Defaulted value of sub-attribute
nextLevel: (levelHistory, item) => levelHistory.currentLevel + 1,
},
},
},
});
Note that there is (and has always been) an ambiguity as to when default
values are actually used, that I hope to solve soon by splitting it into getDefault
, putDefault
, updateDefault
and so on (default
being the one to rule them all). For the moment, defaults
are only used in putItem
commands.
Commands
Now that we know how to design entities, letβs take a look at how we can leverage them to craft commands π
π‘ The beta only supports the PutItem
, GetItem
, and DeleteItem
commands. If you need to run UpdateItem
, Query
or Scan
commands, our advice is to run native SDK commands and format their output with the formatSavedItem
util.
As mentioned in the intro, I searched for a syntax that favored tree-shaking. Here's an example of it, with the PutItem
command:
// v0.x Not tree-shakable
const response = await pokemonEntity.putItem(pokemonItem, options);
// v1 Tree-shakable π
import { PutItemCommand } from 'dynamodb-toolbox';
const command = new PutItemCommand(
pokemonEntity,
// π Correctly typed!
pokemonItem,
// π Optional
putItemOptions,
);
// Get command params
const params = command.params();
// Send command
const response = await command.send();
pokemonItem
can be provided later or edited, which can be useful if the command is built in several steps (at execution, an error will be thrown if no item has been provided):
import { PutItemCommand } from 'dynamodb-toolbox';
const incompleteCommand = new PutItemCommand(pokemonEntity);
// (will return a new command and not mutate the original one)
const completeCommand = incompleteCommand.item(pokemonItem);
// (can be chained by design)
const response = await incompleteCommand
.item(pokemonItem)
.options(options)
.send();
You can also use the .build
method of the entity to craft a command directly hydrated with your entity:
// π We get a syntax closer to v0.x... but tree-shakable!
const response = await pokemonEntity
.build(PutItemCommand)
.item(pokemonItem)
.options(options)
.send();
π‘ As much as I appreciate this syntax, it makes mocking hard in unit tests. I'm already working on a mockEntity
helper, inspired by the awesome aws-sdk-client-mock
. This will probably make another article soon.
PutItemCommand
The capacity
, metrics
and returnValues
options behave exactly the same as in previous versions. The condition
option benefits from improved typing, and clearer logical combinations:
import { PutItemCommand } from 'dynamodb-toolbox';
const { Attributes } = await pokemonEntity
.build(PutItemCommand)
.item(pokemonItem)
.options({
capacity: 'TOTAL',
metrics: 'SIZE',
// π Will type the response `Attributes`
returnValues: 'ALL_OLD',
condition: {
or: [
{ attr: 'pokemonId', exists: false },
// π "lte" is correcly typed
{ attr: 'level', lte: 99 },
// π You can nest logical combinations
{ and: [{ not: { ... } }, ...] },
],
},
})
.send();
βοΈThe "UPDATED_OLD"
and "UPDATED_NEW"
return values options are not fully supported yet so I do not recommend using them for now
GetItemCommand
The attributes
option behaves the same as in previous versions, but benefits from improved typing as well:
import { GetItemCommand } from 'dynamodb-toolbox';
const { Item } = await pokemonEntity
.build(GetItemCommand)
.key(pokemonKey)
.options({
capacity: 'TOTAL',
consistent: true,
// π Will type the response `Item`
attributes: ['pokemonId', 'pokemonType', 'level'],
})
.send();
DeleteItemCommand
The DeleteItem
command is pretty much a mix between the two previous ones, options wise:
import { DeleteItemCommand } from 'dynamodb-toolbox';
const { Attributes } = await pokemonEntity
.build(DeleteItemCommand)
.key(pokemonKey)
.options({
capacity: 'TOTAL',
metrics: 'SIZE',
// π Will type the response `Attributes`
returnValues: 'ALL_OLD',
condition: {
or: [
{ attr: 'level', lte: 99 },
...
],
},
})
.send();
Utility helpers and types
In addition to the SavedItem
and FormattedItem
types, the v1
exposes a bunch of useful helpers and utility types:
formatSavedItem
formatSavedItem
transforms a saved item returned by the DynamoDB client to itβs formatted counterpart:
import { formatSavedItem } from 'dynamodb-toolbox';
// π Typed as FormattedItem<typeof pokemonEntity>
const formattedPokemon = formatSavedItem(
pokemonEntity,
savedPokemon,
// As in GetItem commands, attributes will filter the formatted item
{ attributes: [...] },
);
Note that it is a parsing operation, i.e. it does not require the item to be typed as SavedItem<typeof myEntity>
, but will throw an error if the saved item is invalid:
const formattedPokemon = formatSavedItem(pokemonEntity, {
...
level: 'not a number',
});
// β Will raise error:
// => "Invalid attribute in saved item: level. Should be a number"
Condition and parseCondition
The Condition
type and parseCondition
util are useful to type conditions and build condition expressions:
import { Condition, parseCondition } from 'dynamodb-toolbox';
const condition: Condition<typeof pokemonEntity> = {
attr: 'level',
lte: 42,
};
const parsedCondition = parseCondition(pokemonEntity, condition);
// => {
// ConditionExpression: "#1 <= :1",
// ExpressionAttributeNames: { "#1": "level" },
// ExpressionAttributeValues: { ":1": 42 },
// }
Projection and parseProjection
The AnyAttributePath
type and parseProjection
util are useful to type attribute paths and build projection expressions:
import { AnyAttributePath, parseProjection } from 'dynamodb-toolbox';
const attributes: AnyAttributePath<typeof pokemonEntity>[] = [
'pokemonType',
'levelHistory.currentLevel',
];
const parsedProjection = parseProjection(pokemonEntity, attributes);
// => {
// ProjectionExpression: '#1, #2.#3',
// ExpressionAttributeNames: {
// '#1': 'pokemonType',
// '#2': 'levelHistory',
// '#3': 'currentLevel',
// },
// }
KeyInput and PrimaryKey
Both types are useful to type item primary keys:
import type { KeyInput, PrimaryKey } from 'dynamodb-toolbox';
type PokemonKeyInput = KeyInput<typeof pokemonEntity>;
// => { pokemonClass: string, pokemonId: string }
type MyTablePrimaryKey = PrimaryKey<typeof myTable>;
// => { PK: string, SK: string }
Errors
Finally, letβs take a quick look at error management. When DynamoDB-Toolbox encounters an unexpected input, it will throw an instance of DynamoDBToolboxError
, which itself extends the native Error
class with a code
property:
await pokemonEntity
.build(PutItemCommand)
.item({ ..., level: 'not a number' })
.send();
// β [parsing.invalidAttributeInput] Attribute level should be a number
Some DynamoDBToolboxErrors
also expose a path
property (mostly in validations) and/or a payload
property for additional context. If you need to handle them, TypeScript is your best friend, as the code
property will correctly discriminate the DynamoDBToolboxError
type:
import { DynamoDBToolboxError } from 'dynamodb-toolbox';
const handleError = (error: Error) => {
if (!error instanceof DynamoDBToolboxError) throw error;
switch (error.code) {
case 'parsing.invalidAttributeInput':
const path = error.path;
// => "level"
const payload = error.payload;
// => { received: "not a number", expected: "number" }
break;
...
case 'entity.invalidItemSchema':
const path = error.path; // β error does not have path property
const payload = error.payload; // β same goes with payload
...
}
};
Conclusion
And thatβs it for now! I hope youβre as excited as I am about this new release π
If you have features that I've missed in mind, or would like to see some of the ones I mentioned prioritized, please leave a comment on this article and/or create an issue or open a discussion on the official repo with the v1
label π
See you soon!
Top comments (2)
Thank you for this article. I am real fan of DynamoDB-Toolbox this article will help me to immigrate to the latest version.
Thanks for putting together the information here and updating the framework. Love DynamoDB toolbox!
I have updated this boilerplate which creates the DynamoDB tables using Terraform and handles migrations using Umzug:
github.com/goldstack/dynamodb-boil...
If there is anything that could be worthwhile to add or improve in the builderplate or documentation, would be very happy to know.