Welcome to this part-4 of the key-value database tutorial for codehooks.io serverless Node.js backend. In this part we'll focus on how to work with multiple key-value pairs in a key-value database.
The Codehooks.io key-value database stores all keys as a sorted string index. This is a very useful feature if you need to store and retrieve multiple values together, as a group or as a segment of key-values.
This article is also avaliable at the official codehooks.io website
IoT device - time series data example
To illustrate this feature we'll create an example use-case for capturing data from multiple IoT devices. In this example, each IoT device sends its location, identification and observations to a specific REST API network endpoint.
For example, a temperature sensor device in the "Alpha region West" reports its value each minute. And a wind force sensor in the "Beta region West" reports its values hourly, and so on.
To ensure that we can access and process the observation data sets effectively, we'll create a key string combination that captures all necessary information in one unique key-value pair.
To illustrate this concept lets imagine a series of IoT device observations as shown in the example below.
'/Alpha/West/density/2023-01-09T10:14:00.495Z' => '4.4191'
'/Alpha/West/humidity/2023-01-09T17:01:00.821Z' => '0.7539'
'/Alpha/West/temp/2023-01-09T21:15:00.450Z' => '87.100'
'/Beta/West/wind/2023-01-19T21:57:00.316Z' => '5.9477'
...
Now that we understand the concept for our IoT database, lets begin the development of an example backend API for IoT sensors.
We'll create a serverless REST API endpoint to receive observations from many IoT devices. The API will store each observation with a logic key that effectively creates a time series database with data points that we can explore and use in our application.
Lets get started with creating the first API to receive data observations from a device.
REST API endpoint to POST a single observation
The REST API route takes in 3 parameters for a given device, region, location and device.
And the POST body must contain the actual device observation data point.
For example:
POST /observation/North/Plaza/humid03
{"observation": "4.02"}
The route parameters are combined into a unique key with a time stamp which gives a correct time series for the data points. The following table shows some example device observations.
| IoT devices time series | Data points |
|---|---|
| /East/BuilingA/temp01/2023-02-22T18:30:14.441Z | 21.4 |
| /North/Plaza01/humid3/2023-02-23T05:23:11.306Z | 12.5 |
| /East/BuilingB/temp02/2023-02-22T18:32:38.991Z | 19.0 |
The serverless function below implements a REST API endpoint route that can receive data from a device and insert it to the key-value database.
app.post('/observation/:region/:location/:device', async (req, res) => {
const {region, location, device} = req.params;
const {observation} = req.body;
const key = `/${region}/${location}/${device}/${new Date().toISOString()}`;
// string key e.g. '/North/Plaza/humid03/2023-02-23T05:23:11.306Z'
const conn = await Datastore.open();
const result = await conn.set(key, observation);
console.log(result)
res.status(201).end('Data received ok');
})
Test the REST API with curl.
curl -X POST \
'https://<YOUR_DATABASE_ID>.api.codehooks.io/dev/observation/East/BuilingA/temp01' \
--header 'x-apikey: <YOUR_API_TOKEN>' \
--header 'Content-Type: application/json' \
--data-raw '{
"observation": "21.4"
}'
Use the CLI command
coho info --examplesto inspect your project API endpoint and Curl examples.
Inspecting the key-value database with the CLI
The Codehooks CLI can be used to manage the key-value database basic operations, get, set and del. For example, to retrieve all observations for the East locations you can run the following command:
coho get '/East*'
This will list all matching keys in the key-value database as the example output shows below.
{
key: '/East/BuilingA/temp01/2023-02-23T05:48:19.646Z',
value: '21.4'
}
{
key: '/East/BuilingA/temp01/2023-02-23T05:49:06.614Z',
value: '20.0'
}
2 keys
Endpoint to GET observations subkey
Similiar to the CLI example above, we can use the database API getAll method to retrieve all key-value pairs that match a start segment of a particular key. In this example we are producing CSV data format for easy spreadsheet analysis etc.
app.get('/observations/:prefix', async (req, res) => {
const conn = await Datastore.open();
const {prefix} = req.params;
let count = 0;
res.set('content-type', 'text/csv');
res.write('Device, Data\n');
const stream = conn.getAll(`/${prefix}`);
stream.on('data', (data) => {
const outString = `"${data.key}", ${data.val}\n`;
count++
res.write(outString);
}).on('end', () => {
res.end();
})
})
Testing the API with curl again.
curl -X GET \
'https://<YOUR_DATABASE_ID>.api.codehooks.io/dev/observations/Alpha' \
--header 'x-apikey: <YOUR_API_TOKEN>' \
Device, Data
"/Alpha/Center/decibel/2023-01-09T11:11:31.218Z", 2.3428
"/Alpha/Center/decibel/2023-01-09T11:12:01.050Z", 6.0632
"/Alpha/Center/decibel/2023-01-09T13:13:30.541Z", 0.7196
"/Alpha/Center/decibel/2023-01-09T15:23:00.589Z", 9.7232
"/Alpha/Center/decibel/2023-01-09T15:34:00.520Z", 5.0089
"/Alpha/Center/decibel/2023-01-09T17:04:00.942Z", 9.1861
Complete source code
The complete source code for our IoT backend application is shown below.
// index.js
import { app, Datastore } from 'codehooks-js' // Standard JS lib for express style code
// list a serie of IoT device observations
app.get('/observations/:prefix', async (req, res) => {
const conn = await Datastore.open();
const {prefix} = req.params;
let count = 0;
res.set('content-type', 'text/csv');
res.write('Device, Data\n');
const stream = conn.getAll(`/${prefix}`);
stream.on('data', (data) => {
const outString = `"${data.key}", ${data.val}\n`;
count++
res.write(outString);
}).on('end', () => {
res.write(`Result: ${count}`);
res.end();
})
})
// register a new IoT device observation
app.post('/observation/:region/:location/:device', async (req, res) => {
const {region, location, device} = req.params;
const {observation} = req.body;
// string key e.g. '/North/Plaza/humid03/2023-02-23T05:23:11.306Z'
const key = `/${region}/${location}/${device}/${new Date().toISOString()}`;
const conn = await Datastore.open();
const result = await conn.set(key, observation);
console.log(result)
res.status(201).end('Data received ok');
})
export default app.init(); // Bind functions to the serverless cloud
This part-4 of the key-value store tutorial has shown how you can combine smart keys with the database streaming capabilities to create kind of IoT time series database with codehooks.io.
In our next tutorial [Part-5: Managing data with TTL options(https://codehooks.io/docs/tutorials/key-val-store/part-5-managing-data-with-ttl-options), we'll explore how you can use the time-to-live (TTL) feature to delete old data in the key-value database.
Top comments (0)