DEV Community

Cover image for API CMS: Redis DataSource
Valeria
Valeria

Posted on

API CMS: Redis DataSource

In the previous article of the series we've built REST API, but it only had a health endpoint, assuring us that 🔥 everything is fine 🔥.

There is a lot of things we can add: user management, permission system, and, well, pages themselves, but all of it requires a database.

As stated before, we'll be using Redis as our primary database, which means we'll need to handle all the secondary indexing.

DataSource.md

One more time we start with the documentation. This one, in my opinion, belongs to the api folder, as it is directed to developers of the system and not direct users.

DataSource class provides common CRUD operations as well as listing and indexing. Every entity is stored as hash, with a variety of secondary indexes. Hash parameters, along with everything else in Redis, are stored as strings, therefore to reconstruct the original JSON, one must provide information about its types.

Each entity data source should extend DataSource and provide:

  • collection name, e.g. items
  • id prefix, e.g. itm
  • type information

Type information is stored using the following data type classes, mapped with property name:

  • TBoolean
  • TInt
  • TFloat
  • TString
  • TJSON

These classes provide an interface for encoding and decoding values to and from strings.
Null values are stored as Unicode NUL \u0000

A generic items data source would look the following way:

class Items extends RedisDataSource<{ title: string }> {
  collection = "items";
  prefix = "itm";
  schema = {
    title: TString,
  };
}
Enter fullscreen mode Exit fullscreen mode

DataTypes

Data types implementation is straight forward. We start with the test, paying attention to null values:

//src/api/datatypes.spec.ts
import { describe, it } from "mocha";
import { expect } from "chai";
import {
  DataType,
  NUL,
  TBoolean,
  TFloat,
  TInt,
  TJSON,
  TString,
} from "./datatypes";

describe("data types", () => {
  it("can compose a scheme", () => {
    const schema = {
      title: TString,
      published: TBoolean,
    };
  });

  const checkNull = (Type: any) => {
    const nothing = new Type(null);
    expect(nothing.toString()).to.be.eq(NUL);
    expect(Type.from(NUL)).to.have.property("value", null);
    expect(Type.from("undefined")).to.have.property("value", null);
  };

  it("boolean", () => {
    const yes = new TBoolean(true);
    expect(yes.toString()).to.be.eq("1");
    expect(TBoolean.from("true")).to.have.property(
      "value",
      true,
      "String true"
    );
    expect(TBoolean.from("1")).to.have.property("value", true);
    expect(TBoolean.from("9000")).to.have.property("value", true);
    checkNull(TBoolean);
  });
  it("integer", () => {
    const num = new TInt(123);
    expect(num.toString()).to.be.eq("123");
    expect(new TInt(123.4).toString()).to.be.eq("123");
    expect(TInt.from("-0")).to.have.property("value", 0);
    expect(TInt.from("0")).to.have.property("value", 0);
    checkNull(TInt);
  });
  it("float", () => {
    const num = new TFloat(0.123);
    expect(num.toString()).to.be.eq("0.123");
    expect(TFloat.from("-1.23")).to.have.property("value", -1.23);
    expect(TFloat.from("1.234")).to.have.property("value", 1.234);
    checkNull(TFloat);
  });
  it("string", () => {
    const str = new TString("Hello, World");
    expect(str.toString()).to.be.eq("Hello, World");
    checkNull(TString);
  });
  it("json", () => {
    const obj = new TJSON({ foo: "bar" });
    expect(obj.toString()).to.be.eq('{"foo":"bar"}');
    checkNull(TJSON);
  });
});

Enter fullscreen mode Exit fullscreen mode

And the implementation itself:

//src/api/datatypes.ts
export const NUL = "\u0000";

export abstract class DataType<T> {
  value?: T;
  toString: () => string;
  static from: (value: string) => DataType<any>;
}

const isNullable = (value: any) => {
  return [undefined, "undefined", null, NUL].includes(value);
};

export class TBoolean implements DataType<boolean> {
  constructor(public value?: boolean) {}
  toString() {
    switch (this.value) {
      case true:
        return "1";
      case false:
        return "0";
      default:
        return NUL;
    }
  }
  static from(value: string) {
    if (isNullable(value)) return new TBoolean(null);
    if (["false", "0"].includes(value)) return new TBoolean(false);
    return new TBoolean(true);
  }
}

export class TInt implements DataType<number> {
  constructor(public value?: number) {}
  toString() {
    return this.value?.toFixed(0) ?? NUL;
  }
  static from(value: string) {
    if (isNullable(value)) return new TInt(null);
    return new TInt(parseInt(value));
  }
}

export class TFloat implements DataType<number> {
  constructor(public value?: number) {}
  toString() {
    return this.value?.toString() ?? NUL;
  }
  static from(value: string) {
    if (isNullable(value)) return new TFloat(null);
    return new TFloat(parseFloat(value));
  }
}

export class TString implements DataType<string> {
  constructor(public value?: string) {}
  toString() {
    return this.value ?? NUL;
  }
  static from(value: string) {
    if (isNullable(value)) return new TString(null);
    return new TString(value);
  }
}

export class TJSON implements DataType<object> {
  constructor(public value?: object) {}
  toString() {
    if (!this.value) return NUL;
    return JSON.stringify(this.value);
  }
  static from(value: string) {
    if (isNullable(value)) return new TJSON(null);
    return new TJSON(JSON.parse(value));
  }
}

export type SomeDataType =
  | typeof TBoolean
  | typeof TInt
  | typeof TFloat
  | typeof TString
  | typeof TJSON;

Enter fullscreen mode Exit fullscreen mode

Redis Data Source

Now we can start with the data source implementation.
First, we need to install a package to connect to Redis. My personal favorite is ioredis. It can connect via a secure connection, to a cluster, supports custom LUA scripts, and a bunch of other things.

Let's install this dependency and its types:

yarn add ioredis
yarn add @types/ioredis  --dev
Enter fullscreen mode Exit fullscreen mode

We'll also need to have the actual Redis instance running. You can install it locally or run it with docker. I personally prefer to have it installed locally because it comes with a redis-cli client.

Get

We'll be writing Redis Data Source as we go this time. We can start with:

// src/api/DataSource.spec.ts
import { describe, it, beforeEach, afterEach, after } from "mocha";
import { expect } from "chai";
import Redis from "ioredis";
import { RedisDataSource } from "./DataSource";
import { TBoolean, TString } from "./datatypes";

const redis = new Redis();

class Items extends RedisDataSource<{
  id: string;
  username: string;
  email: string;
  active?: boolean;
}> {
  collection = "items";
  prefix = "itm";
  schema = {
    username: TString,
    email: TString,
    active: TBoolean,
  };
}

describe("DataSource Integration Test", () => {
  beforeEach(() => {
    redis.flushall();
  });
  afterEach(() => {
    redis.flushall();
  });
  after(() => {
    redis.disconnect();
  });
  it("can get item", async () => {
    const rnd = Math.random();
    await redis.hset(
      "items::itm_1",
      "id",
      "itm_1",
      "username",
      "superman" + rnd,
      "email",
      "clark@daily.planet",
      "active",
      "true"
    );
    const source = new Items({ redis });
    const item = await source.get("itm_1");
    expect(item).not.to.be.null;
    expect(item).to.have.property("id", "itm_1");
    expect(item).to.have.property("username", "superman" + rnd);
    expect(item).to.have.property("email", "clark@daily.planet");
    expect(item).to.have.property("active", true);
  });

});

Enter fullscreen mode Exit fullscreen mode

And the actual implementation can be something like:

//src/api/DataSource.ts
import { Redis } from "ioredis";
import { SomeDataType } from "./datatypes";

export abstract class DataSource {
  constructor(protected context: any) {}
}

export abstract class RedisDataSource<
  T,
  I = T,
  P = Partial<I>
> extends DataSource {
  readonly collection: string;
  readonly prefix: string;
  readonly schema: {
    [key: string]: SomeDataType;
  };

  decode = (value: Record<string, string>): T => {
    const result: any = { id: value.id };
    for (let key in this.schema) {
      result[key] = this.schema[key].from(value[key]).value;
    }
    return result as T;
  };

  constructor(protected context: { redis: Redis }) {
    super(context);
  }

  get(id: string) {
    return this.context.redis
      .hgetall(this.collection + "::" + id)
      .then((value) => {
        return this.decode(value);
      });
  }
  create(input: I) {
    return {} as T;
  }
  update(id: string, patch: P) {
    return {} as T;
  }
  delete(id: string) {
    return true;
  }
  list(params: any = {}) {
    return { items: [] as T[], nextOffset: null };
  }
}

Enter fullscreen mode Exit fullscreen mode

So far we implemented only a get method: we simply retrieve all hash keys and values and parse them according to our schema.

Update

Let's add update. Test first:

it("can update item", async () => {
    await redis.hset(
      "items::itm_1",
      "id",
      "itm_1",
      "username",
      "superman",
      "email",
      "clark@daily.planet",
      "active",
      "true"
    );
    const source = new Items({ redis });
    const item = await source.update("itm_1", { username: "Clark Kent" });
    expect(item).not.to.be.null;
    expect(item).to.have.property("id", "itm_1");
    expect(item).to.have.property("username", "Clark Kent");
    expect(item).to.have.property("email", "clark@daily.planet");
    expect(item).to.have.property("active", true);
  });
Enter fullscreen mode Exit fullscreen mode

In most cases, we want to update some parameters and return the updated value. We can perform several operations at once either by using multi method from ioredis, or with a custom LUA script. We could try the first option:

// wrong code
update(id: string, patch: P) {
    const values = this.encode(patch);
    const cid = this.collection + "::" + id;
    return this.context.redis
      .multi()
      .hset(cid, values)
      .hgetall(cid)
      .exec()
      .then((results) => {
        // Throw Errors
        if (results[0][0]) throw results[0][0];
        if (results[1][0]) throw results[1][0];
        // Return results
        return this.decode(results[1][1]);
      });
  }
Enter fullscreen mode Exit fullscreen mode

Functions encode and decode are:

decode = (value: Record<string, string>): T => {
    if (!value) return null;
    const result: any = { id: value.id };
    for (let key in this.schema) {
      result[key] = this.schema[key].from(value[key]).value;
    }
    return result as T;
  };

  encode = (value: Record<string, any>): Record<string, string> => {
    if (!value) throw Error("Value should have properties to be encoded");
    const result: any = {};
    for (let key in value) {
      if (!(key in this.schema)) continue;
      result[key] = new (this.schema[key] as any)(value[key]).toString();
    }
    return result;
  };
Enter fullscreen mode Exit fullscreen mode

However, our logic has a flaw: when there's nothing to update, it would create the item. We don't really want it, so let's add a test for it first:

it("doesnt create items on update", async () => {
    const source = new Items({ redis });
    const item = await source.update("itm_1", { username: "Clark Kent" });
    expect(item).to.be.null;
  });
Enter fullscreen mode Exit fullscreen mode

Now we need to perform three operations at once: check if a hash exists, update specified values and return the hash itself. We still want the operation to be atomic and in this case, we're going to write an LUA script. We can do it by defining a new command like this (e.g. in constructor):

 context.redis.defineCommand("updhash", {
      numberOfKeys: 1,
      lua: `
      local exists = redis.call('exists',KEYS[1]);
      local result = nil
      if(exists==1)
        then
          redis.call('hset',KEYS[1], unpack(ARGV));
          result = redis.call('hgetall',KEYS[1]);
      end;
      return result
      `,
    });
Enter fullscreen mode Exit fullscreen mode

And use it like that:

update(id: string, patch: P) {
    const values = Object.entries(this.encode(patch)).reduce((a, c) => {
      return a.concat(c);
    }, []);
    const cid = this.collection + "::" + id;
    return this.context.redis["updhash"](cid, ...values).then((result) => {
      if (!result) return null;
      return this.decode(this.collect(result));
    });
  }
Enter fullscreen mode Exit fullscreen mode

This time it works as expected.
You probably noticed a collect function in the end. We didn't need it when we used the built-in ioredis command hgetall, but for our custom command, we need to handle its result ourselves. Hashes in are returned as an array or iterating key-value strings, collect function turns them into an object:

collect(values: string[]) {
    if (!values || !values.length) return null;
    const obj = {};
    for (let i = 0; i < values.length; i += 2) {
      const key = values[i];
      const value = values[i + 1];
      obj[key] = value;
    }
    return obj;
  }
Enter fullscreen mode Exit fullscreen mode

Create

To create new items, we need to guarantee it'll always have a unique id. There are several ways how to do it, but I suggest we go with the shortest id: an incremental index.

Every time we will create a new item, we'll increase the value of the items::next key by one so that we can use this value as an id for the next item.

Test first:

it("can create item", async () => {
    const source = new Items({ redis });
    const item = await source.create({
      username: "Clark Kent",
      email: "clark@daily.planet",
    });
    expect(item).not.to.be.null;
    expect(item).to.have.property("id", "itm_1");
    expect(item).to.have.property("username", "Clark Kent");
    expect(item).to.have.property("email", "clark@daily.planet");
  });
Enter fullscreen mode Exit fullscreen mode

And implement:

create(input: I) {
    const values = Object.entries(this.encode(input)).reduce((a, c) => {
      return a.concat(c);
    }, []);
    return this.context.redis["newhash"](
      this.collection,
      this.prefix,
      ...values
    ).then((result) => {
      return this.decode(this.collect(result));
    });
  }
Enter fullscreen mode Exit fullscreen mode

With the following LUA script:

context.redis.defineCommand("newhash", {
      numberOfKeys: 2,
      lua: `
      local next = redis.call('incr',KEYS[1]..'::next');
      local id = KEYS[2]..'_'..next;
      local cid = KEYS[1]..'::'..id;
      redis.call('hset',cid,'id',id, unpack(ARGV));
      return redis.call('hgetall',cid);
      `,
    });
Enter fullscreen mode Exit fullscreen mode

List

We'll probably need different listing options in the future: we might want to have items returned in a specific order or search items by some criteria. For now, let's implement the simplest list possible by using SCAN command.

Our test for a list method:

it("can list items", async () => {
    await redis.hset(
      "items::itm_1",
      "id",
      "itm_1",
      "username",
      "superman",
      "email",
      "clark@daily.planet",
      "active",
      "true"
    );
    await redis.hset(
      "items::itm_2",
      "id",
      "itm_2",
      "username",
      "lexluther",
      "email",
      "lex@luther.corp"
    );
    const source = new Items({ redis });
    const list = await source.list();
    expect(list.items).to.have.length(2);
    expect(list.items[0]?.id).to.be.eq("itm_1");
    expect(list.items[1]?.id).to.be.eq("itm_2");
    expect(list.nextOffset).to.be.a("string");
  });
Enter fullscreen mode Exit fullscreen mode

And the implementation:

list(params: { limit?: number; offset?: string } = {}) {
    const limit = params.limit ?? 10;
    const offset = params.offset ?? "0";
    return this.context.redis["scanhash"](
      this.collection,
      this.prefix,
      offset,
      limit
    ).then((result) => {
      const [nextOffset, items] = result;
      return {
        items: items.map((item) => this.decode(this.collect(item))),
        nextOffset,
      };
    });
  }
Enter fullscreen mode Exit fullscreen mode

And a LUA script to make it happen:

context.redis.defineCommand("scanhash", {
      numberOfKeys: 2,
      lua: `
      local result = redis.call('scan',ARGV[1],'TYPE','hash','MATCH',KEYS[1]..'::'..KEYS[2]..'_*','COUNT',ARGV[2]);
      local keys = result[2];
      local items = {};
      for k,v in ipairs(keys) do
        items[k]=redis.call('hgetall',v);
      end
      result[2]= items;
      return result
      `,
    });
Enter fullscreen mode Exit fullscreen mode

Delete

Last, but not least, a delete method. Yes, test first:

it("can delete items", async () => {
    await redis.hset(
      "items::itm_1",
      "id",
      "itm_1",
      "username",
      "superman",
      "email",
      "clark@daily.planet",
      "active",
      "true"
    );

    const source = new Items({ redis });
    const result = await source.delete("itm_1");
    expect(result).to.have.property("deleted", true);
    const exists = await redis.exists("items::itm_1");
    expect(exists).to.be.eq(0);
  });
Enter fullscreen mode Exit fullscreen mode

And the implementation, this time without LUA:

delete(id: string) {
    return this.context.redis.del(this.cid(id)).then((deleted) => {
      return { deleted: Boolean(deleted) };
    });
  }
Enter fullscreen mode Exit fullscreen mode

Adding Redis service to CI

Now we have a bunch of tests using Redis, we want them to run on CI as well. It's super easy with Github Actions, just add Redis service to .github/workflows/***.yaml:

name: AMP CMS CI
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      redis:
        image: redis
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
    ...
Enter fullscreen mode Exit fullscreen mode

Add, commit, push! Yay us!
Let's take a little break, we definitely deserve it :-)
See you in the next article!
** runs away to fix failing CI test **

Discussion (0)