Part 3: Adding a database

The full code for this part is in branch part-3, found here

In the previous parts we setup our project folders and connected everything together. In this part we will focus on the server package mostly, by adding a database to persist data when the server restarts or is updated.

What database to use?

I went back and forth on this one, as the technology you use has to match your goals for the project. I primarily wanted something simple, easy to deploy and with little extra setup necessary for development.

In the end I want to be able to host the whole project (server/portal/widget) in a single VM somewhere, without having to worry about external connections, databases and such. With that in mind I was looking at some sort of in-memory database with persistence to a local file of some sorts which would be loaded back in upon restarts/updates.

I did want something performant so that I would (hopefully) not run into issues when there are around 100-ish clients connected at the same time. I looked at low-db for a while but did not like that it would JSON.stringify my whole database on every change, which could become a problem when it becomes to big.

So instead I went with lokijs even though I find the documentation of it quite horrible, I have used it before and it works without issues and has lots of features. And I thought it would also be good to show how I use it so others don't have to figure that our on their own 😇

Don't agree with me?

That is of course perfectly fine! In the end it does not matter what database technology you use, feel free to implement it using MongoDB or Firebase whatever you are comfortable with. The only thing that you would need to change is the database initialization and how to save/update/fetch something out of there.

Let's code!

The changes in this section are summarized in this commit

To keep things separated I will put all database related stuff inside the /packages/server/database folder. As the /packages/server/admins.ts we use seed into our database logically belong there, I moved that file into that folder, changing the top line to: import { Admin } from './../types';.

Installing lokijs

To install lokijs run the following commands:

yarn workspace server add lokijs
yarn workspace server add -D @types/lokijs

Initializing the database

I create a packages/server/database/database.ts file with the following:

import { join } from 'path';
import adminSeed from './admins';
import loki from 'lokijs';
import { Admin, Client, Database } from '../types';
const lsfa = require('lokijs/src/loki-fs-structured-adapter');

export default function initDB() {
  return new Promise<Database>((resolve) => {
    const adapter = new lsfa();
    const db = new loki(join(__dirname, './server.db'), {
      adapter,
      autoload: true,
      autosave: true,
      autosaveInterval: 4000,
      autoloadCallback: () => {
        db.removeCollection('admins');
        const admins = db.addCollection<Admin>('admins', {
          autoupdate: true,
        });
        adminSeed.forEach((admin) => {
          admins.insertOne(admin);
        });
        let clients = db.getCollection<Client>('clients');
        if (clients === null) {
          clients = db.addCollection<Client>('clients', {
            autoupdate: true,
            indices: ['id'],
          });
        }
        resolve({ admins, clients });
      },
    });
  });
}

Quick breakdown of what is happening:

1. Lokijs uses so called adapters to handle the persistence to files. We use the fastest and most scalable adapter called the fs-structured-adapter. You can read more about it here
2. We export a initDB function which will setup the database and return a promise, resolving when it is done.
3. Inside the setup we provide some seed data to our database, we repopulate the admins from our seed file every time. Also we check if a collection for our clients exist, and if not, we create one. Collections are logically separated parts of the database, which also get persisted inside their own file.
4. On both collections we use the autoupdate setting, that will persist changes made to the collection automatically. By default you would have to call .update() manually to make sure the data in memory is also saved to file.

Inside our .gitignore file we have to add /packages/server/database/*.db* to make sure our created database files are ignored by git.

Updating packages/server/index.ts

Now we have to use our just created initDB function inside our main entry file. First remove the current database initialization:

• const db: Database ... ❌
• import admins from ./admins; ❌

And add import initDB from './database/database'; at the top somewhere.

Replace the server.listen call with:

let db: Database;
(async function () {
  try {
    db = await initDB();
    server.listen(5000, () => {
      console.log(
        `Server started on port ${5000} at ${new Date().toLocaleString()}`
      );
    });
  } catch (err) {
    console.log('Server failed to start.');
    console.error(err);
  }
})();

Which is our new initialize function that will startup the server once the database is setup.

At this point typescript is probably complaining that the Database type is no longer correct. Lets change packages/server/types.ts:

• add import { Collection } from 'lokijs'; at the top
• update the interface to:

export interface Database {
  clients: Collection<Client>;
  admins: Collection<Admin>;
}

Update handlers

Our code in the packages/server/handlers still expects a plain object as a database, we have to update some code inside adminHandler and clientHandler to use our new database properly:

• Instead of .find((admin) => admin.name === name) we can now use .findOne({name})
• When we want to send all items of a collection we have to to db.clients.find() instead of just db.clients
• When adding a new client we use .insert instead of .push.

There is one gotcha when adding a new message to the clients messages array. As lokijs uses Object.observe on the whole client to determine if something needs updating. This does not work for array mutations (common Vue2 reactivity caveat as well, that got me quite a few times😅). So whenever we add a message we have to update manually by adding db.clients.update(client); afterwards.

Store the client session

Commit found here

When a client connects now it will generate a new random name, and when that client refreshes it's browser window it will create a new client. This is of course not really feasible, we have to store the clients session somewhere, and if that same client reconnects we restore that session.

Generate a random id for clients at the server

Inside packages/server/index.ts we add the following

// Socket middleware to set a clientID
const randomId = () => crypto.randomBytes(8).toString('hex');
io.use((socket, next) => {
  const clientID = socket.handshake.auth.clientID;
  if (clientID) {
    const client = db.clients.findOne({ id: clientID });
    if (client) {
      socket.clientID = clientID;
      return next();
    }
  }
  socket.clientID = randomId();
  next();
});

and add import crypto from 'crypto'; at the top.

This is a piece of middleware that will run for every client that connects to our server. It will check a auth object on the handshake that the socket server does with the client, if the a clientID is present there we set that clientID on the socket object. If not it is a new client and we generate a new random ID.

As we use typescript and we set a clientID property on the socket object it does not know about we have to add that to the type of socket.

To do that we add to packages/server/types.ts:

declare module 'socket.io' {
  interface Socket {
    clientID: string;
  }
}

Using the new clientID inside the clientHandler

Inside the packages/server/handlers/clientHandler.ts we currently define the client and add it to the database. We have to check here if our client already exists and only add a new entry to the database if necessary.

Remove:

• const client: Client = ... ❌
• db.clients.insert(client); ❌

And add:

socket.join(socket.clientID);

socket.emit('client:id', socket.clientID);

let client: Client;
const DBClient = db.clients.findOne({ id: socket.clientID });
if (DBClient) {
  client = DBClient;
  client.connected = true;
  socket.emit('client:messages', client.messages);
} else {
  client = {
    ...data,
    messages: [],
    id: socket.clientID,
    connected: true,
  };
  db.clients.insert(client);
}

Breakdown:

1. By default socket.io will create a user id store it a socket.id and join a room with that specific id. We now have to join the socket.cliendID room as we define our clientID manually.
2. We emit the clientID to the client so that it can store that in localStorage and send it along when reconnecting.
3. We check if the client exists, and if it does not we create and insert that client into the database.
4. If a client is already in the database we send the message history to the client.

Inside the same file we also have to update our disconnect event listener as we have to change the logic that determines if a client is connected. In theory we could have one client opening multiple tabs, which will each establish it's own socket connection. If one of them closes we need to check if there aren't any connections left open for that client before we update the connection status.

Change the socket.on('disconnect') handler to:

socket.on('disconnect', async () => {
  const matchingSockets = await io.in(socket.clientID).allSockets();
  const isDisconnected = matchingSockets.size === 0;
  if (isDisconnected) {
    client.connected = false;
    io.to('admins').emit('admin:client_status', {
      id: client.id,
      status: false,
    });
  }
});

Save the clientID at the widget side

In the widget we will store the id and name of the client in the socket store, and generate a new name there if it is the first time connecting.

Inside the packages/widget/src/stores/socket.ts we add to our state:

id: localStorage.getItem('clientID'),
name: localStorage.getItem('clientName') || '',

and to our actions:

SOCKET_messages(payload: Message[]) {
  this.messages = payload;
},
SOCKET_id(payload: string) {
  localStorage.setItem('clientID', payload);
  this.id = payload;
},
setName() {
  const name = faker.name.firstName();
  this.name = name;
  localStorage.setItem('clientName', name);
},

Also add import faker from '@faker-js/faker/locale/en'; at the top of the file, and remove it from packages/widget/src/App.vue;

Now we have to use the name and id from the store when connecting to the socket server, change const socket = io(URL); to:

const socket = io(URL, {
  auth: {
    clientID: socketStore.id,
  },
});
watch(
  () => socketStore.id,
  (val) => {
    socket.auth = {
      clientID: val,
    };
  }
);
if (!socketStore.name) {
  socketStore.setName();
}

The watcher here is needed when for some reason the server disconnects (for a restart for example) and the socket connection is reset. In that case the socket will reconnect with the correct clientID provided.

In the addClient object change name to name: socketStore.name and add watch to the list of imports from 'vue'.

Handle reconnects at the portal side

Commit found here

The last thing we take care of in this part is handling reconnecting the portal to the server when the server is restarted or otherwise loses it's connection. In the portal currently we only call admin:add when we boot up our application. If the socket connection is lost and restored we have to call admin:add again to register the correct handlers on that socket.

Inside packages/portal/src/boot/socket.ts we change the admin:add call to:

// This will be called on the initial connection and also on reconnects
socket.on('connect', () => {
  socket.emit('admin:add', 'Evert');
});

We have to do the same inside our widget inside packages/widget/src/App.vue change client:add to:

// This will be called on the initial connection and also on reconnects
socket.on('connect', () => {
  socket.emit('client:add', addClient);
});

Fix a small bug in the portal

There is a bug in the portal code that happens when the server reboots and the socket is reconnected. Even if we re-emit the admin:add event, if we already have a client selected, we cannot send new messages to that selected client. That is because when we reconnect we resend the whole client list and in the SOCKET_list action inside packages/portal/src/stores/client.ts we replace the clients array in the state with the newly received value.

However, if we already had a client selected, the clientSelected state pointed to an item in the old array we overwrote. So to keep things working we have to reset the clientSelected in there as well:

if (this.clientSelected) {
  const currentSelectedId = this.clientSelected.id;
  this.clientSelected =
    this.clients.find((client) => client.id === currentSelectedId) ||
    null;
}

Wrapping up

That's it for this one! In the next part I will be adding a login page to the portal and generate a token to also secure the connection from the portal to the server. See you next time! 🚀