This article was originally posted on my personal website.

It was in March 2020 when Anki and I decided to tackle a new project together. After years of abandoning projects in our free time, we were dedicated to spend some time and get this done. I won’t dive deep into this journey, as we‘ve already described it in this blog post if you’re interested. 😄

But we did it: we created a new SaaS which is called TrueQ and also managed to deploy it productively. In this blog post I want to talk about the tech stack we used, decisions we made and what we regret or would do differently nowadays.

Our Background

Let me tell you a bit about our background in software development. We’re both professional software developers being specialized in web development. In general we’re doing full stack development, although we may have more experience in the frontend. But as we strive to work more on our own products in the future, we definitely have a strong passion to build a product in whole.

Before choosing the tech stack in detail, it was clear to us that it will be located in the JavaScript ecosystem, or to be more specific, the TypeScript ecosystem. In our jobs we also worked on different backend applications which were written in Java / Kotlin or C#. But our main experience lies in Node.js. Additionally we are building React Applications since almost 6 years, so this is definitely the frontend framework of our choice (and also will be for future projects).

Requirements for our product

We knew that for our product SEO is essential. TrueQ is all about finding solutions to your day-to-day problems. So these solutions have to be found easily. Therefore we knew that even though we want to build a rich web application, our server still needs to serve the content of our pages in plain HTML to make search engine’s life easier - server side rendering was a requirement for us.

We read about Max Stoibers regrets when he built spectrum, and we were sure that we don’t want to implement SSR on our own. 😅 As we were following Vercel and the development of Next.js, it was the first thing we took a closer look at.

Additionally there was an excellent blog post released at the time we started our project. Loup Topalian wrote about frameworks and libraries he would use to build a webapp in 2020 and in the end we actually adopted most of them.

Next.js

Ok lets begin with the foundation of TrueQ: Next.js. It is a production-ready React framework developed by Vercel and they’re not stopping at the client-side. With Next.js it is possible to write a fully fletched universal webapp which also takes care of server side rendering and other amazing stuff like incremental static site generation.

Very soon it was clear to us that we want to go with Next.js as it perfectly fits our needs. But now the question popped up: how should our backend look like? How should the API be connected with the Next.js application?

We definitely don’t regret the decision of using Next.js. It is a great framework and in the last year we used it, there were major improvements without any bigger breaking changes - Vercel is doing a great job here.

Just our solution of how we connected the backend to Next.js and the decision to host it on a private vServer and deploy it via Ansible wasn’t the best choice for us.

Using a custom Express.js server

So we dived deeper into the topic of how Next.js works and learned from the official docs that there is the possibility to use a custom Express.js server with Next.js where you have the whole freedom of leveraging the power as you would write a standalone Express.js application. It just wraps around Next.js so that you can specify own routes and implement all other sorts of logic with Express.js and all other routes are handled normally by Next.js.

It seemed like the best decision back then. We could move quickly, using a technology we already were familiar with and didn’t have to spin up a separate service for our backend. But for us there are some big cons to this approach which led us to the decision that we wouldn’t go this way anymore in the future.

Cons of the custom Express.js server

With Next.js we were used to fast HMR which allowed us to develop quickly. We also wanted our backend to reload automatically when we do changes to our code so we used nodemon together with ts-node (because all the backend code is written in TypeScript too). The problem here: it wasn't fast anymore. Everytime we changed code in the backend the whole Express.js server rebooted and it took quite a while until it was running again. This also influenced some parts of our frontend, as it included shared files which were also detected by nodemon. We couldn't find a solution for this and it's actually pretty cumbersome.

Additionally you're not able to deploy your Next.js application to Vercel anymore as they only provide deployments for plain Next.js applications. This also led us to custom deployment with Docker + Ansible on a vServer hosted by netcup which we’re going into detail in a later section.

Session handling and authentication

Nevertheless it is how TrueQ is currently built. That means that the session handling and authentication is also completely handled by Express.js. For authentication we use Passport.js which handles our normal E-Mail & Password login, but also third party logins via Google, GitHub and Twitter.

In production we're using a Redis server to persist the sessions (which also keeps them alive after the deployment of new versions).

GraphQL with Apollo

Until then we were used to write REST APIs. We already heard about GraphQL here and then, but never got in touch with it. We got curious and got our hands dirty to spin up an API and see how we like it.

We actually got hooked pretty fast. We love the flexibility to reuse DTOs, but at the same time only serve the fields you need for a specific usecase / view.

We're using Apollo both on the backend as server, but also on the frontend to query the API. Additionally we use graphql-codegen to generate TypeScript models of our DTOs and the React hooks for Apollo. We're very happy with that setup.

Problem with calling the API logic directly when rendering on the serverside

Now that we had an API in place we also needed to make sure that it's callable isomorphically. It should be reachable via the browser, when the Next.js application is in "SPA mode", but also on the server side when the HTML is being built for the first render.

For the browser it is pretty straight forward. It just calls the /api/graphql endpoint to execute queries and mutations. But on the serverside we thought that we somehow directly could execute the Apollo server logic. We didn't managed to get it running like this and that's why we need to do a seperate network request to https://localhost:3000/api/graphql on the serverside, to also be able to make API calls there.

All of this is wrapped in an Helper HoC which takes care of making the API calls isomorphic. Here's the code snippet of how we create the isomorphic Apollo link:

function createIsomorphLink(ctx) {
  if (ctx) {
    const { HttpLink } = require('@apollo/client'); // eslint-disable-line @typescript-eslint/no-var-requires
    // TODO: we need to look into this, as with this we are still doing a network request to our own application, but with apollo-link-schema we don't have our context available on the serverside
    return new HttpLink({
      uri: 'http://localhost:3000/api/graphql',
      credentials: 'same-origin',
      fetch,
      headers: ctx.req && {
        cookie: ctx.req.header('Cookie'),
      },
    });
  } else {
    const { HttpLink } = require('@apollo/client'); // eslint-disable-line @typescript-eslint/no-var-requires
    return new HttpLink({
      uri: '/api/graphql',
      credentials: 'same-origin',
      fetch,
    });
  }
}

Knex.js + Objection.js in connection with Postgres

So we had our API running and implemented the first CRUD operations. But where should the data be stored and be retrieved from? 😄

As I said we already had some experience with Node.js applications back then, but we mostly worked with MongoDB + mongoose for accessing the database. In the last years being employed as software developer we enjoyed to work with relational databases and also thought that it would be a better fit for TrueQ. So we decided for PostgreSQL and searched for solutions how we could easily query our DB.

Pretty soon we stumbled upon Knex.js, a SQL query builder for Node.js. It takes care of the db connection (it also has support for pooling) and gives you the possibility to write SQL queries with a query builder pattern like this:

knex.select('title', 'author', 'year').from('books');

Objection.js

Knex.js even has support for strong typing with TypeScript, but during our research we found Objection.js, an ORM which is built on top of Knex.js.

It gives you the possibility to write Models and execute queries against them with type checking of all the available fields, we're actually pretty happy with it and back then we didn't know of a better solution for handling database access.

Here you can see an example how a model plus a very simple query looks like. For more informations checkout their documentation.

import { Model } from 'objection';

class Topic extends Model {
  id!: number;
  name!: string;

  static tableName = 'topic';
}

const topic = await Topic.query().findOne({ name });

console.log(topic.name);
console.log(topic instanceof Topic); // --> true

Running migrations and seeds

Now when you're running an application productively there are also going to be database changes over the time. Therefore we also needed a solution to create migrations and run them in production.

Luckily Knex.js also got us covered with this one. 🥳 In Knex.js each migration is a JavaScript file which exports an up and a down method to either execute the migration or roll it back. Before deploying a new version of TrueQ we just make sure to execute the latest migrations with the knex migrate:latest command.

Here is an example of the migration of our question table:

exports.up = function (knex) {
  return knex.schema.createTable('question', function (table) {
    table.increments('id').primary();
    table.integer('user_id').notNullable().references('id').inTable('trueq_user');
    table.integer('views').defaultTo(0).notNullable();
    table.timestamp('deleted_at');
  });
};

exports.down = function (knex) {
  return knex.schema.dropTable('question');
};

Additionally Knex also supports Seeds for applying test data in your local environment.

Look out for Prisma

As already mentioned we actually were pretty happy with the Knex.js + Objection.js solution, but in the meanwhile we also found out about Prima. As it recently gained stable support for migrations we really consider to use it in our future applications, as it seems even more straight forward and better maintained.

Our Frontend

After showing you the architecture of our backend, let's take a look at our frontend. As already mentioned we love React, we're writing React applications for a long time already and it is the frontend framework of our choice. Not sure if this still has to be mentioned for React projects created in 2020, but just to cover it: we only make use of functional components together with hooks 😄

But as you might know, in the world of React you can pull in different libraries to solve things like routing or state management in your webapp, there are also some more things to talk about here.

State Management

So we're using Apollo on the client side for fetching data from our GraphQL API. Apollo has a powerful caching mechanism built in which stores the results from your query and it also lets you update this cache manually for optimistic updates. That means for many cases the data is just stored in the Apollo cache.

Additionally we also have quite some local logic, e.g. for our custom built editor. In those cases we're using MobX as a state management library. We love the simplicity which MobX gives you when defining state somewhere in your component tree, but at the same time taking care of only rerendering affected components down the path. It creates performant webapps by purpose.

Maybe I am doing a more in-depth blog post about MobX in the future.

ChakraUI

Of course we also needed a UI library, because we didn't want to write all sort of components on our own. Thanks to the above mentioned blog post we stumbled upon ChakraUI, an accessible and modular UI library for React.

For us ChakraUI is a bit different than other UI libraries. It simplified the way how we develop in the frontend and imho it complements the skills most web developers built up in the last years, which I describe more in detail with this blog post.

Deployment

In the summer of 2020 we came to the point that we already had quite a big part of TrueQ up and running. So we thought about how and where we're going to deploy our newly created webapp.

Due to the custom Express.js app, Vercel wasn't an option straight from the beginning. Back then we absolutely had no knowledge about services like AWS or DigitalOcean. I just had my own vServer running for the Browsergame I developed several years ago. That's why I thought it would be the best idea to also host TrueQ on our own vServer.

Docker

So when we're building TrueQ to deploy it to our test or production environment, we're creating a docker container including the Next.js build output and all necessary files. This docker image is then pushed to our container registry on GitLab.

Ansible

As we said we're deploying everything on a vServer on our own. That includes installing all necessary dependencies on the Linux server, configuring NGINX as our webserver, setting up SSL certificates, setting up the database, ensuring backups, and so on.

Because we didn’t just want to set this up by hand, we chose Ansible as our operator here. With Ansible you can create playbooks which get executed step by step as an automated way to setup your server. You just tell Ansible what to do in it's own DSL written in yaml files. That means that if for whatever reason we need to setup a new server, we just need to execute the Ansible playbook there and the Server would be up and running with TrueQ.

With Ansible it's also much simpler to keep track of the changes you're doing on your server, as all of the playbook files are also versioned via git.

In retrospective we learned very much about this whole process. But one thing we definitely learned is that we don't want to do this stuff on our own anymore. 😅 More about this in the last section of this blog post.

Analytics with the ELK stack

For the analytics we're using the ELK stack also hosted on those vServers. We're collecting logs via filebeat and metrics with metricbeat. Additionally we're having the Kibana APM in place to get even more insights from our Node.js backend application.

Also this was a very interesting step, but again too much hassle to maintain this on our own.

GitLab CI/CD

The process of building a docker image and deploying it with Ansible is all encapsulated in mostly automated steps via our GitLab pipelines.

Everytime we’re merging changes into the master branch, our whole test suite is being executed and if everything succeeds, a new version of TrueQ is being built (the Docker image) und pushed to the image registry on GitLab.

Every night we’re deploying the latest version of our Docker image to our test environment automatically and additionally there is a separate schedule for deploying the latest version to production which can only be executed manually.

During this deployment, we’re executing an Ansible role, which connects to the Server, pulls the latest Docker image there and spins it up.

Here you can see the GitLab stage configuration for deploying trueq:

deploy_trueq:
  stage: deploy_trueq
  script:
    - ansible --version
    - echo $ANSIBLE_VAULT_PASS >> .vault-pass
    - ansible-playbook -i "inventory/$ANSIBLE_INVENTORY" main.yml --tags=trueq --vault-password-file=.vault-pass
    - rm .vault-pass
  only:
    variables:
      - $ANSIBLE_INVENTORY != null
      - $DEPLOY_TRUEQ == "true"

Test Setup

So we now covered the stack we used for developing TrueQ and also how we're deploying it. Now comes a topic which we actually began with pretty soon: writing tests.

TDD is an acronym (standing for Test Driven Development) we heard pretty often in our career before, but never did it by our own. We wanted to give it a try, at least for our backend, and boy was this a good idea. 😄 Every time we began with new features, we created test cases for all the requirements and edge cases we could think of in Jest. And before writing the actual implementation we started to write the failing tests including the assertions of how things should work. After the implementation was done those test should get executed successfully.

It saved us many possible bugs and currently about 80% of our APIs are covered with tests (mostly integration tests) which give us the confidence for larger refactorings and other future changes. The Testing Javascript course by Kent C. Dodds definitely was a huge help with creating our test setup and learning some unknown things about Jest.

In the future we also consider using cypress for e2e tests to gain even more confidence.

What would you do differently nowadays?

Let’s come to an end. And in the end it’s always time for the most interesting question which probably is: what would we do differently in future products?

As already explained in the previous sections the top things we’re complaining about are:

• the custom Express.js server for Next.js
• the deployment which we don’t want to handle on our own anymore
• maybe using Prism instead of Knex.js or another solution for accessing the database

To be honest we’re not completely sure how our future tech stack is going to look like exactly.

For the frontend we’re already very happy and definitely going to stay with React, Next.js and Chakra. Maybe we’re going to switch the Apollo client with React Query.

In the backend there will probably be more changes and depending on what we go with, it will also affect database access and hosting. We’re looking closely at Blitz.js and Supabase and consider deploying on AWS, Vercel, DigitalOcean or Render.

We're very glad that the first MVP of our new product snappify comes with barely any backend logic and is completely hosted on Vercel, but soon we need a more sophisticated solution and I am going to inform you how our updated tech stack looks like as soon as we’ve settled. ✌️