Migrating ToolJet server from Ruby on Rails to Node.js

The background

We recently open-sourced (ToolJet)[https://github.com/ToolJet/ToolJet/]. ToolJet is an open-source no-code platform for building and deploying custom internal tools.

ToolJet has two main components, the client and the server. ToolJet client is a ReactJS application and ToolJet server is a Ruby on Rails API-only application. Whenever a new application is built using ToolJet, the frontend ( client ) generates the definition of the app in JSON and the server persists the versioned definitions on a PostgreSQL database. The server also acts as a proxy that run queries on top of the data sources ( data sources in ToolJet includes databases like MongoDB and API based services like Google Sheets ).

Ruby accounts for only 14.5% of our codebase. When we built ToolJet, we chose Ruby on Rails as the backend because of its ability to quickly prototype and iterate due to its strong adherence to ‘convention over configuration’.

We open-sourced ToolJet with the eventual goal of letting everyone build plugins ( database integrations, API integrations & UI widgets ) for ToolJet easily. But with the current architecture, if a developer wants to build a new data source integration for ToolJet, he/she should be familiar with both Ruby and JavaScript because running queries is handled by Rails and building queries is handled by query editors in the ReactJS frontend. Thus when we migrate the server to Node.js, the developers will be able to build ToolJet plugins using just one language.

TypeScript

The popularity of TypeScript has been exploding in recent years ( JavaScript Flavors ). TypeScript is a superset of JavaScript and provides an optional type system for JavaScript. The freedom of dynamic typing can cause bugs that are hard to detect. With TypeScript, you can avoid classic JS errors like undefined is not a function. We decided to use TypeScript as it helps us to avoid mundane errors and to improve the maintainability of our codebase.

JavaScript analysis paralysis

It's real!
In the Rails world, everything is already decided for you. But in the Node.js world, there is an endless possibility of file structures, naming conventions and tooling. For example, in Rails, we used ActiveRecord but in Node.js, we get to ( or have to) choose ORM ( if any ). When we started looking for a suitable Node.js backend framework. A huge range of frameworks was available to choose from. This can lead to Choice Paralysis but on the bright side, having too many options at our disposal helps us choose what's best for our use case.

Choosing the framework

SailsJS

Since we are migrating from Ruby on Rails, SailsJS was the obvious choice because of its similarity with Rails. We decided not to use SailsJs when we came across issues raised by developers related to the built-in ORM, waterline.

Express

Express is a very minimal yet powerful framework. Express, being an un-opinionated framework, doesn't come with an error handler, body-parser etc. This gives the developer a lot of freedom but with a lot of freedom comes a lot of responsibilities to choose the right way to do something. We did not want to spend a lot of time discussing what framework to use for every single requirement. Hence, we decided not to use Express.

Meteor

Meteor is a powerful full-stack Node.js framework. We did not go ahead with Meteor as it doesn't support PostgreSQL and migrating the database to MongoDB wasn't something we want to spend our time on. ( We came across meteor-postgres but as their documentation says, it is still a work in progress ).

NestJS

NestJS has everything that we were looking for in a backend framework. NestJS is a slightly opinionated framework but gives some level of flexibility by allowing the usage of other libraries. For example, NestJS uses Express under the hood but it can be configured to use Fastify as well.

The new architecture for the server

We decided to go ahead with NestJS because: a) It fully supports TypeScript b) Database agnostic: we can directly use any Node.js database integration libraries or ORM. NestJS documentation explains in detail integrating TypeORM and Sequelize. c) Excellent documentation: everything is explained well.

We started looking for an Object Relational Mapper (ORM) because we do not want to spend our time building and debugging SQL queries. TypeORM and Sequelize were the most mature choices. We chose TypeORM because it is a matured ORM available for TypeScript.

What's next

In the coming days, we will be working on migrating the Ruby on Rails endpoints and query services to Node.js in a way that the users will not have to change the existing data in their PostgreSQL database.

We would love you to check out ToolJet on GitHub: https://github.com/ToolJet/ToolJet/.