Building a scalable system from the ground up

There is a lot to be gained from designing a scalable system from the ground up, especially if you are building it for a startup that is still in its early stages and has the potential to grow. As developers, we are frequently faced with the choice of ignoring scalability and building (version 1) faster or incorporating scalability into the system and moving a little slower. Doing it, either way, has no visible effect at first, but as the system grows larger and begins to do more work, the scalability of the system will be put to test.

At Ceedcap, a start-up studio where I work as a software engineer, the majority of my time is spent designing and developing scalable systems for start-up businesses. It can be difficult to create such a system for projects that must move quickly. But, over time, I was able to decipher the code. To explain my approach, I'll use the most recent project I'm working on (Squareroof) as a case study.

Case study: Squareroof- a platform to easily own shares of rental properties.

Goal: To build a system that enables users to buy shares of properties, administrators to manage the platform activities, and collect and analyze real-estate data.

Timeline (Version 1): 5 weeks

Approach:

1. Use a popular project structure and framework

Photo by Mindinventory

A software project structure is a set of rules that determine how your code, files, and folders will be organized. The project structure makes it easier to find things and for new contributors to understand how your project is organized. It also makes it simple for your team to maintain the same organizational approach from project to project.

Using a popular project structure appears to be the obvious approach, but we are often tempted to create our own, especially when our language of choice does not enforce any. For example, if you use Nodejs, use common project templates provided by frameworks such as Express, if you use Python, use Flasks or Django, and if you use PHP, use Laravel, CodeIgniter, and so on.

The benefit of this is that it makes it easier to onboard new developers who will work on the codebase as the product grows. I chose Laravel Lumen for the backend and VueCLI project structure for the frontend for Squareroof.

2. Choose Microservice over Monolithics:

Photo by Krasamo

Monolithic applications contain everything—front-end, back-end, and databases—in a single location. Because all of the application's components are in one place, they are simple to build and deploy. Microservices is a type of architecture that divides a large computer program into smaller, modular programs. It allows each section of your code to operate independently and focus on its functionality while still communicating with other sections as needed.

I've built several monolithic projects that worked well at first but became a pain as the codebase grew and new features were added. It became more difficult to navigate. This also has an impact on the rate at which new features or changes can be implemented.

Splitting the project components into microservices, on the other hand, made the codebase more manageable and easier to navigate. This is in addition to the benefits of long-term scalability.

Microservices can be difficult and time-consuming to set up. However, with the advent of tools such as Docker and Kubernetes, it is becoming easier. For Squareroof, I decided to devote some of my time to developing a system that makes setting up and publishing a microservice as simple as setting up a monolithic application. This was accomplished by developing a common template for the backend microservices and a separate template for the frontend microservices, and then publishing them using the docker and Kubernetes APIs. This made it simple to quickly spin up a new microservice with only a few steps, without sacrificing scalability.

3. Set up CI/CD (Continuous Integration / Continuous Development) Pipeline

Photo by Wallarm

A CI/CD pipeline is a set of processes that allows for frequent, dependable software releases. The process is known as "continuous integration" and "continuous deployment" because it is automated and can be repeated as many times as needed for new features and updates. It allows developers to automate the process of developing, testing and delivering code.

Initially, I didn't see this as a critical step for Squareroof. To publish a new change, I had the habit of logging into the server and manually running each step. As the codebase grew in size, this became more difficult and time-consuming. Then I decided to implement a CI/CD pipeline by writing all of the manual steps into a script and triggered it whenever I made new changes. This not only gave me more time to code but also eliminated deployment errors and mistakes.

Conclusion

Building a system that can scale with the company can be difficult, but it will save you a lot of time and effort in the long run. You should also recognize that it is a journey. One does not simply build a scalable system and then stop; one must continue to improve on the existing system to fit into solving the problem that the product is addressing.

Let's connect on Twitter @drayfocus and on LinkedIn Akinola Ayomide.

Cheers!