loading...

From Monolith to Microservices in One Year

kristijankanalas profile image Kristijan Kanalaš ・6 min read

So, there I am a year ago, living a carefree life working on a legacy monolith — a codebase written in CodeIgniter 2. Not the best framework in the world, but it did the job, at least on the surface. We had a lot of hacky code, after years and years of making things work for our clients in as little time as possible. Having said that, I think you understand that first sentence was a joke.

After some changes to our business strategy — and a lot of convincing — the IT department got a green light for a no-deadline, full rewrite of our Applicant Tracking System (ATS) — only a part of our monolith, but a big part for sure.

Problems

  1. After adding a new feature, it goes into the manual testing phase. Once it's greenlit, you deploy it to production — but what often happens is you miss a testing scenario, and you end up with lots of bugs in production.
  2. This problem is tightly connected to the previous one: If you have an older codebase, its foundation starts to slowly break down, so adding new code on top of it ends up relying on other code. With that kind of codebase, sometimes you end up breaking code that should be totally irrelevant to what you're currently writing.
  3. When you have 11 people working on the same codebase day in, day out, without any safeties in place whatsoever, problems are bound to arise.

Known solutions

  1. Automated tests — Even if you don't cover every possible scenario, automated tests — in addition to manual testing — give you a pretty good solution to our "bugs in production" problem. One of the biggest benefits of automated tests is that they let you freely upgrade your dependencies . If you have even partial coverage, when you do a "composer update" and run your tests, you can instantly see if a change in a dependency is breaking your code.
  2. Splitting a giant codebase into smaller ones (i.e. microservices). With microservices, you can battle the "old code" problem on two fronts: On the one hand, they allow you to just leave your code sitting there without having it bother anything else, and on the other hand, if you do want to do a rewrite, microservices make it easier by letting you rewrite small parts of your codebase at a time. For example, in our legacy app we use an ancient version of TCPDF, which is one of the things preventing us from migrating it to a recent version of PHP. With microservices, we can put TCPDF inside a container with the specific PHP version it needs, and it's no longer an issue for the rest of the codebase.
  3. One of the best solutions we found for our problem was splitting the codebase between our teams, and doing regular code reviews inside those teams. This way, the whole team knows the code they are working on, and we avoid any potential problems by using merge requests.

How we did it

For the first time ever, we got to do a project of this without having any sort of concrete deadline (Did you think I was kidding when I said that earlier?); even so, we couldn't just rewrite everything remotely related to the ATS, even if we wanted to — so, we decided to settle for rewriting only the parts that communicate with our ATS (in addition to the ATS itself, of course). The first microservice we made was the one that manages our job ads — we call it the job server. This was my first Symfony project; given that, and some of the other problems we found ourselves dealing with later on, I think you'll understand why I had to rewrite that microservice 3 times in the course of this project. And so we arrive at one of the first pieces of advice I can give: don't be afraid of rewriting something you just finished rewriting. Even if you have experience, it's unlikely that you'll get everything right the first time around.

SDKs FTW

If you take a look at the arrow between "ATS" and "Job server," the way it's drawn right now makes it look like they're communicating directly. We knew that the ATS wasn't going to be the only service communicating with the job server. Very early on, we discovered the need for SDKs. There is no better way of communicating with a service other than a service having its SDK. The way we went about this , though, is questionable…

Sidenote: What we refer to as "SDKs" are light, reusable libraries that make it easy to communicate with a specific service. We find these particularly useful because (nearly) all of our backend code is written in one language (PHP), so a service that has an SDK can be easily integrated into any other service.

One day, we found a package called DoctrineRestDriver. Immediately, we were like "OMG, this is awesome." We could make an SDK that gives you an EntityManager and lets you use Doctrine like you're talking directly to a database, even though it actually talks to the job server!

We were so amazed that we immediately went ahead with implementing this package into the Job SDK.

Later in the project, we discovered that it doesn't cover all of our use cases, and also that it's hard to maintain code that uses it. DoctrineRestDriver itself wasn't maintained either, so we had to fork it and spend some long hours digging into and changing the core of the library to make it work for our use case. At the time, we were in too deep to turn back, so we had to continue development with the SDK as-is. The advice I can give on this part of the process is: Don't look for the cheap way out when building SDKs. All the other SDKs we made later on were made with a different approach — Simple, descriptive methods that make a simple HTTP request; basically just thin wrappers around an HTTP client. More code, but far easier to maintain in the long run.

Tests

Up until this point, we'd been working towards splitting our codebase into smaller chunks, and we were doing a pretty good job of it, too; but who was going to test all this when it was done? At the point where the project looked like the diagram above, we had zero tests. (Okay, we actually did have some tests, but I'm not proud of them. Before doing any real research into testing, I started writing tests for the job server. What happened was that, while trying to write what I thought were unit tests, I accidentally wrote integration tests. When I realized what I'd done, I decided to watch some tutorials on how to properly write unit tests. PHPUnit: Testing with a bite from SymfonyCasts is a course that particularly helped me understand testing better.)

After а lot of effort, when all was said and done, we had about 200 tests across all of our microservices. It's still not a lot, all things considered, but we're hoping to gradually increase that number as we go along.

Present Day

After one year, ~50,000 lines of code, and ~2,600 commits, a team of 3 developers — consisting of a senior full stack developer and two medior backend developers — ended up with 18 services. Today, the system powering our ATS looks something like this:

Just imagine having this picture in your head before going to production for the first time. Add to that the inexperience we all had with this technology, and I think you can understand how anxious I was about the prospect of deploying all of this to production. When it came down to it, minor issues aside, we were all surprised by how smoothly the deployment actually went. Today, working in a system like this is amazing, and I wouldn't trade it for anything.

We learned a lot over this past year, and this post contains only a tidbit of the knowledge we came out of this project with. If I'm being honest, probably every microservice in the image above deserves a post of its own, and my team and I are very eager to share all of that knowledge — so, consider this post to be only the first of many.

That's all, folks. I can't wait to hear all your thoughts on this!

About me

Hi, I'm Kristijan! Traveling enthusiast, full time food and beer lover, and occasional software developer. When I'm not planning my next trip to Japan or watching anime, I work at poslovi.infostud.com, a part of Infostud group.

Discussion

pic
Editor guide
Collapse
muhammadtaqi profile image
Muhammad Taqi

Very well written, I'm also working in Micro-services env with symfony. We use Symfony console components to keep the data sync btw the services. Now as the arch is scaling it's becoming bit challanging. I wonder that Cron-Manager is for this thing? Can you explain bit about that also?

Collapse
kristijankanalas profile image
Kristijan Kanalaš Author

Thank you, sorry it took a while to reply.

Sure, I would love to explain it. So if a service has to do some data sync or garbage collection we would write a command inside that service and in Cron Manager we would set up a cron that would send a message to RabbitMQ queue and the consumer inside the said service would trigger the command. For now it proved to be a pretty good solution for scheduling tasks in services.
The cron manager has an option to trigger an endpoint too but it is somewhat cleaner to have stuff separate, in commands and to trigger them with RabbitMQ.

We have a cron manager open sourced but it's still in development and doesn't have documentation but if you want to have a look here is the link github.com/Vodzo/docker-cron-manager

Collapse
spuhpointer profile image
spuhpointer

Check the Enduro/X middleware, it has some nice microservices concept, basically it is C/C++/Go/Java application server, which allows to start several executables where each executable offers services. Number of copies started for each binary is configurable. All calls to services are load balanced. Binaries are governed by Enduro/X, start, restart operations are managed by server. There are transparent clustering options. These concepts comes from XATMI api standart, which has already 20 years and more :)

Collapse
kristijankanalas profile image
Kristijan Kanalaš Author

Thanks for advice I will check it out for sure. Currently we are using kubernetes with rancher and are very happy with it for now, but as I said I will check it out :)

Collapse
fmertins profile image
Fernando Mertins

Hi there, great post, thanks!
Any suggestion for other developers (like me) when moving from CodeIgniter to Symfony? These days I'm using CI 3, Composer and some namespaces with my own classes.
All the best!
Fernando from Brazil

Collapse
kristijankanalas profile image
Kristijan Kanalaš Author

Thank you for reading the post. 😊

Only one major advice comes to my mind.
At my company we often joke how CI is an mvc framework but altho it looks like you are writing object oriented code you end up writing procedural code. So when you jump over to Symfony be patient and give yourself time to get used to writing real object oriented code. Symfony has a good documentation, reading it helps a lot.

Collapse
thevetdoctor profile image
Obafemi

Beautiful and very encouraging. I hope to get a remote job soon. I live in Nigeria.

Collapse
kristijankanalas profile image
Kristijan Kanalaš Author

Thank you! :)

I wish you luck in finding a job and if you ever need some help feel free to contact me, I followed you back.

Collapse
thevetdoctor profile image
Obafemi

Thanks. Am open to remote jobs. I work with JS, Python, PHP. Fullstack. SDLC.

Collapse
jodyshop profile image
Waleed Barakat

Nice story, but what is the final product? And how you planning to monetize it?

Collapse
kristijankanalas profile image
Kristijan Kanalaš Author

Hi Waleed, thanks for the comment.

The final product is a software for applicant tracking. In ideal world we would monetize it as a SaaS but as we are a job board our clients get to use the software for free when they buy a job ad. For now we will try to come up with cool features and eventually have a paid tier for our software.

Demo

Collapse
nebojsac profile image
Nick Cinger

Awesome write up! Love seeing real-world use cases for microservices, especially with a monolith starting point. Thanks for taking the time to write it :)

Collapse
kristijankanalas profile image
Kristijan Kanalaš Author

Thank you for reading it, I hope the information can be useful to you in the future. 🙂