From Monolith to Microservices in One Year

So, there I am a year ago, living a carefree life working on a legacy monolith — a codebase written in CodeIgniter 2. Not the best framework in the world, but it did the job, at least on the surface. We had a lot of hacky code, after years and years of making things work for our clients in as little time as possible. Having said that, I think you understand that first sentence was a joke.

After some changes to our business strategy — and a lot of convincing — the IT department got a green light for a no-deadline, full rewrite of our Applicant Tracking System (ATS) — only a part of our monolith, but a big part for sure.

Problems

1. After adding a new feature, it goes into the manual testing phase. Once it's greenlit, you deploy it to production — but what often happens is you miss a testing scenario, and you end up with lots of bugs in production.
2. This problem is tightly connected to the previous one: If you have an older codebase, its foundation starts to slowly break down, so adding new code on top of it ends up relying on other code. With that kind of codebase, sometimes you end up breaking code that should be totally irrelevant to what you're currently writing.
3. When you have 11 people working on the same codebase day in, day out, without any safeties in place whatsoever, problems are bound to arise.

Known solutions

1. Automated tests — Even if you don't cover every possible scenario, automated tests — in addition to manual testing — give you a pretty good solution to our "bugs in production" problem. One of the biggest benefits of automated tests is that they let you freely upgrade your dependencies . If you have even partial coverage, when you do a "composer update" and run your tests, you can instantly see if a change in a dependency is breaking your code.
2. Splitting a giant codebase into smaller ones (i.e. microservices). With microservices, you can battle the "old code" problem on two fronts: On the one hand, they allow you to just leave your code sitting there without having it bother anything else, and on the other hand, if you do want to do a rewrite, microservices make it easier by letting you rewrite small parts of your codebase at a time. For example, in our legacy app we use an ancient version of TCPDF, which is one of the things preventing us from migrating it to a recent version of PHP. With microservices, we can put TCPDF inside a container with the specific PHP version it needs, and it's no longer an issue for the rest of the codebase.
3. One of the best solutions we found for our problem was splitting the codebase between our teams, and doing regular code reviews inside those teams. This way, the whole team knows the code they are working on, and we avoid any potential problems by using merge requests.

How we did it

For the first time ever, we got to do a project of this without having any sort of concrete deadline (Did you think I was kidding when I said that earlier?); even so, we couldn't just rewrite everything remotely related to the ATS, even if we wanted to — so, we decided to settle for rewriting only the parts that communicate with our ATS (in addition to the ATS itself, of course). The first microservice we made was the one that manages our job ads — we call it the job server. This was my first Symfony project; given that, and some of the other problems we found ourselves dealing with later on, I think you'll understand why I had to rewrite that microservice 3 times in the course of this project. And so we arrive at one of the first pieces of advice I can give: don't be afraid of rewriting something you just finished rewriting. Even if you have experience, it's unlikely that you'll get everything right the first time around.

SDKs FTW

If you take a look at the arrow between "ATS" and "Job server," the way it's drawn right now makes it look like they're communicating directly. We knew that the ATS wasn't going to be the only service communicating with the job server. Very early on, we discovered the need for SDKs. There is no better way of communicating with a service other than a service having its SDK. The way we went about this , though, is questionable…

Sidenote: What we refer to as "SDKs" are light, reusable libraries that make it easy to communicate with a specific service. We find these particularly useful because (nearly) all of our backend code is written in one language (PHP), so a service that has an SDK can be easily integrated into any other service.

One day, we found a package called DoctrineRestDriver. Immediately, we were like "OMG, this is awesome." We could make an SDK that gives you an EntityManager and lets you use Doctrine like you're talking directly to a database, even though it actually talks to the job server!

We were so amazed that we immediately went ahead with implementing this package into the Job SDK.

Later in the project, we discovered that it doesn't cover all of our use cases, and also that it's hard to maintain code that uses it. DoctrineRestDriver itself wasn't maintained either, so we had to fork it and spend some long hours digging into and changing the core of the library to make it work for our use case. At the time, we were in too deep to turn back, so we had to continue development with the SDK as-is. The advice I can give on this part of the process is: Don't look for the cheap way out when building SDKs. All the other SDKs we made later on were made with a different approach — Simple, descriptive methods that make a simple HTTP request; basically just thin wrappers around an HTTP client. More code, but far easier to maintain in the long run.

Tests

Up until this point, we'd been working towards splitting our codebase into smaller chunks, and we were doing a pretty good job of it, too; but who was going to test all this when it was done? At the point where the project looked like the diagram above, we had zero tests. (Okay, we actually did have some tests, but I'm not proud of them. Before doing any real research into testing, I started writing tests for the job server. What happened was that, while trying to write what I thought were unit tests, I accidentally wrote integration tests. When I realized what I'd done, I decided to watch some tutorials on how to properly write unit tests. PHPUnit: Testing with a bite from SymfonyCasts is a course that particularly helped me understand testing better.)

After а lot of effort, when all was said and done, we had about 200 tests across all of our microservices. It's still not a lot, all things considered, but we're hoping to gradually increase that number as we go along.

Present Day

After one year, ~50,000 lines of code, and ~2,600 commits, a team of 3 developers — consisting of a senior full stack developer and two medior backend developers — ended up with 18 services. Today, the system powering our ATS looks something like this:

Just imagine having this picture in your head before going to production for the first time. Add to that the inexperience we all had with this technology, and I think you can understand how anxious I was about the prospect of deploying all of this to production. When it came down to it, minor issues aside, we were all surprised by how smoothly the deployment actually went. Today, working in a system like this is amazing, and I wouldn't trade it for anything.

We learned a lot over this past year, and this post contains only a tidbit of the knowledge we came out of this project with. If I'm being honest, probably every microservice in the image above deserves a post of its own, and my team and I are very eager to share all of that knowledge — so, consider this post to be only the first of many.

That's all, folks. I can't wait to hear all your thoughts on this!

About me

Hi, I'm Kristijan! Traveling enthusiast, full time food and beer lover, and occasional software developer. When I'm not planning my next trip to Japan or watching anime, I work at poslovi.infostud.com, a part of Infostud group.