Packages don't solve distributed systems

Packages and libraries can be a great way to share functionality across different codebases. React, lodash, express - all of these are used across many codebases without issue.

So when it comes time to build out many backend services which operate within a similar domain - packaging logic can seem like a no brainier.

But what if I told you that's a disaster waiting to happen?

The scaling problem 📈

As companies grow, the business domain typically grows in complexity and size. Scaling this becomes a real challenge - both from a technical and organizational perspective.

As problem solvers, we break things down into smaller parts; typically, the business domain is broken into smaller "subdomains" - each with their own teams and services (Conway's law in full effect).

In an ideal world, each of those teams could work in complete isolation, doing what they do best and only having to solve problems within their domain.

In reality, an overlap of the domains of a business is common and significant overhead is introduced due to a need for cross-domain communication.

The same problem applies to technical architecture; while ideally services would be bound by the constraints of their own existence (and not of other services), domain overlap makes this hard to achieve.

Using packages 🎁

At face value, this seems like a very simple problem to solve. "I’ll make a package which abstracts the business logic into on place so all services can use it” said every web developer, ever.

While this seems like a no-brainer, what you end up with is a distributed monolith.

The efforts to solve scaling problems by breaking the domain down into services is undone when we move business logic into packages. We end up with a distributed system - albeit, one where services are coupled.

Coupling 👫

Coupling is the deadly sin of a distributed system. If the goal is to work at maximum velocity without the overhead of cross-communication, coupling achieves the exact opposite.

From the start of our careers as developers, we’re taught that code redundancy is bad, and that abstracting code into functions is good. While this is sometimes the case, it’s easy to get wrong, and if there’s anything worse than redundant code, it’s that one overly-abstracted function that does a milliion different things and is used across the whole system.

When we put business logic into packages, we end up with this exact problem... but on a completely new level.

Runtime coupling

The most obvious issue with distributing domain logic via packages is that, while the business logic might be nice and neatly placed in one place during development time, that won’t be the case at runtime.

Business logic needs to be consistent - so by distributing business logic via a package, how do you ensure consistency? Versioning won’t solve this issue because differing services could be using different versions of a package at any one time.

The solution I’ve seen over and over again is a monolithic release process; leading to slow CI times, a need for coordination amongst many teams, and volitile releases (due to transactional rollouts being outright impossible on a distributed system).

Code coupling

Problems for sharing business logic in packages spread further than just runtime cohesion. As services are tied together by the ones and zeros they use to operate, this can lead to restrictions on what those services can do and how they can operate.

Supported runtimes and dependencies of a package often fail to be updated as nobody is willing to take the risk to update to newer versions; lest it should cause the tightly knit tower of services to crumble.

This can often be a bidirectional problem whereby a service needs to update a package to address a bug, but that would in turn require changes to a shared package (and subsequently, many services).

Ownership 🙋

A less obvious consequence of using packages for business logic is the blurring of lines of ownership. In an architecture where services work within a bounded context (more on this later), all aspects are self contained - logging, metrics, reads/writes etc.

The same cannot be said when business logic from a package is running on any number of services at one time. Something as simple as “monitoring the creation of users” becomes a behemoth effort involving many services.

Before you know it, someone has built a company-wide logging library, further worsening the service coupling problem.

In the case of runtime issues, it is often unclear where the problem is located - the package, or the service. I’ve personally seen this lead to a culture of avoidant engineers who - understandably - don’t want to be accountable for breakages because there is no clarity on who is responsible for what.

Making it work 🤔

So by now, you likely understand why having many services share code can have a spreading impact on the stability, velocity and effectiveness of a distributed system and the teams behind it. What are our options?

Service Oriented Architecture 🏗

While the term itself sounds advanced, it simply refers to building a distributed system using services as the primitive building blocks.

If a distributed system is a jigsaw, services are the pieces which together create the full picture.

Bounded contexts 🔲

A pragmatic solution to coupling, bounded contexts are intended to create distinct boundaries between the different domains of a system.

Following the single responsibility principle, and coined by Eric Evans in Domain Driven Design, the goal is to ensure that any given service has a clearly scoped responsibility.

From a technical perspective, you can think of this as every running service being solely responsible for its existence, deployment, data and anything else relating to its domain.

By having a single source of truth for the different domains within a distributed system, many of the challenges mentioned earlier cease to exist.

Inevitable coupling 🚨

While having clear distinct boundaries sounds nice, in practice, things are never that straight forward. Services are always going to have overlapping concerns - and thats okay. We can solve that logical coupling in a manner that introduces minimal runtime coupling and no code coupling.

The internet

Okay so hear me out - there’s a way to communicate between machines using some wires and electricity - it’s going to take of soon!

GraphQL, REST, SOAP, you name it, any one of these is going to do just fine; and while you might have a better time with some protocols over others, as long as the only point of coupling between services is a (preferably versioned) protocol for communicating, code coupling is out of the question.

It can still go wrong

Excessive runtime coupling is still a possibility; and depending on how services communicate, a messy web of inter-dependent services is not impossible.

That’s a whole other topic in itself - I’d recommend this talk by Jimmy Boggard.

Ownership 🙋

Finally there’s the ownership angle. Those clear boundaries assigned to each service - assign them to people too.

Accountability isn’t just about knowing who broke that thing, it’s about giving teams the ability to have a say in what they do, and take pride in following that through to the end.

Its easy to say “not my problem” when some code that has been touched by 10+ different people breaks, but you’ll be hard pressed to find an engineer who is willing to respond like that when they proudly wrote all those bugs with their bare hands.

Better yet, you’re more likely to see engineers be proactive in maintaining their work when they know they’re responsible for the state of it.

Conclusion

So there you have it - hopefully this helps at least one team from ending up with a distributed monolith!

This is just the tip of the iceberg, so if you're interested in finding out more, I'd recommend the following resources:

• Building Microservices (or basically anything else by Sam Newman)
• You're not actually building micro-services
• Domain driven design
• Avoiding Microservice Megadisasters

---

Thanks for reading!

If you enjoyed this post, be sure to react 🦄 or drop a comment below with any thoughts 🤔.

You can also hit me up on twitter - @andyrichardsonn

Disclaimer: All thoughts and opinions expressed in this article are my own.