[AskDEV] How to learn building scalable distributed systems when its not part of your job?

SO. MUCH. RESEARCH.

However, research alone is an insufficient teacher. Many things I read early on, I did not have enough experience to understand its significance. Here's why: Even if you have a mentor to break down every implication of every single architectural detail of their solution, you would be bored into a coma. Many details do not seem relevant (and will get automatically filtered out by your brain) until you actually face the problems they address. I found myself coming back to articles I tried to read multiple times before. But previously when I started to read them, I could not see the benefit of their approach (and sometimes couldn't even understand what they were trying to do). It wasn't until I had more experience that I understood what conditions made certain tactics necessary. And I'm still learning.

Probably nobody is going to lead you by the hand down the scalable distributed systems path anyway. Most of us are flying by the seat of our pants as it is, while we try to build these things. You have to take the initiative, pursue it as an interest, and press for opportunities. And when you do finally get that chance, it can be quite stressful when the technical success of a product rests on your shoulders. But at some point you get it working, and then you have a base to build on.

As far as where to start, remember this quote:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

• John Gall

Pick a single topic in distributed computing that interests you. (i.e. scalable databases, or service discovery, or event-driven systems, etc.) Research it. Explore the shape and context of it. Heck, even try it. (Spinning up EC2 instances or Lambda functions or w/e for a few hours now and then is quite cheap.) Then work your way out from there into the next topic. Follow the threads until you start to understand how the tapestry is made.

I recommend starting with Martin Kleppman's book titled Designing Data Intensive Systems. It covers all the fundamentals and advanced stuff in one neat package. Saves you hours of scouring multiple websites and cognitive overload.