Building for Scale? Make a R.A.D.I.C.A.L. System!

Renato Cordeiro Ferreira on December 26, 2018

The year is ending, 2019 is almost here and soon the 2010s will come to an end. This decade was marked by services from companies like Netflix, G... [Read Full]
markdown guide
 

Interesting article though I am curious why you chose to go through the list L-to-R rather than R-to-L, it caught me off guard while I was reading it given the acronym you've coined.

Your article did help me realise some terms for things I was already looking into like CEP so thanks for that! :)

 

Hi James, thanks for your comment!

I think this approach is more constructive, that is, it reminds me of the steps I would think about when building a system with these techniques:

  1. Lean inception helps to understand what to build.
  2. Agile methodologies explain how to build it.
  3. CEP deals with how to process the data.
  4. AI/ML handles how to learn from the data.
  5. Distributed environments are about where to run the system.
  6. Automated infrastructure is about how to run the system.
  7. Reactive Microservices Architecture puts everything together in a single structure.

In my mind it's more natural to explain 1 → 2, 3 → 4, and 5 → 6, although I feel I could begin with any of the pairs 1-2, 3-4, and 5-6 since they talk about planning, data and infrastructure respectively. I also like ending with the architecture because it is really what defines the system. In the end this coincided with the acronym backwards :)

I'm glad you learned some interesting new terms! Take a look at the e-book Fast Data Architectures for Streaming Applications by Dean Wampler, it's one of my inspirations for this section about big data and complex event processing.

 

Yeah, I see what you mean when looking at it like that - each point builds upon the next. It is just a shame that order isn't the same order as the actual letters in the acronym - L.A.C.I.D.A.R just doesn't have the same ring to it.

 

It's a good list just that you miss out on the human aspects of it.

You can read books with psychological or ethics to understand human problems like "Hooked" by Nira Eyral and Ryan Hoover so that we don't build stuff that is against ethical issues like Target focusing on Pregnant women or Cambridge Analytical and recent court hearing of Google.

 

Hi Max, thanks for your comments!

I really focused more on the technical side of those systems, the only place where I touch the human aspect was where I listed the agile methodologies -- whose primary concern is not related to these social / ethical / diversity issues you talked about.

I'll take a look on your reading recommendations, thanks!

 

This all seems fairly substantive. I'd caution against making it too buzzwordy, however this might play well in corporate environments.

I approve on the idea.

 

Hi Ben, thanks for your comment!

The text is indeed full of buzzwords. I definitively don't want to promote using them indiscriminately. Applying those techniques together brings all their advantages but also all their drawbacks. And it's by no means easy to do -- maybe something we may target to evolve our system if it ever needs to handle such a large scale.

Nevertheless, I want a way to aggregate interesting concepts to highlight that sometimes they are used together. I really enjoy to try to take this cross-cutting look at system building and that's when the idea of the acronym appeared.

 

Even in corporate environments, there is no generic uniform solution ingeniously making everything better.

Silver bullets exist only inside marketing papers. Real life is always more complicated.

 

Artificial Intelligence and, in particular, Machine Learning

Argh. Machine Learning is by no mean Artificial Intelligence, ask any competent Data Scientist.

Also, would you mind to shed a light where should I put this AI is my very scalable banking software?

Putting a lot of trendy buzzwords all together won’t help to build robust software.

 

Hi Aleksei, thanks for your comment!

You are right: thare are many scalable systems that don't need to use everything I described. As you said in your other comment, there are no silver bullets. Using so many techniques together brings all their advantages and their drawbacks together.

Now about the AI vs. ML discussion, I often find people describing ML as a subset of AI, maybe better described by the name of "statistical learning". In my understanding, AI is a wider field that uses different mathematical frameworks (another example, first order logic and graph theory) to create automated decision making. What are your thoughts?

Here in Brazil we have a quite large fintech ecosystem where more companies, in particular banks, are applying machine learning for risk evaluation, fraud detection, etc. Other interesting use cases I saw in other domains was the use of ML in the infrastructure level to optimize resource usage.

Surely, as you said, many systems don't need to apply everything to be highly scalable (bank systems running in mainframes or chatting systems built with Erlang are nice examples). However I wanted to highlight some systems use all those concepts together and we could give a name to them.

code of conduct - report abuse