Evolutive Design: A new approach to conceive complex system.

This article is a summary of a talk I did in 2023 for the Agile Tour. The talk is available here (French speaking 😉)

For a long time, I have considered that to build a complex system, you had to take a lot of care in the design phase, and this design phase should happen at the beginning of the project. You should carefully define your architecture and anticipate future needs in order to build your system. Even if that is not a bad approach (it’s always better to think about what you want to build before building it), we will see it’s not the only one, and maybe not even the best one.

Darwin and the theory of evolution

Before talking about engineering, let’s talk about biology. I assume you know Charles Darwin and his theory of evolution.

Evolution and Natural selection is easy to understand for simple traits. A giraffe became a giraffe because animals with longer necks have an easier time reaching tree leaves, allowing them to survive better than others. But, how does it work for complex organs ? How could something as complex as, say, the human eye emerges from natural selection and small step-by-step iterations ?

You can find a detailed explanation here, but let summarize the main idea.

Long, long time ago, our life was much simpler than nowadays, and we use to peacefully swim in the oceans of this planet. At that time, our biggest enemy was the Sun. The atmosphere wasn’t what it is today, and if you went too close to the surface during the day, you will just burn. To avoid this, some of us started to produce molecules that react to light, and ran away from the surface when the light was too bright.

Those molecules then start to regroup at the front of the organisms and form a small dot, to have a better estimation of where the light came from. This dot has widened and finally created a small bag with a small hole: a Camera Obscura. Five hundred millions year later, it became what we know today.

The initial need was to runaways from the sun. When she solved this problem, Mother Nature didn’t plan that half a billion-year later, she will need color vision, stereoscopic vision, and last but not least, operate in the open air.

Species evolve by mutating their genetic code, like a software evolve by mutating its source code. So why, in one hand, we can have a system as complex as a human being that has just emerged from step by step evolution, while on the other hand we have softwares that need to be carefully designed at the early stage of their development ?

Predators are drivers of evolution

To evolve, a system requires constrains. Why would you change anything if you don’t have to ? Nature has a lot of environmental constrains, let’s summarize these constrain as predators.

• First, predators are here to force you to evolve. No predators, no evolutions.
• Then, predators are here to constrain your evolution. If you evolve in the wrong way, you will be eaten.
• Finally, predators are here to prevent you from regressing.

What does all this have to do with software engineering ?

If we consider a software as a living entity, what would be its predators ? What prevent a mutation to be accepted ? What prevent a code modification to be deployed in production ? Could it be our tests, more specifically, our test pipeline ?

Modern software strongly relies on their integration pipeline to accept or refuse a Pull Request. Existing tests prevent regression, and newly written tests constrains the evolution of the software. In a sense, they act like software predators.

Tests are software predators

When talking about tests, I talk about automated tests, as it should represent the broad majority of your tests. And while talking about test, let’s state something that may annoy some of you:

Remember the number of time you fixed a bug, a unit test start to fail, and you realized the test was actually validating the buggy behavior. Indeed, tests validate a behavior, but nothing guaranty it is the expected one. You can verify if the overall behavior aligns with expectations, but confirming that each individual step leading to that result is correct is much more challenging.

That bring us to another statement:

Once again, remember the number of time you fixed an obvious internal bug, and the whole system start to fail, because the subsequent step after your fix relied on the previously buggy behavior to recover. When a test fail, the only things we are sure is that something changed.

Ultimately, tests are constraints, not validators. When they fail, they are not saying: “You broke something.”. They say: “It’s not the way it used to be. Are you sure of what you are doing ?”.

And last but not least:

Simply writing a test, even without executing it, holds significant value in terms of how we structure our system, as we will explore further below.

Test Driven Development Design

TDD, or “Test Driven Development”, refer to a development strategy where you write tests before writing code. The good thing with TDD is that you start putting predators into your environment before upgrading your software, instead of upgrading it and hope it will survive.

While TDD is often appreciated for the positive impact it has regarding software compliance with specifications, it is underrated regarding the positive impact it also has on software architecture. That’s why I personally prefer referring to TDD as “Test Driven Design”.

Indeed, one of the characteristics of a well design software is its testability. Good design are easy to test. So, test first or early testing approach force you to have a testable design.

TDD also make you ask “How will I expose this new functionality ?”. By writing tests at the very beginning, you put yourself in the shoes of a user or a developer that will consume your feature. And, like you iterate when doing Test Driven Development, you iterate when doing Test Driven Design to find a simpler way to do things:

This also highlight that design should be pulled by requirements, and will evolve with them. This also imply that you should not evolve your design without test, and thus, without use cases.

Test Driven Design is also simpler than canonical Test Driven Development. By doing Test Driven Design, you focus on the nominal case. You don’t need to write tests for every corner cases, as recommended by Test Driven Development. You focus on the overall architecture and main entry point. Not limit or buggy cases.

Tests prevent Design regressions

Like test prevent behavior regression, they also prevent design regression. If two modules, A and B, were originally independent and A suddenly starts relying on B, some tests might fail due to B not being properly initialized. The tests will complain about why B needs to be initialized while it was functioning perfectly until now.

You might question the necessity of this new dependency and could find a way to avoid it.

Refactoring is not a Crime

With planned design, refactoring is often seen as a failure. You failed in your design phase, and you have to fix your design issues.

With evolutive design, refactoring is part of the development process. It’s not a sign of failure, it’s a sign of maturity. Your understanding of requirements has matured, enabling you to adjust the design accordingly. Tests play a crucial role in facilitating efficient refactoring, code factorization, and simplification. Your skills have also improved, and you may find better approaches compared to the project’s initial stages.

However, refactoring is a double-edged sword. Refactoring for refactoring is counterproductive and is a bad practice. A refactoring should have a measurable outcome. I previously talked about this in a prior article, but the main outcomes of refactoring should be:

• Remove a significant amount of code (Factorization).
• Remove an internal or external dependency (Decoupling).
• Replace a custom solution with a well established, existing one (Don’t reinvent the wheel).
• Any measurable improvement: Performances, memory, scalability, …

If your sole argument is “The system will be cleaner”, then you are just saying “I don’t like the way it is done, I want to do it my way”.

Conclusion

Planned design frequently leads to over-design, as you try to anticipate all the use cases.
The majority of these scenarios will likely never occur, and the rest will unfold differently than initially imagined. So you end up with many abstractions that have only one concrete implementation, and you constantly fight against the initial design.

Evolutive design focus on implementing the simplest solution to address the current requirement, assuming that simpler architectures are easier to evolve than complex ones. You create an abstraction only when you have several similar use cases. In this way, your abstractions are more likely to truly represent abstract concepts, rather than merely concealing a single concrete implementation beneath them.

Nature has produced highly complex systems, relying solely on evolution rather than planning. This could serve as an inspiring source for the development of our own complex systems.