I've been thinking for several years already about the reason of the inefficiency of most long-term companies that built their business around software. It's not uncommon to join teams that are forced to implement solutions in systems with high levels of technical and business debt. Usually in companies with this situation, people complain about the a monster they call The Monolith and whenever possible, try to avoid feeding it with more time and energy.
Usually The Monolith is represented as a big chunk of code that is written in a old stack that is not fancy anymore for teams. Most of the times the solution is coupled, so teams are afraid to write any new code there. Domains are unclear, so teams don't understand their boundaries and there is friction between teams because code ownership is blurred. Usually it doesn't have automated tests, or at least test quality is very low, so adding new features is slow and needs lots of manual testing and even teams dedicated most of the time to it.
That's why organizations with this plague called The Monolith invest a lot of people energy and money to implement a decoupling strategy, or transformation, that would allow teams to increase their performance, because the common understanding is that a more advanced architecture should unleash the potential of teams.
And a better architecture notorably increases the performance of an organization, but it will unlikely remain efficient across the time if we repeat the same failures. Changing the technical solution won't do anything if the backbones of organizational structures remain as they are.
That's why changing an organization is complex and takes years. Because we are not changing just the structure of the delivered solution to our customers, but we are changing the culture of designing and delivering value to the customers and stakeholders.
I personally don't see software architecture as a discipline of just building software blueprints, in any of the formats. It worked ten years ago because teams and solutions were smaller, but nowadays with distributed teams and with worldwide business solutions, it just doesn't work well.
In my experience, software architecture is the discipline of growing and shrinking organizations elastically to build software solutions based on business needs. It's a bit more meta: you are not building blueprints, you are growing teams that build solutions. That's why I like to say that the future of software architecture is organic.
Infusing Organic to Software Architecture empowers a more people-centric vision of systems, with principles that allow organizations to scale up and down, to build success upon failure and leads to a culture of cultivation to teams, so they are ready for the challenges that organizations will be facing in the near future.
So what are the principles of an organic software architecture? At least those are my guidelines when I'm working on a software project.
Explaining everything would take lot of time, so I will like to share just a high-level checklist on each category, and if is there any topic that needs further discussion and explanation, I can write more posts about it.
Business
- Plans should be short-lived, driven by long-term goals.
- Metrics and goals should be transparent and understood by everyone interested.
- Business is expected to change goals based on shared facts.
- Business should embrace failure as a way for learning.
- Business should provide a sandbox for teams to fail.
Teams
- Team performance should be lead by individual growth and collaboration.
- Business change, so do teams.
- Methodologies should be lead by culture, not otherwise.
- Teams decide how they communicate globally in the organization.
- Team vision, expectations, goals and boundaries should be clear.
Solutions
- Together define a technical vision, strategy and principles driven by business needs.
- Don't make cheap code, make code cheap.
- Infrastructure is cheap in terms of budget, but expensive in cognitive load.
- Solutions are driven by the whole team.
Those principles depend on the situation of the company, but at least the decision of dropping one of them should be conscious.
What do you think? I'm open for feedback, experience from other people and ideas.
Top comments (18)
I feel like each point should be a post on it's own! All very important issues.
I think out of everything here (a fantastic post), I'd like this elaborated on more:
Hi James! thanks for your feedback! I will try to elaborate more on each topic with new posts ๐๐ and this one will be next in line.
As a summary, in my experience, I've found that just changing the structure of the software we deliver (for example, from a monolith to microservices with event sourcing) doesn't provide any meaningful value if we, as an organization, do not change how we understand value delivery.
If your practices lead you to have a coupled monolithic organization, even if you invest tons of money to fix it, if you don't change those practices you will eventually go back to the situation with a monolithic architecture (or even worse, with a microservices architecture that is highly coupled, which is more expensive).
Have you ever heard of Conway's law? From Wikipedia : Conway's law is an adage stating that organizations design systems which mirror their own communication structure.
Yes! It's amazing! Most of my work is inspired by the Conway's Law, Chaos Engineering and Lean and Agile Methodologies.
I've seen a very large company fail due to the very same idea you mentioned. The organization wanted to completely change the technology and architecture without changing the organization's mentality, processes, bureaucracy and way of dealing with change.
The plan was to split a huge monolith into an event driven architecture. Bounded contexts were found, teams were split in capabilities and DDD became the de facto approach. Scrum wanted to be adopted and we religiously practiced it within the teams, but we were not autonomous because above us there wasn't an understanding of it nor a willing to adopt it at an organizational level. A canonical model was given to us as a "solution" attempt to model the whole organization and communicate between domains. This canonical model, of course, didn't satisfy anyone bin an attempt to satisfy everyone, so every capability had their own Anti Corruption Layer and a great amount of effort was put into it with requirements to satisfy output canonical model rather than our own domain functional needs.
We had product owners per capability but there was no real product owner per product, no role had the needed overall view of functional needs so these were being dictated by Enterprise architects and their canonical model, not by business.
We soon found out the business mentality struggled with event driven architectures and asynchronous nature. Business kept pushing functional requirements that didn't fit with the way capabilities were split and eventual consistency started becoming a problem because they refused to design requirements around it. We ended up with requirements that required synchronous communication and coupling needs between capabilities to satisfy things that were not really required but "they were like this in the monolith".
We were building distributed monoliths without the benefits of event driven architectures and with the complexity of distributed systems to provide a global view of all the pieces.
When I left the company the plan was restructure things again around the canonical model, embracing it, and have a more monolithic approach.
Nobody really knows what the exact problem was but most of the devs suspect it was due to an approach at the technical level for which the business was not really prepared to make their respective changes. Their mind was often more around power concerns and how the changes would affect their positions. Of course my view and opinion is biased and the reality might have been more complex, but no one ever asked us and probably it would have been wise to get some feedback and take it into consideration.
PS: very interesting article! Thank you
Thanks Diego for your feedback and the story!
I think the story reflects a common case in traditional organizations with hierarchical structures where the decisions of a small committee that act as a single point of failure. Even if cultural changes must be lead by someone, changes need to be done in small steps, guided by a long-term strategy. It's better if those changes are actually driven by teams.
Thanks!
Thank you Kevin! Very interesting topics!
I was immediately thinking about the examples in the comments, on a company that's not even aware that needs to change the business approach, how can we use this architecture discussion to drive the business change?
Congrats for the article!
Hola Ferran! Thanks for your feedback!
Most business stakeholders usually don't see architecture as a competitive advantage, for them it's painful and it's only slowing down teams. Probably they have never seen a good architecture in place and usually there have been attempts to fix team performance before.
However there is an opportunity in those environments because management layers tend to rotate because of the burn-out and the impossibility to fulfill the expectations of growth of those companies.
The approach I would follow (depending on the stakeholder of course) is:
First, understand their goals and motivations, do not focus on the pain points. This will allow you to understand where to start focusing and it will allow you to understand their strategic long-term goals.
Second, make expectations clear. You don't need to fix everything, only whatever is a blocker for their goals. Focus on their strategy and adapt yours. Use this moment to start talking about business solutions and requirements, use trade-off sliders. It's important to mention that you want a long-term goal, so try to align stakeholders on long-term goals. You don't need something really pragmatic, just a vision where everyone agrees.
Third, find a way where both parties are comfortable to start working on their strategy. If you succeed in both earlier points, negotiating time from teams should be easier (not easy), as you are running their strategy and their priorities. This might need restructuring teams, as most of the times, team organizations are not aligned with business goals.
Four, share business goals with teams and propose a short-term plan (with something like a Lean Inception ๐) per team that is important for the business goals. In my experience, something around 1 quarter of work is safe enough to be done and to fit changing business needs. Consider the business strategy always as long-term, but not stable, it can change per quarter.
We can have a deeper conversation if a point is not clear :D.
Thanks!
Brilliant! Thank you!
IT is not โoperationsโ anymore. IT is unibiquituous across the organization, so we are part of the โbusinessโ. If not, we will be solving business problems in a technical way only. It always leads to poor design
I need to say that it is a great post about Software architecture, it help me to look the concept from other. point of view that i like more because you mostly emphasize value for the organization at first.
BTW some of guidelines weren't clear for me, Can you develop the idea behind them?
"Business should provide a sandbox for teams to fail."
โInfrastructure is cheap in terms of budget, but expensive in cognitive load.โ
"Teams decide how they communicate globally in the organization."
Thanks Raul for your feedback! I will try to write new posts with those guidelines so they are more clear. Let me give you a short summary of each of them ๐.
Business should be mature enough to handle failures from teams. For example, if your company incomes are managed by a single team which delivers the core domain, the pressure of delivery on that team makes the organization less reliable and error-prone.
If you build an organization where your stability as a business is distributed across teams, you can actually stand failures and learn from them, which ensures that teams can learn from the business feedback and improve.
Business should understand that you are not building software for today, but for the next 5 years at least.
Infrastructure, by means of any hardware or service that you need to run your software, today is quite cheap for any mid-size company. However, teams that need to handle a lot of infrastructure, suffer in terms of value delivery, because understanding how third-party services communicate and interconnect between them is harder than just a single project with code.
For example, cloud architectures in AWS that depend on several lambdas, S3 buckets, IAM Roles and SQS queues can be cheap (depending on the load) but managing and understanding them for a team is hard. A guideline that I usually have for the design of my software is: a person should be able to understand the solution in less than 1 week and be able to commit in the second day.
There are several team formats and patterns (I recommend the book Team Topologies, which is amazing) but for me the most important decision a team should take around their own organization is how they communicate within the organization.
Most of the times, teams are not alone in an organization, they need to communicate with other teams and stakeholders, and this communication can lead to a lot of trouble if not handled correctly. Helping teams to decide how they communicate based on their needs is as step towards success.
Thanks for your feedback!
Hey been reading a couple of your articles and I'm very interested to hear more from you!
We're struggling with scaling from a < 50 person engineering org to 100+ in a year. Fast start up growth. We already have 2 monoliths and have been talking about splitting things out for a while.
1) As someone who's new to the team and backend development in general, what are some tips for getting a grasp on monoliths, understanding business domains, etc?
2) Can you clarify this line for me? "Don't make cheap code, make code cheap." Maybe an example of two would help.
Hi Daniel! thanks for your feedback ๐. Let me answer your questions:
1) Understanding the business domain through code with lots of technical debt is a challenge that you don't need to face alone. Try to pair or do mob programming with more experienced teammates, make sure that everyone on your team has the same understanding of the business rules (also your PO, don't forget about him/her). Document those rules somewhere.
If you have someone experienced, I would suggest to do several event storming sessions to document and understand business behavior. Usually event storming is used to build new features, but in my experience it works quite well also to share business knowledge.
2) In my experience, one of the biggest issues when refactoring or splitting a code-base is premature code reuse. Most of the times you will be facing code that is reused across components (usually this code is shared through a static method/class, a base class, or utility functions). This means that you need to decouple first from the shared code, and then fix the issue or extract the code to somewhere else.
To avoid this issue, do not refactor to reuse code, refactor to make code safer. Safer means that is easy to change, is observable (good logging, good metrics) and is easy to test (if you do TDD, this will be far easier).
A few recommendations:
Avoid utility classes, if you need to use the same code, use a Value Object. It's cheaper to test and change.
Avoid abstract/generic/reused components. If you are facing that you need to do the same thing (like accessing to the DB) from different several places, find a way to fix the issue with your framework (for example, with Spring, with an ORM or a JDBCTemplate). If it's not possible (most of the times it's possible) create a library that provides this capability and decouple it completely from your business logic. If it's in another deliverable, better, so it's harder to break the boundaries.
Test the application. You don't need automated tests for everything, but at least a test plan. If you can have functional blackbox tests for the happy path, so it's easier to extract, I recommend to write them.
Hope I helped!
Thanks for reading!
Sorry for the very late response. Just saw the post. I believe you are correct. Unfortunately, today's business culture is very biased towards short-term profit over long-term gain.
I'd like to get your opinion on this article:
https://medium.com/nerd-for-tech/composable-services-ec1ced75a8dc?source=friends_link&sk=c0a365d02ff49b843b4b5b028c7a5b68
Thank you.
Cool perspective. Can you elaborate on "Methodologies should be lead by culture, not otherwise"?
Hi Abe!
I see methodologies as patterns, and they are really useful but they are usually misunderstood and can lead to wrong practices. For example, Scrum is actually good, but the usage of Scrum without understanding the principles leads to micromanagement and burn-out of teams.
I personally seen working better to share a common understanding of what principles should the team follow, probably those principles can be chosen by the own team, based on the organization guidelines. Later, the team can decide the methodology and adapt it to their needs (there are workshops, like Roles and Expectations: funretrospectives.com/role-expecta... and an inception).
What do you think?
Ouuuuhh, every sentence describes our Monster application and every sentence forcing me to change it. Go deeper with topics.