Uncertainty: Imperfect or incomplete information

What is uncertainty?

If we step away for a moment from software development and we try to define what uncertainty is, as a concept, we may come up with tons of definitions and synonyms, starting from the obvious, lack of certainty, but also we could go through terms like vagueness, unknown, unobservability, stochastic(random), undefined, unstable, variable, undetermined, unreliable, insecure, doubtful, ambiguous,… all of these terms are states that we cannot describe or outcomes that we cannot predict. Therefore, we can assume that the uncertainty for every situation is determined by two categories:

• Descriptive complexity: the difficulty to make sharp and precise definitions. (current state)
• Complexity by unknown: when selection between one or another option is unspecified. (future state)

When we come back to software development, we still have the same problem when dealing with software projects. We have to deal with uncertainties every day, and most probably, now even more due to all the corona situation. When we have to make decisions and assume risks, we don't like uncertainty, we like things crystal clear, but not always is possible. When uncertainty is around, it is more difficult to make the right decisions, and not just that, given our human condition, we also are bad guessers, so it makes it almost impossible to determine or predict correctly any situation possible. So, what would we do?

A bit of history

In order to have a bit more context and understand the problem better, let's go back to the early days of software development, back then what we did was simply forget the uncertainty. We created processes, methodologies, and tools that completely ignored that uncertainty was there. At the beginning of our young industry, the very first programs provided solutions to specific and small problems with all the requirements and expected outputs defined upfront. They could do that, in the end, they were just a small group of geniuses and other clever people, very methodical and disciplined. For a short while, we created software to solve specific and well-defined problems, but that didn't last much. The problem came later when the software industry grew and expanded literally everywhere and we had to make it work in one way or another. Now the problems were not so easy to model and they were becoming bigger and bigger. In addition, more and more people were entering the industry due to the huge demand, and not all could reach the same levels of excellence that we used to have in the past.

As the sector was growing, we needed a way to control and organize ourselves. So we brought the ways of working and organizing from other industries, such as top-down hierarchy from military or waterfall plan-execution from construction. After all, plan and execute seem like a reasonable thing to do, and in the same way, having the more clever or more experienced people managing and commanding average people makes total sense. We (as an industry) blindly went for this way of working, and after a while, we even adopted complex models like CMMI, so we could evaluate and improve our waterfall development process in our top-down organization.

But it wasn’t until after thousands of companies were broken, millions of projects failed and billions of money were wasted that the industry started to realize that maybe this way of working was not appropriate. Uncertainty is not something that you should ignore, it is something that you should embrace and build your product around it. This is what all these gurus in the Snowbird ski resort signed up for, a better way of building software. It was the agile manifesto that came out from this reunion. We want to build the right product and we want to do it assuming risks of hours or days, not talk about months or even years, that is too dangerous and difficult to make it right.

It turns out that whatever you are trying to build at your software-based company, it is the first time that it is being done in humankind history. This fact makes it impossible to foresee what is coming next for your business, it is normal to have uncertainty there. Nobody went as far as you did, and if somebody is doing something similar, either what you are doing doesn't have any economical interest or: congratulations, you just found your competitors!

Now is when the real talk starts. You have to openly accept the situation that you don't know everything and start working in a way that let us beat our competitors. You don’t need to stop avoiding uncertainty and start embracing it to take the best out of it. This is when Agile methodologies come into place. Agile is meant to develop software around this uncertainty. Organizations do not adopt agile just as a new and cool way of organizing work and managing people. It is way too expensive, we have all these new payrolls from agile coaches, scrum masters, product owners,.. all the time spent in meetings, or even all these new tools that we have to learn and pay to use. Organizations adopt agile to use it as a competitive advantage, if that is not clear for your organization, maybe you should reevaluate your investment.

Agile methodologies, like scrum or kanban, will let your organization build a product clearing the uncertainty that exists in your business. They will go iteration after iteration, collecting feedback and introducing it to improve the product, figuring out the needs of your clients/customers and how your product covers that need. Is this quick feedback loop that agile focuses on what gives your product this advantage in front of your competitors. As this is a competition, the first company that clears the uncertainty and is able to identify what customers really need and want to pay for is the one that will win the prize.

Experimentation Culture

The experimentation culture should be a thing in every agile team. It is by building the product that we continuously learn how to build our product, and also how not to build it. Experiments should take place during the whole project lifecycle. They could be more close or less to the scientific method, more formal or informal, longer or shorter (always timeboxed tho), with simple or complex goals,... But in order to discover which is the best solution, in order to clear the uncertainty, we have to investigate and try out the different options.

Usually, when talking about experimentation, the first thing that comes to mind is an evaluation with users of a new registration form or an A/B test of a certain UI component, but the reality is that it can be applied in any stage of the project lifecycle. Starting from the beginning, did you dedicate the proper effort when defining the navigation between screens? How many prototypes did you build before making the final decision? At team level, the dynamics, the communication, working agreements,... are the right ones? The format and outcomes of your meetings are correct? The tools we are using help us to do a better job? Did you choose the right language/ database/ framework for your problem? Or the only one you knew? Or maybe was the first you found on google? Did you find a better way to implement something but it was already too late to go back? How painful is having to live forever with bad decisions from the past? Take a minute or two, and think about the impact of these decisions you made. Would have been worthy to spend an extra day or even a whole extra week before making that decision? For most of them, you can still run an experiment and try to improve them.

Running experiments it's similar to investing. In order to win, you have to play in the first place, but also, you have to be ready to lose sometimes. Not all experiments end up with huge improvements, some even just confirm that there is no possible solution for that hypothesis you had. But even in those cases, it should never be considered as a failure or a waste. As mentioned before, part of learning how to do it implies that we also learn how not to do it. The experiments are the price you have to pay to obtain certainty in the project.

Uncertainty in code

We can have the right agile mindset in the team and the proper way of working perfectly agreed, but we still have to build and deliver the product, and we have to do it in the right way too. As developers, our duty is to build and deliver the software in a way that lets the business be agile. It is the business that needs to be agile in order to change and adapt to the requirements that appeared by exploring the uncertainty. We need to absorb this feedback in the product and we have to do it fast. This means that our software needs to be flexible, easy to change. Any piece of code should be easily modifiable in a matter of hours if not minutes, we can never talk “in weeks or months” when changing a behavior in our system.

Not tested code == Legacy code == Uncertain code

The only way to achieve speed and flexibility when building and delivering software products is by having a high-quality product, a software that is a cohesive and low coupled piece of code. We have to continuously pursue technical excellence, being disciplined, and taking the best use of the practices. The biggest problem of agile methodologies is that they focus on the process and organization and do not talk about technical disciplines or practices, they take it guaranteed. When we create a piece of code that is not covered (100%) by tests, we are introducing uncertainty in the system, we don't know how that part of the code behaves. If we don’t have a fully automated pipeline, we don’t know how long it will take to deploy. If we don’t estimate the user stories, it makes it impossible to come up with any realistic plan,... We do not do TDD, CI, plan games(like poker), or any other of the best practices and disciplines just for the sake of doing them, we do it because it is the way to achieve high-quality software that is easy to change and fastly deliverable.

If you want to keep reading about this topic, check out my article about the fake agile.

Wrapping up

Instead of running away from uncertainty, instead of avoiding continuously going into the uncomfortable zone, companies and more specifically, product teams have to deep dive in there. Exploring uncertainty creates certainty. Agile provides an excellent framework to explore the unknown, minimizing risks, going one step at a time and validating whether the previous step was right or not, so we can adjust direction for the next one. Is by clearing up all the uncertainty in short iterations (closing the feedback loop) that we can build the best product possible that satisfies the needs of our clients before our competitors. In order to achieve that from the business side, the technical side always needs to be ready to adapt, chase the technical excellence, to build software that is easy to change and fastly deliverable. Is the combination of both, business and technology, working side by side that leads products to success.