On Commercial Software Evolution

Cover image by Wikimedia Commons.

This article models the free market using Stuart Kauffman's Fitness Landscape to understand its influence upon, and from, commercial software products.

Bolded vocabulary terms are used across article sections.

Fitness Landscape

From Wikipedia:

In evolutionary biology, fitness landscapes or adaptive landscapes (types of evolutionary landscapes) are used to visualize the relationship between genotypes and reproductive success.

A few key points:

• a genotype is a collection of features intrinsic to some individual or population.
• reproductive success can be generalized as fitness value.

Here's an example of a fitness landscape visualization:

In this example, the 2D horizontal axes represent the genotype, and the 1D vertical axis represents the fitness value. In reality, the genotype may be more than 2 dimensions. It may be useful to imagine the genotype as an N-dimensional collection of various qualities.

From Wikipedia:

Genotypes which are similar are said to be "close" to each other, while those that are very different are "far" from each other.

This helps us compare different genotypes across the landscape.

Dynamic Structure

Ann evolution function defines the fitness value for every distinct genotype.

Here's one example of an evolution function moving and resizing the hills on a fitness landscape:

A few observations:

• The evolution function doesn't introduce any "cliffs" or "holes" in the fitness landscape.
• The evolution function is perpetual, and does not stop at any time.

We also see a collection of distinct gray species on the landscape's surface.

• Each species corresponds with some unique genotype at any point in time.
• Each species can move across the landscape by changing its genotype.

From now on, the landscape function is the evolution function that changes the surface of the fitness landscape, and the species function is the evolution function that moves species across the surface of the landscape.

---

The Commercial Software Market

The below translation is generally a gross over-simplification, but for the purposes of this article, it'll do just fine.

⬆️ You can ignore this table, it's just for reference.

Some key points:

• A software product is a collection of features
• Consumer behavior affects feature value, and transitively, product value

As with evolutionary biology, producers must evolve their software products to stay in business, or else they die. The ability of a producer to evolve their products over time is more necessary to their long-term survival than having maximally-valuable products.

For this reason, it's important for producers to have some control over their species function so they can remain in business.

Fun fact: some producers are able to routinely ship poor-quality software products, because their species function is just adaptive enough to keep them alive.

Since we're modeling producer and consumer behavior using evolution functions, we can explore the mathematical structure of these functions to better understand the market.

Producers and Consumers

Consumer demand evolves smoothly and continuously over time. Software, however, is shipped and released on a discrete time schedule. Consumers perceive spontaneous "jumps" as new features are introduced or removed from a software product.

Producers dislike this asymmetry, because a set of features under development may become less valuable by the time they are released. When producers cannot react quickly or regularly to consumers, the landscape function has outpaced the species function.

This is evident from the growing popularity of Agile Software Development, a methodology for producers to accelerate their species function and keep pace with the consumer landscape.

Agile Software Development

The four main principles of Agile, according to Wikipedia:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

The critical point is responding to change. Agile places a great emphasis on

• detecting change in the fitness landscape
• nudging your genotype towards areas with a higher fitness

In effect, Agile allows producers to be more confident that whatever features they ship will be in demand by the time they are released.

While this is generally effective, there are some interesting costs associated with having a higher-pace evolution function.

Technical Debt

This section is subjective.

For software, it is faster and easier to expand an existing product, than to build a new one entirely. Producers are generally suspicious of building new products in existing high-fitness areas, and will prefer to push an existing software product towards locally-fit areas whenever possible.

Even when expanding to new consumer markets (new "hills" on the fitness landscape), a producer will opt to Frankenstein-surgery their existing software onto that hill, and will only create a new product if absolutely necessary.

What happens is the species function slows as the number of features becomes larger. The high-paced species function that once benefited our species has since become cancerous, chaining the software product's ability to evolve with consumer demand.

Companies experience this as technical debt, and it's generally difficult to solve because of how immensely complex their software has become.

A Solution

If you've made it this far, thank you.

I believe, but am not certain, there is no "golden pace" for software development to avoid technical debt. I believe that every commercial software product will encounter it in some form, by virtue of the discrete evolution function.

I believe, but am not certain, that we can develop software without technical debt. And for as long as I strive to build better software, I will continue to believe that.

From Sofia Konobievska: How to Sell Technical Debt from a DevOps Perspective?.

Thank you for reading, and enjoy your next day.