I'll start with two questions:
- Is complexity inevitable in software development?
- How much of this complexity is our failure to curb "accidental complexity" which we are the ones to introduce to begin with?
Let us use a couple of popular tools as examples: Rust and React.
For what I've gathered Rust has has been developed out of need. The need to manage systems programming, manual memory management and the development of software that sits very close to the operating system. Its complexity is intrinsic to the problems it had to solve initially.
React has been developed out of need as well. The need to manage a massive frontend code base at Facebook. Its complexity is intrinsic to the problems it had to solve initially.
As any software developer knows, tools intended for a purpose are easily going to be used for anything else but that initial purpose. It's normal, we're humans. It's not like Rust or React creators have a direct line with all the developers out there.
This goes back to the second of my initial questions: is a tool developed and managed by a company, worth several hundreds of billions of dollars, to handle a massive frontend code base going to be accidental complexity when applied to the 99.99% of projects which are infinitely smaller?
If you're not familiar with the difference between essential/intrinsic complexity and accidental complexity I suggest you this post:
Is the push towards React's adoption only a positive or could it have effectively hurt some companies business over the years?
Is using Rust for your average web application a good idea? Is using Rust for anything other than systems programming a good idea? Is using React for anything other than a complex frontend a good idea?
We all know how the wrong tool chosen at the wrong time can affect a company's prospects. So, when people choose a tool before having written any lines of code, are they betting they will be able to manage the complexity it brings or are they really making an informed choice?
Evidence tells us you can build successful products with or without either Rust or React, so that's a given but the problem, as always, isn't in the tool per se.
I think Nat Allison here has a point:
Nat Alison@tesseralis
I've talked with Dan about React's philosophy. React is built for *massive scale*. Facebook-level scale. Sure, it's perfectly fine for your blog or cute app, but it's a tool for megacorporations to influence the world at scale.00:24 AM - 15 Aug 2020
Are we software developers simply bad at making choices for a thousand different reasons?
Top comments (11)
Accidental complexity feels inevitable because we don't have perfect knowledge/understanding of the problems we are solving. We can really only get close to having that after we are done with a project, when it is too late. Just like life... you only know what you should done at the end.
But some approaches seem to grow accidental complexity at a faster rate than others.
Any examples?
Far and away the biggest is focusing on the tools instead of the customer's problem. Accidental complexity ramps up quickly. Been there. I still remember coding and thinking things like... "I don't understand why they want this." or "This requirement is going to be awkward." And instead of asking about it, I filled in my own assumptions. And it didn't turn out well.
Tl;Dr isn't this more about the imperfect 'celebrity' tools you know to exist than the tools that are perfect that you've never heard of?
I don't have any good answers, but I do have a point that adds complexity to your initial query. A 'meta-complexity' if you will.
Do projects like rust and react have extra 'mind share' because of the success of Mozilla and Facebook, and so it's harder to be unaware of them. As a result, we echo chamber the existence of the tools more effectively, and so increase the odds of any one project choosing these technologies for a problem Vs a more home brew, but arguably more suitable technology that exists but has significantly lower awareness around it, as the developers are working on it in their spare time, or its backed by a really small company no one has heard of?
Sure, it's also definitely a problem of reach. But that's true for everything. I cannot care about the perfect tool that's out there if I don't know that it exists. It's a tautology. I do not have opinions on things I don't know I should have opinions about :D
Staying the realm of the known, in my final question, the "a thousand different reasons" I include many of the logical choices (as in "of logic") we make factoring in: time to market, the ability to sell something to colleagues or managers, familiarity, marketing, time itself (as we can't take forever to make a choice), the perceived malleability of software ("well, if it's the wrong tool, I'll change it down the line"), "wisdom of crowd factor" and many others.
I too don't have an aswer. I wonder if it's because there simply isn't one.
I think you're probably right, all the way through :)
PRESENTATION "7 minutes, 26 seconds, and the Fundamental Theorem of Agile Software Development" by Joe Rainsberger, from the Agile Lightning Talks (2013). Which I link here because he talks about accidental complication and how to minimize it. He intentionally uses the term complication rather than complexity, but even so it is within relevance for this discussion. And I really like Joe's talks, and I think his messages are on point. And he says quite a bit in a mere 7 minutes and 26 seconds!
One aspect that was not mentioned yet: I would argue that developers tend to worship complexity (as explained by Eberhard Wolff). Either it's because we as developers simply don't know better but to use the tools we've already worked with (which seems like accidental complexity to me) or because we are arrogant (from the linked blog post: "Our challenges are much greater than those of Amazon or Google.")
Adding this thought from a proper rant: I want off Mr. Golang's Wild Ride:
Here we have a language, like Go, who's marketed as simple and beginner friendly (which it is, it's not a lie) and by its design and its choices inevitably has to hide some complexity. What happens, according to the author and other people he links to, is that some of these choices are made at the expense of correctness.
Is simplicity even achievable if we decide that complexity is inevitable? I think this is why language design is so hard. Software itself is an attempt of managing complexity while trying to solve a problem.
If we say that Go's designer did a terrific work at hiding complexity (and they did, I'm not disputing that), is Go's blanket too short or at least not big enough for stuff to crash and burn when you start exploring those borders?
If you're familiar with Go, the entire discussion around error handling and interface management vs generics is a discussion around simplicity of usage vs complexity of domain.
It takes a few lines of code in Go to make a concurrent program, but it takes roughly the same amount of code to make an incorrect concurrent program.
Where does the real issue lie here?
Every Railsconf keynote DHH gives is generally a rant against software like React. Regardless of individual preferences for certain APIs, I think projects are going to be built to favor the highest needs of its main decision makers.
If I think about the needs of big Rails decision makers, folks like Shopify have certain obvious needs to make the software work for them, and then DHH's camp's needs seem more in the realm of need to be righteous about certain software development topics. This generally seems like a good thing, but it's also just the way things are. If it ever weren't a good thing it would be hard to distinguish.
To speak to the work at Forem, our software has a certain complexity associated with its needs to power a public, global community like DEV— But I'm excited about the forcing factors of smaller communities. DEV needs to be a "powerful" engine in a lot of ways to deal with the complexity of human scale, but smaller communities need more simplicity than power in many ways, especially closed communities which mostly trust all the users implicitly.
If we resigned Forem to only being designed for communities at the scale of DEV or bigger, it might allow us to accept extra complexity or bloat.
True, one of the other aspects in the usefulness of tools is the ability to scale down. But that's probably the center of my argument and it's probaby tangential to the concept of "silver bullet". If a tool that's able to scale up, down, left and right is essentially a silver bullet, so if silver bullets in software are a chimera, aren't we the ones that are bad at labeling those tools?
I quote myself here:
I should have added: "If so, does it matter, if the choice is good enough?"