The Myth of the Spec

As a veteran of the software industry, it is always refreshing to talk to young people interested in becoming software developers. I first explain the standard perks of the job: typically, if you're decently talented, it can earn you the new American dream of a full-salaried, full-benefit position with decent wages, accompanied by that certain amount of autonomy and respect which comes from no one in the highest levels of management quite understanding exactly what it is you do every day, but knowing very well how much the continued success of the entire company depends on it continuing to happen with as few problems as possible. In light of certain socio-economic changes worldwide, it is now far more likely to also be a fully remote job than ever before.

At this point, they're possibly quite excited. Unfortunately, I can't let them leave with that overly rosy picture of the profession.

So I add the following: "And it's always a delightful challenge when a certain percentage of your job involves controlled cognitive dissonance"

If asked to explain this, I point to my go-to example: the prevalence of myths that form the operating capital of the planning of almost every software development team, but which are known (to those in the team itself) as far too simplistic to actually be true. It seems an open secret that this is a large part of what the software team is actually being tasked to balance: that spectrum between the ideal (what was assumed by management in planning) and the actual (the true task at hand)*

*All of these are based solely on my own experience. Your mileage may vary. These views do not reflect dev.to, etc. etc.

Today, I am going to discuss one of these myths: the myth of the 'spec'.

In theory, a "spec" (or "specification") refers to a full description of the intended behavior of a particular software system. The system is expected to work according to its specification. Simple, easy, clean. And a beautiful fiction.

In practice, this is impossible practically, and, in most cases, theoretically, as well. For any sufficiently complicated system that is actively being worked on, the "correct" behavior is changing on a day by day basis. Especially in modern agile practice, there is no spec of any given system. Instead, there is the concept of a set of updates being applied to that system (termed the "product") during the continual execution of sprints as the primary deliverable of each sprint, with any time actually allocated towards documenting the system behavior a happy accident.

Now, I don't want it to be thought that no documentation is being produced. Quite the contrary. A form of documentation is actually being produced at a consistent and regular pace: that documentation is simply not a spec in any meaningful sense.

For, as all devs know, the sprints of agile are all orientated around textual records called "tickets". These are usually created at the request of other departments or upper management, not the dev team. They are, in their core essence, a description in vague terms of what is desired from whoever is requesting the change, made by someone who doesn't really understand how the system works (since they're not part of the dev team). Further, it is well understood these are more "conversation starters" than concrete specifications. As the sprint progresses, these tickets move through various stages, are updated with extra comments that might drastically change what is actually implemented as the developer starts looking into the feasibility of what is being asked, and eventually, they are marked "done"; the process then repeats with a new ticket. Hopefully, each ticket is updated to reflect what was actually done in some way, as well, but that is the core process and documentation of modern software development: the effective management of these tickets.

At best, tickets can be described as the meeting notes towards whatever the spec eventually ends up being. Thus they come with all the accompanying issues that such forms of human communication inevitably entail: they are almost never technically precise enough to describe the intended behavior, may in fact present contradictory information as knowledge of the true task at hand is hashed out and compromises are made, and often bear only a weak connection to what was actually implemented in the final change made. Worse, these are only shallowly linked: a ticket might link to another related ticket, but there is nothing linking all the tickets that relate to any particular feature together, so simply finding the past tickets relevant to the feature you're working on is literally looking for a needle in a haystack.

From these, and examination of the source code history (hopefully the developer last working on the part of the system you're working on left good commits, or at least still works at the company) developers try to piece together how a system is supposed to work, constrained by a need to ensure they don't devote too much time to any one ticket, as they have been tasked with the mission of completing all of them in the timeframe previously committed to when they were assigned.

In other words, a large part of a developer's job is to, repeatedly, derive a partial spec from a system, under highly restrictive time constraints, and present it to management for the purposes of them confirming the partial spec is, at least for the moment, correctly understood.

Only then can they work towards actually fixing whatever that violation is (in the case of "bugs"), or implementing whatever the new feature is (in the far rarer case of "feature requests"), which is the part of the job that is deemed the "real work" of the developer. Developers might be often thought of as mechanics fixing a never quite working system, but they're really tasked with attempting to create the semblance of an impossible to achieve fictional order (a "fully working implementation" - i.e. 100% "correct") out of what they have to work with: bits of chaos (vague notions of what whoever asking for the change believes the system should do, in that given circumstance, without affecting the remaining 99% of a highly interrelated system which is ill-defined; all of which has zero guarantee of even remaining "correct" after the work is done).

As such, for any non-trivial concrete question about the system's intended behavior, usually no one really knows how it works or is supposed to work, no matter how "seasoned" they are in the project: there are no true experts. They couldn't possibly in a system that is undergoing constant change for varied reasons, especially when it is understood that every day a particular "ticket" is left undone is one more than desired. In the trilemma of quick, cheap, or well-designed, quick is always implicitly encouraged even if well-designed is publicly preached.

Until they derive that partial spec, devs simply do not have enough information to make accurate estimates. What estimates they can give are far better represented by relative measures unrelated to time, making the idea that a sprint represents a set of tickets that developers have confidence they'll complete in a given timeframe another too-rosy fiction.

That dev teams manage to complete most sprints with most tickets "done" anyway is a result of highly skilled and intelligent individuals following the business's implicit ask of "whatever it takes" to try to meet whatever timeframe they've been given. This also means software systems, rarely if ever, have any true "Architect" or "Architecture" as those terms are usually understood. In practice, this is impossible anyway, since most modern software development teams, even when"self-organizing" (as agile recommends), have little actual influence on what is implemented or how it is implemented.

At best, software development teams (including the development manager, who often has little actual power, and can best be thought as the cheery face of management to the dev team and vice versa) have the role of first paid advisor, who presents various choices to those who have actual decision-making power and, secondly, faithful executor of the actual decision-makers. The idea that the dev team makes the final decision is a convenient fiction assumed at times, but false on its face. No idea of any import is possible without either stakeholder request or stakeholder "buy-in". Of course, "healthy debate" may be preached, but all developers really know that is of a highly limited form evaporating almost instantly at the mere risk that some particular looming deadline might be missed (which is often).

If you've read this far, I'm opening this question up to my fellow developers in the private salaried sphere especially: what other "myths" have you found consistent across the various development teams and companies you've worked for?

Alternatively, if you work on an open-source software project not driven primarily for-profit, is there more of a place for design discussions when developing, or is it largely the same as the for-profit?