Explain waterfall project management, why did it fall out of style?

Waterfall assumes that the project is complicated, ie not obvious, but certainly understandable, and predictable, and can be addressed linearly. As a consequence of this belief, it comes up with solutions that are totally reasonable in complicated domains. For example, construction is (to my knowledge) a complicated domain, so you might draw the blueprints, construct the structure, get the structure inspected. For software, that might be to specify the code (often called "architecture" and involving UML), hire the coders to implement the code, have the testers test the code. Seems totally reasonable.

The problem is that most projects are complex, not complicated, and this is especially true in software. In a complex domain, there are so many variables that your ability to predict how one thing is affected by another is very limited. Once you've solved a problem, it's fairly easy to look back at the path and see how you got to the solution, but from the starting spot, there may be several equally promising approaches to solve the problem. In a complex domain, you can't plan out a path to a solution, you may not even know what the solution is!

For example you want to make a Slack (internet chat) bot that lets one user send a link to another user. Turns out there's several ways to do almost that but no way to do exactly that, so the feature itself must change: either the link works for any number of users in a channel, or can only be used in a private 1-on-1 conversation. You sit down to format the bot's response thinking you're halfway done but in reality you're about to discover that the project has to pivot because of details in their API.

Now imagine you spent 3 years planning this application and the last 2 years coding, you've got another year left before the testers start testing it when all of a sudden, oops, turns out it's not possible because of this tiny little detail at the bottom. 5 years, tens of millions of dollars, hundreds of people, not a single user, the code never ran once, and finally the whole project gets scrapped.

This happened enough times to enough people for enough cost and frustration and death marches to meet release dates and hacks on hacks to just get it done and drop the mess in someone else's lap that people began to look for other ways of doing things. People who were exploring different ways of doing things wound up converging on some common practices, so they went up to the mountains and wrote a manifesto agilemanifesto.org with twelve principles agilemanifesto.org/principles.html

Since it addresses a real problem that many people were really experiencing, the adoption was "wildly in excess of anything [they] expected" (techbeacon.com/uncle-bob-martin-ag...).

I think it depends on what bubble you're in. Our org uses agile at a certain level towards the operational / technical side of things but at the mid to higher levels waterfall style planning is very much still a thing.

If you're a small startup with a lot of room for error and a decent cash runway, agile is fine, because there's many tolerances to play with there. If you're putting together a multi-billion dollar industrial installation and all of the associated systems (ie, systems engineering proper), at some point very large chunks of work with sometimes 5-10 year leads times need to be co-ordinated using processes that provide stable and predictable interfaces fairly early on.

Even if you completely ignored "waterfall" vs "agile" as methodologies you'd wind up with something very waterfall like when trying to bring together internal resources, EPCM's and the likes over larger timeframes / multi-million dollar budgets with the associated risk at that scale.

Note, this isn't a bash on agile (I use it for most of my recent projects and teams, and think it produces better outcomes), but it doesn't scale up past a certain level of cost and complexity.

I would argue it was never in style. One of the first uses of the phrase "waterfall" was by Royce in 1970, when he held is up as a strawman of the obvious-but-wrong solution.

It keeps showing up for two reasons:

• it maps easily to the methods that people use to produce physical goods
• it maps easily to software-produced-under-contract

Here's one of my favorite episodes of To Be Continuous that talks about this subject in depth. It's a pretty brutal teardown of Waterfall vs Agile and the concept of Gartners Bimodal IT model.

heavybit.com/library/podcasts/to-b...

Waterfall is a straw man argument formally described in an essay written by Dr. Winston W. Royce in 1970 to describe a model of development which has the highest possible likelihood of failure.

cs.umd.edu/class/spring2003/cmsc83...

Companies nowadays, specially the new ones, try to follow "Move fast and Break things." strategy above everything else.
They want to do maximum in the minimum amount of time, with great efficiency.
Waterfall project management just doesn't fit well with this mind set.
Waterfall management technique assumes you have a proper SRS document and everyone is well aware of each and every feature or thing that they are supposed to be working on, many months in advance. But the speed with which new companies are forming today and just the sheer amount of advancement that we are seeing today, begs us to move fast and deliver our products as soon as we can. Therefore, waterfall project management is falling out of style, at least for the start ups and many big and old companies as well, who want to stay in the running. And agile process is taking its place, in which you get the basic requirement and you just start working. The requirements may change at any point of time and you would have to incorporate those changes in your delivery plan.

The basic problem with so-called waterfall software projects was that it took too long to find out if you were actually producing the correct product and to know when it would actually be done. The idea that you could gather all the requirements up front, throw them over the wall to the system analysts, who would then throw their design over the wall to the programmers, meant that no actual product was seen by the client until a lot of work had already been done. Once it finally got into the testers' and clients' hands and something was found to be systemically wrong, schedule and cost overruns were inevitable (and common).

Pipeline is usually way too long. A short feedback loop is critical for "getting it right".

They say you're up shit creek for a reason. It all flows down the stream. Pitty those at the bottom.