Beware the Swift Waterfall

From everyone who has been given much, much will be required, and to whom they entrusted much, of him they will ask all the more.

---

Every software developer will come into contact with agile development at some point in their careers. For me, it came early when I read Practices of an Agile Developer as part of my coursework in college. I was immediately convinced that agile is the right way™ to organize software development cycles. The engineers organize a backlog, plan the work in achievable intervals, and then... just work on it. As technology changes and the new new thing™ appears, your team evaluates and incorporates the good stuff into their designs and training, leaving out the yucky, bad stuff. At the end of every interval, the team gets back together to discover what worked, what didn't, and how to avoid those problems, permanently solving them forever. Over time, the team tightens the development loop until they can write any feature in twelve microseconds.

"It's foolproof and cannot fail," I thought.

---

Then, in 2017, I graduated from college and went into the real world. During my junior year, I accepted a job with an unnamed but easily google-able defense contractor in Arizona. I showed up on my first day, bright-eyed and bushy-tailed, ready to write code with more agility than anyone had ever seen. I had completed the requisite Agile training, and was prepared to tighten the heck out of every development loop that came my way. I was assigned to a team, and told to show up a certain lab. This was it: it was time to check the backlog and pluck out a task. My moment had finally arrived. My first task was... to write a software design document!?

"But where are the software tasks?", I asked the software lead.

"We are preparing for our first software requirements review, so there is no code yet. We're planning on using some older code to bootstrap development for this new project," he said.

I soon found out that the "Agile" training I had done was for everyone, not just software engineers. All of the systems engineers used "Agile" to create their requirements documents, program leadership used "Agile" to interface with the product team leads, even the department managers and VPs were roped into some "Agile" ceremonies. Everyone had daily stand-ups. Everyone had planning poker and backlogs. I started to wonder if even the janitorial staff had their own backlog for cleaning the johns.

To many program graybeards, agile was a four-letter word.

The Software Design Document (SDD), I also found out, is a way of designing software "on paper" before any code is written. And I don't just mean that it's a proposal or RFC; this document has to pass through an extensive design review before work can begin. It also must specifically call out the Software Requirements Specification (SRS) to show how the software design will meet the requirements. So, obviously, the first step will be to acquire the SRS and begin wire-framing the design.

"Ok, so where can I find the SRS?", I asked.

"Oh, the system engineers aren't done developing the SRS yet, they are still waiting on the customer's input for the new system," the software manager said.

It suddenly dawned on me: this is just waterfall design with a different name. I recalled reading that top-down design was the killer of many-a-project, and I immediately understood why. Because the customer was slow in developing their own requirements, the system engineers couldn't complete their design of the system. And without the system design, how could the software engineers even dream of starting work? In less than one day on the job, I was a victim of waterfall design, crushed on the rocks below. The program didn't last more than a couple of months before the plug was pulled.

From then on, I always called this process "swift waterfall." The other programs I worked on used other terms and frameworks (e.g. SAFe), but the result was always the same. We had the daily stand-ups, but never removed the real roadblocks. We had planning poker and never seemed to estimate tasks correctly. We had SDDs, SRSs, TDDs, RVRs, ABCLMNOPs, code reviews, integration test events, postmortems, premortems, scrums, scrum of scrums, product team planning, integrated product team (IPT) planning, program planning, scrum master planning, product owner planning, safety review boards, failure review boards, critical design review, and a couple of team-building BBQ lunches. And every time, in the end, we were all dashed upon the rocks at the bottom of the waterfall as the program sank to the depths.

"How do I escape this?", I thought.

---

Tired of sinking with every ship I boarded, I tried switching departments to work on some adjacent software products. I was desperately trying to find some kind of purchase in this company, rather than being a continuous victim of program planning. I'm a bit old-school in my mentality; I don't like to switch companies for a bigger pile of shiny rocks as a first resort.

Through various machinations, I managed to become the scrum master (gulp!) of a new team, and ran it with my own philosophy: we need to be agile about agile. Our team was tasked with improving a legacy tool for the entire software IPT.

I knew the importance of gathering requirements as quickly and succinctly as possible, so we started there. My team interviewed every Subject Matter Expert in this domain we could find, including the original tool developer. We gathered their requirements and desirements, and immediately followed up with them on whether we got those listed correctly. We didn't. So we fixed them.

I knew the importance of a clean, actionable backlog, so I spent some serious time with the Product Owner turning the requirements into realistic tasks. She left me to then assign "story points" to these tasks. I told her that I would instead give her "done by" dates, and that the work would be done by those dates. She agreed.

Instead of story points, I asked the team how long they thought tasks would take in hours. I then took that number and multiplied it by 3. There isn't a magic formula behind this one, I just have a heuristic I named the rule of threes which has been the lower limit of every bureaucratic nightmare program I've ever seen or been on.

The Rule of Threes

Programs will always deliver at most a third of what was promised, at least three times later than expected, and at least three times the original cost estimate.

(As an aside, I coined this term when I saw California attempt to build a high-speed rail from LA to SF starting in 2008. It still is not complete, and likely never will be.)

It was immediately clear that these requirements would require far more time than we had available for my small team, so I broke up the work into multiple phases and planned out just the first one, which was our Minimum Viable Product (MVP). I divided the work by feature, and subdivided each feature by the tasks we thought it would take to complete it. As soon as I had the high level design written out in our backlog software, I tasked two teammates with prototyping the architecture. It didn't work exactly as I expected. So we fixed it.

We had our first prototype demo with one of the SMEs within a couple weeks, and he also blasted gaping holes in our design. So we fixed those too. And we repeated that process one monkey-bar at a time, grabbing onto the next one before letting go of the previous one. It didn't take long to pick up momentum and keep it going, one bar at a time.

My team was part-time, so we all had other main programs we were working on. But even with this limitation, we hit every planning goal we set. Instead of showing ambiguous story points to the Product Owner at our monthly review, I instead used a pie-chart to show what percentage of tasks under a given feature were done. Because we allocated a certain number of man-hours to each feature, she could tell in one glance exactly which features needed more work, which were nearing completion, and gauge our planning accuracy based on the remaining man-hour budget. She was so impressed that she had the rest of the program begin presenting their progress in this way.

To top it all off, we never missed a deadline. And at the end of the year retrospective, we presented our fully-featured MVP to the IPT at-large. It felt good to get something accomplished for once.

And then we all lived happily ever after...

Is what I would say if it were true, but unfortunately not. Shortly thereafter I was told by my new department manager that I had to move back to my old department because they needed me back over there, even though he had promised me I could stay in his department for at least a year. I was only three months in. And I had no say in the deal, because they needed me really bad. So I did what any normal junior engineer with highly-marketable skills would do: I updated my resumé that night and accepted a new job offer a couple weeks later.

Switching jobs has its trade-offs, but I now work for a company ten-thousand times smaller. I figured that meant they need to trust me with ten-thousand times more responsibility, and I was right. The good news is that there's no better incubator for development than adversity.

But no matter what the job is, or what the project is, or who leads it, the same principle applies: if you misuse agile, your progress will go over the edge of the swift waterfall in a barrel, never to be seen again.

---

Takeaways

• Be agile about agile. Only use the pieces of process which are useful, and forget the ones that aren't.
• Adapt process to your project, not the other way around.
• Triple your time estimates, because you suck at planning. Don't even pretend you don't.
• Don't let leadership bully you into breaking these rules.
• Don't bully leadership into leaving you alone; give them enough information for them to allocate you appropriate resources and align with your planning process.
• Make the most of what you're given, because that's where you have to start.