Technical debt: we need better communication, not better metaphors

Technical debt as a metaphor is not serving our profession well. It was meant to help us talk to business people and make better decisions about our software projects. But it has largely been a failure. Part of the problem is that business people aren't afraid of future interest payments.

In business school, I learned about the power of leverage: what it is, how to get it, how to measure it, and how to manage it. Debt is a tool. My fellow business school grads are comfortable with leverage and debt. So when the programmers come to the business people complaining about technical debt, the business people are unconcerned. And why would they be? The metaphor is misleading; technical debt is only superficially similar to financial debt.

What's at stake

I've noticed that business people tend to be better at convincing programmers to damage the long term health of their code base for short term gains, than programmers are at convincing business people about the importance of having healthy software with few messes and low levels of technical debt.

This is really unfortunate for three reasons:

1. once you allow your project to turn into a mess, it's really difficult to clean it up (in many cases it might be impossible)
2. the decision to take a shortcut or make a mess for short term gain is very often sub-optimal in the long term. If the business people truly understood the risks and costs of these shortcuts, they would likely make better choices
3. trying to maintain and extend a software project full of crappy code is frustrating, soul-crushing work. Many programmers quit their jobs to get away from low quality code--it's that serious

No matter how much technical debt your project has, you can always make it worse. You'll always face both internal and external pressures to take a shortcut here, avoid a refactoring there, and release that new feature without automated tests.

Therefore, we need to do a better job at defending the code with the business people. It's their job to advocate for the business objectives and it's our job, as programmers, to advocate for the code. Don't believe me? Read this: Software Engineering Code of Ethics and Professional Practice

How to advocate for the code

You need to educate stakeholders in your company about the software development process. Your job is not done until you are sure they understand the trade-offs and risks involved in taking shortcuts. I'm going to use the remainder of this post share my tactics.

Find a hook to get business people interested in how your company creates and maintains software

You need to get the business people involved in your process. The more they internalize your struggles and challenges, the better.

Appeal to efficiency

You'd be hard pressed to find a business person who didn't want to discuss an opportunity to have the software developers deliver:

• more features, more quickly
• fewer defects
• faster response time to requests for changes
• more predictable release schedules
• lower development costs
• lower programmer turnover

Your general thrust here could be that your company is missing out on high value opportunities. And that you'd like some advice from the business people on how you can capitalize on those opportunities and make things run more smoothly/efficiently/effectively.

Appeal to risk avoidance

Depending on your audience, you might find another approach to be more persuasive. Business people are trained to manage risk and you can use that to your advantage by framing the situation as one of managing increasing risks.

Here are some risks of making messes in your code:

• competitors release superior software
• defects soar
• costs soar
• unable to meet regulatory requirements
• unable to add new features
• progress grinds to a halt
• data loss/corruption/breaches
• lawsuits/fines
• lose customers
• bad publicity
• lose use of the software completely
• good programmers leave the project
• and here's the biggie: risks interact in unpredictable, non-linear ways

One of these two strategies will likely get the attention of your business people.

Get them involved in your meetings

The best thing my team ever did was adopt SCRUM. Our product owner really 'got it' once he attended a few of these meetings. He also became a captive audience for an hour or so per week during these meetings. I took full advantage of the opportunity to increase his understanding whenever I could.

These meetings make abstract ideas about 'going slow' or 'bad code' more concrete. For example, we look at all the stories where we missed our estimates by a wide margin at our retrospective and talk about why that happened. But it's quite another thing to listen to a programmer take 20 minutes to explain all the reasons why a simple change to the system took 40 hours (10 times the estimate). When that happens sprint after sprint, the business people can't help but buy into it. I find it especially helpful if you get them involved in the root cause analysis and help you formulate solutions to the problems.

The meetings also allow the business people to get a sense of the team. How do the programmers handle themselves? Are they competent? Are they complainers? How's the morale? Are they lazy? Do they care about the business? The answers to these questions aren't nearly as important as allowing the business people to reach the inevitable conclusion that 'this programming thing' is harder than it appears from the outside.

Make a distinction between technical debt and crappy code

Technical debt has expanded from its initial definition to include almost any undesirable quality of code. At my workplace, we are resisting this trend. We make a distinction between technical debt and crappy code.

Technical debt:

is a deliberate decision to take a shortcut to achieve a short term goal at the expense of an unknowable long term cost.

Crappy code:

is any other kind of shortcut that is not technical debt.

It's our policy to only allow technical debt into the code base as a result of a considered decision made by the business people and the programmers. Our software is pretty mature so we only allow technical debt into the code base to deal with emergencies. For example, if we discover a critical defect in production, we might 'hack' in a solution to resolve the problem immediately and call that technical debt. And then we'll make a story to fix it properly later.

Whereas we allow a limited amount of technical debt into our code base, we have a zero-tolerance policy for crappy code. We made this policy after a long series of discussions and root cause analyses. We decided that the only reasonable way to move forward with our project was to stop allowing crappy code into the code base. To that end, all code is reviewed and it doesn't get merged with master until it meets our quality objectives (see our code review checklist). If that means a change takes ten times longer than we originally estimated, so be it.

Of course, it's not as black and white as that. Programmers make judgment calls every day. But our talk, actions, and behaviors are all aligned towards the production of zero lines of crappy code. We've made it part of our culture.

Show them the research on the cost of crappy code

Business people generally have no idea how bad crappy code is for the health of their software. So, I've made it my job to educate them.

I lent my copy of "The Software Project Survival Guide" (Steve McConnell) to two senior people in my company. This book is getting dated but the first 5 chapters are absolutely worth their weight in gold.

I show anyone who will pay attention to me page 4 of "Clean Code" (Robert C. Martin). On this page Uncle Bob describes the total cost of owning a mess. And I especially love pointing to the graph at the bottom of the page (reproduced below) and asking them where they think we are on it.

I also show people my copy of "Facts and Fallacies of Software Engineering" (Robert Glass) and use it to stimulate software engineering discussions.

I advocate aggressively for our code, our team, and our sanity. I'm really trying to get three points across:

1. developing software is hard, expensive, and prone to outright failure so we should take it seriously
2. as a profession, we have some idea about which practices influence the risks, schedule, quality, and cost of software development
3. we should adopt the practices associated with the outcomes we want

Show them examples of crappy code in your project

It's pretty easy to convince people that crappy code is bad in theory. But it's another thing to convince them that they own crappy code.

I've gone as far as literally emailing screenshots from my IDE to our senior people. I found one email I wrote last year that had screenshots showing:

• all the static analysis errors for one of our projects. It showed over a thousand serious warning and errors and thousands of weak warnings and notices.
• the static analysis errors for a single 3,189 line file containing 5 languages and over 500 warnings
• all the errors and warnings for a single file flagged down the right side of the editor window. It was 95% covered with warning indicators

I wrote that email to:

• make our code quality problems visible
• show why our code is so expensive to change
• and advocate for not making the situation worse

Quality and speed are not opposites

I love this one. Steve McConnell published an article titled Software Quality as Top Speed. He argues that:

In software, higher quality (in the form of lower defect rates) and reduced development time go hand in hand.

This article is well argued, written by an expert, and it is backed by research, which is clearly referenced in the foot notes. Read it. Bookmark it. And realize that it is a powerful weapon in your arsenal to combat suggestions that your team would go faster if they'd just stop doing so much QA or upstream planning.

Bend over backwards to make the business successful

All your efforts will be for nothing if the business people think--even for a second--that the best interest of your business are not your top priority. You can't just say that you aren't going to accept any technical debt from now on no matter what. You can't say you are going to do things "the right way" from now on and they'll just have to get over it.

You're all on the same team. And your objective is to make as much money for your company as possible. So when the business people say they need a new feature in 5 weeks and you think it will take 20 weeks, you (collectively) have a problem that needs solving.

And you, as the software expert, have plenty to contribute. Why do they need this feature? Do they need the whole feature all at once? Could it be broken into phases and you just deliver the first phase in 5 weeks? Could you change the spec to make the feature easier to implement? Could you build a completely different feature in five weeks that delivers the same sales and profit impact? Could you integrate with an API and cut your development effort? Could you deliver a minimal version of the feature in five weeks and follow up immediately with an enhanced version? Could you outsource some of the work?

There are lots of possibilities besides hacking something together, and kicking it out the door. And if you've been laying a good foundation with the other steps, perhaps you can keep things from coming to that.

Act like a professional at all times

You need to act like a professional if you want to be treated like a professional. Whining and complaining are not the behavior of professionals. But that's not all there is to it. You need a plan to manage your software development process effectively and deliver results. You need to make honest estimates. You need to hold yourself accountable for the outcomes of your team. You need to be truthful and avoid exaggeration.

For example, I avoid blanket statements about our code base. Some code is bad, some isn't so bad, and some is actually pretty good. So I never say that we can't make a change. But I will say that this change would require substantial modification of that really ugly module without tests I was telling you about last week. The original author is long gone and we don't really know how it works. I would say this change would take a many months of effort; would you like me to work up a detailed estimate?

In a case like this, my estimate would include a range that communicates the uncertainty involved in a change of this magnitude. My estimate for this change might be 3-8 person months with 85% confidence. That means I think there it an 85% chance that we could deliver this change in 3-8 person months and that there is a 15% chance the actual effort could exceed that range on either end.

And no, I can't do better than that. That's the estimate. We can create another estimate if we modify the scope of the change but I always give my best estimate the first time. And I don't allow myself to be bullied into lowering it.

The real wins come when you understand where the business people are trying to go and you can make suggestions and plans to help them get there faster or at least avoid the land mines along the way.

When all else fails, you might have to quit

Sometimes you just won't be able to convince the people you work with that anything other than hack on top of hack on top of hack is possible. They don't care about software engineering research. They don't care about best practices. They don't care if you have to deal with crappy code. They don't care if your life is miserable. They don't want to talk about how you develop software. They just want results, and the cheaper, the better.

If you find yourself in this situation, you have some decisions to make. Let me just say two things. First, there are companies that actually care about the quality of their software and the morale of their developers. And second, high quality software developers are in huge demand so you shouldn't have too much difficulty finding another job if you wanted to make a change.

Wrapping up

When programmers add crappy code to a project with a long life expectancy both research and hard-won experience show they are almost certainly taking on more cost and risk than they realize. You'll face all kinds of internal and external pressures to deliver features faster. But always remember it's your job to advocate for the best interest of your code and your business.