My survey of the computer science literature suggests there are only two economical ways to achieve extremely low defect rates (< 1 defect per KLOC). The first way is to follow the Personal Software Process (PSP), which was created by Watts S. Humphrey at CMU. The second way is to use languages and tools that make it difficult to introduce errors into your code in the first place and easier to detect errors if you do manage to get some into your code. In this post I'm going to briefly discuss these two options and how I plan to explore them to become a better software developer.
But first, why do I want to do this? Because my experience building and maintaining software roughly tracks the data coming out of the research: defect-riddled code and all the testing, debugging, correcting, and rework that go along with it are incredibly wasteful. Our industry is overflowing with failure. There's got to be a better way.
I've already written at length on working on the most important thing, avoiding waste, code review checklists, optimal pull request size, and so forth. So, in this post, I'm going to focus solely on what it takes to produce software with very low defect rates.
Personal Software Process
PSP might best be described as the application of lean manufacturing principles to the activities of an individual developer. Programmers tend have poor insight into the various ways they make mistakes and create waste. They make the same mistakes over and over again and think they are awesome (me included). But the truth is that you are the one who is putting all those errors into your code, writing complicated code, writing unmaintainable code, and so on.
PSP teaches programmers how to track their work and their mistakes. And it provides programmers with strategies they can use to improve their performance over time. From what I hear, people who follow this process can improve dramatically.
Better languages and tools
It's very difficult to write correct code in most languages. They're ambiguous and they allow programmers to do things that are almost certainly errors. Things get more interesting when you switch to a safer language like Ada and/or its ultra-safe subset called SPARK. This fits very well with my idea that programming languages need to become significantly safer by default if we're ever going to improve software security.
Ada
Let's talk about Ada first. Strong typing, ranged types, contracts, and a really smart compiler help you avoid the stupid errors that languages like C/C++ and their derivatives ignore. So, if you choose to program in Ada, you're going to have an easier time detecting and localizing defects that you wouldn't even notice with another language.
SPARK
If you want to take things to another level, you can look into using SPARK, which is a subset of Ada that can apply formal methods to your code.
I know it sounds scary but I've been playing around with SPARK and I think it's awesome. First off, you don't need to know or do any serious math to use SPARK. SPARK can prove (or at least attempt to prove) your Ada contacts and type constraints automatically. If you just stopped right there, SPARK basically acts like that best static analyzer you've ever seen.
But you can go further. You can add assertions to your code and SPARK will attempt to statically prove that they will never be violated. How far you go is up to you but you can get your code--or some important parts of it--to the point that SPARK can demonstrate an absence of run-time errors. That means it can't crash from overflows, divide by zero, etc.
Why would you want to do that? Four reasons:
- Just like with TDD, writing your code in a way that SPARK can prove results in really well designed, simple, easy to read code.
- If SPARK is guaranteeing that whole class of errors can't happen, you don't need to write tests for as many paths through your code.
- There's a strong correlation between absence of run-time errors and the absence of logical errors. That means that code proved by SPARK tends to have fewer logical errors than code that hasn't been proved with SPARK.
- Once you understand what SPARK can do, you can stop thinking about those things when you are coding. It's similar to the way you don't have to agonize about your spelling when you write an email because the spell checker handles that for you. It's really impressive when you see it in action.
Just a word of warning
There's not much information on the web about Ada/SPARK (compared to more popular languages). And much of what's out there is outdated or just wrong. Both Ada and SPARK went through major changes in their latest releases that addressed some real pain points. Plus, Moore's law has made compiling and proving the correctness of your software orders of magnitude faster than they used to be.
My plan
I'm already experimenting with Ada and SPARK. This kind of programming is a little different than I'm used to but I found some good resources and things are starting to click (see resources below). I plan to build a few toy applications with Ada/SPARK solidify my learning.
For a bunch of reasons I won't go into here, it's unlikely that Ada will ever be a mainstream language. So, I'm not under any delusions that I'll get to write production code in it any time soon. But, I can already see that some of the concepts can be applied to any programming language with very little effort. And I'm looking forward to using some of those ideas to improve my code at my day job.
As for PSP, I ordered the book PSP: A Self Improvement Process For Software Engineers by Watts S. Humphrey and I plan to work through it and the accompanying programming exercises straight away. PSP is language agnostic so if you're interested in upping your game immediately, you might want to start with PSP and explore Ada/SPARK later.
Update: 2018-09-03:
I've been studying PSP for the last few months and I've written a summary of what I've learned so far.
Takeaways
Both PSP and Ada/SPARK are more prominent in the safety-critical, mission-critical, and high security sides of our profession. I've never heard any of the web development gurus mention this stuff so, I have no idea how well these tools will apply to my day job as a back-end ecommerce developer. My guess is that almost anything I can do to prevent defects from getting into our code base will be worth doing.
The research is very clear on the rising costs of fixing defects in the later stages of the development lifecycle. So, I can give up some speed on coding to save time on testing, debugging, and unplanned rework. The trick will be finding the sweet spot.
Would you ever consider learning PSP or Ada/SPARK? I'd love to hear your ideas in the comments.
Resources
How to achieve ultra-low defect rates:
Paper: The Fumble Programmer by Rod Chapman
Video: Murphy vs Satan: Why Programming secure systems is still hard by Rod Chapman
Should we trust computers? (video or word doc) by Martyn Thomas
Making Software Correct by construction (video or word doc) by Martyn Thomas
Safety-Critical Systems (video or word doc) by Martyn Thomas
Learn PSP:
Book: PSP: A Self Improvement Process For Software Engineers by Watts S. Humphrey
Video: An Introduction to PSP by Watts S. Humphrey
Learn Ada/SPARK:
Pluralsight Course: An Introductions to Ada 2012
Contract-Based Programming in Ada 2012 (video or slides) by Jacob Sparre Andersen
Website: Introduction to embedded programming with Ada
Book: Building High Integrity Applications with SPARK by John W. McCormick and Peter C. Chapin
Website: Ada Information Clearinghouse
Top comments (6)
I've looked briefly into SPARK, because I also would prefer to avoid introducing mistakes. For now I use typed functional programming. An expressive type system along with pure functions in the money-making code (core business logic) surfaces most errors at design-time. This allows me to avoid it even being possible for data to be in a wrong state. Also property-based testing goes a long way to ensuring the logic handles unexpected inputs properly. (I mainly use standard unit tests against the logic so far. I've been meaning to get into property-based.) Outside of sensitive areas which require formal verification, for me this is enough to prevent most of the mechanical errors of software dev.
So now the problem remains (even if I used formal verification) that sometimes I do a pretty decent job of developing the wrong thing. This is usually even more wasteful than doing something mechanically wrong. This is very common when devs implement "exactly what was asked". The main mitigation is instead of implementing the requested solution, dig deeper to find out the problem that the stakeholder is having, and address that. This can be difficult (to secure stakeholder time, to communicate effectively, etc).
I've shared this before. I had a situation recently where a customer asked for "just a field" added to an entity. But when we dug a little deeper, the field by itself wasn't going to solve the problem they had. We proposed an alternative that was more work, but would directly address the problem. They were pleased with our proposal. And it turned out to be a feature that other customers also wanted. Had we added just the field, it would have become unused cruft.
Thanks for sharing your experiences, Kasey.
Some functional languages (or subsets) can be formally proved/verified for correctness but I don't know much about that.
I love Quickcheck for property-based testing. We don't have a lot of code where it's a good fit but we have a few property-based tests in PHP. However, Quickcheck is much more powerful in a functional language. The minimum counter-example is particularly helpful in exposing errors in an easily digestible form.
Building the wrong thing is always a risk (even if you do careful requirements gathering and analysis). And if you took 5x more time to build it properly/correctly/carefully, it stings that much more when you find out that nobody needs what you built.
I've had experiences similar to yours where the customer presents his solution to the problem. And I've been burned by not digging deeper too. I've saved myself a lot of grief over the years when I started asking "what exactly are you trying to accomplish" when a customer presents a solution instead of a need/problem.
This is interresting and I think you'll learn lot of good things there.
But I'd say at least the language solve only part of the problem. It solve the problem of ensuring that the program you wrote actually does what you told the machine.
It doesn't ensure that you told the machine the right thing about what the program shall be doing from the specifications. It doesn't ensure neither that you understood the real problem you are trying to solve and managed to correctly translate the real world problem into a correct set of specifications...
That last one thing is responsible of 30-45% of all defects in a software and theses kind of defect, actually design defects are among the most costly to solve. In many places people don't even want to accept them as defects... But the real thing stil doesn't solve the real world problem even if it conform to the specs.
I totally agree.
The NASA study I mentioned in the post (also here) shows that the cost of missing or incorrect requirements can be devastating to a project because the costs rise exponentially the longer the defect is in the system.
I restricted my post to coding concerns but getting requirements right is super important too. I'm reading "Software Requirements" by Karl E Wiegers right now and I might write a post on that when I'm done.
Interesting food for thought! I like your dual approach of "better process" and "better language".
What are the safer languages out there?
I like to think Safe-D (a safer subset of the D programming language) is safer. But it is not ultra-safe in the complete and comprehensive static analysis sense that SPARK 2014 can claim.
Thanks.
There are different ways to think about safety. Languages that have automatic bounds checking for example are safer than those without (Java vs C).
Another aspect is whether the language can be proven formally. SPARK is one example and I believe I read that several of the functional languages (or subsets of them) are formally provable.
I read an article that said that Airbus is experimenting with model-driven development for critical systems because making sure that even Ada was correct was too difficult and costly. So, they just want to build a model and have software convert it into correct source code. Obviously that's a very specialized use case that's unlikely to gain widespread use in the near future.
Here's a link to AdaCore's product that might be similar to what Airbus is doing (code generation tools are over my head): adacore.com/qgen