Inattentional Blindness and Scripted Tests

My previous workplace was a large organisation in which many testers were employed to evaluate the quality of the software we developed and maintained. We had a collection of scripted test cases that testers would follow step-by-step. That worked reasonably well, although I was employed to help automate our processes, including testing, which contributed significantly to the quality of our software.

When I started working at my current workplace I found similarly detailed scripted test cases. Part of my responsibilities included manual testing, so I thought I could test the way I was familiar with, and how the rest of my colleagues tested—just follow the test cases. It hasn't worked well. We find bugs, for sure, but as I've grown in experience I've found more and more problems with the software that had been there for a long time, through many versions of the software and through many executions of test cases that should have revealed them.

There are at least a few things that I think explain why we, including my past self, failed to identify problems:

1. Out-of-date test cases. Change happens constantly and we have too many tests with too much detail for our small QA team to keep up with.
2. Treating test cases like an instruction manual. It's relatively easy for an experienced tester to follow the steps of a scripted test case to the letter, assuming the steps are accurate. That was our standard practice. But it's even easier to miss out on opportunities to reveal bugs if you do that.
3. Overwhelming detail. Many test cases are so long, verbose, and complicated that it's very easy to miss important details in the steps and expected results, especially when you're under pressure to get the job done quickly.
4. Unnecessarily specific detail. Often a test case instructs the tester to use a particular element of the UI in a particular way. E.g., "enter a value in the Account text field and click the Validate button at the bottom of the panel." That sort of specificity means the other fields are likely to be ignored, as well as ignoring all the other ways that validation could be triggered. And that problem is in addition to making it hard to keep the test case up to date (because sooner or later, that Validate button is going to move, or be removed entirely).

The last three points have something in particular in common. They all trigger a cognitive phenomenon called inattentional blindness. It's something that we all experience, whether we're aware of it or not¹. A well-known demonstration of the phenomenon comes from a psychological study and you can perform the experiment from the study yourself by watching a video and following the instructions at the start:

If you haven't done so already, I strongly recommend you do the experiment first before you read on—this is something you really only get to experience once, although there are variations of it. Although it's likely something you'll experience again and again in real life.

The study and others like it find that half of the time on average, people fail to notice the unexpected element. They're asked to perform a task and they're focused so intently on it that they fail to perceive something they're looking right at. It's one of the reasons using your phone while driving is so dangerous, even hands-free; if your attention is on the text/app/call it's not on the road.

That kind of inattentional blindness is exactly what can happen when you follow a scripted test. You focus on the steps you have to follow and the results you have to check for and you fail to notice anything else. The software can behave in unexpected ways, but you might miss it if you're only paying attention to what the test case says should happen. Even if you're looking right at the problem. Missing the unexpected becomes more likely the longer you're doing the task and the more anxious you are about completing it quickly.

I'm certainly not the first to draw the link between inattentional blindness and scripted tests. Michael Bolton, for one, has mentioned it a few times on his blog. But it's particularly relevant to me now as I try to improve our testing practices. Whatever changes we make, we need to be aware of the potential for inattentional blindness.

For now it's clear to me that the scripted, highly-detailed test cases we're used to at my workplace are getting in the way of us improving the quality of our software. Part of the solution is a more exploratory approach to testing. One suggestion is that instead of following a test case step-by-step, you could:

glance over the test case; try to discern the task that’s being modeled or the information that’s being sought; then try to fulfill the task without referring to the test case. That puts the tester in command to try the task and stumble over problems in getting to the goal. Since the end user is not going to be following the test case either, those problems are likely to be bugs.

For those interested, there is more information about inattentional blindness on Scholarpedia, including references to the original research publications.

How do you avoid inattentional blindness while you're testing, especially if you have to follow scripted tests?

1. Technically, we're never aware of it. That's the inattentional part.

Originally published on my blog.