Automatically reduce test cases using pairwise methods

When testing an application, the number of test cases depends on which browser is used, which device is used, and which impression method is used.

Let's take a look at this case.

• devices: Mac, Windows, iPhone, iPad, Android
• browsers: Safari, Chrome, Firefox, IE
• login authentication methods: Facebook login, Twitter login, Google login, Instagram login

The number of all test cases is calculated as follows.
This is simple elementary level math.

devices (5) x browsers (4) x login authentication methods (4) = 80 cases

So there are 80 different ways.
Well, I can do it if I share it with my colleagues, but I don't want to if I can help it.

Use pairwise methods to overwhelmingly reduce the number of test cases

A simple explanation of the pair-wise method is: Basically, an app bug is caused by one or two factors. Therefore, even if there are more than three factors, there is no need to check all the combinations.

However, it is a bit difficult to adapt the pairwise method to already existing test cases on your own, so you should use an open source tool called "PICT" published by Microsoft.

Install PICT

You can implement it in the following steps.

$ git clone https://github.com/Microsoft/pict.git
$ cd pict/
$ make
$ sudo install -m 0755 pict /usr/local/bin/pict

Now that the pict command is available, the next step is to write the test case so that pict can read it. The way to write it is super simple and looks like this.

devices: mac,windows,iphone,ipad,android
browser: safari,chrome,firefox,ie
login_authentication_method: facebook,twitter,google,instagram

Save this with a .txt extension.

Run PICT

Now let's see how many test cases have been reduced from the 80 simple calculations made earlier.

pict test_case.txt

result

device  browser login_authentication_method
windows firefox facebook
ipad    safari  twitter
android chrome  google
windows ie  instagram
ipad    firefox google
iphone  ie  google
windows safari  google
iphone  safari  instagram
ipad    ie  facebook
iphone  firefox twitter
mac safari  facebook
android ie  twitter
mac ie  google
mac chrome  twitter
iphone  chrome  facebook
android firefox instagram
windows chrome  twitter
ipad    chrome  instagram
android safari  facebook
mac firefox instagram

Down to only 20 cases (1/4)!!!

For wrong combinations like Mac+IE, please google for details as you can write conditions and drop them.

If you simply want to know the number of cases, you can run it with /s at the end.

$ pict test_case.txt /s
　　　　Combinations:   56
　　　　Generated tests:20
　　　　Generation time:0:00:00

If you are wondering if this is really reliable, please read the paper published by Microsoft.
Pairwise Testing in the Real World: Practical Extensions to Test-Case Scenarios

Advertisement

I am working on a web app called spliito.com that solves the tedious task of recording the money owed and borrowed that occurs when you go on a trip with friends in a spreadsheet and calculating who should pay back how much to whom at the end of the trip.

Created a product used by 50,000 people per month