DEV Community

Cover image for And the Winner Is: Aggregate Model-based Testing
Istvan Forgacs for LambdaTest

Posted on • Originally published at lambdatest.com

And the Winner Is: Aggregate Model-based Testing

In my last blog, I investigated both the stateless and the stateful class of model-based testing. Both have some advantages and disadvantages. You can use them for different types of systems, depending on whether a stateful solution is required or a stateless one is enough. However, a better solution is to use an aggregate technique that is appropriate for each system. Currently, the only aggregate solution is action-state testing, introduced in the book Paradigm Shift in Software Testing. This method is implemented in Harmony.

Check this out: Blackbox Testing Tutorial - A Comprehensive Guide With Examples and Best Practices

Where can we use Action-state testing?

Action-state testing can be used for both stateful and stateless cases or even a mixed solution is possible if a long test case consists of inner states at some test steps, but at some other steps don’t.

The ‘action-state’ model is textual, containing ‘action-state’ steps. Test cases are generated from the action-state model. An action-state step is a triple containing

  • a (user) action,

  • a (system) response,

  • the inner (test) state.

Actions and responses must be implementation-independent and as abstract as possible. Action is described at a high level so that humans can understand. For example, an action can be as:

add items so that price remains below 10
This is more abstract than adding two more concrete actions:

  1. add item for 7

  2. add item for 2

Even more, concrete actions would be:

  1. click on the ‘+’ icon next to Pizza Mexicana

  2. click on the ‘+’ icon next to Coke

However, these are implementation-dependent actions and can be created when the implementation is ready. Making an implementation-independent model is a huge advantage as it can be done before coding and the team can validate it against the specification detecting almost all the problems in it.

Modeling is easy: a subsequent step (step 2) is executed immediately after a step (step 1) if step 2 is indented to the right:

step 1
step 2

If two steps’ indentation is the same then they will be executed in different test cases:

step A
step B.

step A may contain an action and a response (stateless step):

action A => response A

or may contain a triple involving the inner state:

action A => response A STATE state for A

Models can be easily mapped with the requirements:

R1 my requirement A:

action A => response A STATE state for A

One model can embed other models. In this way, very complex systems can also be modeled.

You can easily build a model related to a feature or user story. From this model, the tool can generate the statechart and the test cases immediately. The statechart is another model representation, see later. By applying statecharts, you can easily detect missing actions and states. On the other hand, a statechart contains less information. The information on whether a transition is traversed more times is missing. For example, entering the wrong PIN for the first and second time is modeled in this way:

Add wrong PIN => wrong PIN added STATE waiting for PIN
Add wrong PIN => wrong PIN added STATE waiting for PIN

The related statecharts are the same independently of only once or twice the wrong PIN was added. Action-state models contain more information than statecharts and this is the reason why this technique doesn’t need guard conditions and consequently coding.

However, to avoid coding the models are a little bit larger.

You can also set an appropriate test selection criterion. A tool can offer the missing action-state steps to satisfy the criterion. There may be steps to be offered that are invalid. Fortunately, you can easily recognize an invalid step and it can be rejected. Accepting or rejecting an offered step is the replacement of adding guard conditions and some other coding. It’s much easier and non-coder testers can also do it.

Let’s consider the same requirements as in my previous blog:

The most difficult part is how to find appropriate states. A reasonable solution is that if two actions are the same, but the system calculates the response in different ways, then we arrived at different states. Here is an example. The initial state is ‘No discount, no bike’. From here we can add a car for which we go back to the initial state as no discount happens. Adding a second car, the calculation is the same as a car added to the cart. However, when ad the third car, the calculation will be different as a bike is also added for free. Thus we arrived at a new state ‘Discount, bike added ’. If we start with adding a bike and then adding three cars in a row, then there is another different calculation as the bike becomes free. Thus this is a new state: ‘Discount, bike converted ’. It’s obvious that different calculations can be tested as any may contain an error. In this way, you can select a minimum number of test states.

Adding a bike at the same level as the first car, we go to another state ‘No discount bike included’. All these steps cover requirement R1. This means that covering a single requirement with a single test is usually not enough. Here is the first version of the model:

INITIAL STATE No discount, no bike 
R1 The customer can add cars or bikes one by one: 
    add car => one car STATE No discount, no bike
        add car => two cars STATE No discount, no bike 
    add bike => bike added STATE No discount bike included
Enter fullscreen mode Exit fullscreen mode

and the related statechart:

You can see that the statechart doesn’t show that we added two cars.
For covering requirement R2, we should delete a car/bike that should be a child step of a parent step where at least one car/bike is available:

INITIAL STATE No discount, no bike 
R1 The customer can add cars or bikes one by one: 
    add car => one car STATE No discount, no bike
        add car => two cars STATE No discount, no bike 
          R2 The customer can remove cars or bikes one by one:   
            delete car => one car STATE No discount, no bike     
    add bike => one bike STATE No discount bike included 
      R2 The customer can remove cars or bikes one by one:  
        delete bike => cart empty STATE No discount, no bike
Enter fullscreen mode Exit fullscreen mode

To cover R3a, we should add two cars to the single bike:

add bike => one bike STATE No discount bike included 
      R3a discount for EUR 700 when bike converted free:
        add two cars => two cars and a bike STATE Discount, bike converted
Enter fullscreen mode Exit fullscreen mode

R3b can be covered if we add a third car:

add car => two cars STATE No discount, no bike 
          R3b For discount a bike is added when only cars are in the cart:
            add car => three cars and a bike STATE Discount, bike added
Enter fullscreen mode Exit fullscreen mode

Now it’s reasonable to continue with R4. To cover it we remove the third car:

add car => three cars and a bike STATE Discount, bike added 
              R4 going below the discount limit it is withdrawn:
                delete car => two cars STATE No discount, no bike
Enter fullscreen mode Exit fullscreen mode

Considering R3c, we should add the deleted car again:

delete car => two cars STATE No discount, no bike 
                  R3c reaching the limit again, previous discount back:
                    add car => three cars and a bike STATE Discount, bike added
Enter fullscreen mode Exit fullscreen mode

Our last requirement is R3d. We cover it by deleting the second car, adding a bike, and adding the second car again:

delete car => two cars STATE No discount, no bike 
                  R3d reaching the limit again adding a bike before:
                    delete car => one car STATE No discount, no bike 
                      add bike => one car and a bike STATE No discount bike included 
                        add car => two cars and a bike STATE Discount, bike converted
Enter fullscreen mode Exit fullscreen mode

The whole model is the following:

INITIAL STATE No discount, no bike 
R1 The customer can add cars or bikes one by one: (1)
    add car => one car STATE No discount, no bike
        add car => two cars STATE No discount, no bike 
(2)       R3b For discount a bike is added when only cars are in the cart:
            add car => three cars and a bike STATE Discount, bike added 
(3)           R4 going below the discount limit it is withdrawn:
                delete car => two cars STATE No discount, no bike 
(4)               R3c reaching the limit again, previous discount back:
                    add car => three cars and a bike STATE Discount, bike added
(5)               R3d reaching the limit again adding a bike before:
                    delete car => one car STATE No discount, no bike 
                      add bike => one car and a bike STATE No discount bike included 
                        add car => two cars and a bike STATE Discount, bike converted
(6)       R2 The customer can remove cars or bikes one by one:     
            delete car => one car STATE No discount, no bike       
    add bike => one bike STATE No discount bike included 
(7)   R3a discount for EUR 700 when bike converted free:
        add two cars => two cars and a bike STATE Discount, bike converted   
(6)   R2 The customer can remove cars or bikes one by one:  
        delete bike => cart empty STATE No discount, no bike
Enter fullscreen mode Exit fullscreen mode

Hurray, we covered the requirements with only five test cases (the ones (1) — (5)). Are we ready? Let’s see. The related statechart is here:

Check this out: Usability Testing- A Comprehensive Guide With Examples And Best Practices

It’s a great help as missing actions can be detected easily. For example, from the state ‘Discount, bike added’ we can delete or add a bike. Considering ‘No discount bike included’, there is a missing edge going to itself, when deleting a bike from two. We can also add bike going to itself and going to ‘Discount, bike converted’. Finally, considering ‘Discount, bike converted’, there are several missing actions, i.e., adding/deleting a car and adding/deleting a bike to itself, delete bike going to ‘No discount no bike’ and delete car/bike going to ‘No discount bike included’.

Traversing each valid action/edge is the minimum test selection criterion, thus just covering the requirements is way insufficient. Involving the missing actions, we get the following improved model:

INITIAL STATE No discount, no bike 
R1 The customer can add cars or bikes one by one: 
    add car => one car STATE No discount, no bike
        add car => two cars STATE No discount, no bike 
          R3b For discount a bike is added when only cars are in the cart:
            add car => three cars and a bike STATE Discount, bike added 
              add bike => three cars and two bikes STATE Discount, bike added 
              delete bike => three cars STATE No discount, no bike  
              R4 going below the discount limit it is withdrawn:
                delete car => two cars STATE No discount, no bike 
                  R3c reaching the limit again, previous discount back:
                    add car => three cars and a bike STATE Discount, bike added
                  R3d reaching the limit again adding a bike before:
                    delete car => one car STATE No discount, no bike 
                      add bike => one car and a bike STATE No discount bike included 
                        add car => two cars and a bike STATE Discount, bike converted  
          R2 The customer can remove cars or bikes one by one:     
            delete car => one car STATE No discount, no bike       
    add bike => one bike STATE No discount bike included 
      add bike => two bikes STATE No discount bike included 
        add five bikes => seven bikes STATE Discount, bike converted 
          add bike => eight bikes STATE Discount, bike converted 
            delete bike => seven bikes STATE Discount, bike converted 
          delete bike => six bikes STATE No discount bike included 
        delete bike => one bike STATE No discount bike included  
      R3a discount for EUR 700 when bike converted free:
        add two cars => two cars and a bike STATE Discount, bike converted  
          delete car => one car and a bike STATE No discount bike included 
          add car => three cars and a bike STATE Discount, bike converted  
            delete car => two cars and a bike STATE Discount, bike converted 
          delete bike => two cars STATE No discount, no bike 
          add bike => two cars and two bikes STATE Discount, bike converted 
      R2 The customer can remove cars or bikes one by one:  
        delete bike => cart empty STATE No discount, no bike     
      delete bike => cart empty STATE No discount, no bike
Enter fullscreen mode Exit fullscreen mode

The number of test cases increased from 5 to 13. The new statechart is the following:

Now all the actions are involved. We have seven requirements and we carefully covered by 13 test cases. You can think that 13 tests are too much, but it depends on the risk analysis. If the risk is high, you should test it carefully, otherwise tricky bugs remain undetected. You cannot merge cars and bikes as vehicles as they behave differently, only bikes are added freely or converted for free, and cars are not.

Considering the statechart, if you simply traverse it, some invalid tests are generated. For example, adding a bike from the initial state, then adding five bikes we arrive at ‘Discount, bike converted’. However, having only six bikes we have no discount, a contradiction. The state action-state model consists of only valid steps.

As mentioned, by applying a stronger test selection criterion, new steps are offered. For example, when applying all-transition-pairs criterion the following step is offered:

This is because the edge/action pair (add car, add bike), where add car is the action starts from and arrives at the initial state hasn’t been covered. If the risk is very high you should add these steps as well.

Check this out: Automated Functional Testing: What it is & How it Helps?

Conclusion

Action-state testing is an aggregate MBT method. This is the only one that requires no coding at all, just modeling. The model can be converted to a statechart, by which the original model can be improved and made complete. It can be widely used, and tricky bugs can be detected. From the models, abstract test cases are generated that can be manually executed. The model consists of implementation-independent steps, thus it’s a true shift left MBT method.

Top comments (0)