Negative Testing

As a part of handling failures cleanly, an organization has to do negative testing.

About Negative Testing

There are two main testing strategies in software testing: positive testing and negative testing.

Positive testing determines that an application works as expected. If an error is encountered during positive testing, the test fails.

Negative testing ensures that an application can gracefully handle invalid input or unexpected user behavior. For example, if a user tries to type a letter in a numeric field, the correct behavior, in this case, could be to display a warning. The purpose of negative testing is to detect such situations and prevent applications from crashing. Also, negative testing helps improve the quality of an application and find its weak points.

When you perform negative testing, exceptions are expected ... they indicate that the application handles improper user behavior correctly.

A one-hundred-percent error-free system is not feasible, but if the team has ownership of Software Quality, that team can ensure that they have done everything to prevent a failure.

Negative testing commonly referred to as error path testing or failure testing is generally done to ensure the stability of the code.

Negative testing is the process of validating the application against invalid data. The intended purpose of negative testing is to determine if bad data is handled gracefully.

Bug Testing

When developing and fixing bugs, it is sometimes seen that the bug was caused because there were not enough guards in place. When these guards are put into place, the testing should be augmented with appropriate negative tests to ensure the bug does not return.

Advantages of Negative Testing

• Negative testing is very important to ensure the quality of a product. A good quality product is a zero vulnerability product, to ensure that negative testing is very important.
• Writing negative tests makes sure that all possible cases are covered. Intentionally or unintentionally there is a chance of negative test cases occurring. To ensure all cases are covered we have to do negative testing along with positive testing.
• The client will have more confidence in the product.

Disadvantages of Negative Testing

• In many cases, there is no need for excessive negative testing. Determining conditions in negative test cases are very important. There will be times where we do not have to do negative testing on a particular system.
• Negative testing requires skilled and experienced people to create negative test cases.

Typical Scenarios

The code supporting these scenarios is in a GitHub Repo: HERE

Required Data

Required data means that a function has a parameter that is required. To check this behavior, create a test that leaves the required fields empty, null, or undefined and analyze the response.

function testable001(testField) {
  if (testField === null || testField === undefined) {
    return 'invalid';
  }
  return 'valid';
}

To test this code ...

describe('testable001 [negative inputs]', function() {
  it('expects no parameter to short out correctly', function() {
    var result = testable001();
    expect(result).toEqual('invalid');
  });
  it('expects null to short out correctly', function() {
    var result = testable001(null);
    expect(result).toEqual('invalid');
  });
  it('expects undefined to short out correctly', function() {
    var result = testable001(undefined);
    expect(result).toEqual('invalid');
  });
});

Specific Data Types

This is when a function expects a specific type of data (numeric, date, text, etc.). To verify this code functions properly, create a test that passes incorrect data to the function.

function testable002(testField) {
  if (typeof testField !== 'number') {
    return -1;
  }
  return testField * 2;
}

To test this code ...

describe('testable002 [negative inputs]', function() {
  it('expects string to short out correctly', function() {
    var result = testable002('not this text');
    expect(result).toEqual(-1);
  });
  it('expects null to short out correctly', function() {
    var result = testable002(null);
    expect(result).toEqual(-1);
  });
});

Number Of Characters

Functions sometimes limit the number of characters that can be passed in. To check the behavior of the application, create a test that passes more characters into the function than is allowed.

function testable003(testField) {
  const maxLength = 10;
  return testField.substr(0, maxLength);
}

To test this code ...

describe('testable003 [negative inputs]', function() {
  it('expects length to max out correctly', function() {
    var result = testable003('01234567890123456789');
    expect(result).toEqual('0123456789');
  });
});

Reasonable Data

This encompasses function parameters that have reasonable limits, for example, entering 200 or a negative number as the value for AGE is not allowed. To check this behavior, create a negative test that enters invalid data into the specified field.

function testable004(testField) {
  if (testField <= 0 || testField > 120) {
    return 'invalid age';
  }
  return 'valid age';
}

To test this code ...

describe('testable004 [negative inputs]', function() {
  it('expects age to handle zero correctly', function() {
    var result = testable004(0);
    expect(result).toEqual('invalid age');
  });
  it('expects age to handle -1 correctly', function() {
    var result = testable004(-1);
    expect(result).toEqual('invalid age');
  });
  it('expects age to handle 200 correctly', function() {
    var result = testable004(200);
    expect(result).toEqual('invalid age');
  });
});

Session Testing

Here we are talking about some external requirements, such as the user's login status, that needs to be checked before a function will return a correct value. To check this function correctly, create a test that sets the external value into the incorrect state and check the function's results.

Setup code here ...

var state = {
  loggedIn: true
};

The function under test ...

function testable005(state, testField) {
  if (state.loggedIn !== true) {
    return false;
  }
  return testField;
}

To test this code ...

describe('testable005 [negative inputs]', function() {
  it('expects logged out to be handled correctly', function() {
    state.loggedIn = false;
    var result = testable005(state, 'test');
    expect(result).toEqual(false);
  });
});

Analysis Of Negative Testing Patterns

A look at the code examples shows that the means of determining the appropriate amount of tests can come from several different areas. To determine what areas to cover, we can use Boundary Value Analysis and Equivalence Partitioning.

Boundary Value Analysis

As the name implies, a boundary indicates a limit to something. Hence, this involves designing test scenarios that only focus on the boundary values and validate how the code behaves. Therefore if the parameters supplied are within the boundary values then it is considered to be positive testing and inputs beyond the boundary values are considered to be a part of negative testing.

Equivalence Partitioning

In equivalence partitioning, the test data is segregated into various partitions. These partitions are referred to as equivalence data classes. It is assumed that the various input data (data can be a condition) in each partition behave the same way.

Hence **only one particular situation needs to be tested* from each partition. If one works then all the others in that partition are assumed to work. Similarly, if one condition in a partition doesn't work, then none of the others will work.

Therefore, it is apparent that valid data classes (in the partitions) will be comprised of positive testing whereas invalid data classes will be comprised of negative testing.

Conclusion

When you perform negative testing, exceptions are expected ... they indicate that the application handles improper user behavior correctly.

A one-hundred-percent error-free system is not feasible, but if the team has ownership of Software Quality, that team can ensure that they have done everything to prevent a failure.