(mostly incomplete) List of Test Methodologies

Test methodologies can be of benefit when conceptualizing any test approach -from unit to features- , defining test strategies or building test architectures.

There are in-depth academic areas of study on these. This is not a complete list, only a summary of ones that are more commonly used.

Ad-hoc, Manual, Exploratory testing

Although this is testing at its simplest, there are holistic approaches worth checking out.

Requierements-based (traceability matrix)

Testing based on the requirements of the system. Requirement Coverage is the relationship between test cases and requirements. Requirement Tracability visualizes these relations: 1:1, 1:n, n:1 or n:n.

Scenario / Workflow / Use case / Activity diagrams

Testing based on the activity / sequence of user workflows / events that occur in the system. Usually referred to as use case scenarios. Can be useful while conceptualizing acceptance criteria into BDD context.

Non-Functional Requirement testing

Testing based on aspects that are not directly related to requirements of the system. Per standard, some of these are Performance, Reliability, Security, Maintainability, Usability, Portability etc. The term service-level-agreement (SLA) is also used in related contexts.

Equivalence class partitioning, Boundary value analysis

These methodologies are useful when there is a vast amount of data and we want to reduce the amount of testing needed. In Equivalence class partitioning, the input data is divided into ranges of values that are known to be equivalent. The tests are then created to be representative for each range. In Boundary value analysis, testing is done at the extreme ends / boundaries of these partitions. Usually the two methodologies are used harmoniously.

Decision tree, Classification tree, Cause-effect graphing

A decision tree is a flowchart-like structure where each node represents a test on an attribute and the branches represent the outcomes of the test. In daily testing practices, it can be used in relevance to use case scenarios / workflows, but the applications are not limited to this; machine learning, operations reasearch, decision analysis etc. are some of the other domains it is used.

The high level distinction in a classification tree vs a decision tree is; at node level the attributes vs decisions/scenarios/tests, and at branch level possible values of the attributes vs outcomes.

Cause-effect graphing technique can be used when the combinations of input conditions is a concern. A graph if used to represent the combinations of input conditions, and then the graph is converted to a decision table to obtain the test cases.

Combinatorial testing

The key insight underlying this form of testing is that not every parameter contributes to every failure and most failures are caused by interactions between relatively few parameters. The problem space - which can be anything from configurations, deployments, test suites or test scenarios - is modeled into parameters and values that these parameters can have. The test cases are then generated by feeding this information into tools. Check out this tool and many more like it at pairwise.org.

State-based / Finite State Machines

A finite state machine (FSM) describes the possible states of the system and the transitions that can occur between them triggered by inputs. The test cases are then generated by feeding the state machine into a tool.

1. Understand the various state and transition and mark each valid and invalid state.

2. Defining a sequence of an event that leads to an allowed test ending state.

3. Each one of those visited state and traversed transition should be noted down.

4. Steps 2 and 3 should be repeated until all states have been visited and all transitions traversed.

XState is a well known JS/TS library for creating state machines and statecharts, and interpreting them.

Markov chains (state + statistical)

The distinction between FSMs and Markov chains is that Markov chains are probabilistic and FSMs are deterministic. Markov chains are used to model the behavior of a system, and FSMs are used to model the behavior of a system.

Model based testing

In model based testing, a model is created to describe the system. Generally Activity or State diagrams are used to for the model. In turn, test cases are generated by feeding the model into a tool. While a part of the test community swears by it since decades, others are hindered by test maintenance and the reverse engineering tedium.

Unit: statement, branch, function, line

Familiar terminology from day to day tools we use in development.

Mutation testing

Mutation testing can be used to evaluate the fault-detection capability of a unit test suite. In Mutation testing, artificial "defects" are introduced to the code. Subsequently, different versions of the code -mutants- are tested-against using the test suite at hand. Finally, the efficiency of the test cases is evaluated -mutation score- based on the success rate of finding defects -mutants being killed. In modern web dev, it can be utilized after there is confidence in coverage, to further determine if the testing is adequate. Then, the tests with low mutation score can be enhanced to be more meaningful.

UI e2e testing

The most common terminology being referred to by UI automation. No mocking any dependencies, but testing as a real user would. Can incorporate other test methodologies.

UI (component) Integration testing

These look like UI e2e tests but they fully stub the network, and they are run without hitting a real server. They are faster, and less brittle than traditional UI e2e tests since the network is not a concern. They are great for shift-left approach and to isolate the ui functionality prior to testing on deployments where the backend matters.

UI Component testing

UI component testing paradigm is rapidly changing, but the ultimate goal is to test the component in isolation. Some approaches in the React world are with React Testing Library, Enzyme, and recently with Cypress where components are rendered in the browser as opposed to the terminal. A popular way to build components is with Storybook, and newer approaches are allowing stories to be tested as well.

API testing

In API testing a real server is the request target; this can also be referred to as a contract test sometimes. It can also be that the tests span over multiple apis, whereby this can be referred to as API e2e testing. The goal is to test services or the integration of them at a higher level than isolated modules.

API component/module testing

The distinction from API testing is that all that is external to the component is mocked and the module is tested in isolation. In the NodeJS world, Nock (mock external http) and Supertest or Axios (test only the server) can be used to achieve this kind of testing. With Supertest & Axios cases, the server is served locally, and the db is preferably in a local Docker container.

Consumer driven Contract testing

A type of contract testing that ensures that a provider is compatible with the expectations that the consumer has of it. It is the primary way to shift API testing left, and test the integration of apis/services prior to deploying them to a common environment. Pact is a well known framework on this matter.

Email testing

Typically email testing involves validating email fields (from, to, cc, bcc, subject, attachments), HTML content and links in the email. Email services also allow spam checks and visual checks. The core goal is to enable the last mile of end to end testing, to enable a typical web app to be tested from start to finish.

Visual regression/snapshot testing

A visual regression test checks what the user will see after any code changes have been executed by comparing screenshots taken before and after code changes. It is the primary way to validate regressions in CSS, but it is also useful for covering viewport & cross browser/device combinations, as well as localization.

Performance testing (user xp, load, spike, endurance)

In the web world this is beginning to have 2 paradigms; for UI it is concerned with user experience (tools like Lighthouse), and for the back end it evaluates how a system performs in terms of responsiveness and stability under a particular workload (tools like k6).

Chaos engineering

Related to NFR testing in resiliency, it is the discipline of experimenting on a software system in production in order to build confidence in the system's capability to withstand turbulent and unexpected conditions. Generally speaking it is at the top of the pyramid and applied after some amount of maturity in testing.

Penetration / Security / Vulnerability testing

Related to NFR testing as well, it is performed to evaluate the security of the system, using manual or automated technologies to systematically compromise servers, endpoints, web applications, wireless networks, network devices, mobile devices and other potential points of exposure. A vast topic in its own right, being made approachable to web development via automated test tools that drive front end UIs while assessing vulnerabilities.