Test Environments- What & Why

Why do we have so many/less test environments, why is volume of test data so small on dev environment, where am I going to test critical bug in production without disturbing BAU development, How about testing db cluster setup; where should I test those? All valid questions? Without a further ado let’s get on to answer few of them.

First thing first: What is test environment

Test environment is a set of hardware on which application under test(AUT) and supporting software are deployed in order to assist in testing.

The good part of working in my current organisation is that you get a chance to work on different projects in a very short span of time. Different project means different requirements, different setup, different personalities and different arguments for the same topic. Lately I have been talking to different people about, what all environments do we need in a given project?

Short answer to the question is: “it depends” on the project and current setup of the organisation infrastructure. While this is true but for simplicity we can categorise project in below 3 categories. There are difference Devops maturity model your project can be in. For further reading, refer state of Devops over here:

1. Following waterfall SDLC
2. Projects transitioning to Agile & CI/CD
3. Project already running in CI/CD

As of now I have worked in 2 different types of projects, yet to work on a project following Continuous delivery.

Ok coming back to test environments; with Devops it feels very tempting to have to too many environments and at the same time without Devops we try to manage in whatever test environment is available to us. Both of the approaches comes with pros & cons. Too many test environment comes with overhead of maintaining, deploying application and infrastructure cost. Less test environment might increase testing time and impact quality of scenarios one wants to perform in a project.

Another way to look at the test environment requirement is the number of testing/quality gates and where they will run for successful and timely delivery of software.

Some of the testing activities in typical project will be; Dev testing, Infra testing, Automation testing, Qa Exploratory testing, Integration testing, Business user testing, Compatibility testing, Security testing, Performance testing, Hotfix bug testing/Signoff etc

The bare minimum environment for a typical enterprise project which I have seen so far are Dev, QA, UAT, Perf environments apart from Production but again it varies from project to project(yes, it depends). There are other optional environments also like CI(Automation), Infra, SIT.

One question that comes to mind after seeing all such environments is, do we need so many environments? Lemme know your thoughts in the comments section

Let's get down to what & why of all these major test environments.

Dev environment

Primary owner: Developers

Purpose & Activities:

• Used for dev testing and assist in development activities, can choose to run automated E2E testing
• Automated security scans[Static code does not require any environment though]
• Authentication/Authorisation are relaxed to assist in testing

Environment Setup:

Trimmed down version of production, if there are multiple cluster in production this will typically have 1 cluster setup

Test Data:

• Small size with good enough variation of test data
• Can be mocked for better stability and different scenarios creations

Deployed artefact version: Always latest

QA environment

Primary owner: QAs/ QA Lead

Purpose & Activities:

• Acceptance testing & Story Sign off
• Design/UI testing
• Exploratory testing
• Can be used to run automated regression suite
• Bug bash

Environment Setup:

Trimmed down version of production, if there are multiple cluster in production this will typically have 1 cluster setup

Test Data:

• Should be always integrated with stable backend
• Different shape & size of test data from production with masked PII
• Test data pipeline to update test data with a certain frequency [like once in a month etc]

Deployed artefact version: Runs behind or equivalent to Dev environment

SIT Environment

This environment is usually optional as all of these activities can also be performed in QA environment but typically in enterprise we do have separate SIT environment for smooth working of different teams.

Primary owner: Release manager/ QA Lead

Purpose & Activities:

• Integration testing of releasable artefacts
• All other environments usually connects with dependent artefacts of this environment.
• Manual backward compatibility testing e.g. new backend and older front end [mobile apps]
• Manual regression suite [if applicable]
• Manual Security/Penetration testing
• Roll back strategy testing
• Verbiage verification if any

Environment Setup:

Trimmed down version of production, if there are 5 node setup . This environment will have 3 nodes.

Test Data:

• Should be always integrated with stable backend
• Different shape & size of test data
• No Mocking
• Test data pipeline to update test data with a certain frequency [like once in a month etc]

Deployed artefact version: Typically runs behind QA environment

UAT Environment

Primary owner: Release Manager/ QA Lead

Purpose & Activities:

• Stands for user acceptance testing or SME/Business owner testing: mimicking actual business scenarios.
• Usually requirement miss etc are feedbacks from this environment
• Business owner bug bashes etc
• Hotfix bug testing

Test Data:

• Replica of production with PII masked.

Deployed artefact version: Typically runs behind SIT by a version

Perf Environment

As the name suggests, this environment is used to run various performance & benchmarking tests.

Primary owner: QA lead, Release Manager or Performance team if separate

Purpose & Activities:

• Benchmarking with simulating different load scenarios for certifying speed, scalability & stability of application

Environment Setup: Same as production or very close to production

Test Data: Similar to production in shape & size.

Deployed artefact version:

• Run behind by SIT by a version or equivalent to SIT.

Above list is my experience of working in enterprise accounts, do let me know your setup in comments section. I have seen few variation e.g. having separate automation environment or separate Infra environment for testing roll back strategy or infra scripts but typically above is good enough list. You can find different names of testing environment in different organisation but once we know the purpose we can always reason about the same. Also there are numerous hacks we can put in order to assist in test environment. But general rule of thumb is; all access and setup of test environment should be as close as possible to production; be it opening to internet to test data setup and should assist in verification of Application under test.

Happy Testing!