Enhancing Software Change Impact Analysis

A comprehensive test suite is part of modern software development best practices. Unit tests, integration tests, end-to-end tests, and many others make sure your system keeps working when you need to change the implementation.

But over the lifetime of a software project, you can end up with hundreds of such tests, and every test you add can gradually slow down your CI/CD pipeline. You have high standards for software quality, but with all these tests, your development velocity goes down the drain. How can you get both: quick changes and high quality?

How Does Change Impact Analysis Help?

Change Impact Analysis is one solution to this problem. The goal of it is to minimize the number of tests required to run for a specific change. When you have a massive suite of tests, change impact analysis helps you organize and track which test impacts which source files. Later, when you change one or more of these source files, you can take the change impact analysis list and check which parts of your test suite are impacted by the changes you made.

With change impact analysis, you only run the tests that matter, save time in your CI/CD pipeline, and get results quickly.

Implementing Change Impact Analysis for Node.js

Let’s find out how we can set up change impact analysis for Node.js. For this project, we’ve created an example repository on GitHub that you can clone locally and use as a reference.

This example project is an API that uses the Express framework. It consists of three routes, each having its file and four test files—one for every route and one with tests for two routes.

The unit tests are implemented with the Jest testing framework, which supports code coverage output. This output allows us to check which test file affects which source file.

src/

index.js

route-a.js

route-b.js

route-c.js

tests/

route-a.js

route-b.js

route-c.js

route-a-c.js

Running all tests

If you run the test:all you will execute all tests in the __tests__ directory.

$ npm run test:all

This is the default behavior, and in our example, this is no problem. But, over time, it could grow slower. That’s why we have to use change impact analysis to find out which test file is concerned with which source file.

Analyzing Change Impact

To analyze change impact, we have to complete the following steps:

1. Check which test files exist
2. Run each test file on its own
3. Check the resulting code coverage of each test run
4. Write down which source files the test touched

To find out what source files were touched, you need to configure Jest to output code coverage data; in this project, that’s done inside the package.json file.

This is the code for the Jest configuration:

 "jest": {
  "collectCoverage": true,
  "coverageDirectory": "coverage",
  "coverageReporters": [
    "json"
  ]
},

In the scripts directory is a init-tia.js file, which implements these steps. Let’s look at the important parts of this file:

 const testFileNames =
  fs.readdirSync("./__tests__")
const testImpact = {}
for (const testFileName of testFileNames) {
  await jest.run(testFileName)
  const absoluteFilePaths = Object.keys(
    JSON.parse(
      fs.readFileSync(
        "./coverage/coverage-final.json"
      )
    )
  )
  testImpact[testFileName] =
    absoluteFilePaths.map((f) =>
      f.replace(process.cwd(), ".")
    )
}
fs.writeFileSync(
  "./scripts/tia.json",
  JSON.stringify(testImpact, null, 2)
)

The script gets all of the test files and loops through them. Each loop iteration executes Jest with just one test file and reads the disk’s coverage report. Next, the script adds an entry to the change impact analysis list that contains the test file name and the corresponding source file names. Before writing the source file

names, it transforms the file’s absolute path into a relative one.

Finally, the script writes the change impact analysis list as JSON to disk, so a future execution of the test suite can use it as a filter. The list looks like this:

{
  "route-a-c.js": [
    "./src/route-a.js",
    "./src/route-c.js"
  ],
  "route-a.js": [
    "./src/route-a.js"
  ],
  "route-b.js": [
    "./src/route-b.js"
  ],
  "route-c.js": [
    "./src/route-c.js"
  ]
}

We can see here that most tests only affect one source file, but theroute-a-c.js test affects two source files. As a result, the tia.json file must be checked into the source control and updated every time the test suite changes.

Running Only Impacted Tests

Now, we can run only essential tests. But let’s look at the run-tia.js script before we run it.

 const requiredTestFiles = []
  const allChangedFiles = JSON.parse(process.argv[2])
  const { FORESIGHT_JEST_ARGUMENTS } = process.env
  if (FORESIGHT_JEST_ARGUMENTS) {
    const foresightArgs =
      FORESIGHT_JEST_ARGUMENTS.split(" ")
    requiredTestFiles.push(...foresightArgs)
  }
  const changedSourceFiles = allChangedFiles
    .filter((file) => /src\/.*.js/gi.test(file))
    .map((file) => "./" + file)
  for (const sourceFile of changedSourceFiles)
    for (const testFile of Object.keys(tia))
      if (tia[testFile].includes(sourceFile))
        requiredTestFiles.push(testFile)
  if (requiredTestFiles.length < 1)
    return console.log(
      "No tests cover the changed files. Aborted."
    )
await jest.run(requiredTestFiles)

The idea here is that the script will get a list of changed files from the CI/CD pipeline as an argument. It will then check which tests cover the files and pass the tests to the Jest library. This way, only tests that are related to the changes will be executed.

If we look at the GitHub Action definition, we can see where the changed files come from.

 steps:
  - id: files
    uses: jitterbit/get-changed-files@v1
    with:
      format: json
  - uses: actions/checkout@v2
  - name: Use Node.js $0
    uses: actions/setup-node@v2
    with:
      node-version: $0
  - name: Install dependencies
    run: npm ci
  - name: Run tests
    run: npm run test:tia -- '$'
    if: $

The files step will gather all files changed in the push or PR that triggered the action. The Run tests step will only execute if a file in the src directory has changed and passed a JSON array of these to our test:tia script.

To run the tests, make a change inside a file in the src directory, commit it, and push it to GitHub. GitHub Actions will start automatically right after the push.

Getting More Insight with Foresight

It’s nice to filter out tests that aren’t needed, but we also want to make the most of the tests we keep and continue to execute—and there’s no better tool for optimizing your tests than Foresight. It can be integrated simply by installing Foresight's GitHub app on the GitHub Marketplace and changing the “Run tests” step in the run-test-tia.yml file.

First, create a Foresight account and connect your pipeline by simply installing the GitHub app. After installing the app successfully, you will see the project creation and repositories screen. You should name your first project and select the repositories you want to monitor.

In order to gain insights about our tests such as grouping tests, test suites along with their logs, screenshots, and more to understand why even the most complex integration test failed; we need to update our YAML file with Foresight's report uploader step and troubleshoot our test failures easily.

We can change the GitHub action for our tests to use the Foresight integration.

steps:
      - id: files
        uses: jitterbit/get-changed-files@v1
        with:
          format: json
      - uses: actions/checkout@v2
      - name: Use Node.js $0
        uses: actions/setup-node@v2
        with:
          node-version: $0
      - name: Install dependencies
        run: npm ci
    - name: Foresight Test Report Uploader
          if: always()
      uses: actions/upload-artifact@v2
      with:
        name: test-result-jest
        path: ./target
     - name: Run TIA tests
        run: "npm run test:tia -- '$'"
        if: $

That’s it. Now we can change a file in the src directory again, commit it, and push it to GitHub to see if everything is working.

It will take a few minutes to get the results in Foresight, but after that push, you can go to the Foresight web console, open your project, and wait for the results.

Foresight provides full visibility and deep insights into the health and performance of your tests and CI/CD pipelines. You can assess the risk of changes, resolve bottlenecks, reduce build times, and deliver high-quality software at speed.

Conclusion

Big test suites give us peace of mind when modifying our code, but they also slow down the delivery of new releases—the bigger the test suite, the slower the pipeline. With change impact analysis, we have a solution to that problem. If we only run tests related to our changes, we can save time while still checking the crucial parts of our code. Sign up for Foresight to improve your test runtimes and gain insight into your tests.

Save the date! 🕊️
On the 28th of September, we are launching Foresight on Product Hunt 🎉🍻
Take your seat before we launch on Product Hunt 💜