Cover image for Enhance Your CI/CD Pipeline With Automated API Tests

Enhance Your CI/CD Pipeline With Automated API Tests

ginomessmer profile image Gino Messmer Originally published at Medium on ・5 min read

A Journey in Automated API Tests (2 Part Series)

1) Run your first automated API test in 5 minutes 2) Enhance Your CI/CD Pipeline With Automated API Tests

In my previous post, I looked into running API tests and automate them. For this, I stumbled upon Insomnia and Loadmill. This time, I want to integrate these tests into my CI/CD pipeline as easily as possible.

Let’s depart and check out some possibilities.

Framework specific tools

ASP.NET Core has a test server extension (it’s called TestServer) by Microsoft, a dependency that neatly integrates into ASP.NET Core and exposes an in-memory web host. In addition to that, it also provides a pre-configured HTTP client that we can use.

This extension does also integrates well into test frameworks like xUnit or NUnit. That sounds interesting enough so let’s give it a shot without overwhelming you too much.

Let’s take my work in progress homepage API again as a sample. For now, I maintain a simple project structure with a single API controller in it.


(yes I know, Homepage.API sounds generic & boring so I rolled with this name instead)

In my next step, I create a simple test project powered by xUnit, pull the test host dependency (dotnet add Microsoft.AspNetCore.TestHost), reference my API project, and modify the default test class to something like this:

public BlogControllerHttpIntegrationTest()
    // For testing purposes, I initialize my dependencies in here.
    // I recommend to use a test fixture instead though.

    // Set up local configuration
    Configuration = new ConfigurationBuilder()
        // ...

    // Set up web host
    var builder = new WebHostBuilder()
        .ConfigureAppConfiguration(builder =>

    // Set up test server
    _server = new TestServer(builder);
    _client = _server.CreateClient();

    _client.DefaultRequestHeaders.Add(HeaderNames.Authorization, $"Bearer {Configuration["Jwt"]}");

And our test method looks like this:

public async Task Create_NewPost_Successfully()
    var response = await _client.PostAsync("/blog/posts", new StringContent(JsonConvert.SerializeObject(new
        id = "test-post"
        // ... trimmed for brevity


This goes without saying, but you can run this integration test in any CI/CD environment. Visual Studio can also run tests while you are developing (‘live tests’). It sounds like a great way to ensure that your code’s quality is on a high bar next to your typical unit and integration tests. Pretty cool.

This method is a bit farther away from production because it runs in in-memory, but it helps you to test with confidence in development and staging. Not bad.

In the realms of CI/CD

Since I utilize Loadmill for evaluation purposes lately, I thought I’d continue this path and see how we can use it in this scenario. I stumbled upon their official npm package for CI which looks promising. In my case, I’m using Azure DevOps, but this works with any other CI tool.

Loadmill’s npm package allows you to run a pre-configured Test Suite from remote. I have an existing Azure Pipelines definition that releases the new build to my server. Let’s modify this definition so it can also run the npm package with the same test suite I created last time.

For this, we create a new stage that depends on the release stage.

- stage: PostDeployTest
  displayName: 'Post Deployment Tests'
  dependsOn: Deploy
  condition: variables.deploymentEnabled
  - job: TestProd
    displayName: 'Run Tests in Production'
    - task: CmdLine@2
      displayName: Run Loadmill Test
        script: 'npx loadmill $(loadmill.testsuite) -s -w -t $(loadmill.token)' # <-- Here's where the magic happens

In the npm step, I simply copy-pasted the command from my Test Suite’s CI section. Before I did that I created a few additional Flows and toggled the CI flag. Their package works great as a node module or CLI tool. It can do both at the same time. If the test fails, the tool will return a nonzero exit status. Our pipeline essentially fails if that’s the case. You can have a look at my full pipeline definition here if you like to.

Pipeline definition

In case I want to test a different environment other than production, then there’s a way to provide or override parameters with the CLI tool. In my case, I could simply change the ${baseUrl} parameter to a different host.

One of the biggest advantages of this is that you can keep your test configuration separate from your CI pipeline. Other than that, I make use of something that I’ve already configured and used before, so there’s no repetitive work involved. This is a massive time saver while you gain a huge benefit in CI.

If you want to make changes to the API tests, then you will just need to access your Loadmill project and perform everything else from there. This also helps you to be confident in production because the test gets executed just like you would do locally. You’re not only testing with HTTP requests, but you are also testing whether or not your host is reachable. In other words, whether or not your deployment step has worked successfully without any silly fuzz in the middle of the deployment journey.

I can recommend this extensive article by Cindy Sridharan on Medium who goes in-depth about different phases, types, and approaches of testing in production; it’s worth a read during an afternoon at work if you’d like to know more about release techniques.


In the end, you should go with an approach that suits your whole workflow during development and release phases (these include pre- and post-release).

While I wrote this article I found it super useful to keep tests that are tailored for development/staging and production separate. The TestHost extension is perfect for ASP.NET and works for live testing or tests in CI while the binaries are being built.

Before I deploy to production, I don’t want to test my business logic solely. I also want to test if the actual server has acceptable response times, doesn’t mess with cache, works well with load balancing and so on. I want to test the whole infrastructure as a whole in production, I don’t want to restrict it to my business logic. After that, I’m most likely confident to depart to full-production. Loadmill or similar third-party tools help you to easily integrate and manage it.

This was the second post of my journey in automated API tests. Please let me know what you think. You can also follow me on Medium or dev.to for future posts.


A Journey in Automated API Tests

Further references

  1. Integration Tests in ASP.NET Core: https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests
  2. Loadmill’s npm package: https://www.npmjs.com/package/loadmill
  3. Build and Deployment Automation Case Study for World Wide Time Keeping: https://docs.microsoft.com/en-us/azure/devops/migrate/build-deployment-best-practices
  4. Phased Out Release Best Practices: https://docs.microsoft.com/en-us/azure/devops/migrate/phase-rollout-with-rings
  5. Testing in Production, the safe way: https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1

Photo by Brian Suman on Unsplash

A Journey in Automated API Tests (2 Part Series)

1) Run your first automated API test in 5 minutes 2) Enhance Your CI/CD Pipeline With Automated API Tests

Posted on Mar 2 by:

ginomessmer profile

Gino Messmer


Full-stack software engineer & student // .NET, Azure, Angular, React, DevOps // I build and sometimes break stuff


markdown guide