Setup and Teardown of Tracetest Tests with Test Suites

A customer recently asked what the best way to run setup or tear down steps before running a test. Since this is a common pattern, we felt a pair of articles discussing it would be helpful. This article will focus on using the test suite capability to chain together and pass variables between Tracetest tests.

We also support using the @tracetest/client NPM package to programmatically execute setup steps and call the Tracetest test as part of a JavaScript or TypeScript program.

Want to write your setup process in JavaScript or TypeScript instead? Read “Automating Tracetest Tests via TypeScript or JavaScript”.

What is Trace-based Testing and Why You Should Care?

Before we dive in, let me answer the first question: “What the heck is Tracetest?”

It’s an observability-enabled testing tool for Cloud Native architectures. Distributed tracing was needed to empower SREs to troubleshot the complex, interconnected systems being used in microservices and FaaS-based architectures. Tracetest can be leveraged in your preproduction environments by DevOps, QAs, or SREs to test critical flows. Since it uses your existing observability tooling, its tests have more visibility into the overall system. This technique is called trace-based testing, and has several compelling advantages:

• Makes it easier to write tests, as you can see all the systems involved in the flow.
• Tests can be more powerful and assert on conditions that are not visible to black box tests.
• When troubleshooting failed tests, you have the full distributed trace available, decreasing MTTR.

How to Handle Setup and Teardown of Tracetest Tests with Test Suites?

Tracetest’s codeless approach for this requirement utilizes test suites to chain together several tests into one comprehensive flow. Tracetest tests have the ability to trigger activity in the system under test, triggering activity based on the following types of calls:

• HTTP
• GRPC
• Message queue

Using this capability, you can run “setup tests” before the main test and “clean up” tests after it.

This article describes a test scenario, the two tests involved, and provides instructions so you can download and run the examples yourself.

The Test Scenario

This test scenario is a common one. Adding an entity so you can then test the deletion of it.

The system we will be testing if a Pokemon was deleted successfully. You’ll add and delete a Pokemon with two tests. The tests are:

• Adds an entity as a setup step. For this example, we are adding the Pokemon Fearow!
• Run a delete test to verify that the deletion removes the entity we added in the setup step. This is the main flow we are interested in testing. Hopefully we can say “Goodbye Fearow!”.

We are going to use a pre-deployed version of our Pokeshop demo which is sending its telemetry information to Jaeger. This instance is publicly available so you can run tests against it.

You can even view the UI, here. The code for this Pokeshop Demo is available at on GitHub at kubeshop/pokeshop.

To follow the examples and see the code below, clone our main repo:

git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/setup-of-tracetest-tests

The Delete Test

The delete test is contained in the file named delete_pokemon.yaml. This test does a DELETE call against a specific Pokemon based on the id of the Pokemon. We are passing the id of the Pokemon into this test using an environment variable using the syntax ${env:pokemon_id}.

This trace-based test has four test specifications. The first is a typical assertion you might see with any black box test framework:

• Delete returns a 200 status code: Ensures the REST API returns a successful status.

The next 3, however, leverage the visibility enabled by trace-based testing to look deeper into the execution, allowing you to assert on what actually happened by looking at the data contained in the captured distributed trace:

• Ensure we are deleting from the Redis cache also: A more insightful assertion, and one you cannot make with a normal, black box test.
• All Database Spans: Processing time is less than 10ms: Check performance as part of the test so it will fail if a slow query is introduced.
• Check that number of deleted rows from Postgres is one: Verify the delete actually occurs.

Here is the delete_pokemon.yaml test:

type: Test
spec:
  id: delete-pokemon
  name: Delete Pokemon
  trigger:
    type: http
    httpRequest:
      method: DELETE
      url: https://demo-pokeshop.tracetest.io/pokemon/${env:pokemon_id}
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[tracetest.span.type="general" name="Tracetest trigger"]
    name: Delete returns a 200 status code
    assertions:
    - attr:tracetest.response.status = 200
  - selector: span[tracetest.span.type="database" db.system="redis" db.operation="del" db.redis.database_index="0"]
    name: Ensure we are deleting from the redis cache also
    assertions:
    - attr:tracetest.selected_spans.count = 1
  - selector: span[tracetest.span.type="database"]
    name: "All Database Spans: Processing time is less than 10ms"
    assertions:
    - attr:tracetest.span.duration < 10ms
  - selector: span[tracetest.span.type="database" name="delete pokeshop.pokemon" db.system="postgres" db.name="pokeshop" db.user="ashketchum" db.operation="delete" db.sql.table="pokemon"]
    name: Check that number of deleted rows from Postgres is one
    assertions:
    - attr:db.result = 1

Now, lets look at the setup step which we will run before to add the Pokemon.

Using Another Tracetest Test to Set up the Data

We can define a new test to create the Pokemon, and then chain the creation and deletion test together within a test suite. First, let’s look at the setup_pokemon.yaml test. It is fairly basic, with 3 primary areas of interest:

• It is doing a POST against the /pokemon endpoint and passing body data to create the Pokemon.
• We are checking for a 201 response from this POST to ensure the creation of the Pokemon worked. This is just a basic sanity check so the entire test suite will fail if something goes wrong in this setup step.
• We are getting the id of the Pokemon by selecting data returned in the trace, parsing it with a json_path statement, and assigning it to the output variable pokemon_id. This is enabled with Tracetest’s test output capability.

type: Test
spec:
  id: setup-pokemon
  name: Setup Pokemon
  trigger:
    type: http
    httpRequest:
      method: POST
      url: https://demo-pokeshop.tracetest.io/pokemon
      body: "{\n  \"name\": \"fearow\",\n  \"type\":\"normal,flying\",\n  \"imageUrl\":\"https://raw.githubusercontent.com/PokeAPI/sprites/master/sprites/pokemon/22.png\",\n  \"isFeatured\": false\n}\n"
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[tracetest.span.type="general" name="Tracetest trigger"]
    name: Check to see the post worked
    assertions:
    - attr:tracetest.response.status = 201
  outputs:
  - name: pokemon_id
    selector: span[tracetest.span.type="general" name="Tracetest trigger"]
    value: attr:tracetest.response.body| json_path '$.id'
  skipTraceCollection: true

Combining the Two Tests into a Test Suite

We can then create a test suite, include the setup_pokemon.yaml test as the first to run, then call our main test, delete_pokemon.yaml. Creating test suites is covered in our docs.

When you have finished creating a test suite, the automate tab provides the YAML definition for the suite. Below is the definition for the delete_test_suite.yaml test suite we created:

type: TestSuite
spec:
  name: Delete with Setup
  description: A test of the deletion end point with setup test which adds the pokemon as a setup step.
  steps:
  - setup-pokemon
  - delete-pokemon

The setup-pokemon test outputs the pokemon_id variable, and the delete-pokemon test expects a variable with this name via the ${env:pokemon_id}. With this syntax, when you run them as part of a test suite the first test will automatically pass the value and the second test will consume it. This lets the delete-pokemon test know to delete the Pokemon that was created in the setup-pokemon setup test.

How do we Run the Example?

Enough talking - lets see this test suite run! First, clone the repo if you did not earlier:

git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/setup-of-tracetest-tests

We are going to do several steps so we can see the test suite run:

• Join the Tracetest pokeshop-demo environment so you can run and view your tests
• Install the Tracetest command line tool (CLI) so you can upload the two tests
• Use the CLI to run the test suite and view the test output
• Open the test suite in Tracetest and examine the results from the suite and the delete test.

Let’s Do It!

Let’s run the test suite inside of the pokeshop-demo environment which we have available as part of our tracetest-demo organization at app.tracetest.io. In order to connect and run tests in this environment, you will need to join it with this link:

➡︎ Join the Pokeshop Demo ⬅︎

You will be promoted to sign in with either your GitHub or Google account. Once logged in, click the “Join organization” button to get access to the Pokeshop demo environment.

Great - you are now a member of the organization with access to the Pokeshop demo environment! We will be running the test suite against this environment from the command line interface (CLI) tool, so you will need to install the CLI. The instructions are documented here, but for Mac, you can just type:

brew install kubeshop/tracetest/tracetest

Now, tell the command line what organization and environment we want to issue commands against by typing:

tracetest configure

You will need to select the tracetest-demo organization and the pokeshop-demo environment as shown below:

You’re ready to add the two tests to this environment so they can be run by the test suite. The tests are part of the repo you cloned, so just type the following:

tracetest apply test -f setup_pokemon.yaml
tracetest apply test -f delete_pokemon.yaml

Now that the definitions for the two tests have been created in the test environment, you can run the test suite, which chains the two tests together and executes them.

tracetest run testsuite -f delete_test_suite.yaml

Here is the output you should see after running the test suite:

This output shows that the “Delete with Setup” test suite was run successfully. It also shows the two tests that were run as part of the suite. Both the suite and the two tests show links which will allow you to view the results.

And here is what you will see if you click on the Delete Pokemon test results and click to see the “Test” tab results. It shows the 4 test specs you are running as part of the delete test.

Congratulations! You have walked through the process needed to create two tests, combine them into a test suite, and pass a test output from one to another. This is an easy way to chain tests together in order to perform setup tasks.

What’s next?

While I did not show an example, you can use the same test suite technique to run a teardown or cleanup process after a test. Rename the processes “teardown” rather than “setup” and run them after the test! 😃

Now that you have seen how to run setup steps with test suites, you are now ready to create your own tests! Follow these steps:

• Create a new organization and environment.
• Use the environment setup wizard to enable your Tracetest environment to run tests against and collect distributed trace information from your system under test.
• Start creating tests!

To learn how to automate this same setup process in JavaScript or TypeScript instead, check out part 2 of called “Automating Tracetest Tests via Typescript or Javascript”.

Have questions? The devs and I hang out in the Tracetest Slack channel - join, ask, and we will answer!