Load testing your applications with Artillery

In this article, I will show how to use Artillery to load test your application.

What is Artillery?

Artillery is a modern, powerful & easy-to-use solution for load testing and functional testing.

It´s built with NodeJS and it´s open source.

Unlike other load testing tools that have complicated GUIs, Artillery is a simple CLI tool, making it very easy to use and to integrate into any CI environment.

It supports testing applications that use HTTP, Socket.io or Websockets.

One of the biggest points of Artillery is that it allows defining test scenarios using YAML, making it possible to version the tests together with the application code.

Getting started

Artillery can be installed like any other Node package, using NPM:

npm install -g artillery 

Doing a quick test

Artillery can be used to do one-off tests to a specific URL in a similar way of tools like Apache Benchmark.

That is done using the quick command.

artillery quick -c 10 -n 20 https://httpbin.org/get

This will launch 10 virtual users that will do 20 requests each, to the specified URL.

The quick command supports some other flags to control how the load will be distributed, like rate, which can be used to define the number of new users per second, or the duration, which defines a fixed time for the test.

For example, the following command will run the test during 10s, with 2 new users arriving each second, resulting in around 20 requests in the total.

artillery quick https://httpbin.org/get -r 2 -d 10

You can combine all these flags, to achieve the desired pattern of load for your test.

In terms of other options, I found this command to be pretty limited as you can´t configure almost anything more of the request, like defining request headers. It also only allows GET requests.

So, if you just want to do some one-off test directly from the command line and need a little more flexibility configuring the request, there are probably better tools like Apache Benchmark.

But the power of Artillery lies in being able to simulate realistic user behavior with scenarios and flows, using declarative YAML files.

Scnearios and YAML configuration

The most basic test case can be defined like this:

config:
  target: 'https://httpbin.org/'
  phases:
    - duration: 60
      arrivalRate: 20
scenarios:
  - flow:
    - get:
        url: "/get"

The config section allows defining some generic configuration about the test, like the target URL, the duration of the test and the way users will generate load.

In this example, we define one phase, which will last 60 seconds with 20 new virtual users (arriving every second (on average)).

The scenarios section defines the scenarios this test will cover. Each scenario has a flow associated with it, which is a set of steps that each user will do in your application.

We will see this in more detail, later in this article.

In this particular case, we have defined one single scenario, with a single step, which is a GET request to the https://httpbin.org/get endpoint.

You can run this test using the run command:

artillery run <path_to_file>.yml

This is very similar to the "quick" example, just a request to a specific endpoint.
But unlike the "quick" command, using a configuration allows you to define a lot more params of the request, like HTTP method, headers, cookies, request payload, etc. You can even load your payload dynamically from CSV files.

Here is a more advanced example demonstrating how to do a POST request with payload data loaded from a CSV file.

  config:
    payload:
      # path is relative to the location of the test script
      path: "users.csv"
      fields:
        - "username"
        - "password"
  scenarios:
    - flow:
        - post:
            url: "/auth"
            json:
              username: "{{ username }}"
              password: "{{ password }}"

This test will do a "POST" request to the "/auth" endpoint, with a JSON body containing the fields "username" and "password", which values are loaded from a CSV file "users.csv", from the specified fields.

You can find out about all the possibilities in the Documentation.

A more complete user flow

We have seen how to define a scenario with a simple flow.
Next, we will see how you can define an entire user journey.

Considering a typical E-commerce site. You might want to simulate a user flow, starting from the catalog page, selecting a product, and add it to the cart.

This can be achieved with the following configuration:

config:
  target: "https://staging1.local"
  phases:
    - duration: 60
      arrivalRate: 5
  payload:
    path: "keywords.csv"
    fields:
      - "keywords"
scenarios:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          body: "kw={{ keywords }}"
          capture:
            json: "$.results[0].id"
            as: "id"
      - get:
          url: "/details/{{ id }}"
      - post:
          url: "/cart"
          json:
            productId: "{{ id }}"

Some things we haven´t seen before.

A flow can consist of multiple steps, representing the user journey in the application. Each user spawned by Artillery will execute all the defined steps sequentially.

In this particular example, The first step is doing a POST request to the "/search" endpoint with a body, containing keywords, loaded from a CSV file.

- post:
          url: "/search"
          body: "kw={{ keywords }}"
          capture:
            json: "$.results[0].id"
            as: "id"

The "capture" block allows us to get a value from the response and save it as a variable for later use. You can use "JSONPath", "XPath", "Regex" and more to parse the response and get the value you want.

For example, to get a URL form an anchor tag in HTML response you could do as following:

- get:
    url: "/some/page"
    capture:
      - selector: "a[class^=productLink]"
        attr: "href"
        as: "productUrl"

Back to our E-commerce flow, We will use the captured "id" variable from the "search" step, to go to the "product details" page and to add the product to cart:

- get:
     url: "/details/{{ id }}"
- post:
    url: "/cart"
    json:
       productId: "{{ id }}"

As you can see with a few lines of YAML we can define an entire user flow to test.

Artillery supports many more functions to work with HTTP requests and responses. You can find then in the http reference section of the documentation.

Multiple Scenarios

In the previous example, we saw how to define a single scenario with multiple steps.
But you can also define multiple scenarios to simulate users doing different things at your application at the same time.

Let´s say you want to simulate users searching and buying products at the same time. You can create two different scenarios to test both flows. Note that each user will only execute one of the scenarios.

The scenario supports a "weight" property which you can use to indicate how likely it is that scenario to be picked by a particular user.

By default, each scenario will have the same probability. If you define a scenario with weight 2, that scenario is twice as likely to be picked as the one with 1 (which is the default value).

Setting success conditions

It´s possible to configure Artillery to return a non-zero exit code if the test run doesn't comply with specified conditions based on a set of parameters like error rate, minimum, maximum and percentile based latency.

This is really useful in a CI environment as you can make the test fail if it doesn´t meet your performance requirements.

The following configuration will ensure that at least 95% of requests are executed below 200ms, otherwise, the command will exit with an error.

config:
  ensure:
    p95: 200

Extend Artillery

Hooks

Artillery has support for "hooks", which allow for custom JS functions to be executed at certain points during the execution of a scenario.

The following extension points are available:

• beforeScenario and afterScenario - called before/after a virtual user executes a scenario
• beforeRequest - called before a request is sent. request parameters can be customized here.
• afterResponse - called after a response has been received. The response can be inspected and custom variables can be set here.
• function - which can be inserted as a step at any point in a scenario.

You can use these hooks, to log extra information you need or to modify the request of some sort before doing a request.

this article shows a very good example of this, by leveraging Faker.js to generate random data for the tests.

Plugins

Artillery supports custom plugins that you can write using Javascript.

Artillery has some official plugins like artillery-plugin-expect that can be used to add expectations/assertions to your HTTP scenarios, allowing you to do some basic functional tests together with your load tests or the artillery-publish-metrics, which allows sending metrics about your test runs to external services like Datadog, InfluxDB or StatsD.

Here is an example of the artillery-plugin-expect plugin in use. See The "expect" section:

scenarios:
  - name: Get pets
    flow:
      - get:
          url: "/pets"
          capture:
            - json: "$.name"
              as: name
          expect:
            - statusCode: 200
            - contentType: json
            - hasProperty: results
            - equals:
              - "Tiki"
              - "{{ name }}"

You can build your own to add extra functionality.

Go distributed with Artillery Pro.

Besides the Open source version, Artillery has a Pro version starting at $199/month.

The pro version offers deep integration with AWS services and allows us to run distributed load tests using AWS infrastructure, and includes other integrations designed to support testing of modern enterprise applications & modern DevOps workflows.

The open-source version should be enough for most people, but if your scale is pretty big, and you reached the limit of load that a single machine can generate, these extra features can be valuable.

You can find more about the Pro plan here.

Conclusion

There are many load testing tools out there, some a lot more popular like Jmeter or Gatling.

Where Artillery really shines for me compared with these tools, is the balance between simplicity and ease of use and functionality.

Being a simple CLI tool makes it very easy to install and use in every environment, from the local developer machine to the CI server.

It´s built with Node, which makes it a lot lighter these Java-based tools.

The tests are declarative in YAML files, so they are easily readable by everyone and can be versioned together with the application code.

Scenarios and flows provide great flexibility and allow you to go beyond testing a single endpoint at a time and test entire user journeys.

"Success Conditions" helps to keep your performance requirements under control, by integrating with your CI tool and make the build fail if it doesn't meet your performance criteria.

Hooks and plugins make possible to extend the tool with extra functionality.

There are no more excuses not to write load tests for your applications!