Load Testing GraphQL Performance With k6 and StepZen

For many companies, performance is the main reason to go with GraphQL. But is that a valid argument? Often developers compare GraphQL to REST APIs and see the N+1 requests (or over-fetching) as an important reason to choose GraphQL. Let's put that to the test and explore if GraphQL APIs can outperform existing REST APIs. For this, we'll take two GraphQL-ized REST APIs (IP-API and Frankfurter) and load test the performance of GraphQL, nested data in GraphQL, and compare it with the original REST APIs. We'll be using the tool k6, one of the most popular load testing tools today, to do this performance test.

Explore a GraphQL API

To load test this GraphQL API, we’ll be using k6, an open-source load testing tool. You can run k6 on your local machine by installing it from the GitHub repository or using k6’s cloud offering. Using this tool, you can test any API that accepts HTTP requests. The most straightforward test you can run uses the http.post function from k6:

import http from 'k6/http';

const query = `
  query GetLocation {
    ipApi_location(ip: “8.8.8.8”) {
      id
      city
      country
    }
  }
`;

const headers = {
  'Content-Type': 'application/json',
};

export default function () {
  http.post(
    'https://graphqldd.stepzen.net/api/dd1cf47f51ac830fe21dc00ec80cee65/__graphql',
    JSON.stringify({ query }),
    { headers },
  );
}

This k6 script can now hit the GraphQL API using the query to load test its performance. Because the IP address is static, k6 will make the same request repeatedly in your performance test. Furthermore, as the GraphQL API will cache the results, the tests will get less realistic as different users will use different IP addresses when hitting the GraphQL API. Therefore you should make use of dynamic variables in your GraphQL query. The same query with a dynamic value for ip will look like this:

query GetLocation($ip: String!) {
  ipApi_location(ip: $ip) {
    ip
    city
    country
  }
}

When sending the request, you need to append a JSON object containing a value for ip alongside your GraphQL query. If you visit the GraphiQL interface, you can use the tab "query parameters" for this:

Using the dynamic query parameters, you could use the http.batch function from k6 to send multiple requests with different values for ip to the GraphLQ API to simulate a more realistic testing scenario:

import http from 'k6/http';

const query = `
  query GetLocation($ip: String!) {
    ipApi_location(ip: $ip) {
      ip
      city
      country
    }
  }
`;


const headers = {
  'Content-Type': 'application/json',
};


export default function () {
  http.batch([
    [
      'POST',
      'https://graphqldd.stepzen.net/api/dd1cf47f51ac830fe21dc00ec80cee65/__graphql',
      JSON.stringify({ query, variables: { ip: '8.8.8.8' } }),
      { headers },
    ],
    [
      'POST',
      'https://graphqldd.stepzen.net/api/dd1cf47f51ac830fe21dc00ec80cee65/__graphql',
      JSON.stringify({ query, variables: { ip: '12.0.1.3' } }),
      { headers },
    ],
    [
      'POST',
      'https://graphqldd.stepzen.net/api/dd1cf47f51ac830fe21dc00ec80cee65/__graphql',
      JSON.stringify({ query, variables: { ip: '95.120.0.0' } }),
      { headers },
    ],
  ]);
}

Running this k6 script will send a batch of requests to the GraphQL API with different IP addresses. These requests are sent in parallel, giving you a more realistic scenario than sending just one. You could also create an array of IP addresses and loop over this to create a new array that you pass to the http.batch function. The next section will use these scripts to performance test this GraphQL API with k6.

Load testing a GraphQL Query

With the k6 scripts set up, we can now do a performance test on the GraphQL API. We can run two tests, one with a static IP address and another with batched requests and dynamic IP addresses. Running these tests only requires you to have k6 downloaded and installed on your local machine, or you should have a k6 cloud account.

To run the first test, you need to save the script in a file called simple.js so you can run the test with:

k6 run --vus 10 --duration 30s simple.js

This command runs k6 with 10 VUs (Virtual Users) for 30 seconds. To get more information on running k6 go here.

The results show that the GraphQL API was hit almost 2500 times in 30 seconds, with an average duration of 122ms. Which is very close to the average duration of the hits for 95% of all requests meaning there are no outliers.

By looking at the results, we can also test the scalability of the GraphQL API that runs on StepZen. Therefore we need to have a closer look at the number of iterations that the GraphQL API handled:

iterations.....................: 2472 82.082529/s

When we ran the k6 script for 30 seconds with ten concurrent VUs, you can see that k6 hit the GraphQL API almost 2500 times - or 82 times per second. If the GraphQL API is perfectly scalable, it should be able to handle ten times more iterations when we increase the number of concurrent VUs to 100. Let's try this out:

k6 run --vus 100 --duration 30s simple.js

This results in the following:

As expected for a perfectly scalable service, the number of iterations isn't 820 but 798, which is only a 3% difference. The GraphQL API isn't perfectly scalable but is getting pretty close to being so.

Next to testing the simple query with a static IP address, we can also run the script with the dynamic IP address by placing it in a file called batch.js:

k6 run --vus 10 --duration 30s batch.js

The iterations to the GraphQL API in this test are sent in batch, meaning that every iteration results in three HTTP requests - the number of requests added to the http.batch function. As we learned previously, the GraphQL API is almost perfectly scalable.

The number of iterations the GraphQL API can handle in this test should be roughly the same, while the number of HTTP requests should be around three times larger. When I ran the test, the number of iterations resulted in:

http_reqs......................: 7251   240.737555/s
iteration_duration.............: avg=124.43ms min=116.53ms med=121.91ms max=509.1ms  p(90)=126.39ms p(95)=129.13ms
iterations.....................: 2417   80.245852/s

With 2417 versus 2500 requests, the number of iterations is comparable, and the number of HTTP requests is three times larger than the number of iterations.

Now we know k6 can test the performance of GraphQL APIs and the GraphQL API is scalable. Let's proceed by testing a heavier GraphQL query in the next section.

Load testing different data sources

The ability to determine the shape of the data isn't the only reason developers adopt GraphQL as the query language for their APIs. GraphQL APIs have just one endpoint, and the queries (or other operations) can also handle nested data. You can request data from different database tables (like with SQL joins) or even various data sources in one request. This is different from REST APIs, where you typically have to hit multiple endpoints to get data from other sources.

In the GraphiQL interface for the StepZen GraphQL API we're testing, you can explore what other queries are available. One of those queries will get data from the Frankfurter REST API, an open-source API containing current and historical exchange rate data published by the European Central Bank. This REST API is converted to GraphQL using StepZen in the same way as the IP-API was. To get the current conversion rate from Euros to US Dollars, you can query:

query GetConversion {
  frankfurter_latest_rates(from: "EUR", to: "USD") {
    amoun
    base
    date
    rates
  }
}

The query above gets the conversation rate from 1 EUR to USD on the current date. Because the schema for this API is a combination of the data from IP-API and Frankfurter, there is more you can do. Using this combination, you can get the current location based on the IP address and convert the local currency of that location to USD. This currency conversion is available on the field priceInCountry. You can see the result by visiting the GraphiQL interface, or in this image below:

Besides the IP address location, this query also lets the GraphQL API convert Euros to the local currency of that location. In this case, it means converting Euros to Dollars again.

To get this data, the GraphQL API will do the following:

• Send a request to the underlying IP-API REST API to get the location and currency based on the IP address;
• Convert the currency of that location to Euros using the Frankfurter REST API.

We can use this nested query in a k6 script to do another performance test of the GraphQL API. This query has a different depth than the query we've used in the previous section, as the data is now nested as it comes from different sources. You can put the following k6 script in a new file called nested.js:

import http from 'k6/http';

const query = `
  query GetConversion($ip: String!, $amount: Float!, $from: String!) {
    ipApi_location(ip: $ip) {
      ip
      city
      country
      currency
      priceInCountry(amount: $amount, from: $from)
    }
  }
`;

const headers = {
  'Content-Type': 'application/json',
};

export default function () {
  http.post(
    'https://graphqldd.stepzen.net/api/dd1cf47f51ac830fe21dc00ec80cee65/__graphql',
    JSON.stringify({
      query,
      variables: {
        amount: 1,
        from: 'EUR',
        ip: '8.8.8.8',
      },
    }),
    { headers },
  );
}

And run it under the same circumstances as the previous tests:

k6 run --vus 10 --duration 30s nested.js

The results of this performance test are similar to the test with the batched requests. The batching isn't occurring in the k6 script but at the GraphQL API, which handles the two requests to the underlying REST APIs. As the GraphQL API is built to be performant, the difference between the initial query to get just the location of the IP address and this query to get both the IP address location and the currency conversion is minimal. Something you can check in the output of the k6 load test below:

The test we've just run shows GraphQL is perfectly able to get your data from different sources all in one request. Let's break down the GraphQL query to the REST API requests they are doing under the hood in the final section to prove this point.

Compare GraphQL to REST performance

You've already learned how to performance test GraphQL using k6 and how that differs from testing a REST API. The last GraphQL query we've tested is calling two different REST API endpoints. There are two scenarios we can try to compare GraphQL and REST performance. Either set up two separate tests to compare the performance of the individual REST endpoints against their corresponding GraphQL queries or recreate the complete behavior of the GraphQL API by directly calling the two REST API endpoints from a k6 test.

The latter is the most interesting to test, as both REST endpoints need to return data for the GraphQL query to resolve. The GraphQL query behavior we're testing is:

query GetConversion($ip: String!, $amount: Float!, $from: String!) {
  ipApi_location(ip: $ip) {
    ip
    cit
    country
    currency
    priceInCountry(amount: $amount, from: $from)
  }
}

This sends requests to the REST endpoints:

• To get the location and currency for IP address 8.8.8.8 from IP-API: http://ip-api.com/json/8.8.8.8?fields=city,country,currency
• To convert the currency from the IP address location to Euros from Frankfurter: https://api.frankfurter.app/latest?amount=1&from=EUR&to=USD

To test this using k6 you need to set up the following script in a new file, let's call it rest.js:

import http from 'k6/http';
import { check } from 'k6';

const headers = {
  'Content-Type': 'application/json',
};

export default function () {
  const response = http.get(
    'http://ip-api.com/json/8.8.8.8?fields=city,country,currency',
    null,
    { headers },
  );

  check(response, {
    'currency should be returned': (res) => res.json().currency === 'USD',
  });

  http.get(
    'https://api.frankfurter.app/latest?amount=1&from=EUR&to=USD',
    null,
    { headers },
  );
}

Not only will this k6 performance test call the two REST endpoints. But in a real scenario, it would make no sense to reach the second endpoint to convert the locations' currency to Euros if the first request doesn't return the local currency of the IP address. Running the test above with the same conditions as we did for the other tests:

k6 run --vus 10 --duration 30s rest.js

Gives the following results:

The first thing that stands out is that only 50 iterations with two HTTP requests per iteration are complete in 30 seconds. Only 3 VUs have been able to send requests in these tests, and 20% of all requests have failed. Compared to the GraphQL API, these test results are disappointing. The GraphQL-ized versions of the two REST APIs can handle many more requests and resolve them faster. Why is this? The GraphQL API created using StepZen applies caching where the REST API itself doesn't seem to apply any caching at all. The requests to the IP-API endpoint seem to fail twenty percent of the time. Furthermore, the GraphQL API will also batch requests or optimize for N+1 requests when needed. For example, you need to request the same data from the currency REST API twice.

Conclusion

GraphQL usage has been on the rise, especially amongst front-end developers who want a more declarative way to handle data. In this post, we've explored how to performance test a GraphQL API using k6. This GraphQL API was created with StepZen by converting the open-source REST APIs from IP-API and Frankfurter. The test results showed the GraphQL API was close to being perfectly scalable and performant in all scenarios. In contrast, the individual REST API endpoints had significant performance issues in the tests. This is due to the StepZen GraphQL API's performance optimizations, such as caching. For a complete performance test script for GraphQL, see the StepZen GraphQL Benchmark tool in this repository.

Want to learn more about StepZen? Try it out here or ask any question on the Discord here.