DEV Community

Cover image for GraphQL Schema Test with GraphQL Inspector
Katelyn Truong for Beanworks

Posted on with Divya rakhiani

GraphQL Schema Test with GraphQL Inspector


We were nearing the alpha release of one of our new modules which includes the release of a new mobile application. Our web app, new mobile app and our desktop app, all consume our graphQL API.
To increase the reliability of our platform, we needed to formalize and implement a backwards compatibility and deprecation policy for our GraphQL API.
The rules to add or delete from schema were

  • Only add new fields.
  • If you want to remove a field, deprecate it and remove it when it’s no longer in use.
  • If you want to change an existing field, create a new field, deprecate the old field and remove it once it’s no longer in use.

However, since the mobile app has a much larger footprint as users might be using the app with the older version with old graphQL schema, having a test for it seemed inevitable. Our existing e2e automation on mobile did not cover all graphQL endpoints, and it would have been too expensive and time consuming to cover all endpoints from a user interface perspective. Moreover, we wanted backward compatibility test to be a test that runs before we run e2e tests to detect breaking changes earlier in the cycle.

Since we needed something light weight and graphQL compatible, we decided to use GraphQL Inspector Tool to ensure backward compatibility and prevent schema breaking changes.

Tech Stack

  • Bitbucket (Git)
  • Codeship Pro for CI/CD workflow
  • Docker
  • GraphQL

Code Structure

Alt Text

Why GraphQL Inspector Tool?

We decide to use GraphQL Inspector Tool for graphQL backward capability test because of following reasons:

  • GraphQL compatibility
  • Ease of implementation with custom rules
  • Easy CLI command for CI integration

Steps to Implement

Step1: Create a dockerfile for GraphQL Inspector
  • Dockerfile template can be found here at graphQL inspector docker hub
  • Dockerfile can be named as Dockerfile.graphqlInspector in ./codeship folder
FROM node:10-slim


RUN apt update 
RUN apt install -y ssh
RUN yarn global add @graphql-inspector/cli@2.1.0 graphql
RUN apt install -y git-all

# Copy the decrypt ssh private key for Bitbucket Readonly User to allow codeship access to 
# Our repository from codeship container  
RUN mkdir -p /root/.ssh
RUN echo $PRIVATE_SSH_KEY >> /root/.ssh/id_rsa
RUN chmod 600 /root/.ssh/id_rsa
RUN ssh-keyscan -H >> /root/.ssh/known_hosts

RUN mkdir /app
Step2: Create an entrypoint file
  • This file will be triggered after docker image is built
  • Name file in ./codeship folder
    • To add git remote for our repository
      • Note: it uses remote name as api because origin already taken and it doesn't work in codeship
    • To fetch the master branch which will use for graphql inspector command in codeship
git remote add api<owner_name>/<repo_name>.git
git fetch api master
exec "$@"
Step3: Create graphQL inspector test service
  • graphql-inspector-test service will be placed in ./codeship-services.yml
  • Build from Dockerfile.graphqlInspector
  • Include entrypoint link with to trigger the command line after finishing pulling and building the docker image from dockerfile
  • For volumes, it can be different depending on your code structure or same as created above in dockerfile.
    • How to find which volume to mount
    • Install jet if you have not (for locally running commands for codeship)
    • cd <repo>
    • jet run SERVICE_NAME pwd
    • Example: jet run graphql-inspector-test pwd
    • With pwd, it will show the path of the volume where it should be mounted
      context: ./
      dockerfile: ./codeship/Dockerfile.graphqlInspector
      encrypted_args_file: <path_to_encrypted_git_ssh_key>
    encrypted_env_file: <path_to_encrypted_git_ssh_key>
    entrypoint: ["sh", "./codeship/"]
    cached: true
      - .:/app
Step4: Create graphql-inspector-test step
  • GraphQL Schema Inspector Test test step under Jest Tests in ./codeship-steps.yml
  • Use graphql-inspector-test service
  • To trigger command graphql-inspector diff git:api/master:./src/generated/graphql/schema.graphql ./src/generated/graphql/schema.graphql --rule ./src/generated/graphql/custom-rule.js
- type: serial
  name: Continuous Integration
  encrypted_dockercfg_path: dockercfg.encrypted
  - name: Tests
    type: parallel
    exclude: ^(production)
    - name: Jest Tests
      type: serial
      - name: Jest Tests for E2E Tests
        service: e2e-runner
        command: yarn test
      - name: ESLint for E2E Tests
        service: e2e-runner
        command: yarn run lint
      - name: Graph QL Test
        service: graphql-test
        command: yarn run graphql-test
      - name: Graph QL Schema Inspector Test
        service: graphql-inspector-test
        command: graphql-inspector diff
                 ./src/generated/graphql/schema.graphql --rule 
  - name: Deploy to Dev and Production
Step5: How to create custom rules for graphQL inspector test
  • For details, you can see graphql inspector custom rules
  • We use custom-rules to skip certain attributes that we don’t want the test to run
    • Example: the field may not be available to mobile apps yet and still under development, so there may be many changes to the field.
  • Create file name custom-rules.js in the folder where you have your schema.graphql
// custom-rules.js
 * @summary exceptionSet is an array of schema attributes that can be skipped during graphql schema diff test
 * for breaking changes. 
 * !important: by adding to exceptionSet, there is a risk of graphql schema test will skip these attributes
 * example: new Set(['Form.user]);
const exceptionSet = new Set([]);

 * @summary changes is an object array of all breaking changes [{error1},{error2}]
 * Each error object contains these key value pairs as follow 
 * critical: { level: 'BREAKING', reason: "some reason"}
 * message: "Field 'XYZ' was removed from interface 'ABC'"
 * path: graphql schema attribute, eg: 'Comment.description' 
module.exports = ({ changes }) => {
    const included = [];
    const excluded = [];
    for (const change of changes) {
        if (exceptionSet.has(change.path)) {
    if (excluded.length > 0) {
        const warnColor = '\x1b[33m';
        console.log('The following changes have been excluded from breaking changes test:');
        for (const change of excluded) {
            // print something sensible - should show that the field was excluded
            // and what potential problem we're ignoring ;-)
            const message = `!!! ${change.path}: ${change.message}`;
            console.log(warnColor, message);
    return included;

How to execute test locally

npm install --global @graphql-inspector/cli graphql
cd <repo_folder> 
graphql-inspector diff git:origin/master:./src/generated/graphql/schema.graphql ./src/generated/graphql/schema.graphql --rule ./src/generated/graphql/custom-rule.js

Alt Text


We were successfully able to add graphQL backward capabilities test that is easy to maintain, and is platform agnostic by using GraphQL Inspector Tool. We are able to gain a better understanding of how API should be handled for projects, where the same API is used across multiple platforms (i.e. web, desktop and mobile).

Top comments (4)

kamilkisiela profile image
Kamil Kisiela

We would love to give the same experience to Bitbucket as it is today on GitHub with our GitHub Application but the API of Bitbucket is super awkward to deal with.

I recommend to use a branch created by bitbucket (not sure if there's one like on github refs/pull/1234/merge) where you get the combination of the target branch and source branch. This way you won't end up in a state where Pull Request is behind the target branch.

mschelstastic profile image

Very cool!

paleka profile image

Very nice! Lots of useful details! Thanks!