GraphQL Schema Test with GraphQL Inspector

Background

We were nearing the alpha release of one of our new modules which includes the release of a new mobile application. Our web app, new mobile app and our desktop app, all consume our graphQL API.
To increase the reliability of our platform, we needed to formalize and implement a backwards compatibility and deprecation policy for our GraphQL API.
The rules to add or delete from schema were

• Only add new fields.
• If you want to remove a field, deprecate it and remove it when it’s no longer in use.
• If you want to change an existing field, create a new field, deprecate the old field and remove it once it’s no longer in use.

However, since the mobile app has a much larger footprint as users might be using the app with the older version with old graphQL schema, having a test for it seemed inevitable. Our existing e2e automation on mobile did not cover all graphQL endpoints, and it would have been too expensive and time consuming to cover all endpoints from a user interface perspective. Moreover, we wanted backward compatibility test to be a test that runs before we run e2e tests to detect breaking changes earlier in the cycle.

Since we needed something light weight and graphQL compatible, we decided to use GraphQL Inspector Tool to ensure backward compatibility and prevent schema breaking changes.

Tech Stack

• Bitbucket (Git)
• Codeship Pro for CI/CD workflow
• Docker
• GraphQL

Code Structure

Why GraphQL Inspector Tool?

We decide to use GraphQL Inspector Tool for graphQL backward capability test because of following reasons:

• GraphQL compatibility
• Ease of implementation with custom rules
• Easy CLI command for CI integration

Steps to Implement

Step1: Create a dockerfile for GraphQL Inspector

• Dockerfile template can be found here at graphQL inspector docker hub
• Dockerfile can be named as Dockerfile.graphqlInspector in ./codeship folder

FROM node:10-slim

ENV LOG_LEVEL "debug"
ARG PRIVATE_SSH_KEY

RUN apt update 
RUN apt install -y ssh
RUN yarn global add @graphql-inspector/cli@2.1.0 graphql
RUN apt install -y git-all

# Copy the decrypt ssh private key for Bitbucket Readonly User to allow codeship access to 
# Our repository from codeship container  
RUN mkdir -p /root/.ssh
RUN echo $PRIVATE_SSH_KEY >> /root/.ssh/id_rsa
RUN chmod 600 /root/.ssh/id_rsa
RUN ssh-keyscan -H bitbucket.org >> /root/.ssh/known_hosts

RUN mkdir /app
WORKDIR /app

Step2: Create an entrypoint file

• This file will be triggered after docker image is built
• Name file entrypoint-graphql-inspector.sh in ./codeship folder
  • To add git remote for our repository
    • Note: it uses remote name as api because origin already taken and it doesn't work in codeship
  • To fetch the master branch which will use for graphql inspector command in codeship

#!/bin/bash
git remote add api git@bitbucket.org:<owner_name>/<repo_name>.git
git fetch api master
exec "$@"

Step3: Create graphQL inspector test service

• graphql-inspector-test service will be placed in ./codeship-services.yml
• Build from Dockerfile.graphqlInspector
• Include entrypoint link with entrypoint-graphql-inspector.sh to trigger the command line after finishing pulling and building the docker image from dockerfile
• For volumes, it can be different depending on your code structure or same as created above in dockerfile.
  • How to find which volume to mount
  • Install jet if you have not (for locally running commands for codeship)
  • cd <repo>
  • jet run SERVICE_NAME pwd
  • Example: jet run graphql-inspector-test pwd
  • With pwd, it will show the path of the volume where it should be mounted

services:
  graphql-inspector-test:
    build:
      context: ./
      dockerfile: ./codeship/Dockerfile.graphqlInspector
      encrypted_args_file: <path_to_encrypted_git_ssh_key>
    encrypted_env_file: <path_to_encrypted_git_ssh_key>
    entrypoint: ["sh", "./codeship/entrypoint-graphql-inspector.sh"]
    cached: true
    volumes:
      - .:/app

Step4: Create graphql-inspector-test step

• GraphQL Schema Inspector Test test step under Jest Tests in ./codeship-steps.yml
• Use graphql-inspector-test service
• To trigger command graphql-inspector diff git:api/master:./src/generated/graphql/schema.graphql ./src/generated/graphql/schema.graphql --rule ./src/generated/graphql/custom-rule.js

- type: serial
  name: Continuous Integration
  encrypted_dockercfg_path: dockercfg.encrypted
  steps:
  - name: Tests
    type: parallel
    exclude: ^(production)
    steps:
    - name: Jest Tests
      type: serial
      steps:
      - name: Jest Tests for E2E Tests
        service: e2e-runner
        command: yarn test
      - name: ESLint for E2E Tests
        service: e2e-runner
        command: yarn run lint
      - name: Graph QL Test
        service: graphql-test
        command: yarn run graphql-test
      - name: Graph QL Schema Inspector Test
        service: graphql-inspector-test
        command: graphql-inspector diff
                 git:api/master:./src/generated/graphql/schema.graphql             
                 ./src/generated/graphql/schema.graphql --rule 
                 ./src/generated/graphql/custom-rule.js
  - name: Deploy to Dev and Production

Step5: How to create custom rules for graphQL inspector test

• For details, you can see graphql inspector custom rules
• We use custom-rules to skip certain attributes that we don’t want the test to run
  • Example: the field may not be available to mobile apps yet and still under development, so there may be many changes to the field.
• Create file name custom-rules.js in the folder where you have your schema.graphql

// custom-rules.js
/**
 * @summary exceptionSet is an array of schema attributes that can be skipped during graphql schema diff test
 * for breaking changes. 
 * !important: by adding to exceptionSet, there is a risk of graphql schema test will skip these attributes
 * example: new Set(['Form.user]);
 */
const exceptionSet = new Set([]);

/**
 * @summary changes is an object array of all breaking changes [{error1},{error2}]
 * Each error object contains these key value pairs as follow 
 * critical: { level: 'BREAKING', reason: "some reason"}
 * type: 'FIELD_TYPE_CHANGED' or 'FIELD_REMOVED' or 'ENUM_VALUE_REMOVED', etc... 
 * message: "Field 'XYZ' was removed from interface 'ABC'"
 * path: graphql schema attribute, eg: 'Comment.description' 
 */
module.exports = ({ changes }) => {
    const included = [];
    const excluded = [];
    for (const change of changes) {
        if (exceptionSet.has(change.path)) {
            excluded.push(change);
            continue;
        }
        included.push(change);
    }
    if (excluded.length > 0) {
        const warnColor = '\x1b[33m';
        console.log('The following changes have been excluded from breaking changes test:');
        for (const change of excluded) {
            // print something sensible - should show that the field was excluded
            // and what potential problem we're ignoring ;-)
            const message = `!!! ${change.path}: ${change.message}`;
            console.log(warnColor, message);
        }
    }
    return included;
};

How to execute test locally

Prerequisite

npm install --global @graphql-inspector/cli graphql

Steps

cd <repo_folder> 
graphql-inspector diff git:origin/master:./src/generated/graphql/schema.graphql ./src/generated/graphql/schema.graphql --rule ./src/generated/graphql/custom-rule.js

Conclusion

We were successfully able to add graphQL backward capabilities test that is easy to maintain, and is platform agnostic by using GraphQL Inspector Tool. We are able to gain a better understanding of how API should be handled for projects, where the same API is used across multiple platforms (i.e. web, desktop and mobile).