An Introduction to API Based Documentation Automating

TLDR; The formal api documentation specifications ecosystems have tools that can help create documentation more easily and use the documentation to automatically validate e.g. Dredd, Swagger and StopLight

I'm working on an API for The Pulper, and I want to write documentation in a formal spec and use tools to validate it. Sounds Easy.

The Pulper is a simple CRUD app that is part of my TestingApp, a collection of applications to act as sample apps to practice testing on:

• https://github.com/eviltester/TestingApp

A version without the API is online on heroku if you want to play with it.

• https://thepulper.herokuapp.com

I'm adding an API so that it can support API exploration and so I can hook the GUI to the API backend with JavaScript and allow a whole new class of emergent bugs to appear.

One of the tings that makes The Pulper slightly different is that in the Admin drop down menu, you will find that there are multiple versions of the application that you can switch between. These offer slightly different functionality, potentially different bugs, and a different GUI experience if you are automating.

Once I've added an API I can have different versions of that, with different bugs etc.

Documenting and Testing

The issue with the API is that I wanted to do it a little better than my REST Listicator test app, which you can also play with online, or download as part of the TestingApp.

• https://rest-list-system.herokuapp.com/

The documentation for this is hand crafted - which is good in that it allows errors to creep in, which have to be tested for, but isn't the easiest thing to read to understand the API.

I suspect version 1 of the The Pulper API might have hand written documentation for this reason.

Standard Documentation Formats

There are standard documentation formats for APIs. The two most popular seem to be:

• Swagger's OpenAPI
• API Blueprint

You can find information about OpenAPI at

• https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md
• https://swagger.io/docs/specification/about/

And the API Blueprint at https://apiblueprint.org/

Tools to convert between the various formats seem to exist so I don't think it matters which one you start with.

Testing Documentation

Documentation forms one of the inputs into our testing models.

• Does the stuff in the documentation exist?
• Can we do what the documentation says?
• Does the system look and operate like the documentation?
• etc.

Formal documentation formats offer the possibility of tooling to help out.

And the tooling eco system around the API formats offers the tantalising prospect of being able to automatically test the API from the formal specification.

Testing Interprets Documentation

The tooling can help, but mostly they help 'validate' the requests and responses against the spec, rather than test it.

I haven't explored the tool space enough yet to see how far they can go.

The first tool I looked at was Dredd

Dredd

https://dredd.org/en/latest/

Out of the box, Dredd can take an API Blueprint Spec or a Swagger spec:

• lint it to check that the spec is a valid format
• issue all 2xx status code associated requests

Issuing all 2xx status code requests isn't quite as helpful as it seems since it tries to issue POST requests to receive a 201, but does so without the data so you get a failing test. If you write the schema files well then Dredd may pick up examples in the spec but I haven't experimented with this.

But I found it quite useful to see, out of the box with no configuration:

• the list of requests issued
• actually see some passing
• seeing some valid errors where the API didn't match the spec

I think it adds value out of the box.

Dredd Hooks

Dredd has hooks to allow scripting and I experimented with that to add payload bodies into to requests and skip any response codes I don't want to see failing. That worked well.

To find out the hook transaction names you use the --names command line parameter

~~~~
dredd swagger.json http://localhost:4567 --names
~~~~

I added a simple hooks.js for using Dredd. This:

• adds a payload for my POST books to create an item and trigger a 201 status.
• skips a transaction I haven't coded for yet

~~~~
var hooks = require('hooks');

hooks.before('/apps/pulp/api/books > Create or amend a single or multiple books > 201 > application/json', (transaction) => {
transaction.request.body=JSON.stringify({
"books":[
{
"title": "The Land of Little People Terror",
"publicationYear": 1980,
"seriesId": "Issue 1",
"authors": [
{
"id": "4"
}
],
"series": {
"id": "1"
},
"publisher": {
"id": "1"
}
}
]
});
});
hooks.before('/apps/pulp/api/books > Create or amend a single or multiple books > 200 > application/json', (transaction) => {
transaction.skip=true;
});
~~~~

Dredd looks like it has a good set of lightweight augmentation approaches for adding extra information to allow the untouched documentation to help drive some automated execution.

Tooling

I found writing the swagger specification quite time consuming with the online swagger editor http://editor.swagger.io

But it was much faster with stoplight.io

https://stoplight.io/

My current api documentation work in progress is here, but this is subject to massive change.

https://next.stoplight.io/eviltester-1/thepulper

I'm going to experiment more with the formal api documentation specifications and tooling around it to see if there are any more useful tools and approaches I can add to my API Testing processes.

---

If you are interested in testing and automating APIs then you might find my book "Automating and Testing a REST API" useful. It covers testing and automating APIs from scratch and uses tools like cURL, Proxies, Postman, RestAssured and discusses abstraction layers for automating.