Short note: HTTP request recording (vcr) vs higher level mocks

You are implementing your hundredth binding for some useful third party rest API and you are thinking about testing. You aren't sure if you should record traces of the http requests, mock at a level higher than http, or even drop a level lower and build a mock server.

I would consider the following:

• How frequently does the API change?
• How frequently does your request producing / response parsing logic change?

If the rates of change are low, then I would default to implementing a higher level mock unless you have a compelling reason to have http level traces. Higher level mocks require less credential configuration, they are easier to tweak, involve less work to isolate test data when updating tests, and they involve managing/synchronizing fewer files. Essentially, there is some management overhead to using http recording and the overhead isn't worth incurring unless you need to update tests frequently.

Implementing a mock server seems like strictly more work than recording traces of http requests from a real server. A mock server might save a little effort in not having to manage recording files and knowing the test data is isolated since there is no persistence of generated test data in a live server.

When you implement a higher level mock, you regain some lost test coverage by adding tests for the behavior which produces the request and the behavior that parses the response. There will still be one gap. You won't know if your actual request and expected response will actually function/match the behavior of the third party service. You will need to interactively test against a real service to cover this final piece. Http request recording and a mock server can exercise this final piece, but they each can still be susceptible to falling out of sync with the real service. For the interactive tests, you can leave some documentation on how to interactively test and default the real implementation to use a sandbox service when it is run interactively in non production mode.