Recording and Replaying HTTP Interactions with Ease: A Guide to VCR.py

Introduction

Testing code that interacts with external services can be tricky. You need to make sure that your code is handling all the possible responses correctly, and that it's not making too many unnecessary requests. But how do you test this kind of code reliably and efficiently?

That's where VCR.py comes in. The idea behind VCR.py is simple: it records HTTP interactions made by your code, and then replays them when your tests run. That way, your tests can simulate HTTP requests without actually hitting the external service. Pretty neat, right?

In this article, we'll take a closer look at VCR.py and some of its features. We'll cover how to use VCR.py to record and replay HTTP interactions, how to customize its behavior with matchers and filters, and how to work with the cassette file that VCR.py uses to store recorded interactions.

Getting Started

To get started with VCR.py, you'll need to install it first. You can do this using pip:

pip install vcrpy

Once you have VCR.py installed, you can start using it in your tests. Here's a simple example:

import requests
import vcr

with vcr.use_cassette('test_api.yaml', record_mode='once'):
    def test_api():
        response = requests.get('https://api.example.com')
        assert response.status_code == 200

In this example, we're using VCR.py to intercept a GET request to https://api.example.com. The first time this test is run, VCR.py will record the HTTP interaction in a cassette file named test_api.yaml. On subsequent runs, VCR.py will replay the recorded interaction from the cassette file instead of making a new request.

Record_mode that passed in with the VCR cassette determines how the interactions with an external API are recorded. There are four record modes available:

1. once: This mode will record the interactions with the external API the first time the test is run. Subsequent runs of the test will use the recorded interactions instead of making new requests to the API.

2. new_episodes: This mode will record any interactions with the external API that haven't been previously recorded. If an interaction has already been recorded, it will be replayed instead of making a new request.

3. all: This mode will record all interactions with the external API, regardless of whether they have been previously recorded or not. This is useful for building up a complete set of recordings for an API.

4. none: This mode will turn off recording altogether and will only replay previously recorded interactions. This mode is useful when you want to ensure that your tests are only using recorded interactions and not making any new requests to the external API.

In the previous example, the record_mode argument is set to 'once', which means that the interactions with the external API will be recorded the first time the test is run and subsequently replayed on all subsequent runs.

Working with the Cassette File

The cassette file is where VCR.py stores recorded HTTP interactions. By default, the cassette file is a YAML file, but you can also use JSON or SQLite. The cassette file can be easily read and edited, which makes it easy to inspect the contents of the file and make changes if necessary.

Here's an example of a cassette file:

- request:
    body: !!binary |
      eyJ0ZXN0IjoidGVzdCJ9
    headers:
      Content-Type: application/json
    method: POST
    uri: https://api.example.com/foo
  response:
    body: !!binary |
      {"id": "123", "foo": "bar"}
    headers:
      Content-Type: application/json
    status:
      code: 200
      message: OK

In this example, we have a single interaction that was recorded. The interaction consists of a POST request to https://api.example.com/foo with a JSON request body, and a response with a JSON body containing an ID and a foo value.

One important thing to keep in mind when working with the cassette file is that it can become quite large if you have a lot of interactions recorded. To help with this, VCR.py provides several options for controlling how interactions are recorded and stored in the cassette file.

Matchers

By default, VCR.py matches HTTP interactions based on the HTTP method, URL, and request body. But what if you want to match on other criteria, like headers or query parameters?

That's where matchers come in. Matchers allow you to specify custom criteria for matching HTTP interactions. Here's an example:

import vcr

def custom_matcher(request1, request2):
    # Your custom matching logic here
    return True

with vcr.use_cassette('my_test.yaml', 
                      match_on=['method', 'uri', custom_matcher]):
    def test_my_function():
        # Code that makes HTTP requests

In this example, we define a custom matcher function custom_matcher that takes two requests and returns True if they match based on our custom criteria. We then use this custom matcher in our use_cassette call, along with the default matchers for HTTP method and URI.

Filters

Sometimes you might want to filter or modify requests or responses before they are recorded or played back. For example, you might want to filter out sensitive data from the response before it gets stored in the cassette file. Or you might want to modify the request headers before the request is sent.

VCR.py provides two options for filtering: before_record and before_playback. These options allow you to specify functions that are called before requests are recorded or played back, respectively. Here's an example:

import vcr

def filter_request(request):
    request.headers['Authorization'] = 'Bearer xxxxxxx'
    return request

def filter_response(response):
    response.headers.pop('Set-Cookie', None)
    return response

with vcr.use_cassette('my_test.yaml',
                      before_record=filter_request,
                      before_playback=filter_response):
    # Your test code here

In this example, we define two filter functions filter_request and filter_response that modify the request and response respectively. We then pass these filter functions to the before_record and before_playback options of use_cassette, respectively.

Placeholders

Sometimes you might have dynamic data in your responses that you don't want to hard-code in your tests. For example, if you have a test that creates a new resource and returns a unique ID, you might want to use a placeholder to represent that ID in the cassette file.

VCR.py supports placeholders in the cassette file. Placeholders are strings that are replaced with actual values during playback. Here's an example:

- request:
    body: !!binary |
      eyJ0ZXN0IjoidGVzdCJ9
    headers:
      Content-Type: application/json
    method: POST
    uri: https://api.example.com/foo
  response:
    body: !!binary |
      {"id": "{{ ID }}", "foo": "bar"}
    headers:
      Content-Type: application/json
    status:
      code: 200
      message: OK

In this example, we're using the {{ ID }} placeholder in the response body to represent the unique ID that is generated by the API. When VCR.py replays the response, it will substitute the placeholder with the actual ID value that was returned by the API during the recording.

Basic Illustration

Let's assume that we have a simple Flask web application that acts as an external web service API. It returns a JSON response when we make a GET request to the /hello endpoint:

from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/hello')
def hello():
    return jsonify({'message': 'Hello, World!'})

if __name__ == '__main__':
    app.run()

Now, let's write a vcrpy test to test this API. We'll use the requests library to make the HTTP request to the API:

import vcr
import requests

def test_hello_api():
    with vcr.use_cassette('hello_api.yaml', record_mode='once'):
        response = requests.get('http://localhost:5000/hello')
        assert response.status_code == 200
        assert response.json() == {'message': 'Hello, World!'}

In this test, we're using the use_cassette context manager to wrap the HTTP request to our Flask app. We're specifying that we want to use the cassette file hello_api.yaml to record the interactions with the API, and we're using the record_mode argument to specify that we only want to record the interactions once (i.e., on the first run of the test).

When we run this test for the first time, vcrpy will record the interaction with the Flask app and save it to the hello_api.yaml file. On subsequent runs of the test, vcrpy will replay the interaction from the cassette file instead of making a new HTTP request to the Flask app.

To run this test, we can simply execute the test file using a test runner such as pytest:

$ pytest test_hello_api.py

This will run the test and output the results of the assertions. If the test passes, we should see output similar to the following:

============================ test session starts ============================
collected 1 item

test_hello_api.py .                                                   [100%]

============================= 1 passed in 0.13s =============================

If the test fails, we'll see an error message indicating which assertion failed.

here's what the hello_api.yaml cassette file might look like after vcrpy records the HTTP interaction with the Flask app (ie. our external API):

interactions:
- request:
    body: null
    headers:
      Accept-Encoding: identity
      Connection: keep-alive
      Host: localhost:5000
      User-Agent: python-requests/2.25.1
    method: GET
    uri: http://localhost:5000/hello
  response:
    body:
      encoding: utf-8
      string: '{"message": "Hello, World!"}'
    headers:
      Connection: Keep-Alive
      Content-Length: '25'
      Content-Type: application/json
      Date: Mon, 05 Jul 2023 12:34:56 GMT
      Keep-Alive: timeout=5, max=100
      Server: Werkzeug/2.0.2 Python/3.9.5
    status:
      code: 200
      message: OK
  recorded_at: 2023-07-05T12:34:56.789012Z
  recorded_with: vcrpy/4.1.1

This file contains a YAML representation of the HTTP interaction between the test and the Flask app. The interactions section contains a list of all the HTTP interactions that were recorded, in this case just one. The request and response sections contain the request and response headers and bodies, respectively. The recorded_at and recorded_with fields indicate when and with what version of vcrpy the interaction was recorded.

Conclusion

VCR.py is a powerful tool for testing code that interacts with external services. Its ability to record and replay HTTP interactions makes it easy to write tests that are reliable and efficient. With VCR.py, you can customize how interactions are matched and filtered, and work with the cassette file to inspect and modify recorded interactions.

I hope this article has given you a good introduction to VCR.py and its features. If you're interested in learning more, I encourage you to check out the official documentation and start experimenting with VCR.py in your own tests.