Two Methods for Testing HTTPS API Calls with Python and pytest and Also Communicating with the in-Laws

API endpoints and web URLs are, thankfully, more secure than ever, usually requiring encrypted HTTPS. Python works well as an HTTPS client, and pytest simplifies testing Python-based tools or libraries. Tools like VCR.py or the combination of pytest-httpserver and trustme provide an additional testing layer that is fast and convenient, and well-suited for HTTPS work. This will help with family gatherings. Let me show you.

A potentially awkward social situation, solved by HTTPS

Imagine this scenario: you recently married into a wonderful family, but your spouse's parents only speak pig Latin. Unfortunately, you are not fluent in this exotic language, and you are worried about the next family gathering. As you can guess, this is a common situation for many, so, thankfully, there is an API for that.

Being the enterprising Python developer that you are, you write a Python client to serve as a realtime translator.

It looks like this:

import json
from urllib.parse import urlencode
from urllib.request import urlopen


def translate(text, server="api.funtranslations.com"):
    data = {"text": text}
    form_encoded_data = urlencode(data).encode()
    url = f"https://{server}/translate/piglatin.json"
    with urlopen(url, form_encoded_data) as response:
        result = json.load(response)
        print(result)
    return result["contents"]["translated"]

Less than a dozen lines of code for marital bliss. Seems worthwhile.

You might save the above as pypiglatin.py in a directory of your choosing.

The script and associated tests can be found in the companion Github repo.

The tests

If you are like me, you have already tried the above enough times to receive a 429 "Too Many Requests" error. The funtranslations.com service allows you to sign up and become a paying customer in order to remove the rate limiting, but the anonymous limits certainly make a point. Your many tests are using up someone's server bandwidth and compute cycles. Maybe we should stop doing that. Even if the API you are accessing isn't rate limited.

Instead, find ways of testing locally. It will be faster, and kinder. Tests will make sure that the pig Latin translation is working smoothly, in advance of any high-pressure family gatherings.

I believe tools for HTTP client testing should:

• As already mentioned, easily replicate API endpoints locally, for performance and respect of API limits.
• Locally test the same protocol (HTTPS or HTTP) that is used in production. Usually API endpoints are HTTPS only, and so the URLs, even in testing, should start with "https://"
• HTTP client library flexibility. Yes, requests pairs well with responses or requests-mock, and HTTPX has RESPX or pytest_httpx. Those testing helpers are an excellent match for the corresponding library, and should certainly be recommended. However, I don't always want to face the risk of rewriting all my tests if I replace the client library some day. And sometimes I am using an altogether different tool (even though both requests and HTTPX are quite awesome) such as urlopen, urllib3, httplib2, tornado, or aiohttp.

Let's explore two methods for meeting the above goals. Both assume the use of pytest:

1. An easy-to-configure HTTP server using pytest-httpserver, coupled with trustme for quick SSL certificate setup.
2. While not an actual HTTP server, pytest-vcr records and then mocks HTTP requests using the VCR.py library.

pytest-httpserver

It is hard to beat an actual HTTP server for testing reassurance, and pytest-httpserver provides a quick way to configure responses based on expected requests. The web application library Werkzeug is used to build the server.

Here is an example that tests if a request to the "/hello" endpoint indeed returns "Hello, World!" as expected:

from urllib.request import urlopen

import pytest


@pytest.fixture(scope="session")
def httpserver_listen_address():
    return ("localhost", 8888)


def test_hello(httpserver):
    body = "Hello, World!"
    endpoint = "/hello"
    httpserver.expect_request(endpoint).respond_with_data(body)
    with urlopen(httpserver.url_for(endpoint)) as response:
        result = response.read().decode()
    assert body == result

To use the above test, save it in your current directory as test_http.py or similar, then install pytest and pytest-httpserver in your preferred manner (pip install pytest pytest-httpserver works well in a Python virtual environment). Then run pytest from the command line in that directory.

While not strictly necessary, I like defining the httpserver_listen_address fixture from pytest-httpserver to explicitly define the server and port. This makes it easy to define URLs later, as the port is known. Otherwise, pytest-httpserver will assign a random port.

The heart of the test server setup is the line

httpserver.expect_request(endpoint).respond_with_data(body)

We tell the server what sort of request to expect, and the data with which to respond to the expected request. This is quite configurable, on both ends. See the docs for more advanced examples. For instance, request and/or response headers can be defined, the method (GET, POST, PUT, DELETE) specified, and various formats of data can be expected or returned, such as JSON.

Adding SSL with trustme

What the sample test above is missing is HTTPS. Thankfully, pytest-httpserver allows a Python ssl.SSLContext to be specified by defining the httpserver_ssl_context fixture.

However, building a functioning certificate authority and the rest of the cert chain is not exactly a trivial matter. Enter trustme, a Python-based tool for enabling SSL for testing purposes. You can install it in your virtual environment with pip install trustme.

Here is a test mechanism for our previously-built translate function:

import ssl
from urllib.parse import urlencode

import pytest
import trustme

import pypiglatin

SERVER = "localhost"
PORT = 8888
ENDPOINT = "/translate/piglatin.json"


@pytest.fixture(scope="session")
def httpserver_listen_address():
    return (SERVER, PORT)


@pytest.fixture(scope="session")
def httpserver_ssl_context():
    ca = trustme.CA()
    client_context = ssl.SSLContext()
    server_context = ssl.SSLContext()
    server_cert = ca.issue_cert("test-host.example.org")
    ca.configure_trust(client_context)
    server_cert.configure_cert(server_context)

    def default_context():
        return client_context

    ssl._create_default_https_context = default_context

    return server_context


def test_post(httpserver):
    text = "Thank you for your hospitality"
    translated = "ank-Thay ou-yay or-fay our-yay ospitality-hay "
    response = {
        "success": {"total": 1},
        "contents": {
            "translated": translated,
            "text": text,
            "translation": "pig-latin",
        },
    }

    httpserver.expect_request(
        ENDPOINT, method="POST", data=urlencode({"text": text})
    ).respond_with_json(response)
    assert pypiglatin.translate(text, f"{SERVER}:{PORT}") == translated

A few notables:

• We take advantage of the httpserver_ssl_context fixture to not only build a server SSL context and return it, but also a build a client context that trusts the server cert. We then make ssl._create_default_https_context() return that client context. Thus, any HTTPS client that seeks to utilize the default context from the ssl module will respect our new cert authority.
• If your client does not use the ssl module to provide an SSL context, you will want to find another way to pass the client SSL context to your HTTP client. If need be, see the trustme documentation on how to write the certs to files so they can be utilized later.
• The expected request is a little more complex than our test_hello example. It expects application/x-www-form-urlencoded POST data (encoded with urllib.parse.urlencode()).
• The response is a Python dict populated with a known API response, then converted to JSON by the respond_with_json() method.
• In my tests, the server SSL context seemed to work as the client SSL context as well. It just didn't seem like the right way to do it, so I separated them. Feel free to post advice in the comments.
• This is only one test. We can add tests to make sure incorrect requests elicit an error, and other messages are translated correctly. Neither the production API server nor the in-laws will know how fast and how often we fail. Beauty.

The end result is a live local HTTPS service intended to behave very similarly to the remote production service when called with specific data. The primary difference is that the URL is "https://localhost:8888/translate/piglatin.json" when testing, but "https://api.funtranslations.com/translate/piglatin.json" in production.

For a nice introduction to pytest-httpserver, see the tutorial. For a deeper dive, see the how-to.

VCR.py

I love the convenience and realism that pytest-httpserver provides. The only downside is the mild tedium in recording and setting up responses and expected requests manually.

An alternate approach involves using VCR.py, a tool that records HTTP interactions in YAML files, then intercepts future HTTP requests and plays back the recorded responses. In this tutorial, we will use pytest-vcr to interface with VCR.py, although pytest-recording is another good option for doing the same.

While this doesn't provide an actual HTTPS server like pytest-httpserver does, the URLs used in the client code do not need to change. They can still start with "https://"; in fact, because the HTTPS calls are intercepted, URLs can use the same domains as production, without fear that calls will be emitted to the production servers (after the initial recording is saved).

Unfortunately, VCR.py only supports a certain number of HTTP client libraries. Thankfully, some very popular implementations are included: the Python built-in urllib and http.client, as well as requests, urllib3, and aiohttp. And, while it may not be listed in the docs at the time of this writing, the changelog notes that HTTPX is also supported. Sadly, pycurl is not supported.

To use VCR.py with pytest, start by installing pytest-vcr in your virtual environment:

pip install pytest-vcr

Implementing a test for the translate() function we built is quite simple using pytest-vcr:

import pytest

import pypiglatin


@pytest.mark.vcr()
def test_translate():
    text = "Thank you for your hospitality"
    translated = "ank-Thay ou-yay or-fay our-yay ospitality-hay "
    assert pypiglatin.translate(text) == translated

You might save this as test_vcr.py in the current directory. Now we can run pytest.

Afterward, there should be a cassettes directory with a test_translate.yaml file in it. Feel free to take a look: you will get an idea for the many facets of an HTTP request and response. The file is also easily editable, which could be useful, for instance, to munge authentication/authorization information (just do that before saving the file to version control, of course).

If you run pytest again, the test should run much faster, using the existing cassette.

This YAML cassette can be saved to version control, alongside the tests, so that later runs on any machine will not need to hit the live server at all.

And there it is. VCR.py is actually this easy.

Improved family dynamics

Hopefully, you are now gaining fluency in pig Latin. Meanwhile, you can be confident in your fallback plan: a thoroughly-tested translator, available at a moment's notice. Feel the reliability.

Please feel free to leave comments with your experiences, questions, or marital advice!

Alternatives

Thes two methods are solid and flexible.

Of course, as noted earlier, there are other options out there, especially if you use requests or HTTPX:

For requests:

• responses
• requests-mock

For HTTPX:

• RESPX
• pytest_httpx

Also of interest

If this article was helpful, you may also be interested in some of my other articles:

• Using pytest_httpx with HTTPX
• Using pytest in a Python dev environment