Python API Client Using Magic Methods

Python "magic methods" allow for some very fun and powerful code.

Magic methods refers to all those special methods in Python classes that start and end with __.

The documentation for Special method names says:

A class can implement certain operations that are invoked by special syntax (such as arithmetic operations or subscripting and slicing) by defining methods with special names.

For instance the __init__ method is called when a new instance of a class is instantiated. __str__ is a method that returns a string representation of the class. As stated in the docs, these are not methods that generally get invoked directly on the class. For example, you don't call my_object.__str__() but __str__ would be called by print(my_object).

In this post, I'm going to demonstrate using the __getattr__ and __call__ magic methods to build a dynamic API client.

Let's say you're building a client for an API with endpoints like:

• GET https://some-api.net/user
• GET https://some-api.net/user/permissions
• POST https://some-api.net/article
• GET https://some-api.net/article/

There are a lot of ways to build a client for this. One might look like:

import logging
import requests

class APIClient:
    BASE_URL = "https://some-api.net"

    def make_request(method, url, params=None, data=None):
        response = requests.request(
            method,
            f"{self.BASE_URL}/{url}"
            params=params,
            json=data
        )

        response_body = {}
        try:
            response_body = response.json()
        except ValueError:
            log.warning("Unexpected Response '%s' from '%s'",
                        response.content, response.url)
        return response_body

    def get_user(self):
        return self.make_request("GET", "user")

    def get_user_permissions(self):
        return self.make_request("GET", "user/permissions")

    def create_article(self, data):
        return self.make_request("POST", "article", data=data)

    def get_article(self, id):
        return self.make_request("GET", f"article/{id}")

We would then use the client:

client = APIClient()

user = client.get_user()
permissions = client.get_user_permissions()
new_article = client.create_article({"some": "data"})
existing_article = client.get_article(123)

And this is fine.

But what if the API is changing often? Or if there are dozens of endpoints and you don't know which you need to call? If later on you discover you need to call GET https://some-api.net/user/roles or GET https://some-api.net/some_other_thing?some_param=foo, you'll need to go back to you client and add matching methods.

It would be nice to have things a bit more dynamic, and one way to do that is with magic methods. Specifically, I'll be using:

• __getattr__ which is "Called when the default attribute access fails with an AttributeError" (see the docs). This means that if you call my_object.some_missing_attr, then __getattr__ will get invoked with "some_missing_attr" as the name parameter.
• __call__ which is "Called when the instance is "called" as a function" (see the docs), eg. my_object()

Using these, we can build something like:

import logging
import requests

log = logging.getLogger(__name__)

class APIClient:
    BASE_URL = "https://some-api.net"

    def __init__(self, base_url=None):
        self.base_url = base_url or BASE_URL

    def __getattr__(self, name):
        return APIClient(self._endpoint(name))

    def _endpoint(self, endpoint=None):
        if endpoint is None:
            return self.base_url
        return "/".join([self.base_url, endpoint])

    def __call__(self, *args, **kwargs):
        method = kwargs.pop("method", "GET")

        object_id = kwargs.pop("id"))
        data = kwargs.pop("data", None)

        response = requests.request(
            method,
            self._endpoint(object_id),
            params=kwargs,
            json=data
        )

        response_body = {}
        try:
            response_body = response.json()
        except ValueError:
            log.warning("Unexpected Response '%s' from '%s'",
                        response.content, response.url)
        return response_body

This allows us to call the API like so:

client = APIClient()

user = client.user()
permissions = client.user.permissions()
new_article = client.article(method="POST", data={"some": "data"})
existing_article = client.article(id=123)

How does this work?

Well in the case of client.user.permissions(), .user is an attribute and since the client doesn't have that attribute, it calls __getattr__ with name="user", which then returns an APIClient instance with the name appended to the endpoint URL. The same happens when .permissions is invoked on the APIClient instance that was returned by .user, in turn giving us another APIClient instance, now with a path of https://some-api.net/user/permissions. Finally this APIClient instance is called by (), which invokes the __call__ method. This method makes the actually HTTP call to the constructed URL based in any parameteres passed in, but defaults to a GET request.

In the case of the article calls, client.article(id=123) works much the same way, but the APIClient is called with an id parameter. This id gets appended to the url by the same internal _endpoint() method that __getattr__ uses, which results in a GET call to https://some-api.net/article/<id>.

For client.article(method="POST", data={"some": "data"}), we override the method and add a data payload to the POST.

As you can see, the client itself can handle a large variety of API calls without any significant changes, allowing the utilizing application a lot of flexibility.

In the example of the newly required calls, GET https://some-api.net/user/roles and GET https://some-api.net/some_other_thing?some_param=foo, no changes to the client are needed:

roles = client.user.roles()
thing = client.some_other_things(some_param="foo")