Python entry points

Recently I became interested in using entry points to "invade" the namespace of a package. We'll define a base package which will allow different functionality through modules. We'll then expose the entry points and load any and all modules which adhere to our definitions.
We'll simulate a video/audio encoding tool. This should help make thing a bit more realistic. This will NOT be a tutorial on encoding/decoding, so if you're looking for something like that, this is the wrong article for you.

The base package

We need three modules in our base package: the main module which will expose the functionality to the users, a default module which will demonstrate the API and a plugins module which will expose a function to load plugins.
We'll call our base package "ender" (encoder + decoder), because it's fun and I'm also a fan of the books by Orson Scott Card. We'll create a plugin module "ender_mp3" which will use entry points to provide its functionality.
The file structure for the base package is as follows:

ender/
├── ender
│   ├── __init__.py
│   ├── empty.py
│   ├── main.py
│   └── plugins.py
├── poetry.lock
├── pyproject.toml
├── README.md

Now let's define the main part of our core package. We'll create a class to manage all modules and run them.

# in main.py
class Ender:
    def __init__(self):
        self.modules = []

    def work(self):
        for module in self.modules:
            module.work()

def main():
    e = Ender()
    e.work()
    return 0


if __name__ == '__main__':
    # behave like a nice CLI and return 0 when successful
    exit(main())

As you can see from the code above, the module is very simple. It initializes a list of modules and exposes a method work. When that method is called, all the registered modules will do their thing. We'll extend this later, but for now it's going to be enough to demonstrate how to register modules and use them.
You can run the main file, and you'll see nothing, because we haven't registered any modules.

poetry run python ender/main.py

Let's fix that and register a default empty encoder/decoder. First create the module where it will live:

# ender/empty.py
class EmptyEncoderDecoder:
    def work(self):
        print(f"'{type(self).__name__}'", "Default empty encoder/decoder working...")

    def register(self, core):
        core.modules.append(self)


def register(core):
    m = EmptyEncoderDecoder()
    m.register(core)

We've defined the empty encoder/decoder and we've included a register method, which we'll use to add our module to the core. We've also exposed a register function which will be called by our plugin loading logic. Let's go implement that:

# in ender/plugins.py
import importlib.metadata


def load_plugins():
    entry_points = importlib.metadata.entry_points()
    plugins = entry_points.select(group='ender.plugins')

    plugin_modules = []
    for plugin in plugins:
        plugin_modules.append(plugin.load())
    return plugin_modules

Let's break down what's going on in this file. We've declared a function load_plugins which can be imported by the core module and used to fetch all registered plugins. The line plugins = entry_points.select(group='ender.plugins') loads any module from any package which has an entry point named 'ender.plugins' defined either in a setup.py file or in the pyproject.toml. After that the module is actually loaded and then passed down to the list which is returned from the load_plugins function. Let's take a look at the entry point registration in pyproject.toml (managed by poetry):

# in the pyproject.toml of the base ender package
[tool.poetry.plugins."ender.plugins"]
"default" = "ender.empty"

Note: For other plugin management tools this section will look differently.

You can see we've registered a "default" module within the "ender.plugins" entry point, which points to the actual module 'ender.empty', which just maps to ender/empty.py. Now when the load_plugins function is run, the module will be loaded. Let's wire that up in our main.py:

# in ender/main.py
from ender.plugins import load_plugins

def main():
    e = Ender()
    # load and register plugin modules
    plugins = load_plugins()
    for plugin in plugins:
        plugin.register(e)

    e.work()
    return 0

If we run the code now, we'll get the output:

$ poetry run python ender/main.py
'EmptyEncoderDecoder' Default empty encoder/decoder working...

Take it one step further

Now we have made the local default encoder/decoder work, let's go and make a separate package. Create a plugin package using poetry:

poetry new ender_mp3

The file structure we get is:

ender_mp3/
├── ender_mp3
│   ├── __init__.py
├── pyproject.toml
├── README.md
└── tests
    └── __init__.py

Let's create our module in the ender_mp3 directory and add an mp3 encoder/decoder:

# in ender_mp3/ender_mp3/mp3_module.py
class Mp3EncoderDecoder:
    def register(self, core):
        core.modules.append(self)

    def work(self):
        print(f"'{type(self).__name__}'", "Encoding/Decoding MP3.")


def register(core):
    m = Mp3EncoderDecoder()
    m.register(core)

Then add an entry point to your pyproject.toml:

# in ender_mp3/pyproject.toml
[tool.poetry.plugins."ender.plugins"]
"mp3" = "ender_mp3.mp3_module"

The same as before, we've registered our plugin. Notice that we're not making use of the name we've given our plugin, here 'mp3', but it is indeed assigned to our EntryPoint. Depending on the use-case it might be needed or not.

Let's install it now and see it all come together. First go back to your ender directory and run:

poetry add ../ender_mp3

This is assuming you've put the two directories next to each other:

.
├── ender
│   ├── ender
│   │   ├── __init__.py
│   │   ├── empty.py
│   │   ├── main.py
│   │   └── plugins.py
│   ├── poetry.lock
│   ├── pyproject.toml
│   ├── README.md
└── ender_mp3
    ├── ender_mp3
    │   ├── __init__.py
    │   └── mp3_module.py
    ├── pyproject.toml
    ├── README.md

Now when you run your main package, you'll get more output:

$ poetry run python ender/main.py
'EmptyEncoderDecoder' Default empty encoder/decoder working...
'Mp3EncoderDecoder' Encoding/Decoding MP3.

Beyond the basics

There are of course packages which have dealt with all the tricky parts of this workflow. We'll take a look at 'pluggy' which takes this concept to another level.

Let's rewrite our core package and then we'll go on to rewriting the plugin.

Install pluggy

poetry add pluggy

First create a spec file for specific hooks:

# in ender/hookspec.py
import pluggy

hookspec = pluggy.HookspecMarker('ender.plugins')


@hookspec
def ender_plugins_load_module():
    pass

Now let's make our default implementation use the pluggy specs:

# in ender/empty.py
from ender.hookimpl import hookimpl


class DefaultModule:
    def work(self):
        print(f"'{type(self).__name__}'", "Default empty module")

    def register(self, core):
        core.modules.append(self)


@hookimpl
def ender_plugins_load_module():
    return DefaultModule()

So we've replaced our register function with a hook implementation annotated function which just returns the module.

Now let's go ahead and modify our main.py:

# in ender/main.py
import pluggy
from ender import hookspec, default_module


class Ender:
    def __init__(self):
        self.modules = []

    def work(self):
        for module in self.modules:
            module.work()


def main():
    ender = Ender()
    pm = get_plugin_manager()
    modules = pm.hook.ender_plugins_load_module()
    for module in modules:
        module.register(ender)

    ender.work()
    return 0


def get_plugin_manager():
    pm = pluggy.PluginManager("ender.plugins")
    pm.add_hookspecs(hookspec)
    pm.load_setuptools_entrypoints("ender.plugins")
    return pm


if __name__ == '__main__':
    exit(main())

We've now introduced a get_plugin_manager function, which will create a plugin manager object for the "ender.plugins" entry point. We'll tell it about our hook specification and then we'll load the entry point.

In our main we'll again initialize Ender and then we'll get the objects we receive from the ender_plugins_load_module hook, loop as before and register them with the ender instance.

Note that we haven't done a lot of work, just changed a couple of lines in some files. The pyproject.toml definition stays the same.

Let's go and prepare our plugin:

from ender.hookimpl import hookimpl


class Mp3EncoderDecoder:
    def register(self, core):
        core.modules.append(self)

    def work(self):
        print(f"'{type(self).__name__}'", "Encoding MP3")


@hookimpl
def ender_plugins_load_module():
    return Mp3EncoderDecoder()

Now we're doing something a bit weird here. We're importing the hookimpl from the core package, although we haven't installed it. You'll probably get some errors because of this in your IDE, but just ignore them. This will work just fine.

Now replace the register function with the annotated implementation and you're done. Install the plugin:

# let's just remove the old package from our virtualenv
# from the ender/ project directory
poetry remove ender_mp3
poetry add ../ender_mp3

Now when you run your project you'll see the same message as before. Voila, we've moved our crude plugin implementation to this magnificent much more advanced version. The awesome part is that now we're able to define multiple hooks for our core package, use attributes when running the hooked functions and generally achieve much more, with not a lot of extra work.

Let's go even further

Ok so now we're actually able to move some logic around, because pluggy is just that powerful. We can now easily split the encoders and the decoders and load them separately into different objects.

For that we'll rename our spec to load_encoders and add another one called load_decoders.

# in ender/hookspec.py
import pluggy

hookspec = pluggy.HookspecMarker('ender.plugins')


@hookspec
def ender_plugins_load_encoders():
    pass


@hookspec
def ender_plugins_load_decoders():
    pass

Note that the hook specifications MUST be named after the marker. In our case the marker is ender.plugins which would translate to ender_plugins. This is important and the whole setup won't work if you don't follow this convention.

Let's quickly update our mp3 encoder/decoder and make sure it will work.

# in ender_mp3/mp3_module.py
from ender.hookimpl import hookimpl


class Mp3Encoder:
    def work(self):
        print("mp3 encoder working...")


class Mp3Decoder:
    def work(self):
        print("mp3 decoder working...")


@hookimpl
def ender_plugins_load_encoders():
    return Mp3Encoder()


@hookimpl
def ender_plugins_load_decoders():
    return Mp3Decoder()

As you can see we've split our single module into two separate classes, each responsible for it's own task, e.g. encoding or decoding.

Let's update the default empty module as well:

# in ender/empty.py
from ender.hookimpl import hookimpl


class DefaultEncoder:
    def work(self):
        print("Default encoder working")


class DefaultDecoder:
    def work(self):
        print("Default decoder working")


@hookimpl
def ender_plugins_load_encoders():
    return DefaultEncoder()


@hookimpl
def ender_plugins_load_decoders():
    return DefaultDecoder()

And finally let's update the main package:

# in ender/main.py
import pluggy
from ender import hookspec, default_module


class Ender:
    def __init__(self, hook):
        self.hook = hook
        self.encoders = []
        self.decoders = []
        encoders = self.hook.ender_plugins_load_encoders()
        for encoder in encoders:
            self.encoders.append(encoder)

        decoders = self.hook.ender_plugins_load_decoders()
        for decoder in decoders:
            self.decoders.append(decoder)

    def work(self):
        for enc in self.encoders:
            enc.work()

        for dec in self.decoders:
            dec.work()


def main():
    pm = get_plugin_manager()
    ender = Ender(pm.hook)
    ender.work()
    return 0


def get_plugin_manager():
    pm = pluggy.PluginManager("ender.plugins")
    pm.add_hookspecs(hookspec)
    pm.load_setuptools_entrypoints("ender.plugins")
    return pm


if __name__ == '__main__':
    exit(main())

Let's start near the bottom. We've kept the get_plugin_manager function intact and we load it before we initialize the Ender class. We'll just add a hook argument and pass it to the instance.

In the __init__ method we accept the hook and use it to load the encoders and decoders and save them to their respective property for later use.

We'll also update our work logic to use the encoders and decoders and we're done. Just reinstall the plugin and take a look:

poetry remove ender_mp3
poetry add ../ender_mp3

Run the application

$ poetry run python ender/main.py
mp3 encoder working...
Default encoder working
mp3 decoder working...
Default decoder working

You've probably noticed that the order of the modules is now switched. This is because the modules are loaded in LIFO order. If you want to restore the order you can just sort the encoders and decoders after loading them.

Further thoughts

We've seen how to go from the simple idea of loading plugins through the awesome entry points feature, and we explored how to scale up to use something like pluggy to achieve more complex setups. With all that in place we'd be capable of performing a more complex setup when we load the encoders and decoders. We should expand our main application to allow the user to select if they want to encode or decode, add some basic implementations for certain audio and video types, overwrite them when loading plugins, or at least give the user the opportunity to do so. These are just some ideas. Feel free to experiment on this basis.