Extending Python with Go

One of the most interesting features of CPython is the ability to add new built-in modules to Python written in C and C++. Python provides its API to do that, a set of headers and core types for writing extensions.

In this tutorial I’ll give you an overview of how to extend Python 3 with C, and then how to do the same thing using a modern language like Go.

Go is a valid alternative to C if you want to extend Python: is easier than C, has a Garbage Collector, provides great performances and executing code in parallel is really straightforward with Go routines.

Setup your environment

This tutorial is based on:

• Python 3.8.1
• Golang 1.13.5

Setup LIBRARY_PATH and PKG_CONFIG_PATH on your system. I run the code under macOS Catalina 10.15.1 so I have to set

export PKG_CONFIG_PATH=/Library/Frameworks/Python.framework/Versions/3.8/lib/pkgconfig/
export LIBRARY_PATH=/Library/Frameworks/Python.framework/Versions/3.8/lib

Extending Python with C

Suppose you want to create a new module to make sums, something easy like

from newmath import sum

print (sum(5,4))

Following the Python section in the docs, you can write the first extension newmath.c

#define PY_SSIZE_T_CLEAN
#include <Python.h>

static PyObject *sum(PyObject *self, PyObject *args) {
    const long a, b;

    if (!PyArg_ParseTuple(args, "LL", &a, &b))
        return NULL;

    return PyLong_FromLong(a + b);
}

static PyMethodDef MathMethods[] = {
    {"sum", sum, METH_VARARGS, "Add two numbers."},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef newmathmodule = {
    PyModuleDef_HEAD_INIT, "newmath", NULL, -1, MathMethods
};

PyMODINIT_FUNC PyInit_newmath(void) {
    return PyModule_Create(&newmathmodule);
}

And compile it as a shared library

gcc -shared -o newmath.so `pkg-config --cflags --libs python3` `python3-config --libs --embed` newmath.c

The result is a new module newmath.so that can be used in Python code as a module.

Do the same using Go

Go SDK comes with an amazing toolset called cgo which allows Go programs to interoperate with C and enables you to build shared libraries from Go.

Thanks to the magic C.* namespace is possible to use anything from the C world, so the C code above can be rewritten in Go in this way

package main

// #cgo pkg-config: python3
// #cgo LDFLAGS: -L/Library/Frameworks/Python.framework/Versions/3.8/lib -lpython3.8 -ldl -framework CoreFoundation
// #define PY_SSIZE_T_CLEAN
// #include <Python.h>
import "C"

//export sum
func sum(self, args *C.PyObject) *C.PyObject {
    var a, b C.longlong
    return C.PyLong_FromLongLong(a + b)
}

func main() {}

In this case cgo needs a preamble before import “C”: it may contain any C code, functions, includes statements, variables declarations and definitions and may then be referred to from Go code as though they were defined in the package “C”. Compile everything with

go build -buildmode=c-shared -o newmath.so

And newmath module is ready to be used in Python

from newmath import sum

print(sum(2, 40))

It works! ✌🏻

Anyway this is not the method I prefer to extend Python with Go because there are some limitations: cgo doesn’t support variadic functions! In this piece of code I don't use PyArg_ParseTuple because variadic functions like that need to be wrapped in another function.

int PyArg_ParseTuple_LL(
    PyObject * args,
    long long * a,
    long long * b
) {
    return PyArg_ParseTuple(args, "LL", a, b);
}

Macro functions need to be wrapped as well, e.g. PyLong_Check (source: include/python3.8/listobject.h).

int is_a_long(PyObject * p) {
    return PyLong_Check(p);
}

Another way to extend Python with Go is moving all the Python stuff into C and just call the Go function inside C. To do that, define sum function as a normal Go function using export sum preamble.

package main

import "C"


//export sum
func sum(a, b int) int {
    return (a + b)
}

func main() {}

and then compile everything with

go build -buildmode=c-archive -o libnewmath.a

This command generates two files:

• libnewmath.a we need to link later in the final step
• libnewmath.h containing some definitions, including

extern GoInt sum(GoInt p0, GoInt p1);

Now rewrite _newmath.c (prepend _ to exclude this file from go build) including libnewmath.h and using sum function we created with Go

#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "libnewmath.h"


static PyObject *sum_wrapper(PyObject *self, PyObject *args) {
    const long a, b;

    if (!PyArg_ParseTuple(args, "LL", &a, &b))
        return NULL;

    return PyLong_FromLong(sum(a, b));
}

static PyMethodDef MathMethods[] = {
    {"sum", sum_wrapper, METH_VARARGS, "Add two numbers."},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef newmathmodule = {
    PyModuleDef_HEAD_INIT, "newmath", NULL, -1, MathMethods
};

PyMODINIT_FUNC PyInit_newmath(void) {
    return PyModule_Create(&newmathmodule);
}

And then generate the module newmath.so, linking libnewmath

gcc _newmath.c -shared -o newmath.so `pkg-config --cflags --libs python3` `python3-config --libs --embed` -L . -lnewmath

Easier! 😉

Performances

Having to deal with different universes has a cost in terms of performances: when a Go function is used from another runtime, it spins up the Go runtime in parallel with the caller's runtime getting the goroutine threads, GC and all that other nice stuff that would normally be initialized up when running a Go program on its own.

Once you cross the boundary, try to do as much on the other side as you can! If you call a Go function inside a Python or C "for" cycle you spin up and destroy Go env on any iteration, it's more performant writing a single Python interface that wraps the execution of the entire "for" cycle inside Go.

When extending Python with Go is faster than using Python? The answer is parallel execution!

I tried to make a simple application to countdown from 25000000 to 0 with a thread and do the same thing with another thread. In Python I get the following results:

• Using simple threads ~3.7 sec.
• Using Multiprocessing Pools ~2.4 sec.
• Using Python JobLib ~2.2 sec.

Extending Python with Go and doing the parallel computation inside Go takes only ~0.02 sec.! Python is not performant running CPU-bound multithread programs, it's limited by Global Interpreter Lock - GIL.

You can check the code on Github

More on cgo

• cgo documentation
• Adventures With cgo: Part 1 — The Pointering by Sean Allen if you have to deal with pointers between C and Go.
• cgo is not Go by Dave Cheney.