Simply put, dependency injection is a collection of programming techniques that enables software components to have their dependencies replaced, thereby increasing re-usability, for example by allowing a database module to depend on a connection string in order that it can connect to multiple databases. Dependency injection also improves testability because complicated dependencies such as database connections can be easily replaced with mocks.
In this post we'll study dependency injection in Python. We'll see how it can be made completely type safe using functional programming, specifically with a modern functional programming library called pfun.
Dependency injection in Python is typically implemented using monkey patching. In fact, monkey patching as a form of providing dependencies is so common that the unittest.mock.patch
function was added to the standard library in Python 3.3.
While monkey patching is a simple technique, it often leads to rather complex patching scenarios, where it's tricky to figure out what and how to patch. Moreover, it can be tricky to achieve complete type safety with monkey patching on both the dependency consumer and provider side. Finally, there are no straight-forward ways of making sure that the dependency provider provides all required dependencies, often leading to statements such as if dependency is not None
.
An alternative (and potentially even simpler) method for dependency injection is the following: function arguments. In fact, this approach is so simple that using the term "dependency injection" to describe it seems almost pretentious. An attractive feature of implementing dependency injection with functions that take arguments is that it is completely type safe by default (provided that you use type annotations of course). Moreover, you can tell a function's dependencies simply by reading its signature, making it totally clear what needs to be provided, and in fact making it impossible to not provide all required dependencies.
The main drawback from using function arguments as your "injection" mechanism, is that functions that call functions that have "injected dependencies" now need to take those dependencies as arguments themselves.
For example, consider a function connect
that returns a connection to a database given a connection_str
as argument:
def connect(connection_str: str) -> Connection:
...
To achieve the promise of "dependency injection", every function that calls connect
must now take a connection_str
parameter in addition to its other parameters, in order that the calling function itself can be used against different databases:
def get_user(connection_str: str, user_id: int) -> User:
connection = connect(connection_str)
return connection.get_user(user_id)
For functions with many dependencies, this quickly becomes very tedious.
Thankfully, we can use functional programming to improve the situation by a margin: Notice how the only use get_user
has of connection_str
is to pass it to connect
. With functional programming we can abstract this pattern of functions taking arguments only to pass them to other functions into some general 'plumbing' functions. This will alow us to write get_user
without mentioning the connection_str
at all!
To keep things nice and readable (and save us some typing), lets define a type-alias that represents functions that require dependencies:
from typing import TypeVar, Callable
R = TypeVar('R')
A = TypeVar('A')
Depends = Callable[[R], A]
In words, Depends[R, A]
is a function that depends on something of type R
to produce something of type A
. For example, our previous connect
function is a Depends[str, Connection]
value: it's a function that depends on a str
to produce a Connection
.
Simple enough, but how do we use the Depends
represented by connect
in get_user
without explicitly taking connection_str
as a parameter and calling connect
? To do that, let's introduce our first plumbing function map_
:
B = TypeVar('B')
def map_(d: Depends[R, A], f: Callable[[A], B]) -> Depends[R, B]:
def depends(r: R) -> B:
return f(d(r))
return depends
map_
precisely implements the "passing of parameters" plumbing we were talking about earlier: it creates a new Depends
value that calls d
, and passes the result to a function f
. This allows us to write get_user
as follows:
def get_user(user_id: int) -> Depends[str, User]:
return map_(connect, lambda con: con.get_user(user_id))
And voila: we have type-safe dependency injection using only Depends
(which is just a type-alias for functions of 1 argument) and map_
(which is just a function that composes Depends
values with functions). Realise that we could keep applying map_
to Depends
values to return a final Depends
all the way back to where we make desicions about what concrete connection strings to pass in (probably in the main section of our program). The pattern we've discovered here seems simple enough that it would be surprising if we were the firsts to think of it, and sure enough it's in fact a common pattern in functional programming called the reader effect.
That's all well and good, but what happens when we want to map_
functions that return new Depends
values? For example, let's imagine that in addition to reading user data from databases, our application also needs to call an HTTP api with the user data. Doing so requires authentication credentials that we want to inject. The function that calls the api might look like this:
from typing import Tuple
Auth = Tuple[str, str]
def call_api(user: User) -> Depends[Auth, bytes]:
def depends(auth: Auth) -> bytes:
...
return depends
We might try to use map_
to pass the result of the Depends
value returned by get_user
to call_api
. But in doing so we would end up with a Depends[str, Depends[Auth, bytes]]
:
if __name__ == '__main__':
user_1: Depends[str, User] = get_user(user_id=1)
call_api_w_user_1: Depends[str, Depends[Auth, bytes]] = map_(user_1, call_api)
When we want to call call_api_w_user_1
, we need to first supply the connection string, and then the auth information:
result: bytes = call_api_w_user_1('user@prod')(('prod_user', 'pa$$word'))
This might not seem like big issue in this example, but notice that we might apply map_
over several functions that produces Depends
values with the same dependency type, resulting in a situation where we have to pass in the same dependency many times. We might also have a varying number of dependencies if we use map_
in a loop, leading to a situation where we don't know how many times we need to call the final Depends
!
To fix this situation, let's introduce another plumbing function that can pass dependencies to both a Depends
value, and a Depends
value returned by a function that's being mapped. We'll call it and_then
since it chains together results of Depends
values with functions that return new Depends
values:
from typing import Any
R1 = TypeVar('R1')
def and_then(d: Depends[R, A], f: Callable[[A], Depends[R1, B]]) -> Depends[Any, B]:
def depends(r: Any) -> B:
return map_(d, f)(r)
return depends
The observant reader will notice that there's a big problem with the typing of our and_then
function: it returns a Depends[Any, B]
! Here you might object: 'But wait a minute, I though you said this was "completely type safe"'! The reason for this epic fail is that the r
parameter passed to the depends
function must be passed to both d
, which means it must be of type R
, and the Depends
returned by f
, which means it must be of type R1
. In other words, the dependency must be of both type R
and R1
at the same time. In our example, using and_then
to combine get_user
and call_api
should result in a type that is both a str
and a Tuple[str, str]
:
call_api_w_user_1: Depends[?, bytes] = and_then(get_user(user_id=1), call_api)
Such a type is called an intersection type and unfortunately it's not supported by the Python typing
module (yet). So without introducing third party libraries, this is the best we can do.
This is where the library pfun comes into the picture. In pfun
, map_
and and_then
are instance methods of a Depends
type, but the idea is exactly the same:
from pfun import Depends
def connect() -> Depends[str, Connection]:
...
def get_user(user_id: int) -> Depends[str, User]:
connect().map(lambda con: con.get_user(user_id))
def call_api(user: User) -> Depends[Auth, bytes]:
...
call_api_w_user_1 = get_user(user_id=1).and_then(call_api)
pfun
provides mechanisms for combining Depends
values with different dependency types when using MyPy with one small requirement: the dependency type must be a typing.Protocol
. The reason is that intersections of protocol types are simply a new protocol that inherits from both:
from typing import Protocol
class P1(Protocol):
pass
class P2(Protocol):
pass
class Intersection(Protocol, P1, P2):
pass
This means that when using and_then
with Depends
values that have protocol types as dependencies, pfun
can infer an intersection type automatically. Let's rewrite our example to take advantage of this feature:
from typing import Protocol, Tuple
class HasConnectionStr(Protocol):
connection_str: str
class HasAuth(Protocol):
auth: Tuple[str, str]
def connect() -> Depends[HasConnectionStr, Connection]:
...
def get_user(user_id: int) -> Depends[HasConnectionStr, Connection]:
connect().map(lambda con: con.get_user(1))
def call_api(user: User) -> Depends[HasAuth, bytes]:
...
call_api_w_user_1 = get_user(user_id=1).and_then(call_api)
The type of call_api_w_user_1
in this example will be Depends[pfun.Intersection[HasConnectionStr, HasAuth], bytes]
. The dependency type pfun.Intersection
ensures that call_api_w_user_1
must be
called with an argument that fulfills both the HasConnectionStr
protocol and the HasAuth
protocol. In other words, this would be a type error:
class Dependency:
def __init__(self):
connection_str = 'user@prod'
call_api_w_user_1(Dependency()) # MyPy Error!
This compositional nature of and_then
means that complex dependency types are built from simple dependency types automatically, which means you can always figure out which dependencies a part of your application has, simply by inspecting it's return type. To learn more about functional programming with pfun
check out some of the other posts in the series, the documentation or the github repo.
Top comments (1)
Thanks Sune, this is a nicely written article which demonstrates how to compose functions in Python while taking care of the environment/dependencies in a type-safe manner.
The astute reader will notice that this ressembles mecanisms found in ZIO in Scala or effect-ts in TypeScript.