Write Unbreakable Python

In this article, I’ll show you how to write Python with no runtime exceptions. This’ll get you partway to writing code that never breaks and mostly does what it’s supposed to do. We’ll do this by learning how to apply functional programming to Python. We’ll cover:

• ensure functions always work by learning Pure Functions
• avoid runtime errors by return Maybes
• avoid runtime errors in deeply nested data using Lenses
• avoid runtime errors by return Results
• creating pure programs from pure functions by Pipeline Programming

Why Care?

Writing code that doesn’t break is much like ensuring your teeth remain healthy. Both have proven ways to prevent problems, yet people still act irresponsibly, either by eating bad foods and not brushing, or in programming being lured by “getting things working quickly with what they know”.

For example, to prevent cavities in your teeth, you should avoid food high in sugar, drink public water supplies with fluoride, and brush your teeth twice a day. At least in the USA, many still have teeth problems. The access problem for dental hygiene can be an economic one. For others, they just aren’t responsible and don’t keep up on teeth health.

For programming, I can help with the access problem. This article will provide proven ways to equip you with the knowledge to help yourself.

For the responsibility, however, I empathize it’s harder. I’ll teach you the techniques, but you have to practice them. Brushing your teeth everyday, you can master it quite quickly. Utilizing functional programming techniques in a non-functional programming language takes a lot more practice and hard work.

Your teeth thank you for brushing often. Your code will thank you by consistently working.

What Does “Never Break” Mean Exactly?

The “Never Break” means the function will always work, will always return a value, won’t ever raise an Exception, nor will it ever exit the Python process unless you want it to. It doesn’t matter what order you call these functions or program in, how many times you call it, they are always predictable, and can be depended on.

What Does “Mostly Work” Mean Exactly?

Just because your functions work all the time doesn’t mean your software works. Pass the wrong environment variable for a Docker container, a downstream API is down, or perhaps you just did the math wrong. You still need unit, fuzz, feature tests, formal methods if you’re able, and good ole manual testing to verify your software works. Not having to worry about your software exploding randomly allows you to focus 100% on those.

Also, no amount of FP will save you from badly formatted code. You’ll still need things like PyLint to ensure you wrote print("Sup") instead of print("Sup)

If this is so obvious, when do I _not_ do this?

In order of priority:

1. When you’re exploring ideas.
2. When you’re committing code amongst those who aren’t trained in FP.
3. When you’re on a deadline.
4. When you’re modifying legacy code with no unit tests.

The allure and draw of Python is it is a dynamic language. While not as forgiving as JavaScript, Lua, or Ruby, it still allows a lot of freedom to play with ideas, various data types, and provide a variety of ways to run your code efficiently on various infrastructure architectures. With types only enforced (mostly) at runtime, you can try various ideas, quickly run them, correct mistakes you find after running them, and repeat this until you’ve locked down an implementation. While you _can_ use FP concepts for this, if you’re still learning, they can slow you down. Other times, this is a fun time to learn.

Commiting FP code to a Github repo where others aren’t familiar with FP, or have no clue why you’re coding things that don’t seem PEP Compliant can really cause problems. Typically a team adopts their own rules, patterns, and styles… and they don’t always have reasonable reasons. It’s best to learn why they code the way they do things, and adopt those standards. If you’re in a position to teach the team, great, but FP is quite alien, already has a reputation for being obtuse with horrible evangelists. Tread slowly here. Breaking trust in the team is one way to ensure your software never works correctly. Bad working relationships result in horrible software.

If you’re on a deadline, learning anything new can slow you down and risk not hitting it. Alternatively, it’s also the guaranteed way to ensure you learn something quickly, heh.

FP or not, you shouldn’t add or modify code in a large codebase you’re responsible for if it doesn’t have unit tests. Otherwise, you have no good idea if you’ve broken something, sometimes for days or weeks. Add tests first, THEN refactor things.

Functions That Always Work: Writing Pure Functions

Pure functions always work. They reason they always work is that they always return values. They don’t raise Exceptions. They’re technically not supposed to have side effects. They’re desired because they always return the same value, so are close to math in that you can depend on their result, or “answer”. Unit testing them also doesn’t require mocks, only stubs.

The 2 official rules go like this:

1. same input, same output
2. no side effects

Returning None is ok. The first function most Python devs are introduced too, print doesn’t appear to return value, much like console.log in JavaScript. However, it does: None:

result = print("Sup")
print(result == None) # True

Typically a function that returns no value, or None in Python’s case, is considered a “no operation” function, or “noop” for short (pronounced no-op). Noop’s are usually a sign the function has side effects. Noops are not pure functions. We know that print does produce side effects; the whole point of calling it is to produce the side effect of writing to standard out so we can see what the heck our code is doing.

For classes, however, it’s more subtle and now that you know the rules, you’ll see what’s wrong. Here’s how you stop verifying SSL certs for rest calls using a class to wrap urllib:

import request.rest
import ssl

req = request.rest()
req.disable_ssl()
res = req.get("https://some.server.com")

Note the disable_ssl() class method. It takes no parameters, and returns no value. Why? Probably because like most classes, it’s changing a setting internally in the class instance to turn off SSL stuff so the next person who does a REST call won’t have to have certs validated.

You do the complete opposite in functional programming. Although, in this case, it’s probably ok to call disable_ssl() multiple times without any harm. Things like get are more tricky.

So, impure function:

ssl = enabled

def get(url):
  return requests.get(url, ssl_enabled=ssl)

And a pure function:

def get(ssl, url):
  return requests.get(url, ssl_enabled=ssl)

And one that’s even more pure, and unit testable:

def get(requests, ssl, url):
  return requests.get(url, ssl_enabled=ssl)

And the most pure function you can possibly write in Python in a reasonable way:

def get(requests, ssl, url):
  try:
    result = requests.get(url, ssl_enabled=ssl)
    return result
  except Exception as e:
    return e

Write functions like that, and you’re well on your way to understanding how Golang is written.

Avoiding None by using Maybes

Python doesn’t offer a lot of guarantee’s, that’s why risk takers like it. If you’re writing software, you might not want risk. There are 3 main places this comes from when calling functions:

1. getting data from Dictionaries (cause you’re using them now, not Classes, right?)
2. getting data from Lists
3. getting data from outside of Python

Safer Dictionaries: Maybe Tuple

Let’s talk about dictionaries.

Dictionaries can work:

person = { firstName: "Jesse" }
print(person["firstName"]) # Jesse

Dictionaries can also fail:

print(person["lastName"])
# KeyError: 'lastName'

Wat do!?

You need to change how you think Python in 2 ways. First, how you access objects safely, such as using key in dictionary or lenses. Second, how you return values from functions, such as using the Golang syntax by returning multiple values which lets you know if the function worked or not, or a Maybe/Result type.

You can safely access dictionaries by creating a getter function:

def get_last_name(object):
  if "lastName" in object:
    return (True, object["lastName"], None)
  return (False, None, f"lastName does not exist in {object}")

This function is pure, and safe and will work with any data without blowing up. It also uses a nice trick Python has when you return a Tuple (read only List) from a function; you can destructure it to get 3 variables out in a terse syntax, making it feel like it is returning multiple values. We’ve chosen something similar to the Golang syntax where they return value, error, we’re returning didItWork, value, error. You can use the Golang syntax if you wish, I just don’t like writing if error != None.

ok, lastName, error = get_last_name(person)
if ok == False:
  return (False, None, f"Failed to get lastName from {person}")

So this is your first Maybe at a raw level. It’s a Tuple that contains if the function had your data or not, the data if it did, and if not why. Note if ok is False, your program is probably done at this point.

Developers are encouraged to create Exceptions for everything that isn’t necessarily exceptional, and raise them so others can catch them or different types of them and react accordingly, usually higher up the function chain. The problem with this is you can’t easily read your code and see errors as they could be coming from completely different files. Using maybes in this way, it’s very clear what function failed, and your don’t have to wrap things in try/catch “just in case”.

Safer Dictionaries: Maybe Type

Tuples are ok, but are verbose. A shorter option is the Maybe type. We’ll use PyMonad’s version because they did a lot of hard work for us. First import it:

from pymonad.Maybe import *

Then, we’ll create our getLastName function to return a Maybe type instead of a Tuple like we did before:

def get_last_name(object):
  if "lastName" in object:
    return Just(object["lastName"])
  return Nothing

I say the word “type”, but in Python, it feels like a function. Replace (True, data, None) with Just(data) and (False, None, Exception('reason')) with Nothing. You can then use it:

lastNameMaybe = get_last_name(person)

You first instinct will be “cool, if it’s a Just, how do I get my data out?”. Well, you don’t.

“Wat!?”

Trust me, we’ll cover this in Pipeline Programming below, for now, just know this function will never fail, and you’ll always get a Maybe back, ensuring your code doesn’t throw errors, and is more predictable and testable.

Speaking of which, here’s Pytest:

def test_get_last_name_happy():
  result = get_last_name({'lastName': 'cow'})
  assert result == Just('cow')

“Wat…”

😁 Ok, till you get more comfortable, try this:

def test_get_last_name_happy():
  result = get_last_name({'lastName': 'cow'})
  assert result.value == 'cow'

Safer Lists: Maybe Type

The same thing can be said for Listsin Python.

people = [{'firstName': 'Jesse'}]
first = people[0]

Cool.

people = []
first = people[0]
# IndexError: list index out of range

Uncool. There are many ways to do this; here’s the quickfix you’ll end up repeating a lot:

def get_first_person(list):
  try:
    result = list[0]
    return Ok(result)
  except Exception:
    return Nothing

You’ll see this implemented in a more re-usable way as nth, except instead of returning None, it’ll return a Maybe:

def nth(list, index);
  try:
    result = list[index]
    return OK(result)
  except Exception:
    return Nothing

Pure Deeply Nested Data Using Lenses

You know how to make unbreakable functions by making them pure. You know how to access Dictionaries and Lists safely using Maybes.

In real-world applications, you usually get larger data structures that are nested. How does that work? Here’s some example data, 2 people with address info:

people = [
  { 'firstName': 'Jesse', 'address': { skreet: '007 Cow Lane' } },
  { 'firstName': 'Bruce', 'address': { skreet: 'Klaatu Barada Dr' } }
]

Let’s get the 2nd person’s address safely using a Maybe:

def get_second_street(list):
  second_person_maybe = nth(list, 1)
  if isinstance(second_person_maybe, Just):
    address_maybe = get_address( second_person_maybe.value)
    if isinstance(address_maybe, Just):
      street_maybe = get_street(address_maybe.value)
      return street_maybe
  return Nothing

Yeahh…… no. Gross. Many have taken the time to do this with a better, easier to use API. PyDash has a get method:

from pydash import get

def get_second_street(list):
  return get(list, '[1].address.skreet')

Cool, eh? Works for Dictionaries, Lists, and both merged together.

Except… one small issue. If it doesn’t find anything, it returns a None. None will cause runtime Exceptions. You can provide a default as the 3rd parameter. We’ll wrap it with a Maybe; less good looking, but MOAR STRONG AND PURE.

def get_second_street(list):
  result = get(list, '[1].address.skreet')
  if result is None:
    return Nothing
  return Just(result)

Returning Errors Instead of Raising Exceptions

Dictionaries and Arrays not having data is ok, but sometimes things really do break or don’t work… what do we do without Exceptions? We return a Result. You have 2 options on how you do this. You can use the Tuplewe showed you above, doing it Golang style:

def ping():
  try:
    result = requests.get('https://google.com')
    if result.status_code == 200:
      return (True, "pong", None)
    return (False, None, Exception(f"Ping failed, status code: {result.status_code}")
  except Exception as e:
    return (False, None, e)

However, there are advantages to use a true type which we’ll show later in Pipeline Programming. PyMonad has a common one called an Either, but Left and Right make no sense, so I made my own called Result based on JavaScript Folktale’s Result because “Ok” and “Error” are words people understand, and associate with functions working or breaking. Left and Right are like… driving… or your arms… or dabbing… or gaming… or anything other than programming.

def ping():
  try:
    result = requests.get('https://google.com')
    if result.status_code == 200:
      return Ok("pong")
    return Error(Exception(f"Ping failed, status code: {result.status_code}"))
  except Exception as e:
    return Error(e)

You don’t have to put Exceptions in Error; you can just put a String. I like Exception’s because they have helpful methods, info, stack trace info, etc.

Pipeline Programming: Building Unbreakable Programs

You know how to build Pure Functions that don’t break. You can safely get data that has no guarantee it’ll be there using Maybes and Lenses. You can call functions that do side effects like HTTP requests, reading files, or parsing user input strings safely by returning Results. You have all the core tools of Functional Programming.. how do you build Functional Software?

By composing functions together. There are a variety of ways to do this purely. Pure functions don’t break. You build larger functions that are pure that use those pure functions. You keep doing this until your software emerges.

Wait… What do you mean by “Composing”?

If you’re from an Object Oriented Background, you may think of Composing as the opposite of Inheritance; uses class instances inside of another class. That’s not what we mean here.

Let’s parse some JSON! The goal is to format the names of humans from a big ole List of Dictionaries. In doing this you’ll learn how to compose functions. Although these aren’t pure, the concept is the same.

Behold, our JSON string:

peopleString = """[
    {
        "firstName": "jesse",
        "lastName": "warden",
        "type": "Human"
    },
    {
        "firstName": "albus",
        "lastName": "dumbledog",
        "type": "Dog"
    },
    {
        "firstName": "brandy",
        "lastName": "fortune",
        "type": "Human"
    }
]"""

First we must parse the JSON:

def parse_people(json_string):
  return json.loads(json_string)

Next up, we need to filter only the Humans in the List, no Dogs.

def filter_human(animal):
  return animal['type'] == 'Human'

And since we have a List, we’ll use that predicate in the filter function from PyDash:

def filter_humans(animals):
  return filter_(animals, filter_human)

Next up, we have to extract the names:

def format_name(person):
  return f'{person["firstName"]} {person["lastName"]}'

And then do that on all items in the List; we’ll use map from PyDash for that:

def format_names(people):
  return map_(people, format_name)

Lastly, we need to upper case all the names, so we’ll map yet again and start_case from PyDash:

def uppercase_names(people):
  return map_(people, start_case)

Great, a bunch of functions, how do you use ’em together?

Nesting

Nesting is the most common.

def parse_people_names(str):
  return uppercase_names(
    format_names(
      filter_humans(
        parse_people(str)
      )
    )
  )

Oy… that’s why you often hear “birds nest” being the negative connotation to describe code.

Flow

While PyDash and Lodash call it flow, this is the more common way to build larger functions out of smaller ones via composing them, and gives you your first insight into “pipeline” style programming.

parse_people = flow(parse_people, filter_humans, format_names, uppercase_names)

Pipeline: PyMonad Version

Now Flow is quite nice, but hopefully you saw some problems. Specifically, none of those functions are super pure. Yes, same input, same output and no side effects… but what happens when one of them returns a Nothing? When happens if you’re doing dangerous stuff and one returns a Result that contains an Error instead of an Ok?

Well, each of those types are tailor made to pipe together. You saw how the functions I made for flow worked together; they just needed 3 rules:

1. be a mostly pure function
2. have a single input
3. return an output

The Maybe and Result can wired together too, but they have a few extra special features. The only 4 we care about for this article are:

1. if a Maybe gets a Just, it’s smart enough to get the Just(thing).value and pass it to the next function. The Result is the same unwrapping the value in the Ok and and passing it to the next function.
2. Each expects you to return the same type back. If you chain Maybe‘s together like you do in flow, then it’s expected you return your Just(thing) or Nothing.
3. Both handle bad things. If a chain of Maybe‘s suddenly gets a Nothing, the entire thing will give you a Nothing. If any of the functions you’ve wired together get a Result and suddenly one gets an Error in the chain, the entire chain gets an Error.
4. They have flow built in; but instead of calling it, you use weird, new, non-Python symbols to confuse you, look impressive, and make the code feel less verbose despite increased brain activity.

That’s a lot, ignore it. Just look at the example:

def parse_people_names(str):
  return parse_people(str) \
  >> filter_humans \
  >> format_names \
  >> uppercase_names

Goodbye! 👋🏼

If that’s a bit alien and strange, that’s because:

• you’re doing Functional Programming in a Object Oriented + Imperative language; you rock!
• >> isn’t Pythonic nor PEP Compliant®
• Most Python devs see a \ and think “Aw man, this code is too long…”
• “… wait, you’re putting functions there, but not calling them, nor giving them parameters, what in the…”

This is why many Functional Programmers are nice even if they are not-so-great evangelists. Many non-FP destined people see that kind of code, freakout and leave. This makes many FP’ers lonely, thus they are more than happy to be cordial and polite when people talk to them in the programming community.

Manual Pipeline

Remember that nasty example of getting deeply nested properties before you learned Lenses? Let’s replace that with a pipeline using a Maybe; it’ll give you a better sense of how these things are wired together, like flow is above.

def get_second_street(list):
  second_person_maybe = nth(list, 1)
  if isinstance(second_person_maybe, Just):
    address_maybe = get_address( second_person_maybe.value)
    if isinstance(address_maybe, Just):
      street_maybe = get_street(address_maybe.value)
      return street_maybe
  return Nothing

Gross. Ok, let’s start from the top, first we need lastName (btw, I’m glad you ignored the ‘Goodbye’ and are still here, YOU GOT WHAT IT TAKES, OH YEAH):

def get_second_person(object):
  return nth(object, 1)

Cool, next up, get the address:

def get_address(object):
  if 'address' in object:
    return Just(object['address'])
  return Nothing

Finally, get dat skreet skreet!

def get_street(object):
  if 'skreet' in object:
    return Just(object['skreet']
  return Nothing

Now let’s compose ’em together.

def get_second_street(object):
  second_person_maybe = get_second_person(object)
  if isinstance(second_person_maybe, Nothing):
    return Nothing

  address_maybe = get_address(second_person_maybe.value)
  if isinstance(address_maybe, Nothing):
    return Nothing

  street_maybe = get_street(address_maybe)
  if isinstance(street_maybe, Nothing)
    return Nothing

  return street_maybe

Essh… ok, let’s test her out:

second_street_maybe = get_second_street(people)

Note a couple things. Each time you call a function, you then inspect if the return value is a Nothing. If it is, you immediately just return that and stop running everything else. Otherwise, you call the next one in the chain and unwrap the value. Here we’re doing that manually via maybe.value. Also, the return street_maybe at the end is a bit redundant; no need to check for Nothing there, just return it, but I wanted you to see the repeating pattern 3 times.

That pattern is what the >> does for you: checks for Nothing and aborts early, else unwraps the value and gives to the function in the chain. Rewriting it using that bind operator:

def get_second_street(object):
  return get_second_person(object) \
  >> second_person_maybe \
  >> get_address \
  >> get_street

It’s easy to forget the \. This is because Python is super strict on whitespace and only occasionally lets you do line breaks with code. If you don't want to use that, just put her all on 1 line:

def get_second_street(object):
  return get_second_person(object) >> second_person_maybe >> get_address >> get_street

Manual Result Pipeline

Let’s do some dangerous shit. We’re going to load our encryption key from AWS, then fetch our OAuth token, then load in a list of Loan products from an API that requires that token to work, and finally snag off just the ID’s.

Dangerous stuff? Potential Exceptions? A job for Result.

Get dat key:

def get_kms_secret():
  try:
    result = client.generate_data_key(
      KeyId='dev/cow/example'
    )
    key = base64.b64decode(result['Plaintext']).decode('ascii')
    return Ok(key)
  except Exception as e:
    return Error(e)

Don’t worry if you don’t know what the heck KMS is, it’s just Amazon’s encryption stuff, and gives you keys that you’re only allowed to get. If you’re not, that function will fail. It just gives us a temporary private encryption key as text. We’ll use that to get our OAuth token.

Next up, get our OAuth token via the requests library:

def get_oauth_token(key):
  try:
    response = requests.post(oauth_url, json={'key': key})
    if response['status_code'] == 200:
      try:
        token_data = response.json()
        return Ok(token_data['oauth_token'])
      except Exception as parse_error:
        return Error(parse_error)
    return Error(Exception(response.text))
  except Exception as e:
    return Error(e)

You have a style choice here. If you look above, there are 4 potential failures: status code not being a 200, failing to parse the JSON, failing to find the oauth token in the JSON you parsed, or a networking error. You can just “handle it all in the same function” like I did above. The point of mashing it together is “Did I get a token or not?”. However, if you don’t like the nested ifs and trys then you can split that into 4 functions, each taking a Result as well to wire them together in order.

Now that we have our token, let’s call the last API to get a list of Loan products. We’ll get a list potentially, but all we want is the id’s, so we’ll map to pluck those off:

def get_loan_ids(token):
  try:
    auth_header = {'authentication': f'Bearer {token}'}
    response = requests.get(loan_url, headers=auth_header)
    if response.status_code == 200:
      try:
        loans = response.json()
        ids = map_(loans, lambda loan: get(loan, 'id', '???'))
        return Ok(ids)
      except Exception as e:
        return Error(e)
    return Error(Exception(response.text))
  exception Exception as overall_error:
    return Error(overall_error)

If everything goes well, you’ll get a list of strings. Otherwise, and Error. Let’s wire all 3 up:

def get_loan_ids():
  return get_kms_secret() \
  >> get_oauth_token \
  >> get_loan_ids

When you go:

loan_ids_result = get_load_ids()

Either it works, and that loan_ids_result is an Ok, or it failed somewhere in there and it’s an Error containing the Exception or error text.

… now, one cheat I did, and you’re welcome to mix and match Maybes and Results together. You see when we attempt to get the loan id?

get(loan, 'id', '???')

That 3rd parameter is a default value if the property isn’t there or is None. The _right_ thing to do is use a Maybe instead. You can be pragmatic like this if you wish, just be aware, these are the types of things you’ll see where “the code has no errors but doesn’t work” 🤪. Also, I tend to like Errors more than Nothing in these scenarios because they give you an opportunity to provide an Exception or text with a lot of context as to WHY. Why is huge for a programmer, especially when you’re close to the error and know why it probably failed in a large set of functions that could all possibly break.

Conclusions

Functions are fine, but Exceptions can cause a lot of indirection. Meaning, you think you know the 3 ways a function, or even a program can fail, but then another unexpected Exception comes from some other deeply nested function. Completely removing these, and locking down the ones you know, or don’t know, will blow up using try/catch via pure functions removes that problem for you. You now know for a fact what paths your functions take. This is the power of pure functions.

However, that doesn’t protect you from using null (i.e. None) data. You’ll get all kinds of runtime Exceptions in functions that were expecting good data. If you force all of your functions to handle that eventuality via Maybes, then you never get null pointers. This includes using Lenses to access deeply nested data.

Once you go outside of your program via side effects such as making REST calls, reading files, or parsing data, things can go wrong. Errors are fine, and good, and educational. Having a function return a Result instead of raising an Exception is how you ensure the function is pure. You can also abort early if there is a problem vs having cascading effects with other functions not having their data in a state they need. You also, much like Golang and Rust, have the opportunity to document what the error is where it happens with helpful context vs. guessing at a long stack trace.

Finally, once you’ve written pure functions, you can build larger pure functions that compose those together. This is how you build pure software via pipeline programming (also called railroad programming, a style of streaming, etc).

Python provides the PyDash library to give you the basics of pure functions with working with data and lists. PyMonad provides the basics in creating Maybe and Either (what I call Result). If you’re interested, there is the Coconut programming language that integrates and compiles to Python, allowing you to write in a more Functional Programming style. Here’s a taste re-writing our example above:

def get_loan_ids():
  return get_kms_secret()
  |> get_oauth_token
  |> get_loans
  |> map( .id )

Don’t fret if all of this seems overwhelming. Functional Programming is a completely different way of thinking, Python is not a functional language, and these are all super advanced concepts I just covered the basics of. Just practice the pure functions and writing tests for them. Once you get the basics of that, you’ll start to get a feel, like a Spidey Sense, of when something is impure and has side effects. With Maybes, you’ll love them at first, then realize once you start using them responsibly, how much extra work you have to do to avoid null pointers. Eithers/Results can be challenging to debug at first as you won’t be wading through stacktraces, and you learn what error messages are best to write, and how to capture various failures. Stick with it, it’s ok if you use only a little; the little you use will still help your code be more solid.