Exceptions in Python are a great part of the language. They're easy to use, easy to subclass, and pretty fast. Unfortunately, convenience can lead to sloppy thinking and sloppier programming. In my 5+ years of Python programming in various startups, I've encountered a lot of bad exception handling, so here are some recommendations for dos and don'ts.
Towards the bottom of the post I linked to above, the author provides evidence that
catch can be slower than
if statements when the errors are frequent. His somewhat flippant advice is "to look into more reliable infrastructure," which is helpful in some contexts. I've found, however, that the convenience of Python exceptions, coupled with the dynamically-typed nature of the language itself, leads to developers either taking exceptions for granted, over-using
catch, or both.
One advantage of statically-typed languages like Haskell and Rust is that they force the programmer to think more deliberately about errors, and provide them with the language constructs to do so in the form of
Either (in Haskell) or
Result (in Rust.)
[For those unfamiliar,
Option are ways to avoid
None aka Hoare's 'billion-dollar mistake'; in Haskell, a value of type
Maybe a can either be
Just a (e.g.
Just "a friend"), or
Nothing. Similarly, a value of type
Either a b can be either (get it?)
Left a (e.g.
Left "for dead") or
Right b (e.g.
Right 1). Both languages use one half of their
Either-like types for "success" values, and the other for "error" values; it's common to see things like
Result<Int,String> in Rust or
Either String Int in Haskell. (I know why Haskell orders it the way it does (ask me in the comments if you care), I don't know why Rust switched the order.)]
Both of these languages have "real" exceptions, but most Haskell and Rust programmers eschew them in favor of
Result. Why? Because using these types allows you, and people reading your code, to differentiate between errors and exceptions. Errors are anticipatable, semi-routine, unfortunate but not disastrous. Exceptions are exceptional—they shouldn't happen, their very presence represents some serious derivation from the usual course of events.
Even in Python, which doesn't have anything like
Either, it's still worth thinking about which of your "exceptions" are actually exceptional, and which are merely errors. Viz:
try: middle_name = user['middle_name'] except KeyError: middle_name = '' try: birthday = user['birthday'] except KeyError: birthday = ''
We're treating two pieces of missing information the same way, but they're actually completely different. Lots of people don't have middle names; everybody has a birthday. (Of course, some users may be wary of divulging such information to strangers on the internet, but let's assume you've earned their trust.) It's much clearer to treat different cases differently:
middle_name = user.get('middle_name', '') birthday = user['birthday']
Now we're treating errors and exceptions with the relative alarm they each deserve.
This is an extension of the previous point. You can use multiple
try, and if there's cause to, you should.
# instead of this... try: do_whatever() except SomeException as err: if isinstance(err, FooException): handle_foo_exception(err) elif isinstance(err, BarException): handle_bar_exception(err) # ...do this try: do_whatever() except FooException as err: handle_foo_exception(err) except BarException as err: handle_bar_exception(err)
You can facilitate and extend the pattern above by writing and throwing your own exception subclasses. It's easy and aids in clarity.
# instead of this... def check_birthday(user): if not user.birthday: raise ValueError('User birthday missing!') elif user.birthday.year > 2002: raise ValueError('User is too young!') elif user.birthday.year < 1900: raise ValueError('User is probably dead!') try: check_birthday(user) except ValueError as err: msg = str(err) if 'missing' in msg: redirect_to_birthday_input() elif 'young' in msg: redirect_to_site_for_teens() elif 'old' in msg: handle_probable_fraud() # ...do this class UserBirthdayException(ValueError): pass class UserBirthdayMissing(UserBirthdayException): pass class InvalidUserBirthday(UserBirthdayException): def __init__(self, problem, *args, **kwargs): super().__init__(*args, **kwargs) self.problem = problem def check_birthday(user): if not user.birthday: raise UserBirthdayMissing elif user.birthday.year > 2002: raise InvalidUserBirthday(problem='young') elif user.birthday.year < 1900: raise InvalidUserBirthday(problem='old') try: check_birthday(user) except UserBirthdayMissing: redirect_to_birthday_input() except InvalidUserBirthday as err: if err.problem == 'young': redirect_to_site_for_teens() else: handle_probable_fraud()
Sure, it's more boilerplate, but the added clarity and control is worth it. It's especially worth considering if you're doing a lot of validation, or find yourself repeatedly handling a set of errors from an API—write some custom error classes and a response handler function, and your days of grepping through your codebase to find all the places you need to change your error handling because some third-party has switched from colons to tabs in their API responses are over.
Like everything else in Python, exceptions are objects, and they have their own classes and inheritance.
catch will handle subclasses of errors appropriately.
class AError(Exception): pass class BError(AError): pass class CError(AError): pass try: raise BError except CError: print('Caught CError') # won't happen except AError: print('Caught AError') # will happen
Familiarize yourself with Python's builtin exceptions, and don't be shy about using inheritance when defining your own error classes.
Quick, tell me which of these functions could throw
def f(x): try: a = do_something(x) b = do_something_else(a) c = lastly_do(b) log_or_whatever(c) return c except SomeException as err: log_exception(err) return None
Give up? Me too. How about here:
def f2(x): a = do_something(x) try: b = do_something_else(a) except SomeException as err: log_exception(err) return None else: c = lastly_do(b) log_or_whatever(c) return c
Better, no? Keep the scope of your
catch blocks as narrow as possible. Yes, nested
else blocks can get annoying, but why not just refactor each of the potential culprits into separate functions?
This should go without saying—by me, because your tooling (
pylint, etc.) should be saying it at you instead. But I'm saying it anyway, because it bears repeating (and because I've seen—and written!—far too many
pylint: disable=bare-except comments.)
A common excuse I've seen for
catch Exception is ignorance—there's a 3rd-party API with bad documentation, or some complicated code with multiple points of failure. Unfortunately, as with the law, ignorance of potential exceptions is not a valid excuse. Do as much research as you can, add logging, and then refactor to catch the actual exceptions you want to catch.
And whatever you do, DO NOT
catch BaseException. Trust me.
Computers are incredibly complex; the world is a terrifying, unpredictable place; chaos is the only constant. Stuff (as it were) will happen. You don't have to
catch everything; you don't even have to try. Think about what what handling every exception would even mean: if your DB can't be reached, or a config file doesn't exist, or Jeff Bezos decides overnight to get out of cloud storage and into self storage, what's the point of carrying on like your user can go about their business as normal? Five 9s is the goal, but Potemkin-oriented programming serves no-one. Print that TB, return that 500, move on.