Exceptions in Python are a great part of the language. They're easy to use, easy to subclass, and pretty fast. Unfortunately, convenience can lead to sloppy thinking and sloppier programming. In my 5+ years of Python programming in various startups, I've encountered a lot of bad exception handling, so here are some recommendations for dos and don'ts.
Do think about your exceptions
Towards the bottom of the post I linked to above, the author provides evidence that try
/catch
can be slower than if
statements when the errors are frequent. His somewhat flippant advice is "to look into more reliable infrastructure," which is helpful in some contexts. I've found, however, that the convenience of Python exceptions, coupled with the dynamically-typed nature of the language itself, leads to developers either taking exceptions for granted, over-using try
/catch
, or both.
One advantage of statically-typed languages like Haskell and Rust is that they force the programmer to think more deliberately about errors, and provide them with the language constructs to do so in the form of Maybe
/Either
(in Haskell) or Option
/Result
(in Rust.)
[For those unfamiliar, Maybe
/Option
are ways to avoid null
, aka undefined
aka None
aka Hoare's 'billion-dollar mistake'; in Haskell, a value of type Maybe a
can either be Just a
(e.g. Just "a friend"
), or Nothing
. Similarly, a value of type Either a b
can be either (get it?) Left a
(e.g. Left "for dead"
) or Right b
(e.g. Right 1
). Both languages use one half of their Either
-like types for "success" values, and the other for "error" values; it's common to see things like Result<Int,String>
in Rust or Either String Int
in Haskell. (I know why Haskell orders it the way it does (ask me in the comments if you care), I don't know why Rust switched the order.)]
Both of these languages have "real" exceptions, but most Haskell and Rust programmers eschew them in favor of Maybe
/Either
/Option
/Result
. Why? Because using these types allows you, and people reading your code, to differentiate between errors and exceptions. Errors are anticipatable, semi-routine, unfortunate but not disastrous. Exceptions are exceptional—they shouldn't happen, their very presence represents some serious derivation from the usual course of events.
Even in Python, which doesn't have anything like Maybe
/Either
, it's still worth thinking about which of your "exceptions" are actually exceptional, and which are merely errors. Viz:
try:
middle_name = user['middle_name']
except KeyError:
middle_name = ''
try:
birthday = user['birthday']
except KeyError:
birthday = ''
We're treating two pieces of missing information the same way, but they're actually completely different. Lots of people don't have middle names; everybody has a birthday. (Of course, some users may be wary of divulging such information to strangers on the internet, but let's assume you've earned their trust.) It's much clearer to treat different cases differently:
middle_name = user.get('middle_name', '')
birthday = user['birthday']
Now we're treating errors and exceptions with the relative alarm they each deserve.
Do handle specific errors appropriately
This is an extension of the previous point. You can use multiple catch
es per try
, and if there's cause to, you should.
# instead of this...
try:
do_whatever()
except SomeException as err:
if isinstance(err, FooException):
handle_foo_exception(err)
elif isinstance(err, BarException):
handle_bar_exception(err)
# ...do this
try:
do_whatever()
except FooException as err:
handle_foo_exception(err)
except BarException as err:
handle_bar_exception(err)
Do write your own exception subclasses
You can facilitate and extend the pattern above by writing and throwing your own exception subclasses. It's easy and aids in clarity.
# instead of this...
def check_birthday(user):
if not user.birthday:
raise ValueError('User birthday missing!')
elif user.birthday.year > 2002:
raise ValueError('User is too young!')
elif user.birthday.year < 1900:
raise ValueError('User is probably dead!')
try:
check_birthday(user)
except ValueError as err:
msg = str(err)
if 'missing' in msg:
redirect_to_birthday_input()
elif 'young' in msg:
redirect_to_site_for_teens()
elif 'old' in msg:
handle_probable_fraud()
# ...do this
class UserBirthdayException(ValueError):
pass
class UserBirthdayMissing(UserBirthdayException):
pass
class InvalidUserBirthday(UserBirthdayException):
def __init__(self, problem, *args, **kwargs):
super().__init__(*args, **kwargs)
self.problem = problem
def check_birthday(user):
if not user.birthday:
raise UserBirthdayMissing
elif user.birthday.year > 2002:
raise InvalidUserBirthday(problem='young')
elif user.birthday.year < 1900:
raise InvalidUserBirthday(problem='old')
try:
check_birthday(user)
except UserBirthdayMissing:
redirect_to_birthday_input()
except InvalidUserBirthday as err:
if err.problem == 'young':
redirect_to_site_for_teens()
else:
handle_probable_fraud()
Sure, it's more boilerplate, but the added clarity and control is worth it. It's especially worth considering if you're doing a lot of validation, or find yourself repeatedly handling a set of errors from an API—write some custom error classes and a response handler function, and your days of grepping through your codebase to find all the places you need to change your error handling because some third-party has switched from colons to tabs in their API responses are over.
Do Make use of exception class hierarchies
Like everything else in Python, exceptions are objects, and they have their own classes and inheritance. catch
will handle subclasses of errors appropriately.
class AError(Exception):
pass
class BError(AError):
pass
class CError(AError):
pass
try:
raise BError
except CError:
print('Caught CError') # won't happen
except AError:
print('Caught AError') # will happen
Familiarize yourself with Python's builtin exceptions, and don't be shy about using inheritance when defining your own error classes.
Do keep it tight
Quick, tell me which of these functions could throw SomeException
:
def f(x):
try:
a = do_something(x)
b = do_something_else(a)
c = lastly_do(b)
log_or_whatever(c)
return c
except SomeException as err:
log_exception(err)
return None
Give up? Me too. How about here:
def f2(x):
a = do_something(x)
try:
b = do_something_else(a)
except SomeException as err:
log_exception(err)
return None
else:
c = lastly_do(b)
log_or_whatever(c)
return c
Better, no? Keep the scope of your try
/catch
blocks as narrow as possible. Yes, nested try
/catch
/else
blocks can get annoying, but why not just refactor each of the potential culprits into separate functions?
Don't over-generalize
This should go without saying—by me, because your tooling (pep8
/flake8
/pylint
, etc.) should be saying it at you instead. But I'm saying it anyway, because it bears repeating (and because I've seen—and written!—far too many noqa
/pylint: disable=bare-except
comments.)
A common excuse I've seen for catch Exception
is ignorance—there's a 3rd-party API with bad documentation, or some complicated code with multiple points of failure. Unfortunately, as with the law, ignorance of potential exceptions is not a valid excuse. Do as much research as you can, add logging, and then refactor to catch the actual exceptions you want to catch.
And whatever you do, DO NOT catch BaseException
. Trust me.
Lastly,
Don't be afraid to let it ride
Computers are incredibly complex; the world is a terrifying, unpredictable place; chaos is the only constant. Stuff (as it were) will happen. You don't have to catch
everything; you don't even have to try. Think about what what handling every exception would even mean: if your DB can't be reached, or a config file doesn't exist, or Jeff Bezos decides overnight to get out of cloud storage and into self storage, what's the point of carrying on like your user can go about their business as normal? Five 9s is the goal, but Potemkin-oriented programming serves no-one. Print that TB, return that 500, move on.
Top comments (6)
Thanks for the info. I started learning Python and found this a good source of tips for proper error handling. I have a question:
Why do I need the else? Can I simply write it like this?
technically you don't, in this case, because of the
return
. but if you don't have areturn
you still need theelse
, and i'd argue it's clearer/more informative either way.So I will end up having nested try? As in
?
potentially! but like i said, it might be worth refactoring those cases into separate functions
So
Am I getting there?
yeah, depending on what you're trying to do—if everything's decoupled and
bar
doesn't depend on the success offoo
, then this approach is the way to go