Iterable gotcha's in Python

What are Iterables?

Typehints have been introduced in
Python since Python 3.5. As part of this introduction, a new module typing was
introduced which contained various types for built-in collections and Abstract
Base Classes. Iterable is a type that is prominantly used in the PEP for Type hints and is implemented by many of the most used Python collections. Since Python 3.9, this type can be imported from collections.abc and can be used as a type as well as an Abstract Base Class.

When to use Iterables?

If you enthousiastically started using type hints in Python and have been following best practices to accept the most generic type possible in your arguments, you might have been using Iterable. Iterable can be used at any place where your code expects the variable to implement __iter__, in other words, where you code wants to iterate through a collection of some sorts.

Say you want to send an email to multiple receivers:

class MailApi:
    def sendmail(self, sender: str, receiver: str, message: str) -> None:
        ...


api = MailApi()


def send_emails(message: str, sender: str, receivers: Iterable[str]) -> Optional[T]:
    if not receivers:
        raise ValueError("you need atleast one receiver")

    for receiver in receivers:
        api.sendmail(sender, receiver, message)

If you send an email using the following code you will send an email:

send_emails(
    "Hello",
    "me@example.com",
    ["steve@example.com", "larry@example.com", "elon@example.com"],
)

If you use the following code, you will get an exception:

send_emails("Hello", "me@example.com", [])

Pitfalls

But what if you did this?

send_emails(
    "Hello",
    "me@example.com",
    filter(
        lambda x: not x.endswith("example.com"),
        ["steve@example.com", "larry@example.com", "elon@example.com"],
    ),
)

You would intuitively expect an exception, because the filter will remove all matching emails, but Nothing will happen!

filter returns a valid Iterable and MyPy correctly asserts that there are no type errors. What is going on?

The pythonic way, to check if a collection is empty, is to use if not <collection_name. This works because Python returns False when __len__ returns 0. However, Iterable is not required to implement __len__ as well as __bool__. The default truth value is True, so even though your code is perfectly Type safe, it will not do what you might expect. A solution to this problem is not simple, but you could prevent similar bugs by using a slightly less generic type such as Collection, which does require __len__ to be implemented, if you want to catch these errors with a type checker.

Conclusion

The new Python type hints are a welcome addition to the language and boost maintainability, reduce bugs and allow better editor support for Python. However, due to historic design decsicions there are subtle cases which can result in bugs in ways you wouldn't expect from type safe code. If you use Iterable in Python, consider using Collection.