Upcoming Python Features Brought to You by Python Enhancement Proposals

Before any new feature, change or improvement makes it into Python, there needs to be a Python Enhancement Proposal, also knows as PEP, outlining the proposed change. These PEPs are a great way of getting the freshest info about what might be included in the upcoming Python releases. So, in this article we will go over all the proposals that are going to bring some exciting new Python features in a near future!

Syntax Changes

All these proposals can be split into a couple of categories, first of them being syntax change proposals which are bound to bring interesting features.

First in this category is PEP 671 which proposes syntax for late-bound function argument defaults. What does that mean, though?

Well, functions in Python can take other functions as arguments. There's however no good way to set default value for such arguments. Usually None or sentinel value (global constant) is used as default which has disadvantages, including inability to use help(function) on the arguments. This PEP describes new syntax using => (param=>func()) notation to specify functions as default parameters.

# Current solution:
_SENTINEL = object()

def func(param=_SENTINEL):
    if param is _SENTINEL:
        # default_param holds expected default value
        param = default_param

# New solution:
def func(param=>default_param):
    ...

This change looks reasonable and useful to me, but I think we should be cautious of adding too many new syntax notations/changes. It's questionable whether small improvement like this one warrants yet another assignment operator.

Another syntax change proposal is PEP 654, which proposes except* as a new syntax for raising groups of exceptions. The rationale for this one is that Python interpreter can only propagate one exception at the time, but sometimes multiple unrelated exceptions need to be propagated as the stack unwinds. One such case is concurrent errors from asyncio from concurrent tasks or multiple different exceptions raised when performing retry logic, e.g., when connecting to some remote host.

try:
    some_file_operation()
except* OSError as eg:
    for e in eg.exceptions:
        print(type(e).__name__)

# FileNotFoundError
# FileExistsError
# IsADirectoryError
# PermissionError
# ...

This is a very simple example of using this new feature. If you take a look at the examples of handling Exceptions Groups in the PEP you will find a lot of wild ways to use this, including recursive matching and chaining.

Typing

Next category - which has been very heavily present in recent Python releases - is typing/type annotations.

Let's start with PEP 673 - which doesn't require extensive knowledge of Python's typing module (as is usually the case). Let's explain it with an example: Let's say you have a class Person with method set_name, which returns self - that being - instance of type Person. If you then create subclass Employee with same set_name method, you'd expect it to return instance of type Employee rather than Person. That's however not how type checking currently works - in Python 3.10 type checker infers return type in subclass to be that of base class. This PEP fixes that, by allowing us to use the "Self Type" with the following syntax that will help type checker to infer types correctly:

class Person:
    def set_name(self, name: str) -> Self:
        self.name = name
        return self

If you've run into this issue, then you can look forward to using this feature fairly soon, because this PEP is accepted and will be implemented as part of Python 3.11 release.

Another typing change is presented in PEP 675 which is titled Arbitrary Literal Strings.

Introduction of this PEP stems from the fact that it's currently not possible to specify that type of function argument can be an arbitrary literal string (only specific literal string can be used, e.g. Literal["foo"]). You might be wondering why is this even a problem and why would someone need to specify that parameter should be literal string and specifically not f-string (or other interpolated string). It's mostly a matter of security - requiring that parameter is literal helps avoid injections attacks, whether it's SQL/command injection or for example XSS. Some examples of these are shown in PEP's appendix. Having this implemented would help libraries like sqlite to provide warnings to users when string interpolation is used where it shouldn't be, so let's hope this one gets accepted soon.

Debugging

Next up are PEPs that help us debug our code a bit more efficiently. Starting off with PEP 669, titled Low Impact Monitoring for CPython.This PEP proposes to implement low cost monitoring for CPython that would not impact performance of Python programs when running debuggers or profilers. This is not something that will greatly impact end-users of Python, considering that there aren't big performance penalties when doing basic debugging. This can however be very useful in some niche cases, such as:

• Debugging an issue that can only be reproduced in production environment while not impacting application performance.
• Debugging race conditions where timing can affect whether the issue will occur or not.
• Debugging while running a benchmark.

I personally probably won't benefit that much from this, but I'm sure people who are trying to improve performance of Python itself would definitely appreciate this change, as it would make it easier to debug and benchmark performance issues/improvements.

Next one is PEP 678 which suggests that __note__ attribute should be added to BaseException class. This attribute would be used to hold additional debugging information which could be displayed as part of traceback.

try:
    raise TypeError('Some error')
except Exception as e:
    e.__note__ = 'Extra information'
    raise

# Traceback (most recent call last):
#   File "<stdin>", line 2, in <module>
# TypeError: Some error
# Extra information

As shown in the example above, this is especially useful when re-raising exception. As the PEP describes, this would be also useful for libraries that have retry logic, adding extra information for each failed attempt. Similarly, testing libraries could use this to add more context to failed assertions, such as variable names and values.

Final debugging-related proposal is PEP 657 which wants to add additional data to every bytecode instruction of Python programs. This data can be used to generate better traceback information. It also suggests that API should be exposed which would allow other tools such as profilers or static analysis tools to make use of the data.

This might not sound that interesting, but it's actually - in my opinion - the most useful PEP presented here. The big benefit of this PEP will definitely be having nicer traceback information, such as:

Traceback (most recent call last):
  File <example.py>", line 5, in <module>
    value * (a - b * c)
             ~~^~~~~~~
TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'

Traceback (most recent call last):
  File <example.py>", line 10, in <module>
    some_dict['first']['second']['third']
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable

I think this is an amazing improvement to traceback readability and in general to debugging, and I'm really happy that this got implemented as part of Python 3.11, so we will get to use it very soon.

Quality of Life Changes

Final topic is dedicated to PEPs that bring certain "quality of life" improvements. One of those is PEP 680, which proposes to add support for parsing TOML format to Python's standard library.

TOML as a format is by default used by many Python tools, including build tools. This creates a bootstrapping problem for them. Additionally, many popular tools such as flake8 don't include TOML support citing its lack of support in standard library. This PEP proposes to add TOML support to standard library based on tomli which is already used by packages such as pip or pytest.

I personally like this proposal, and I think it makes sense to include libraries for common/popular formats in standard library, especially when they're so important to Python's tooling and ecosystem. The question is then, when will we see YAML support in Python standard library?

Last but not least, is PEP 661, which is related to so-called "sentinel values". There's no standard way of creating such values in Python. It's usually done with _something = object() (common idiom) as shown with PEP 671 earlier. This PEP proposes specification/implementation of sentinel values for standard library:

# Old:
_sentinel = object()

# New:
from sentinels import sentinel
some_name = sentinel("some_name")

Apart from the new solution being more readable, this can also help with type annotations as it will provide distinct type for all sentinels.

Closing Thoughts

From the proposals presented above, it's clear that a lot of great stuff is coming to Python. Not all these features will however make it into Python (at least not in their current state), so make sure to keep an eye on these proposals to see where they are going. To keep up-to-date with above PEPs as well as any new additions, you can take an occasional peek at the index or get notified about every new addition by subscribing to PEP RSS feed.

Additionally, I only included PEPs that propose some new feature/change to the language, there are however other ones that specify best practices, processes or oven Python's release schedule, so make sure to check the above mentioned index, if you're interested in those topics.