Using Environment Variables in Python for App Configuration and Secrets

Learn how experienced developers use environment variables in Python, including managing default values and typecasting.

As a developer, you’ve likely used environment variables in the command line or shell scripts, but have you used them as a way of configuring your Python applications?

This guide will show you all the code necessary for getting, setting, and loading environment variables in Python, including how to use them for supplying application config and secrets.

Not familiar with environment variables? Check out our ultimate guide for using environment variables in Linux and Mac.

Why use environment variables for configuring Python applications?

Before digging into how to use environment variables in Python, it's important to understand why they're arguably the best way to configure applications. The main benefits are:

• Deploy your application in any environment without code changes
• Ensures secrets such as API keys are not leaked into source code

Environment variables have the additional benefit of abstracting from your application how config and secrets are supplied.

Finally, environment variables enable your application to run anywhere, whether it's for local development on macOS, a container in a Kubernetes Pod, or platforms such as Heroku or Vercel.

Here are some examples of using environment variables to configure a Python script or application:

• Set FLASK_ENV environment variable to "development" to enable debug mode for a Flask application
• Provide the STRIPE_API_KEY environment variable for an Ecommerce site
• Supply the DISCORD_TOKEN environment variable to a Discord bot app so it can join a server
• Set environment specific database variables such as DB_USER and DB_PASSWORD so database credentials are not hard-coded

How are environment variables in Python populated?

When a Python process is created, the available environment variables populate the os.environ object which acts like a Python dictionary. This means that:

• Any environment variable modifications made after the Python process was created will not be reflected in the Python process.
• Any environment variable changes made in Python do not affect environment variables in the parent process.

Now that you know how environment variables in Python are populated, let's look at how to access them.

How to get a Python environment variable

Environment variables in Python are accessed using the os.environ object.

The os.environ object seems like a dictionary but is different as values may only be strings, plus it's not serializable to JSON.

You've got a few options when it comes to referencing the os.environ object:

# 1. Standard way
import os
# os.environ['VAR_NAME']

# 2. Import just the environ object
from os import environ
# environ['VAR_NAME']

# 3. Rename the `environ` to env object for more concise code
from os import environ as env
# env['VAR_NAME']

I personally prefer version 3 as it's more succinct, but will stick to using os.environ for this article.

Accessing a specific environment variable in Python can be done in one of three ways, depending upon what should happen if an environment variable does not exist.

Let's explore with some examples.

Option 1: Required with no default value

If your app should crash when an environment variable is not set, then access it directly:

print(os.environ['HOME']
# >> '/home/dev'

print(os.environ['DOES_NOT_EXIST']
# >> Will raise a KeyError exception

For example, an application should fail to start if a required environment variable is not set, and a default value can't be provided, e.g. a database password.

If instead of the default KeyError exception being raised (which doesn't communicate why your app failed to start), you could capture the exception and print out a helpful message:

import os
import sys

# Ensure all required environment variables are set
try:  
    os.environ['API_KEY']
except KeyError: 
    print('[error]: `API_KEY` environment variable required')
    sys.exit(1)

Option 2: Required with default value

You can have a default value returned if an environment variable doesn't exist by using the os.environ.get method and supplying the default value as the second parameter:

# If HOSTNAME doesn't exist, presume local development and return localhost
print(os.environ.get('HOSTNAME', 'localhost')

If the variable doesn't exist and you use os.environ.get without a default value, None is returned

assert os.environ.get('NO_VAR_EXISTS') == None

Option 3: Conditional logic if value exists

You may need to check if an environment variable exists, but don't necessarily care about its value. For example, your application can be put in a "Debug mode" if the DEBUG environment variable is set.

You can check for just the existence of an environment variable:

if 'DEBUG' in os.environ:
    print('[info]: app is running in debug mode')

Or check to see it matches a specific value:

if os.environ.get('DEBUG') == 'True':
    print('[info]: app is running in debug mode')

How to set a Python environment variable

Setting an environment variable in Python is the same as setting a key on a dictionary:

os.environ['TESTING'] = 'true'

What makes os.environ different to a standard dictionary, is that only string values are allowed:

os.environ['TESTING'] = True
# >> TypeError: str expected, not bool

In most cases, your application will only need to get environment variables, but there are use cases for setting them as well.

For example, constructing a DB_URL environment variable on application start-up using DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, and DB_NAME environment variables:

os.environ['DB_URL'] = 'psql://{user}:{password}@{host}:{port}/{name}'.format(
    user=os.environ['DB_USER'],
    password=os.environ['DB_PASSWORD'],
    host=os.environ['DB_HOST'],
    port=os.environ['DB_PORT'],
    name=os.environ['DB_NAME']
)

Another example is setting a variable to a default value based on the value of another variable:

# Set DEBUG and TESTING to 'True' if ENV is 'development'
if os.environ.get('ENV') == 'development':
    os.environ.setdefault('DEBUG', 'True') # Only set to True if DEBUG not set
    os.environ.setdefault('TESTING', 'True') # Only set to True if TESTING not set

How to delete a Python environment variable

If you need to delete a Python environment variable, use the os.environ.pop function:

os.environ.pop['MY_KEY']

To extend our DB_URL example above, you may want to delete the other DB_ prefixed fields to ensure the only way the app can connect to the database is via DB_URL:

Another example is deleting an environment variable once it is no longer needed:

auth_api(os.environ['API_KEY']) # Use API_KEY
os.environ.pop('API_KEY') # Delete API_KEY as it's no longer needed

How to list Python environment variables

To view all environment variables:

print(os.environ)

The output of this command is difficult to read though because it's printed as one huge dictionary.

A better way, is to create a convenience function that converts os.environ to an actual dictionary so we can serialize it to JSON for pretty-printing:

import os
import json

def print_env():
    print(json.dumps({**{}, **os.environ}, indent=2))

Why default values for environment variables should be avoided

You might be surprised to learn it's best to avoid providing default values as much as possible. Why?

Default values can make debugging a misconfigured application more difficult, as the final config values will likely be a combination of hard-coded default values and environment variables.

Relying purely on environment variables (or as much as possible) means you have a single source of truth for how your application was configured, making troubleshooting easier.

Using a .env file for Python environment variables

As an application grows in size and complexity, so does the number of environment variables.

Many projects experience growing pains when using environment variables for app config and secrets because there is no clear and consistent strategy for how to manage them, particularly when deploying to multiple environments.

A simple (but not easily scalable) solution is to use a .env file to contain all of the  variables for a specific environment.

Then you would use a Python library such as python-dotenv to parse the .env file and populate the os.environ object.

To follow along, create and activate a new virtual environment, then install the python-dotenv library:

python3 -m venv ~/.virtualenvs/doppler-tutorial      # 1. Create
source ~/.virtualenvs/doppler-tutorial/bin/activate  # 2. Activate
pip install python-dotenv                            # 3. Install dotenv

Now save the below to a file named .env (note how it's the same syntax for setting a variable in the shell):

API_KEY="357A70FF-BFAA-4C6A-8289-9831DDFB2D3D"
HOSTNAME="0.0.0.0"
PORT="8080"

Then save the following to dotenv-test.py:

# Rename `os.environ` to `env` for nicer code
from os import environ as env

from dotenv import load_dotenv
load_dotenv()

print('API_KEY:  {}'.format(env['API_KEY']))
print('HOSTNAME: {}'.format(env['HOSTNAME']))
print('PORT:     {}'.format(env['PORT']))

Then run dotenv-test.py to test the environment variables are being populated:

python3 dotenv-test.py
# >> API_KEY:  357A70FF-BFAA-4C6A-8289-9831DDFB2D3D
# >> HOSTNAME: 0.0.0.0
# >> PORT:     8080

While .env files are simple and easy to work with at the beginning, they also cause a new set of problems such as:

• How to keep .env files in-sync for every developer in their local environment?
• If there is an outage due to misconfiguration, accessing the container or VM directly in order to view the contents of the .env may be required for troubleshooting.
• How do you generate a .env file for a CI/CD job such as GitHub Actions without committing the .env file to the repository?
• If a mix of environment variables and a .env file is used, the only way to determine the final configuration values could be by introspecting the application.
• Onboarding a developer by sharing an unencrypted .env file with potentially sensitive data in a chat application such as Slack could pose security issues.

These are just some of the reasons why we recommend moving away from .env files and using something like Doppler instead.

Doppler provides an access controlled dashboard to manage environment variables for every environment with an easy to use CLI for accessing config and secrets that works for every language, framework, and platform.

Centralize application config using a Python data structure

Creating a config specific data structure abstracts away how the config values are set, what fields have default values (if any), and provides a single interface for accessing config values instead of os.environ being littered throughout your codebase.

Below is a reasonably full-featured solution that supports:

• Required fields
• Optional fields with defaults
• Type checking and typecasting

To try it out, save this code to config.py:

import os
from typing import get_type_hints, Union
from dotenv import load_dotenv

load_dotenv()

class AppConfigError(Exception):
    pass

def _parse_bool(val: Union[str, bool]) -> bool:  # pylint: disable=E1136 
    return val if type(val) == bool else val.lower() in ['true', 'yes', '1']

# AppConfig class with required fields, default values, type checking, and typecasting for int and bool values
class AppConfig:
    DEBUG: bool = False
    ENV: str = 'production'
    API_KEY: str
    HOSTNAME: str
    PORT: int

    """
    Map environment variables to class fields according to these rules:
        - Field won't be parsed unless it has a type annotation
        - Field will be skipped if not in all caps
        - Class field and environment variable name are the same
    """
    def __init__(self, env):
        for field in self.__annotations__:
            if not field.isupper():
                continue

            # Raise AppConfigError if required field not supplied
            default_value = getattr(self, field, None)
            if default_value is None and env.get(field) is None:
                raise AppConfigError('The {} field is required'.format(field))

            # Cast env var value to expected type and raise AppConfigError on failure
            try:
                var_type = get_type_hints(AppConfig)[field]
                if var_type == bool:
                    value = _parse_bool(env.get(field, default_value))
                else:
                    value = var_type(env.get(field, default_value))

                self.__setattr__(field, value)
            except ValueError:
                raise AppConfigError('Unable to cast value of "{}" to type "{}" for "{}" field'.format(
                    env[field],
                    var_type,
                    field
                )
            )

    def __repr__(self):
        return str(self.__dict__)

# Expose Config object for app to import
Config = AppConfig(os.environ)

The Config object exposed in config.py is then used by app.py below:

from config import Config

print('ENV:      {}'.format(Config.ENV))
print('DEBUG:    {}'.format(Config.DEBUG))
print('API_KEY:  {}'.format(Config.API_KEY))
print('HOSTNAME: {}'.format(Config.HOSTNAME))
print('PORT:     {}'.format(Config.PORT))

Make sure you have the .env file still saved from earlier, then run:

python3 app.py 
# >> ENV:      production
# >> DEBUG:    False
# >> API_KEY:  357A70FF-BFAA-4C6A-8289-9831DDFB2D3D
# >> HOSTNAME: 0.0.0.0
# >> PORT:     8080

You can view this code on GitHub and if you're after a more full-featured typesafe config solution, then check out the excellent Pydantic library.

Summary

Awesome work! Now you know how to use environment variables in Python for application config and secrets.

Although we're a bit biased, we encourage you to try using Doppler as the source of truth for your application environment variables, and it's free to get started with our Community plan (unlimited projects and secrets).

We also have a tutorial for building a random Mandalorion GIF generator that puts into practice the techniques shown in this article.

Hope you enjoyed the post and if you have any questions or feedback, we'd love to chat with you over in our Community forum.

Big thanks to Stevoisiak, Olivier Pilotte, Jacob Kasner, and Alex Hall for their input and review!