While .env files are still commonly used and were an improvement upon storing secrets in source code, the security risks and impact on developer productivity are only now being fully realized.
To understand what’s driving the next evolutionary shift in secrets management, let's first revisit the history of environment variables and why .env files were a good idea in theory, but created more problems than they solved.
From Hard-Coded Secrets to Environment Variables
The Twelve-Factor App methodology was instrumental in defining modern best practices for developing and deploying applications. Arguably the most important “factor”, was replacing hard-coded config and secrets from source code with dynamic values loaded from environment variables.
The usage of environment variables was a huge win for security as previously, anyone that had access to the code could view the secrets as well. Because environment variables are supported in every operating system and programming language, no special code or third-party libraries were needed to access them.
"A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials." - Twelve-Factor App III - Config
In addition, because configuration was now supplied dynamically, applications could be run in any environment, from local development through to production with no code changes necessary.
While the security and deployment benefits of using environment variables was clear, something was missing—How would environment variables be managed, created, updated, removed, and synced between environments? Who would be responsible for managing them? And what was the source of truth should an application be misconfigured?
Let's delve into the history of environment variable management to how .env files tried to answer these questions.
How .env Files Were Born
The Twelve-Factor App was originally drafted by developers at Heroku in 2012 who built a feature called Config Vars into their application dashboard. These Key/Value pairs were then injected during deployment as environment variables, enabling any language supported by Heroku to consume them.
This proved environment variables were the ideal delivery format for app config and secrets with a simple and straightforward UI for editing. Heroku even added the feature of automatically reloading the application whenever the Config Vars changed.
So Heroku had solved this problem for their customers, but what about the multitude of other hosting providers and platforms?
Shell scripts could be written to set an application’s environment variables during deployment before the app runs, but this approach is application and platform specific, with different shell syntax required for Linux and Windows.
That’s where .env files come in. A plain text file of key/value pairs that is actually just valid Linux shell syntax for defining variables.
DATABASE_URL=postgres://user:pass@db.host:5432/app_db
AUTH_API_KEY=adb54299-f8dc-4726-bd3f-9fe2726bfd7a
This simple format (restricted to be valid Linux shell syntax) enabled .env (dotenv) file parsing libraries to be created for every major programming language such as Python dotenv and Node dotenv.
An application would then use the dotenv library to parse the contents of a .env file and automatically populate the environment variables object for that programming language prior to the application starting up.
The .env file format seemed to have it all!
It provided the syntax, packaging, and a cross-platform solution with a similar conceptual model to Heroku’s Config Vars, just in a plain-text file on a server. And thanks to the open source community, dotenv packages for every major language became readily available.
The usage of .env files is what made it possible for every language and framework to adopt the best practice of using environment variables for application config and secrets. It seemed like a perfect solution.
Unfortunately, .env files created a new set of security and developer productivity issues that only recently, have started to become well publicized.
Let’s now explore why .env files weren’t the silver bullet we all wanted them to be.
The Problem With .env Files
While they were a step up from hard-coded secrets, the growing number of security risks and issues related to the manual management of .env files across different environments and cloud providers are causing engineering leaders to take notice.
Here is just a sample of the challenges teams face when using .env files:
- Sharing of unencrypted secrets in .env files over Slack when new secrets are added or changed breaks the principle of least privilege if secrets are exposed to users unauthorized to view them.
- Local development environments constantly break when required updates to an .env file aren't proactively communicated, e.g. a new secret required after merging a pull request.
- The human element in manually managing .env files across environments and different cloud providers can easily lead to typos and misconfiguration errors, posing a risk to production stability.
- The inconsistent environment variable syntax between languages and platforms can easily cause issues, e.g. Docker and GitHub require values to be unquoted while Python and Node.js dotenv packages work with quoted or unquoted values. The issue is so prevalent that many teams have been forced to adopt an .env file linter to combat these syntactical issues.
- As .env files are stored in plain-text (not just environment variables in memory), they are at risk of being read by unauthorized users with no audit trail in terms of access and changes made.
Not to mention there is a constant stream of bots scanning the web for accidentally exposed .env files in public webroot folders and S3 buckets.
It's clear .env files have serious implications for application security, but what about their impact on development velocity?
The constant papercuts experienced from manually managing .env files has for most teams become so normalized, it’s no longer seen as an issue. Just a given cost of practices associated with modern application development.
Teams that fail to quantify the costs of managing .env files are losing valuable time and money that could be saved if a centralized secrets manager was employed to automate secrets syncing across environments.
Interruptions caused by .env file related issues also have the significant and harder to quantify cost of breaking the concentration of a developer in a flow or deep work state. Assuming the research estimating an average of 23 minutes is lost per serious interruption is correct, the time and momentum lost due to .env file related interruptions is more serious than previously thought.
High performing teams are beginning to realize the true cost of relying on .env files for managing application config and secrets.
They're looking to a new breed of secret management platforms, designed to bring automation to the management of environment variables for every cloud, platform, and hosting environment.
We Need a Secrets Manager for Environment Variables
We’ll always have .env files to thank for bringing the use of environment variables for application config and secrets to the mainstream. It was a simple and straightforward solution and solved the immediate problem of how to move hard-coded secrets out of source code and into a universal format that works for every language.
While they may have served us well in the past, it’s time to acknowledge the serious shortcomings of .env files in order to embrace the next evolutionary step: Using a secrets manager to provide centralized and secure storage for secrets.
That’s why we built Doppler—Removing the need for .env files by providing a dashboard for managing environment variables that works with every cloud provider and platform.
Top comments (0)