Doing much better than your .env file

Six months ago, I advocated for everyone to stop using .env files in favor of approaches using secret management platforms. Since then, I've learned and experienced much more about how to manage environment variables effectively. In this article, I argue for why we should reduce the practice of storing all your application environment variables in a .env file and move to a more sophisticated approach for managing them in local development.

Before you roast me, I want to make clear that my position is not that you have to ditch the .env file entirely. I'm asserting that your application's environment variables don't need to be stored directly in a .env file. This doesn't mean, however, that you can't store a token in it that pulls in the rest of your environment variables at runtime.

I also, by the way, want to make clear that this article is intended for software development teams and not security and devops teams that already have this figured out; this article is also not intended for solo developers who can feel free to use .env files.

Don't fix what's not broken

In the beginning, developers hardcoded environment variables into their codebase. After realizing that hardcoding them into source control was suboptimal, we introduced .env files for separation of concerns that is to split sensitive data from the rest of code.

In practice, we'd create a .env file, add environment variables to it, and .gitignore the file. We'd start up our applications and read the environment variables into them in local development. Unfortunately, we still teach developers this as a staple of software development and proponents tend to say "don't fix what's not broken."

If you're one of these stubborn proponents, then stop reading; if you're open to improvement and breaking tradition, go on.

Developers have unmet needs

At its core, .env files are broken and we've all experienced it to some degree depending on the scale of our applications and infrastructure; folks deep in the security and devops department know it. In order to understand my argument for why the current usage of .env files is broken, it's important to unlearn tradition and break the topic of managing environment variables locally down to first principles. Let's explore what developers need:

Reliability: We want to make sure that developers have access to the right environment variables at all times. Oftentimes, developers introduce new environment variables and forget to update other developers on the team about the new values. As a result, developers start up their applications and … crash.

Security: We want to keep our environment variables safe and out of source control. While we want to keep bad actors away from our environment variables, since this article focuses on .env files that are typically used in development and contain less critical environment variables, I make this my second point.

Note that my points above generalize across the entire development cycle from development to production and these needs grow with the scale and complexity of application infrastructure. In any case, we seek reliability and security when dealing with environment variables. We want the right environment variables in the right hands and places safely.

Okay so what's wrong with .env files?

While .env files prevent developers from hardcoding and committing environment variables to source control, they fail the reliability criteria.

In a two person team, it's easy to maintain .env files because you don't have to share environment variables with many other developers. As your team size grows, however, you start to run into the secret sprawl problem that I mentioned earlier and it becomes unwieldly to manually share environment variables.

Take for example a microservices architecture where now you have many services, each demanding their own set of environment variables. Here, developers frequently face a phenomenon known as secret sprawl which is when environment variables get scattered across infrastructure and it all becomes unwieldy to manage.

Developers need to centralize their environment variable management

Naturally, the solution is to store environment variables in one centralized location and have developers fetch them from that place back to their applications. It turns out that centralization is indeed what bigger teams do but even then there is still variation. Sometimes, teams store environment variables in password managers; other times, teams encrypt them and store them directly in a private repository on GitHub. 

These methods work but they typically present an extra step of friction that is having to manually grab or trigger refetching environment variables before starting up an application; you could build a custom script that does the job of pulling secrets and injecting them into an application but that in itself presents a step of friction for you.

What developers need is a frictionless tool that automatically pulls and injects environment variables into an application. For this, smarter teams use a dedicated tool that's called a secret manager, built to manage environment variables. These solutions like Infisical, Vault, AWS Parameter Store, Google Secret Manager help securely store environment variables and deliver them back to your infrastructure reliably. When set up correctly, these tools can serve environment variables efficiently across any complex microservices infrastructure from local development through CI/CD pipelines and to production.

Most developers don't know better

Unfortunately, beyond the security and devops crowd, most developers have never heard of secret managers and the proof is in the number of YouTube videos and views for the topic of secret management versus anything else. Let's face it, secret management isn't considered to be a sexy topic the way artificial intelligence and ChatGPT is.

The reason secret managers aren't popular is because they are unwieldy. That's right, the tools built to solve the secret sprawl problem I outlined earlier have histoically introduced new problems with undesirable characteristics. On one hand, there is difficulty in getting started; no one wants to watch a course and read through tens of pages of documentation just to learn the basic concepts of how store and fetch back environment variables from a secret manager. On the other hand, the easy tools are closed-source and, well, how can we trust them?

Okay, I know that I'm going to get a lot of backlash now from die-hard Vault fans but let's be real here. If any secret management tool was as simple as installing the dotenv package, importing it, and starting up your app, then we'd be seeing mass adoption of secret managers everywhere.

But we're not.

Instead, we're stuck with 30M+ downloads per week of the dotenv package for Node.js alone and a sad statistic of 10% of organizations worldwide having adopted secret managers.

Developers would love secret managers, if only they were easy to use and trustworthy

At the beginning of this article, I made clear that my position is not that you have to ditch the .env file entirely. I'm asserting that your application's environment variables should not be stored directly in a .env file.

While I have many tips and strategies for managing environment variables, the major one I'll allude to for this article is a hybrid approach that is to use a secret manager in conjunction with a .env file.

Instead of storing your application environment variables directly in a .env file, you can store them in a dedicated secret management platform, preferably one that's easy to use with little overhead. Then, store an access token in the .env file that your application can use to fetch environment variables from the secret manager at runtime. That's a simple adjustment you can make to improve the reliability of receiving the right set of environment variables at all times across your team.

That's enough, show me what you're suggesting, gimme the code

To keep things simple, I'm going to use Infisical, an open-source, end-to-end encrypted secret management platform that you can store environment variables with. You can also use another secret manager like Vault or AWS Parameter Store that each have their own SDKs that you can use to achieve the recommendation.

I'm picking Infisical because it's self-hostable on your own infrastructure, well-documented, and beautiful. It's also worth noting that it's self-hostable on Fly.io and Render as well as AWS, Kubernetes etc.

Anyways, we'll be using the Node SDK to fetch back environment variables at runtime with this example, though there are other SDKs and methods available.

Getting started

Before we can fetch environment variables back into your Node application, you need to add them to a project in Infisical Cloud or in a self-hosted instance of Infisical.

Okay, let's get started.

First, install the infisical-node package in your project:

$ npm install infisical-node --save

Next, import the SDK and create a client instance with your Infisical Token:

import InfisicalClient from "infisical-node";

const client = new InfisicalClient({
  token: "your_infisical_token"
});

To ensure optimal performance, I'd recommend creating a single instance of the Infisical client and exporting it to be used across your entire app. The reason is because the Node SDK caches every secret and updates it periodically, reducing excessive calls; this built-in caching makes syncing environment variables seamless at scale.

I'd also recommend storing the Infisical Token in a .env file in local development or as the only environment variable in production. This way, you don't have to hardcode it into your application and can use it to fetch the rest of your environment variables.

Now you can use the client to fetch secrets for your application on demand:

app.get("/", async (req, res) => {
  const name = await client.getSecret("NAME");
  res.send(`Hello! My name is: ${name.secretValue}`);
});

That's it!

Now whenever your application needs an environment variable, it can request it from Infisical on demand. You're now able to view all the environment variables for your Node application from one central place and avoid any missing environment variables.

How do I trust a secret manager?

Adopting a secret manager can be daunting at first and rightfully so. You have to trust that the solution you're working with is secure, that the vendor won't view or tamper with your data, and that the underlying infrastructure will work and deliver environment variables reliably when needed. In many ways, it does feel easier to store environment variables in a decentralized fashion or perhaps centralized but encrypted in a private repository on GitHub.

There is, however, a right tool for every task and using dedicated tooling to manage environment variables is surely better than any manual process that's either not efficient or prone to error; this is especially true in cases where your infrastructure is sprawled out.

The analogy I like to use is password managers like 1Password and Bitwarden. Before using one, I was content with managing dozens of passwords manually and occasionally forgetting them. The minute I started to have many different passwords across different services, each satisfying a different requirement, I faced a password sprawl that compelled me to sign up for a password manager because I couldn't keep hitting the "forget my password" button.

The same is true for secret managers. When you're a solo developer, you can use manual approaches to store your environment variables. The minute you start to scale above five engineers, you should start thinking about maintaining your secret management hygiene. While there is no 100% guarantee for the reliability of any process, you can get close to 99.99% and my suggestions for selecting a secret manager are as follows:

• Pick a solution that's either open source or made by the big cloud providers; it will be more resilient that way and you can inspect and, in the open source case, make changes to the code if needed.

• Pick something that provides a good developer experience for the size of your team. If you have site reliability engineers and teams for process, you can afford to go with more complex tools. Otherwise, if you're a smaller team, go frictionless and focus on building the rest of your application.

Conclusion

For the longest time, developers defaulted to storing all their environment variables in a .env file largely because it was the most frictionless way to get up and running locally. Alternatives like other secret management platforms were too cumbersome to adopt and never became mainstream because they were too difficult to use/learn and intended for the security and devops crowd.

In this article, however, I make clear that the approach of storing all of your environment variables in a .env file lacks reliability when it comes to keeping your team's environment variables in sync. By using a dedicated secret manager, preferably one with little overhead, and instead keeping a single token in your .env file that your application can consume and use to fetch the rest of your environment variables, you can preserve the security benefits of separating sensitive data from your codebase whilst ensuring reliability of environment variables across your infrastructure.