How do you store private keys?

My projects are based on Google Cloud Platform and over the years I've used two different approaches:

1. Encrypt .env files and store them in the git repo. Manage encryption keys via Cloud KMS. This approach has an advantage that secrets are stored with the code and it's easy to distribute them.
2. Use Secret Manager to store arbitrary (.env, .json) configurations and pull those directly from google. This way it's easier to rotate the configurations and it provides audit logs for secrets access.

You can use dotenv and encrypt the file .env by git-crypt or git-secret. There are other similar tools. Or you can encrypt just some file secrets.env and during the deploy process append it to the .env file.

I work with Serverless framework mainly, but it can apply to any serverless functions. Since I mostly use AWS, I store my secrets on Secret Manager. I use the same key (but different value) for both staging and production and when I deploy to any stage it will go look for the secret using that key regardless of the environment and retrieve the corresponding value. Note: each stage is a seperate AWS account, if you have different stages in the same account/region, then you might need to have different secret keys e.g. private_key_dev and private_key_production

Example:

Go to Secrets manager and create a new secret in the UI (or cli).. and call this secret "my_app_secrets". The object in this secret is key,value type of object. e.g
private_key: "secret_value"

And then in serverless.yml

custom:
   myAppSecrets: ${ssm:/aws/reference/secretsmanager/my_app_secrets~true}

functions:
 myFunctionWithASecret:
   handler: src/handlers.myFunction
   environment: 
     privateKey: ${self:custom.myAppSecrets.private_key}

There are also other ways for example if you are using a CI/CD tool like Jenkins you can use the Jenkins Credentials to store the secrets there especially for secrets to deploy the apps to the services.

I usually find a package to encrypt my files and then decrypt them before deploying
For python based apps i use pyAesCrypt for nodejs you don't need anything the crypto package already available with nodejs comes to rescue.

Django Based Example
NodeJS Express API Example

How is everyone working with keys in a shared development environment? For example we have a blog page that pulls data from Contentful but we don't want to share the keys with someone that is contributing and may want to work on the blog page for our OSS. Otherwise keys are on Netlify at build time.

I have different approaches depending on the platform where I deploy. If I have control over the server (i.e. if I'm using Digital Ocean) I have one (or multiple) .env files that I use to store private information. I have a local version of the files, such as these ones that can be used as a reference for others even though they are not comprehensive, and a production copy, which I store on my machine+backed up, but never on VCS. I upload the production files to the server 'manually'. Those files are sourced so that the variables become environment variables within the context of the app. This works equally well either using Docker or directly on a server.

I have used something like dotenv in some projects, however having environmental variables defined allowed me to work directly connected to the server (i.e. perform maintenance tasks, data dumps, etc.) replicating the same conditions that the app is operating under. Therefore the chances of running a script, or triggering a batch of e-mail messages, using the wrong access keys is very low.

If, however, an internal process requires a 3rd party to access private keys, I shared them encrypted. It is important to share only the strictly necessary keys. So, for example, in this repo, Travis can decrypt a key that is later used to push some changes to a server. This puts a lot of trust on Travis security, and that is why I believe limiting the scope of such approach is important. If there's ever a breach at Travis (after all, it becomes a very attractive target), there'll be not enough reaction time to limit revoke access to those keys.

Finally, if I'm deploying on something like Heroku, I only use environment variables, so no files ever leave my computer.

There's the other aspect, however, on how to store and backup keys in your dev machine, so that you can guarantee access to the server even after your laptop gets destroyed, but that is not the point of your question, I believe.

I am using AWS Secrets Manager (AWS Systems Manager) for some systems.
I have also used KMS encryption keys for storing them in S3 for systems running on ECS which can decrypt them with IAM Role access to the KMS keys.

Excellent question!
And some great responses too.

I am using Azure Keyvault to store secrets and the managed identity of the web application to regulate access to the vault.
Im memory I keep them as short as possible and then in a SecureString.

Never lose your Docker trust keys! I use Gitlab CI to build Docker images and sign them with Harbor Notary, which is the private image registry too.

Everything's automated to integrate with Vault's API to fetch and store secrets, and also encrypted files like a TOFU signature. TOFU, or "Trust on First Use" is the underlying framework that the docker api uses to establish authenticity of images, just like signing a commit but with a whole set of challenges like maintaining your own keyserver. This is why I programmatically keep the signatures in Vault and utilize some for loops to download, sign and upload to the key value store. I could go on, but hopefully you get the idea.

I use a lot of managed AWS services for this. I wish DigitalOcean had a secret manager service, too, I use them for all of my personal stuff.

@madza i do feel questions you ask are good but I also do feel that more focus should be on writing articles than asking questions around on daily basis which might overshadow someone's article.

This would make a great "Ask HN"