Storing Secrets in AWS

Secrets can be keys, authorization tokens, passwords, and everything else that should be kept private and in this context be represented as a string.

We can store secrets in multiple ways:

• Encrypting environment variables
• Store them in an encrypted file on S3
• Storing secrets in dedicated service

We will look into each of these options shortly but before let's be more specific on "Storing secrets in dedicated service". There are numerous products available to store secrets, i.e. HashiCorp's Vault, or strongDM to name a few, however, here we just want to focus on what solutions AWS offers. Therefore, our "dedicated services" include the Secrets Manager and the System Manager Parameter Store.

Encrypting environment variables

AWS Key Management Service (AWS KMS) is an encryption and key management service scaled for the cloud. AWS KMS keys and functionality are used by other AWS services, and you can use them to protect data in your own applications that use AWS.

KMS is the ideal tool to create and manage cryptographic keys which can be used to encrypt and decrypt secrets. KMS is so deeply rooted within AWS that it's used when encrypting files on S3 or anywhere AWS offers encrypted storage. Important to note is that KMS only stores the key used for decrypting/encrypting the secret, not the secret itself. In the context of AWS Lambda, the secret would still be stored within an environment variable and be decrypted during the Lambda runtime adding to your execution time and bill. As KMS is just meant for encrypting and decrypting the secrets, it has no built-in mechanism to rotate secrets. With the secrets themselves, still stored as environment variables, rotating the secrets, you have to encrypt the secret before you store it in an environment variable. You have to repeat this step for every function using the secret and for every encryption key used, in case you use more than one. Also, keep in mind that KMS encrypts the environment variables at rest, meaning when you deploy a Lambda function, and the secrets are "in transit" they will not be encrypted.

Store them in an encrypted file on S3

While this may suffice at first and is very easy to implement, it is hard to keep your secrets secret and not be over-permissive or have other resources, i.e. Lambda functions, access secrets they are not using. I wouldn't recommend it.

Storing secrets in AWS System Manager Parameter Store

Parameter Store, a capability of AWS Systems Manager, provides secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, database strings, Amazon Machine Image (AMI) IDs, and license codes as parameter values. You can store values as plain text or encrypted data. You can reference Systems Manager parameters in your scripts, commands, SSM documents, and configuration and automation workflows by using the unique name that you specified when you created the parameter.

Storing secrets inside Parameter Store as key-value pairs leverage KMS to encrypt the secrets at rest. A parameter is a key-value pair, where the value can be one string, a list of strings, or even a JSON object in string representation. Parameter Store allows to store SecureStrings encrypting them with KMS, but its default is to store plaintext parameter values.

The parameter key should follow a unique structure accepted by all admins of the Parameter Store within your account. The key must start with a slash (/) and could be followed by your department or team name, the service to which the secret belongs, and a parameter name e.g. /acme/serviceA/username. One reason for this is to limit access to parameters by key paths (/acme/serviceA/*). Another one is, that Parameter Store allows us to read all parameters within a hierarchy. With this, we can get all parameters to serviceA by retrieving all parameter keys starting with /acme/serviceA/.

Parameter Store does not charge anything for storing parameters with up to 4 kB size, if you want to store larger parameters you get charged 0.05 USD per parameter and month.

Rolling back secrets in Parameter Store is a piece of cake because Parameter Store provides a history of the parameter where you can not only see the previous value of the secret but also who changed it.

Together with CloudTrail this provides you with full control over your parameter stores. As Parameter Store is not region agnostic, the parameters get stored in a specific region. You can use them from other regions, of course, but if you want to replicate your parameters across regions, there's no built-in way to do so. However, with CloudTrail you could have a Lambda executed every time a parameter got changed.

Storing secrets in AWS Secrets Manager

AWS Secrets Manager helps you to securely encrypt, store, and retrieve credentials for your databases and other services. Instead of hardcoding credentials in your apps, you can make calls to Secrets Manager to retrieve your credentials whenever needed. Secrets Manager helps you protect access to your IT resources and data by enabling you to rotate and manage access to your secrets.

AWS Secrets Manager complies with many international security standards. The service is dedicated to storing all kinds of secrets and supports a default set of supported resources for which secret management is even simpler. Therefore, everything within Secrets Manager is stored encrypted with a KMS key.

You can store secrets as key-value pairs. A secret can consist of multiple key-value pairs. This is especially useful as you get charged 0.40 USD per stored secret and month. When you want to store an API key, username, password, and maybe even the URL, storing them in separate secrets would cost you 1.60 USD, whereas, if you store them together, similar to a JSON object you just pay 0.40 USD. Take care, that you need to parse the object, however.

For the secret's name, it's also recommended to follow some structure, though retrieving secrets hierarchically is not possible like it is in Parameter Store, you can still benefit from a somewhat structured key when setting up IAM permissions.
Like in other AWS services a tag can be helpful in the context of billing, i.e. separating the bill by a specific tag.

Resource permissions are particularly helpful when you want to share secrets across AWS accounts.

Similar to Parameter Store, Secrets Manager keeps the secrets within the AWS region they were created in. With "Replicate Secrets" you have the option to automatically have your secrets copied to other regions as read-only copies. The secret in the origin region is the only one you can edit. Read the getting started guide to learn more.

For your own secret types (API key, username, passwords, ...) Secret Manager does not provide automatic secret rotation. To rotate your secrets in this case, set up a Lambda function dedicated to creating new secrets and invalidating the old ones. Be careful in this Lambda function on what you log, to not accidentally leak your secrets. You can renew your secrets once a day or up to once a year, it's up to your use case.

When setting up Secrets Manager please consider the best practices.

Conclusion

You can mix and match KMS with Secrets Manager or Parameter Store or only use one of the services to keeping the secrets completely safe.
Secrets Manager and Parameter Store are in many parts very similar to one another. Parameter Store will likely be your cheaper solution. Unless you need an automatic rotation of secrets or secret replication across regions and fancy convenience, Parameter Store will likely be your preferred solution. It does not have to be one or the other as Secrets Manager and Parameter Store can be used side by side. Make sure to take advantage of CloudTrail and set up carefully drafted IAM roles for your users.

In the next part of the series, we look into how we can use these services to secure secrets when using AWS Lambda.

References

• https://www.manning.com/livevideo/production-ready-serverless
• https://seanjziegler.com/a-complete-guide-to-using-the-aws-systems-manager-parameter-store
• https://dhwaneetbhatt.com/blog/using-aws-parameters-store
• https://openupthecloud.com/kms-aws-lambda/
• https://medium.com/@ZeroDeth/aws-secrets-manager-vs-parameter-store-606d4ee2af88
• https://orca.security/aws-lambda-secrets/
• https://serverless-stack.com/chapters/storing-secrets-in-serverless-apps.html