From a security perspective, there is no need to explain why rotating your AWS access keys regularly is "key" ;), regardless of whether you are bound by laws and regulations. In any case, you can read a quick rationale in this AWS post .
On the other hand, if you have worked with AWS IAM for a while, you have probably struggled with access key rotation. This isn't a tough task at all. In fact, the AWS post above shows how easily we can rotate a user's AWS keys from the CLI. No additional complication doing so from the AWS console. As long as we have the required permissions, this is easy peasy. The problem comes when we have to automate this for all IAM users in our organization and share their new access keys in a secure fashion.
There are other solutions out there to solve this problem. From using AWS native services like CloudWatch and Lambda (where you still have the new-key-secure-share issue with the user), to custom scripts that you execute and rotate the access key on the user's machine. The latter meaning that we have to rely on the user to update their keys, something that my experience has taught me to avoid :)
In this post, I will explain how to automate key rotation for IAM users with access keys older than 90 days using GitHub Actions and a python script. I won't get into the weeds about how I have accomplished this for my organization's specific needs, but you'll get the idea ;)
First, like for any GitHub Workflow, we need to define our trigger. In our case, we want the workflow to run daily.
name: Daily AWS access key rotation check
on:
#Every day at 5:55 AM UTC (23:55 CST) '55 5 * * *'
schedule:
- cron: '55 5 * * *'
Then, we will need to define our elevated account's access keys as environment variables, which will be used to rotate every IAM user's access keys on its behalf. I have those variables stored as secrets in GitHub Secrets.
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
And finally, we get to the fun part of the workflow, where we write the required steps to enable the GitHub MacOS runner to execute our python script smoothly. We need to checkout our code so the runner can access it, inject the secrets into our script, and install python and boto3.
jobs:
AWS-key-rotation:
name: Quarterly AWS access key rotation
runs-on: macos-latest
steps:
- name: Checkout Code
uses: actions/checkout@v2
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.DEVOPS_TF_AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.DEVOPS_TF_AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- uses: actions/setup-python@v2
with:
python-version: 3
- name: Install boto3
run: |
pip install boto3
- name: Secret Injection MAILBOX credentials
run: |
sed -i "" 's/MAILBOX_EMAIL/${{ secrets.MAILBOX_EMAIL }}/' scripts/rotate-aws-keys.py
sed -i "" 's/MAILBOX_PASSWORD/${{ secrets.MAILBOX_PASSWORD }}/' scripts/rotate-aws-keys.py
cat scripts/rotate-aws-keys.py
- name: Run AWS key rotation script
run: |
chmod +x scripts/rotate-aws-keys.py
python3 scripts/rotate-aws-keys.py
The complete GitHub Workflow will look like this:
name: Daily AWS access key rotation check
on:
#Every day at 5:55 AM UTC (23:55 CST) '55 5 * * *'
schedule:
- cron: '55 5 * * *'
env:
AWS_ACCESS_KEY_ID: ${{ secrets.DEVOPS_TF_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.DEVOPS_TF_AWS_SECRET_ACCESS_KEY }}
jobs:
AWS-key-rotation:
name: Quarterly AWS access key rotation
runs-on: macos-latest
steps:
- name: Checkout Code
uses: actions/checkout@v2
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- uses: actions/setup-python@v2
with:
python-version: 3
- name: Install boto3
run: |
pip install boto3
- name: Secret Injection MAILBOX credentials
run: |
sed -i "" 's/MAILBOX_EMAIL/${{ secrets.MAILBOX_EMAIL }}/' scripts/rotate-aws-keys.py
sed -i "" 's/MAILBOX_PASSWORD/${{ secrets.MAILBOX_PASSWORD }}/' scripts/rotate-aws-keys.py
cat scripts/rotate-aws-keys.py
- name: Run AWS key rotation script
run: |
chmod +x scripts/rotate-aws-keys.py
python3 scripts/rotate-aws-keys.py
That was straightforward, right? We already have a workflow that will be running our python script every day. Let's now tell the python script what to do!
The following libraries need to be imported.
import datetime
from datetime import date
import dateutil
from dateutil import parser
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
import boto3
from botocore.exceptions import ClientError
iam_client = boto3.client('iam')
The following steps detail what the script does:
- Iterate through all the IAM users in our AWS account. We need to use pagination because the default AWS API call only returns 100 items per page.
- Check the key age for each user with an active access key.
- If the key has been active for 90 days, rotate it (delete the old key and create a new one).
- If the key has been active for 83 days, send a one-week email reminder.
- If the key has been active for 89 days, send a one-day email reminder.
- Check the key age for each user with an active access key.
try:
marker = None
paginator = iam_client.get_paginator('list_users')
# Need to use a paginator because by default API call only returns 100 records
for page in paginator.paginate(PaginationConfig={'PageSize': 100, 'StartingToken': marker}):
print("Next Page : {} ".format(page['IsTruncated']))
u = page['Users']
for user in u:
keys = iam_client.list_access_keys(UserName=user['UserName'])
for key in keys['AccessKeyMetadata']:
active_for = date.today() - key['CreateDate'].date()
# With active keys older than 90 days
if key['Status']=='Active' and active_for.days >= 90:
print (user['UserName'] + " - " + key['AccessKeyId'] + " - " + str(active_for.days) + " days old. Rotating.")
delete_key(key['AccessKeyId'], user['UserName'])
create_key(user['UserName'])
# Send a notification email 7 days before rotation
elif key['Status']=='Active' and active_for.days == 83:
send_email("MAILBOX_EMAIL", "MAILBOX_PASSWORD", "recipient_email", subject_1_week, body_1_week)
print ("Email sent to " + user['UserName'] + " warning of key rotation in a week.")
# Send a notification email 1 day before rotation
elif key['Status']=='Active' and active_for.days == 89:
send_email("MAILBOX_EMAIL", "MAILBOX_PASSWORD", "recipient_email", subject_1_day, body_1_day)
print ("Email sent to " + user['UserName'] + " warning of key rotation tomorrow.")
except ClientError as e:
print("An error has occurred attempting to rotate user %s access keys." % user['UserName'])
Since I wanted to be a nice guy, I will be sending a reminder to the users a week before and a day before their key will expire (you don't have to, and can shorten your script by skipping those conditional statements). That means that I will be sending a few emails, thus I defined a send_email function for such task. In my case, I used Office365.
def send_email(sender, password, recipient, subject, body):
mimemsg = MIMEMultipart()
mimemsg['From']=sender
mimemsg['To']=recipient
mimemsg['Subject']=subject
mimemsg.attach(MIMEText(body, 'html'))
connection = smtplib.SMTP(host='smtp.office365.com', port=587)
connection.starttls()
connection.login(sender,password)
connection.send_message(mimemsg)
connection.quit()
At this point, we are only missing what the delete_key and create_key functions do.
The delete_key function couldn't literally be easier.
# Delete an specified access key for a user
def delete_key(access_key, username):
try:
iam_client.delete_access_key(UserName=username, AccessKeyId=access_key)
print("%s has been deleted." % (access_key))
except ClientError as e:
print("The access key with id %s cannot be found" % access_key)
create_key isn't much harder. We simply need to collect the newly created AWS access key id and secret access key, as well as the IAM user email address (depending on your organization, you may retrieve it from the username, as a tag...), to share the rotated keys with the corresponding end users via email.
# Create a new AWS access key and share it with the user via email
def create_key(username):
access_key_metadata = iam_client.create_access_key(UserName=username)['AccessKey']
access_key = access_key_metadata['AccessKeyId']
secret_key = access_key_metadata['SecretAccessKey']
recipient_email = "end_user_email"
subject = "AWS API Key rotation"
body = """\
<html>
<head></head>
<body>
<p>Include your message body here, including your access_key and secret_key</p>
</body>
</html>
"""
send_email("MAILBOX_EMAIL", "MAILBOX_PASSWORD", "recipient_email", subject, body)
print("New access key (%s) created for %s" % (access_key, username))
As promised, I wasn't going to get into the weeds with org specifics. But, as you can imagine, you will need email account credentials and an encryption tool to send the updated keys securely with the user. In my scenario, I've used Virtru and Office365 email rules. You might also want to include some instructions in your email body explaining the end users how they can copy and paste the new access key in their ~/.aws/credentials file to avoid running into Access_Denied errors.
At this point:
- You are complying with security regulations and best practices by periodically rotating AWS access keys
- You aren't chasing your developers to rotate their keys (and praying that these keys don't get compromised in the meantime!)
- Developers won't be running into permission issues attempting to rotate their keys themselves
- It becomes the developers' responsibility to update their local credentials file with the shared keys
And that's all folks! Hope this solution helps you relieve your AWS access key 90 day rotation headache.
Maintaining a secure cloud environment isn't that complicated if you are determined to keep it secure :)
Top comments (0)