Amazon Web Services (AWS) S3 objects are private by default. Only the object owner has permission to access these objects. Optionally we can set bucket policy to whitelist some accounts or URLs to access the objects of our S3 bucket.

Recently, while working on a project, I came across a scenario where I wanted to make objects of my bucket public but only to limited users. Consider some of the scenarios as follows:

• You store some user-specific files on S3 which is accessible to your respective authenticated users via your frontend. But what if a user wants to share this file with the people outside of your web application user accounts (who are not part of your application or simply unauthorized users)?
• You have some user reports stored on the S3 bucket. And you want to email these reports to your users and some other email ids that have been allowed by the user? An email has a limit for the size of the attachment and hence not feasible to attach all reports in one email.

So how can we allow these S3 objects to be accessed by public users (users not part of our system)? The answer is presigned url. The objects can be shared with others by creating a presigned url, to grant time-limited permission to download (view) the objects. Anyone who receives the presigned url can then access the object. For example, if you have a video in your bucket and both the bucket and the object are private, you can share the video with others by generating a presigned url.

Out of all possible ways to generate a presigned url of an S3 object, let's discuss the 2 most important ones:

1. S3 presigned url method
2. Amazon CloudFront's signed url method

✔️ Prerequisite

• Before we begin, we are assuming that we have an S3 bucket idr-aws-bucket with an object test.gif on AWS.
• This bucket is private i.e. when we try to access the object we get the following error.
• We are gonna use boto3 - AWS SDK for Python for code demonstration. For any other language, the process is the same but the code would defer. Links provided at the end of the post.

Let's roll in!🍟

1. S3 presigned url method

A user who does not have AWS credentials or permission to access an S3 object can be granted temporary access by using a presigned url.

A presigned url is generated by an AWS user who has access to the object. The generated url is then given to the user without making our bucket private. The presigned url can be entered in a browser or used by a program or HTML webpage. The credentials used by the presigned url are those of the AWS user who generated the url.

This method is pretty easy but has got some points to consider. Let's implement this and then look into the shortcomings of using this method.

Step 1. Configuring IAM user credentials required for AWS SDK

Anyone with valid security credentials can create a presigned url. However, to successfully access an object, the presigned url must be created by someone who has permission to perform the operation that the presigned url is based upon. Yes, you read that correctly! You can perform any operation such as GetObeject, PutObject, etc while generating the presigned url (will see later in code below).

Step 2. Setup AWS SDK

Once we have our AWS credentials by which we gonna create presigned url, we need to set up AWS SDK to access the method. You can find the documentation here.

Step 3. The Code

from typing import Optional

# import the AWS library
import boto3
from botocore.exceptions import ClientError


def create_presigned_url(
        bucket_name: str, object_name: str, expiration=3600) -> Optional[str]:
    """Generate a presigned URL to share an s3 object

    Arguments:
        bucket_name {str} -- Required. s3 bucket of object to share
        object_name {str} -- Required. s3 object to share

    Keyword Arguments:
        expiration {int} -- Expiration in seconds (default: {3600})

    Returns:
        Optional[str] -- Presigned url of s3 object. If error, returns None.
    """

    # Generate a presigned URL for the S3 object
    aws_profile = <your-aws-profile-name>
    s3_client = boto3.session.Session(
        profile_name=aws_profile).client('s3')
    try:
        # note that we are passing get_object as the operation to perform
        response = s3_client.generate_presigned_url('get_object',
                                                    Params={
                                                        'Bucket': bucket_name,
                                                        'Key': object_name
                                                    },
                                                    ExpiresIn=expiration)
    except ClientError as e:
        logging.error(e)
        return None

    # The response contains the presigned URL
    return response

This is a wrapper function that returns the presigned url when called with an S3 bucket and object name. Also, note that the expiration can also be set which allows the url to remain valid up to expiration (in seconds).

# function that calls above function to create presigned url
def generate_presigned_url():
    bucket_name = <your-s3-bucket-name>
    bucket_resource_url = <your-s3-bucket-resource-url>
    url = create_presigned_url(
        bucket_name,
        bucket_resource_url
    )
    return {
        'url': url
    }

// Output of above function
{
  "url": "https://<your-aws-bucket>.s3.amazonaws.com/test.gif?AWSAccessKeyId=AKIA5UAF6XOYXYXLIIVG&Signature=ztJYSThTOmpqU4nVSertoNuNYLE%3D&Expires=1584881770"
}

👇 Points to remember

• This method will allow our object to be accessible via the generated url and hence, the presigned url is the actual object path of our AWS S3 bucket, which is now public in the url.
• This method will not modify any bucket policy or any other bucket permissions. It will be validated at the time we visit the url.
• Presigned url is signed by the credentials we are using to create it and hence any change in the url will fail the authentication resulting in an unauthorized request by AWS.
• The credentials that we can use to create a presigned url include:
  • AWS Identity and Access Management (IAM) instance profile: Token can be valid up to a maximum of 6 hours.
  • AWS Security Token Service: Token can be valid up to a maximum of 36 hours when signed with permanent credentials, such as the credentials of the AWS account root user or an IAM user.
  • IAM user: Token can be valid up to a maximum of 7 days when using AWS Signature Version 4.

Hence, by this method, we can create presigned url valid up to a maximum of 7 days. For higher expiration value, we need to move to the second approach i.e. by using CloudFront

2. Amazon CloudFront's signed url method

We have looked into how easy it is to share our S3 object with the outside world using the AWS SDK. Though it is relatively easy with no extra setup in AWS, there are limitations:

• Our S3 object url is now visible to the person we shared presigned url with
• URL can be valid for a maximum of 7 days (if created via IAM user credentials) Hence, for more flexibility, AWS CloudFront is the second approach to go for. It is not that easy but if you stick to the post, we will get it there. ☺️

Let us understand how it works before implementation:

1. AWS CloudFront provides 2 methods to allow users to access our private content - signed urls or signed cookies. For simplicity, we will look into the implementation via signed urls.
2. Firstly, we will generate CloudFront key-pairs for an AWS account to allow the account to create a signed url. For simplicity, I am using my root account for the same. Optionally, you can create a user for the same and add the account id of the user in the AWS CloudFront distribution set up process below.
3. Next, we will have to create the AWS CloudFront distribution to allow our bucket to be accessible via AWS CloudFront signed url.
4. Finally, we will write the code to use the private key created in step 1 above to create signed url for an S3 object.

1. Generating CloudFront key-pairs

Remember, I am creating a key-pair for my root IAM user here. Click on the My Security Credentials option.

Next, as highlighted in the image below, click on the button Create New Key Pair

This will open up a pop-up where you can download both your private and public keys and also make a note of your access ID as well.

2. Creating CloudFront Distribution

1. Create CloudFront Distribution
   On the Cloudfront dashboard, click on Create Distribution.
   

2. Select a content delivery method
   As we are gonna access our S3 content via the web (HTTP), we will choose the method web (as highlighted in the image).
   

3. Add distribution settings
   In the Origin Settings section, we will select an Amazon S3 bucket that we want to make available for users via signed url, and make sure you select rest of the options as in the image below:
   
   📕 Two important points to note here, I have enabled Restrict Viewer Access and in Trusted Signers I have checked Self. Trusted Signers are the AWS Accounts which are authorized to create signed urls. Checking the Self option means that I am using my AWS root IAM user (for which we create key-pair in step 1) to generate the signed url. Rest all of the settings are self-explanatory.

4. Make a note of your distribution domain
   

3. Finally, the code to create signed url 😌

from datetime import datetime

from botocore.exceptions import ClientError
from botocore.signers import CloudFrontSigner
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import padding

def rsa_signer(message: str) -> str:
    # cloudfront-pk.pem is the private key generated in step 1 for IAM user
    with open('.cloudfront-pk.pem', 'rb') as key_file:
        private_key = serialization.load_pem_private_key(
            key_file.read(),
            password=None,
            backend=default_backend()
        )
    return private_key.sign(
        message, padding.PKCS1v15(), hashes.SHA1())


def create_cloudfront_signed_url(
        object_name: str, expiration_date: datetime) -> str:
    # cloudfront key-par access ID generated in step 1
    key_id = <your-cloudfront-key-par-access-ID>
    # your cloudfront distribution domain created in step 4 of distribution creation steps
    cloudfront_domain = <your-cloudfront-distribution-domain>

    url = '{cloudfront_domain}/{object_name}'.format(
        cloudfront_domain=cloudfront_domain,
        object_name=object_name
    )

    cloudfront_signer = CloudFrontSigner(key_id, rsa_signer)

    # Create a signed url that will be valid until the specfic expiry date
    # provided using a canned policy.
    signed_url = cloudfront_signer.generate_presigned_url(
        url, date_less_than=expiration_date)
    return signed_url

Once you execute the above function, you will get the out as shown in the image below.

from datetime import datetime, timedelta

# function that calls above function to create presigned url
def generate_presigned_url():
    expire_date = datetime.utcnow() + timedelta(days=2) # expires in 2 days
    bucket_resource_url = <your S3 bucket resource url>
    url = create_cloudfront_signed_url(
        bucket_resource_url,
        expire_date
    )
    return {
        'url': url
    }

// Output of above function
{
  "url": "https://<your-cloudfront-id>.cloudfront.net/test.gif?Expires=1585148785&Signature=AeRhPIEAfWX4g-5rpfMkyQD1-pjNdUj2sUG8yyL-mBEcYG6Vhk2jCTpQMJ2m6oUkz-6zZXSqT-3v2KKB-HSHqG984ZBq8rKe~ZWMC3k3I3gqW9x8u-S-Xl20Fj6CyydunW6pDWILxyLZeU5s2x2aYZok2PKyns0eUgBpls0oPb6jX3WVdRafmIIU98EZXEqx5mTWi0se62cqDrI8MPk5Em7gGJ8vhh5X7PNebMEVQcqf189Yj~3RSPxTwIKylzRc6ZyMU5wq8pGeKkrwZK~iQO5DXqHRXOFzI8JM5XL7608F7aFl6uSlboMvKZWEh0FGqmp8GZpR45sbUOKT38AJ3g__&Key-Pair-Id=<key-pair-id>"
}

📢 Final Words

Congratulations! 👏 We have successfully managed to solve one of the key challenges in AWS of serving private S3 content to users without making it public with the help of signed url. Do not forget to check the Useful Links section to find links to read more about this topic.

Also, I have created a small Flask project on GitHub demonstrating the code to implement both the approaches we have talked about here.

⭐ Useful Links

• My GitHub repository implementing both of the above approaches
• S3 presigned url links
  • AWS documentation on presigned url
  • AWS boto3 code reference
  • Presigned url expiration issues
  • Upload objects using presigned url
• CloudFront signed url links
  • AWS boto3 code reference
  • Code examples in other languages
  • AWS Docs link

If you find this helpful or have any suggestions, feel free to comment. Also, do not forget to hit ❤️ or 🦄 if you like my post.

See ya! until my next post 😋