loading...
Cover image for Share Your AWS S3 Private Content With Others, Without Making It Public

Share Your AWS S3 Private Content With Others, Without Making It Public

idrisrampurawala profile image Idris Rampurawala Updated on ・9 min read

Amazon Web Services (AWS) S3 objects are private by default. Only the object owner has permission to access these objects. Optionally we can set bucket policy to whitelist some accounts or URLs to access the objects of our S3 bucket.

Recently, while working on a project, I came across a scenario where I wanted to make objects of my bucket public but only to limited users. Consider some of the scenarios as follows:

  • You store some user-specific files on S3 which is accessible to your respective authenticated users via your frontend. But what if a user wants to share this file with the people outside of your web application user accounts (who are not part of your application or simply unauthorized users)?
  • You have some user reports stored on the S3 bucket. And you want to email these reports to your users and some other email ids that have been allowed by the user? An email has a limit for the size of the attachment and hence not feasible to attach all reports in one email.

So how can we allow these S3 objects to be accessed by public users (users not part of our system)? The answer is presigned url. The objects can be shared with others by creating a presigned url, to grant time-limited permission to download (view) the objects. Anyone who receives the presigned url can then access the object. For example, if you have a video in your bucket and both the bucket and the object are private, you can share the video with others by generating a presigned url.

Out of all possible ways to generate a presigned url of an S3 object, let's discuss the 2 most important ones:

  1. S3 presigned url method
  2. Amazon CloudFront's signed url method

βœ”οΈ Prerequisite

  • Before we begin, we are assuming that we have an S3 bucket idr-aws-bucket with an object test.gif on AWS.
  • This bucket is private i.e. when we try to access the object we get the following error. Private S3 Object Access Error
  • We are gonna use boto3 - AWS SDK for Python for code demonstration. For any other language, the process is the same but the code would defer. Links provided at the end of the post.

Let's roll in!🍟


1. S3 presigned url method

A user who does not have AWS credentials or permission to access an S3 object can be granted temporary access by using a presigned url.

A presigned url is generated by an AWS user who has access to the object. The generated url is then given to the user without making our bucket private. The presigned url can be entered in a browser or used by a program or HTML webpage. The credentials used by the presigned url are those of the AWS user who generated the url.

This method is pretty easy but has got some points to consider. Let's implement this and then look into the shortcomings of using this method.

Step 1. Configuring IAM user credentials required for AWS SDK

Anyone with valid security credentials can create a presigned url. However, to successfully access an object, the presigned url must be created by someone who has permission to perform the operation that the presigned url is based upon. Yes, you read that correctly! You can perform any operation such as GetObeject, PutObject, etc while generating the presigned url (will see later in code below).

Step 2. Setup AWS SDK

Once we have our AWS credentials by which we gonna create presigned url, we need to set up AWS SDK to access the method. You can find the documentation here.

Step 3. The Code

from typing import Optional

# import the AWS library
import boto3
from botocore.exceptions import ClientError


def create_presigned_url(
        bucket_name: str, object_name: str, expiration=3600) -> Optional[str]:
    """Generate a presigned URL to share an s3 object

    Arguments:
        bucket_name {str} -- Required. s3 bucket of object to share
        object_name {str} -- Required. s3 object to share

    Keyword Arguments:
        expiration {int} -- Expiration in seconds (default: {3600})

    Returns:
        Optional[str] -- Presigned url of s3 object. If error, returns None.
    """

    # Generate a presigned URL for the S3 object
    aws_profile = <your-aws-profile-name>
    s3_client = boto3.session.Session(
        profile_name=aws_profile).client('s3')
    try:
        # note that we are passing get_object as the operation to perform
        response = s3_client.generate_presigned_url('get_object',
                                                    Params={
                                                        'Bucket': bucket_name,
                                                        'Key': object_name
                                                    },
                                                    ExpiresIn=expiration)
    except ClientError as e:
        logging.error(e)
        return None

    # The response contains the presigned URL
    return response

This is a wrapper function that returns the presigned url when called with an S3 bucket and object name. Also, note that the expiration can also be set which allows the url to remain valid up to expiration (in seconds).

# function that calls above function to create presigned url
def generate_presigned_url():
    bucket_name = <your-s3-bucket-name>
    bucket_resource_url = <your-s3-bucket-resource-url>
    url = create_presigned_url(
        bucket_name,
        bucket_resource_url
    )
    return {
        'url': url
    }
// Output of above function
{
  "url": "https://<your-aws-bucket>.s3.amazonaws.com/test.gif?AWSAccessKeyId=AKIA5UAF6XOYXYXLIIVG&Signature=ztJYSThTOmpqU4nVSertoNuNYLE%3D&Expires=1584881770"
}

S3-object-access-presigned-url

πŸ‘‡ Points to remember

  • This method will allow our object to be accessible via the generated url and hence, the presigned url is the actual object path of our AWS S3 bucket, which is now public in the url.
  • This method will not modify any bucket policy or any other bucket permissions. It will be validated at the time we visit the url.
  • Presigned url is signed by the credentials we are using to create it and hence any change in the url will fail the authentication resulting in an unauthorized request by AWS.
  • The credentials that we can use to create a presigned url include:
    • AWS Identity and Access Management (IAM) instance profile: Token can be valid up to a maximum of 6 hours.
    • AWS Security Token Service: Token can be valid up to a maximum of 36 hours when signed with permanent credentials, such as the credentials of the AWS account root user or an IAM user.
    • IAM user: Token can be valid up to a maximum of 7 days when using AWS Signature Version 4.

Hence, by this method, we can create presigned url valid up to a maximum of 7 days. For higher expiration value, we need to move to the second approach i.e. by using CloudFront


2. Amazon CloudFront's signed url method

We have looked into how easy it is to share our S3 object with the outside world using the AWS SDK. Though it is relatively easy with no extra setup in AWS, there are limitations:

  • Our S3 object url is now visible to the person we shared presigned url with
  • URL can be valid for a maximum of 7 days (if created via IAM user credentials) Hence, for more flexibility, AWS CloudFront is the second approach to go for. It is not that easy but if you stick to the post, we will get it there. ☺️

Let us understand how it works before implementation:

  1. AWS CloudFront provides 2 methods to allow users to access our private content - signed urls or signed cookies. For simplicity, we will look into the implementation via signed urls.
  2. Firstly, we will generate CloudFront key-pairs for an AWS account to allow the account to create a signed url. For simplicity, I am using my root account for the same. Optionally, you can create a user for the same and add the account id of the user in the AWS CloudFront distribution set up process below.
  3. Next, we will have to create the AWS CloudFront distribution to allow our bucket to be accessible via AWS CloudFront signed url.
  4. Finally, we will write the code to use the private key created in step 1 above to create signed url for an S3 object.

1. Generating CloudFront key-pairs

Remember, I am creating a key-pair for my root IAM user here. Click on the My Security Credentials option.
CloudFront My Security Credentials

Next, as highlighted in the image below, click on the button Create New Key Pair
CloudFront Create New Key Pair
This will open up a pop-up where you can download both your private and public keys and also make a note of your access ID as well.

2. Creating CloudFront Distribution

  1. Create CloudFront Distribution
    On the Cloudfront dashboard, click on Create Distribution.
    aws-cloudfront-create-distribution

  2. Select a content delivery method
    As we are gonna access our S3 content via the web (HTTP), we will choose the method web (as highlighted in the image).
    aws-cloudfront-select-delivery

  3. Add distribution settings
    In the Origin Settings section, we will select an Amazon S3 bucket that we want to make available for users via signed url, and make sure you select rest of the options as in the image below:
    aws-cloudfront-settings
    πŸ“• Two important points to note here, I have enabled Restrict Viewer Access and in Trusted Signers I have checked Self. Trusted Signers are the AWS Accounts which are authorized to create signed urls. Checking the Self option means that I am using my AWS root IAM user (for which we create key-pair in step 1) to generate the signed url. Rest all of the settings are self-explanatory.

  4. Make a note of your distribution domain
    aws-cloudfront-domain-name

3. Finally, the code to create signed url 😌

from datetime import datetime

from botocore.exceptions import ClientError
from botocore.signers import CloudFrontSigner
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import padding

def rsa_signer(message: str) -> str:
    # cloudfront-pk.pem is the private key generated in step 1 for IAM user
    with open('.cloudfront-pk.pem', 'rb') as key_file:
        private_key = serialization.load_pem_private_key(
            key_file.read(),
            password=None,
            backend=default_backend()
        )
    return private_key.sign(
        message, padding.PKCS1v15(), hashes.SHA1())


def create_cloudfront_signed_url(
        object_name: str, expiration_date: datetime) -> str:
    # cloudfront key-par access ID generated in step 1
    key_id = <your-cloudfront-key-par-access-ID>
    # your cloudfront distribution domain created in step 4 of distribution creation steps
    cloudfront_domain = <your-cloudfront-distribution-domain>

    url = '{cloudfront_domain}/{object_name}'.format(
        cloudfront_domain=cloudfront_domain,
        object_name=object_name
    )

    cloudfront_signer = CloudFrontSigner(key_id, rsa_signer)

    # Create a signed url that will be valid until the specfic expiry date
    # provided using a canned policy.
    signed_url = cloudfront_signer.generate_presigned_url(
        url, date_less_than=expiration_date)
    return signed_url

Once you execute the above function, you will get the out as shown in the image below.

from datetime import datetime, timedelta

# function that calls above function to create presigned url
def generate_presigned_url():
    expire_date = datetime.utcnow() + timedelta(days=2) # expires in 2 days
    bucket_resource_url = <your S3 bucket resource url>
    url = create_cloudfront_signed_url(
        bucket_resource_url,
        expire_date
    )
    return {
        'url': url
    }
// Output of above function
{
  "url": "https://<your-cloudfront-id>.cloudfront.net/test.gif?Expires=1585148785&Signature=AeRhPIEAfWX4g-5rpfMkyQD1-pjNdUj2sUG8yyL-mBEcYG6Vhk2jCTpQMJ2m6oUkz-6zZXSqT-3v2KKB-HSHqG984ZBq8rKe~ZWMC3k3I3gqW9x8u-S-Xl20Fj6CyydunW6pDWILxyLZeU5s2x2aYZok2PKyns0eUgBpls0oPb6jX3WVdRafmIIU98EZXEqx5mTWi0se62cqDrI8MPk5Em7gGJ8vhh5X7PNebMEVQcqf189Yj~3RSPxTwIKylzRc6ZyMU5wq8pGeKkrwZK~iQO5DXqHRXOFzI8JM5XL7608F7aFl6uSlboMvKZWEh0FGqmp8GZpR45sbUOKT38AJ3g__&Key-Pair-Id=<key-pair-id>"
}

S3 object access by CloudFront presigned url


πŸ“’ Final Words

Congratulations! πŸ‘ We have successfully managed to solve one of the key challenges in AWS of serving private S3 content to users without making it public with the help of signed url. Do not forget to check the Useful Links section to find links to read more about this topic.

Also, I have created a small Flask project on GitHub demonstrating the code to implement both the approaches we have talked about here.

GitHub logo idris-rampurawala / aws-s3-access-demo

This project is an implementation to show various approaches to share your AWS S3 private object with others without making it public.

AWS S3 Object Access Demo

Share Your AWS S3 Private Object With Others Without Making It Public

This project is an implementation to show various approaches to share your AWS S3 private object with others without making it public. You can find an in-depth article on this implementation here.

Background

Amazon Web Services (AWS) S3 objects are private by default. Only the object owner has permission to access these objects. Optionally we can set bucket policy to whitelist some accounts or URLs to access the objects of our S3 bucket.

There are various instances where we want to share our S3 object with users temporarily or with some specific expiration time without the need to make our S3 bucket private. This project aims to solve that problem by creating presigned url with some code examples.

Prerequisites

  • Python 3.7.2 or higher
  • Install pip
  • AWS account with an S3 bucket and…

⭐ Useful Links


If you find this helpful or have any suggestions, feel free to comment. Also, do not forget to hit ❀️ or πŸ¦„ if you like my post.

See ya! until my next post πŸ˜‹

Posted on by:

idrisrampurawala profile

Idris Rampurawala

@idrisrampurawala

A Full Stack Developer specializes in Python (Django, Flask) & JavaScript technologies (Angular, Node.js). Experience designing, planning, building complete web applications with backend API systems

Discussion

markdown guide
 

Hi thanks for the article.I am quite new to this.But in boto3,where should I specify my access_key,secret_key etc?
One can't simply generate a signed right?
I am new to this.So I maybe wrong!

 

Hey, it is not recommended to use AWS secrets for any AWS SDK (boto3 in your case). The recommended method is to install AWS CLI in your local machine and configure it with your IAM secret keys. Now, in this project you only need to add this profile-name (gets generated by AWS CLI setup) in .env against AWS_PROFILE_NAME key and your AWS SDK should work then.