Context
Amazon S3 is an object storage service that, along with many other features, allows you to create presigned URLs that enable external users to access objects within a given S3 bucket.
While there is a possibility to set an expiration date of the URL, there is no native built-in capability of making it a one-time use resource.
The following is an example of how one could add such capability using building blocks exposed by AWS.
High-level architecture
Implementation
I will be using AWS CDK as my IAC tool of choice and TypeScript as a programing language for the implementation.
First, the AWS DynamoDB table for storing information about the usage of a given presigned URL and the Amazon S3 bucket for holding objects.
import * as dynamo from "@aws-cdk/aws-dynamodb";
import * as s3 from "@aws-cdk/aws-s3";
const entriesTable = new dynamo.Table(this, "entriesTable", {
partitionKey: { name: "pk", type: dynamo.AttributeType.STRING },
billingMode: dynamo.BillingMode.PAY_PER_REQUEST,
removalPolicy: cdk.RemovalPolicy.DESTROY
});
const bucket = new s3.Bucket(this, "assets-bucket", {
removalPolicy: cdk.RemovalPolicy.DESTROY
});
Next, the AWS Lambda fronted with Amazon API Gateway. This part of the infrastructure is responsible for generating the presigned URLs.
import * as lambda from "@aws-cdk/aws-lambda-nodejs";
import * as apigw from "@aws-cdk/aws-apigatewayv2";
import * as apigwIntegrations from "@aws-cdk/aws-apigatewayv2-integrations";
const urlLambda = new lambda.NodejsFunction(this, "urlLambda", {
entry: join(__dirname, "./url-lambda.ts"),
environment: {
BUCKET_NAME: bucket.bucketName
}
});
bucket.grantRead(urlLambda);
const api = new apigw.HttpApi(this, "api", {
corsPreflight: {
allowMethods: [apigw.CorsHttpMethod.GET]
}
});
api.addRoutes({
integration: new apigwIntegrations.LambdaProxyIntegration({
handler: urlLambda
}),
path: "/get-url",
methods: [apigw.HttpMethod.GET]
});
It is crucial for the urlLambda (implementation reference) to return presigned URL with the right domain.
By default, if you are using the Amazon S3 SDK, the presigned URLs contain the Amazon S3 domain. In our case, the domain has to be swapped to the one exposed by Amazon CloudFront. This will allow us to run code (Lambda@Edge) whenever the URL is requested.
For the last piece, the Amazon CloudFront distribution with the Lambda@Edge (implementation reference) that will record the usages of the presigned URLs and decide if the request should be allowed or not.
/**
* Lambda@Edge does not support environment variables.
* To forward the `entriesTable` name generated by CloudFormation,
* an SSM parameter is created.
*
* This parameter will be fetched during the runtime of the `edgeLambda`.
*/
const entriesTableParameter = new ssm.StringParameter(
this,
"URL_ENTRIES_TABLE_NAME",
{
stringValue: entriesTable.tableName,
parameterName: "URL_ENTRIES_TABLE_NAME"
}
);
const edgeLambda = new lambda.NodejsFunction(this, "edgeLambda", {
entry: join(__dirname, "./edge-lambda.ts")
});
entriesTable.grantReadWriteData(edgeLambda.currentVersion);
entriesTableParameter.grantRead(edgeLambda.currentVersion);
const distribution = new cloudfront.Distribution(this, "distribution", {
defaultBehavior: {
origin: new origins.S3Origin(bucket),
cachePolicy: new cloudfront.CachePolicy(this, "cachePolicy", {
maxTtl: cdk.Duration.seconds(1),
minTtl: cdk.Duration.seconds(0),
defaultTtl: cdk.Duration.seconds(0),
// QueryStrings from presigned URL have to be forwarded to S3.
queryStringBehavior: cloudfront.CacheQueryStringBehavior.all()
}),
edgeLambdas: [
{
eventType: cloudfront.LambdaEdgeEventType.VIEWER_REQUEST,
functionVersion: edgeLambda.currentVersion,
includeBody: false
}
]
}
});
urlLambda.addEnvironment("CF_DOMAIN", distribution.domainName);
Usage
All variables used in the example commands below are available as stack outputs.
After deployment, let us upload an object to the assets-bucket. I'm going to upload an image of a cat.
aws s3 mv cat.jpeg s3://<assetsBucketName>
With the object uploaded, we can request the presigned URL.
curl -XGET 'https://<getPresignedUrlEndpoint>?key=cat.jpeg'
Now we have everything in place to test our solution. In theory, the first GET request for URL returned from the previous command should succeed. All subsequent requests should fail with the 403 status code.
Sadly, when we make the request, an error will be returned
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message>
<ArgumentName>Authorization</ArgumentName><ArgumentValue>AWS4-HMAC-SHA256 Credential=AKIAIJPQUQ6PR4TR73SQ/20210516/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f87f653cce70a3dd3260c1e5d187bc62aa8768ed606f59b29d811febe6880b20</ArgumentValue>
<RequestId>SBKQ3G6QDEJSJTRX</RequestId><HostId>8oCArh0jBSNahbv5A8keMVuNk2HFDF5ud52YHurBB7tMCxnEhNmqZU/4wJiaQ7WR70NhJLfH934=</HostId></Error>
Fixing the auth mechanism
Due to the nature of how high-level some of the AWS CDK constructs used in the IaC portion of the code are, the error message might not be helpful at all. We are not setting the Authorization header anywhere in our code explicitly, so what is going on?
It turns out that the origins.S3Origin construct creates so-called Origin Access Identity (OAI). This identity is used whenever Amazon CloudFront communicates with a given origin, in our case the assets-bucket.
With OAI in place, Amazon CloudFront will add an Authorization header for each request to a given origin. As for us, this behavior creates a conflict between authorization-related information contained within the query parameters of the presigned URL and the Authorization header.
As the S3Origin construct, to my best knowledge, does not allow us to configure whether we want to create OAI or not, we can leverage one of the AWS CDK escape hatches to modify the underlying CloudFormation template directly - effectively removing the created OAI.
const distribution = new cloudfront.Distribution(this, "distribution", {}); // Defined previously.
const cfnDistribution = distribution.node
.defaultChild as cloudfront.CfnDistribution;
cfnDistribution.addPropertyOverride(
"DistributionConfig.Origins.0.S3OriginConfig.OriginAccessIdentity",
""
);
With the OAI removed for the assets-bucket origin and the stack re-deployed, the previous GET request should work as intended.
If you are curious how the
node.defaultChildworks, here is a great video you can watch.
Summary
I've created this architecture as a way for me to brush up my knowledge of Amazon CloudFormation. The underlying implementation is simplified to be easily digestible.
Please do not treat it as the only possible way to achieve the underlying goal, there definitely might be cheaper ways to do so! (Lambda@Edge is relatively costly compared to regular AWS Lambda).
- I'm on twitter - @wm_matuszewski
- Code used for this article
Thanks 👋
Top comments (5)
Hello, I am trying this approach and running into issues after I replace the S3 domain with the Cloudfront Domain.
With the S3 Domain I'm able to access the file successfully, however, with the replaced CF domain I am getting an error with the Signature.
NOTE: both the URLs are identical with the exception of the domain
Did you run into any similar issues ?
dev-to-uploads.s3.amazonaws.com/up...
I sadly do not remember but I think I did. Have you tried looking at my implementation -> github.com/WojciechMatuszewski/one... maybe you missed a step?
Hello, Thanks for the article it is pretty interesting.
BTW with that solution, we can still download documents multiple times using the bucket URL instant of the Cloudfront URL.
Hello @wojciechmatuszewski
Which tool did you use to build that diagram please ?
miro.com/app/