Upload large files to AWS S3 using Multipart upload and presigned URLs

If you need to upload large files to AWS S3 from the client, the best way to do this is with multipart uploads and presigned URLs.

There are some articles out there that cover this process already, so why am I writing this?

None of the existing articles are written specifically for node with typescript (or at least none that are up to date). I had to trawl the AWS documentation for the specific functions, methods and types I needed to perform an upload in node. Hopefully this article will save you some of my pain.

Big thanks to the author of this article, which I was able to use as an overview of the steps I needed to take in this process.

Why presigned URLs?

This article is a nice introduction to why you might want to use presigned URLs.

Prerequisites

• Basic familiarity with AWS
• An AWS account with an existing S3 bucket and an environment with access and permissions to access and upload to that bucket.
• Basic familiarity with a node server-side framework and client-side framework.
• The AWS-SDK v3 configured on your local machine.

These are the specific AWS-SDK library versions I used in node:

"@aws-sdk/s3-request-presigner": "3.534.0",
"@aws-sdk/client-s3": "3.534.0",

Steps

1. Start a multipart upload on the server side.
2. Generate presigned URLs on the server side.
3. Split your file into chunks and use each presigned URL to upload each chunk.
4. Complete the multipart upload on the server side.

Step 1: Start a multipart upload

This should be run on the server side in whatever back end node framework you're using.

import {
    S3Client,
    CreateMultipartUploadCommand
} from '@aws-sdk/client-s3';

...

const client = new S3Client({ region: yourRegionHere });

const createMultipartUploadCommand = new CreateMultipartUploadCommand({
    Bucket: yourBucketNameHere,
    Key: yourNewS3KeyHere,
});

const startUploadResponse = await client.send(createMultipartUploadCommand);

const uploadId = startUploadResponse.UploadId;

This instructs S3 to start an upload. The response will return an uploadId, which will be used in generating presigned URLs in the next step to tie them all together.

NOTE: This CreateMultipartUploadCommand is where you set the ACL for the uploaded object, amongst other things. See the AWS-SDK documentation for more details.

Step 2: Generate Presigned URLs

This should be run on the server side in whatever back end node framework you're using.

import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { UploadPartCommand } from '@aws-sdk/client-s3';

...

// Calculate the number of parts to split the file into.
// I am going to split the file into ~10mb chunks.
// AWS requires it to be above 5mb, for all but the last part.
const numberOfParts = Math.ceil(fileSize / 10000000);

let presignedUrls: string[] = [];

for (let i = 0; i < numberOfParts; i++) {
    const presignedUrl = await getSignedUrl(
        client,
        new UploadPartCommand({
            Bucket: bucketName,
            Key: s3Key,
            UploadId: uploadId,
            PartNumber: i + 1,
        }),
        {},
    );

    presignedUrls.push(presignedUrl);
}

This will generate a presigned URL for each chunk of the file to upload.

These will need to be returned to the client to be used to upload the file chunk by chunk.

Step 3: Split your file and use presigned URLs to upload

This part should be run on the client side.

async function uploadPart(
    fileChunk: Blob,
    presignedUrl: string,
    partNo: number,
): Promise<ApiResponse<CompletedPart>> {
    const uploadResponse = await fetch(presignedUrl, {
        method: 'PUT',
        body: fileChunk,
    });

    return { value: { ETag: uploadResponse.headers.get('ETag') ?? '', PartNo: partNo } };
}

...

const numberOfParts = Math.ceil(file.size / 10000000);

const uploadsArray: Promise<ApiResponse<CompletedPart>>[] = [];

for (let i = 0; i < numberOfParts; i++) {
    const start = i * 10000000;
    const end = Math.min(start + 10000000, file.size);
    const blob = file.slice(start, end);

    uploadsArray.push(uploadPart(blob, presignedUrls[i], i + 1));
}

const uploadResponses = await Promise.all(uploadsArray);

This splits the file up into ~10mb chunks (you can split files into smaller or larger chunks to your preferences) and then uses each presigned URL to upload it to S3.

Note that I have collected the ETag and PartNo from each upload because we will need to pass these to the server to use when completing the multipart upload in the next step.

Note: Make sure that the presigned URLs are used in correct part number order otherwise S3 will not be able to construct the file properly.

Aside: CORS Settings for AWS S3

The ETag, which is returned from the presigned URL PUT requests as a header, is required to complete the multipart upload.

Your browser will probably block access to the header due to CORS. If this is the case, you will need to expose it explicitly in your S3 CORS settings, using the ExposeHeaders property.

In the AWS console, these are found on the permissions tab, and will look something like this:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "PUT",
            "POST"
        ],
        "AllowedOrigins": [
            ...
        ],
        "ExposeHeaders": [
            "ETag"
        ]
    }
]

NOTE: You may also need to add the PUT method in AllowedMethods as above.

Step 4: Complete the multipart upload

This part should be run on the server side.

import {
    S3Client,
    CompleteMultipartUploadCommand,
} from '@aws-sdk/client-s3';

...

const client = new S3Client({ region: yourRegionHere });

const completeUploadInput = {
    Key: s3Key,
    Bucket: bucketName,
    UploadId: uploadId,
    MultipartUpload: {
        Parts: completedParts.map((x) => {
            return { ETag: x.ETag, PartNumber: x.PartNo };
        }),
    },
};

const completeMultipartUploadCommand = new CompleteMultipartUploadCommand(completeUploadInput);

await client.send(completeMultipartUploadCommand);

Once you have uploaded the file from the client side using all of your presigned URLs, you'll need to complete the command on the server side.

You'll need the uploadId returned from the CreateMultipartUploadCommand request in part 1, and you'll need the part numbers and corresponding ETag values from all of the uploaded parts.

The presigned URL upload happened on the client side so you will need to send the ETag/PartNumber pairs from the client so that you can complete the upload.

With the command finished, you're done! The file should be available in your S3 bucket.

Conclusion

You should now have a working multipart file upload using presigned URLs.

Hopefully this saves somebody the pain of trawling through the leviathan of AWS-SDK documentation trying to find the specific types and functions required to perform this multipart upload!