Uploading files in Remix to a S3 compatible service

#remix #webdev #react

Remix is a new react-based framework designed to be a full stack application. With many applications, you will need to be able to store files, sometimes in an S3 compatible service. Here is how I was able to accomplish this. I took heavy influence from this dev.to article.

Create a file named uploader-handler.server.ts with the following contents:

import { s3Client } from './s3.server';
import type { UploadHandler } from '@remix-run/node';
import type { PutObjectCommandInput } from '@aws-sdk/client-s3';
import { GetObjectCommand } from '@aws-sdk/client-s3';
import { PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';

const uploadStreamToS3 = async (data: AsyncIterable<Uint8Array>, key: string, contentType: string) => {
  const BUCKET_NAME = "my_bucket_name";

  const params: PutObjectCommandInput = {
    Bucket: BUCKET_NAME,
    Key: key,
    Body: await convertToBuffer(data),
    ContentType: contentType,
  };

  await s3Client.send(new PutObjectCommand(params));

  let url = await getSignedUrl(s3Client, new GetObjectCommand({
    Bucket: BUCKET_NAME,
    Key: key,
  }), { expiresIn: 15 * 60 });

  console.log(url);

  return key;
}

// The UploadHandler gives us an AsyncIterable<Uint8Array>, so we need to convert that to something the aws-sdk can use. 
// Here, we are going to convert that to a buffer to be consumed by the aws-sdk.
async function convertToBuffer(a: AsyncIterable<Uint8Array>) {
  const result = [];
  for await (const chunk of a) {
    result.push(chunk);
  }
  return Buffer.concat(result);
}

export const s3UploaderHandler: UploadHandler = async ({ filename, data, contentType }) => {
  return await uploadStreamToS3(data, filename!, contentType);
}

Next, you will need to create the actual route to be able to upload a file. I have the following file: ~/routes/api/storage/upload.tsx with the following contents

import type { ActionFunction } from "@remix-run/node";
import { unstable_parseMultipartFormData } from "@remix-run/node";
import { auth } from "~/server/auth.server";
import { s3UploaderHandler } from "~/server/uploader-handler.server";

export const action: ActionFunction = async ({ request }) => {
  await auth.isAuthenticated(request, { failureRedirect: '/login' });

  const formData = await unstable_parseMultipartFormData(request, s3UploaderHandler);

  const fileName = formData.get('upload');

  return {
    filename: fileName,
  }
}

Now that you have the supporting files in place, let's upload a file.

<Form method="post" action={'/api/storage/upload'} encType="multipart/form-data">
    <Input type="file" name="upload" />
    <Button type="submit">Upload</Button>
</Form>

There you have it!

Version of sdks used:

@remix-run/node: 1.6.5
@remix-run/react: 1.6.5
@aws-sdk/client-s3: 3.145.0
@aws-sdk/s3-request-presigner: 3.145.0

Top comments (8)

Jon Callahan • Dec 30 '22

Very helpful, thank you. I am using a file input inside a bigger form so I needed to handle the other form fields as well. It was hard for me to figure out so if anyone finds this and sees the other form fields coming back undefined, it's because you need to return a serialized value from the upload handler for every form field. To do this I used a TextDecoder like this

export const uploadHandler = unstable_composeUploadHandlers(
  async (formField) => {
    if (formField.name === "attachments") {
      return await uploadStreamToR2(formField); // Returns the key
    }
    // We are uploading the attachment and returning the key for storage and retrieval. Everything else needs to be serialized using TextDecoder.
    return new TextDecoder().decode(await convertToBuffer(formField));
  }
);

Austin Gil • Mar 6 '23

I could be wrong here, but in doesnt the function convertToBuffer pull the entire file into a Buffer? And wouldn't that store the entire file contents in memory? And if a large file was uploaded, couldn't that crash the server?

Async iterators are weird, so I'm not positive, but I think that may be the case. I've been working on an alternative approach with streams, but if this works, this is much simpler.

Orji Chidi Matthew • Jun 3 '23

I think you're on to something. I'm not too familiar with async iterators myself.
If you could share your approach with streams it'd be helpful.

Thanks

Austin Gil • Jun 4 '23

I did a lot more reading and ultimately went with streams then wrote a whole series around file uploads (in general, not just Remix).

austingil.com/upload-to-s3/

Streams are the way to go, in my opinion, but there is another popular approach using signed URLs. In fact, it seems like most people do that. A major benefit is that you don't sent the file through your backend, so you don't have to pay for the transfer of the file in and out of your server.

I still prefer streams for the benefit of having more control over the file, and making it possible to support progressive enhancement.

Austin Gil • Jun 4 '23

Although, I should add that the post I shared does it in Node, and although Remix can run in Node, they do some funny stuff with their file uploads. It takes a whole big workaround. I was able to figure it out, but I did not write anything down about it yet, so if you're working with Remix, it's not nearly as easy.

Orji Chidi Matthew • Jun 4 '23

Hey thanks for the link. Makes for a good reference.

That said, I personally prefer signed urls as well for the very reason that the user's machine handles the upload itself thus you won't run into server problem with multiple people uploading simultaneously.

I guess every situation has to be weighed to see which method is vest suited for the problem at hand.