Upload File to S3 Using NestJS Application

Introduction

In this post, I will show how to upload a file to S3 using a NestJS application and Multer. S3 is Amazon's Simple Storage Service (S3). In old systems, one could upload files to a database. However, with storage services, it is easier to upload and retrieve files. This is also more performant.

If you are a beginner with the NestJS framework, I would recommend reading their documentation. I have written some posts about NestJS previously here, here, and here.

Simple Storage Service

Simple storage service is also known as S3. Irrespective of what kind of application you build, you have to store static files somewhere. AWS offers a simple and effective service called S3. As previously mentioned in the post, application developers used to save files in the database, but that is not very performant.

With S3, now you can store the file in a storage service and create a file link that you can save in the database.

To understand S3 is to understand a hash table. Usually, when you store any file on S3, the service generates a random string as a key to identify that file. The file is the blob format data.

S3 is targeted towards application builders to individual users. The advantages of S3 are scalability, high availability, performance, and security. S3 can also be used for redundancy. One thing to remember is that you can not use S3 to host a static website, especially if you want to use it with HTTPS.

Setting up an S3 bucket

For this demo, let's set up an S3 bucket in the AWS S3 service. I would assume that you have an AWS account with enough permissions to manage resources like creating or deleting buckets. If you don't, you can always create an account for AWS with a free tier. Over the years, AWS has done a great job to help developers play with their services and improve them.

Once you have your user, make sure to download the AWS Access Key and AWS Secret Key. We will need these keys to call AWS services programmatically.

Let's create a bucket in S3.

NestJS Application

Let's create a nestjs application. If you do not have nest-cli downloaded, I would recommend use npm i @nestjs/cli.

1. Set up Application

Create a new application

nest new fileuploaddemo

Once we have the facade of the application, we will create an API, service, and a database table. If you are wondering, why a database table, then to answer that - We will use a database to store file metadata information including the link and when it was created.

Just like previous articles, we will use prisma as our ORM for creating this nestjs application.

Install Prisma dependencies

npm install prisma --save-dev

npm install @prisma/client

Create initial Prisma set up

npx prisma init

If you are using Windows environment, you might run into an error while running above command. So set environment variables

set PRISMA_CLI_QUERY_ENGINE_TYPE=binary

set PRISMA_CLIENT_ENGINE_TYPE=binary

Add the database table details in schema.prisma file

// This is your Prisma schema file,
// learn more about it in the docs: https://pris.ly/d/prisma-schema

generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "mysql"
  url      = env("DATABASE_URL")
}

model FileEntity {
  id Int @default(autoincrement()) @id
  fileName String
  fileUrl  String 
  key      String
  createdAt   DateTime @default(now())
  updatedAt   DateTime @updatedAt
}

And make sure to set DATABASE_URLas the environment variable before running Prisma migrations.

Run prisma migrations

npm prisma migrate dev

2. Set up environment

We will need a few environment variables to make sure our application is functioning.

So create .env file in your application and add the following variables.

AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=accessKey
AWS_SECRET_ACCESS_KEY=secretKey
AWS_BUCKET_NAME='nestjsfileuploaddemo'
DATABASE_URL="mysql://user:password@localhost:3306/fileuploaddemo?schema=public"

3. Upload API

Usually, a client application will send us file details through form-data. For this demo, we will create an API that the client application can call. This will be our upload API.

import { Controller, Post, UploadedFile, UseInterceptors } from "@nestjs/common";
import { FileInterceptor } from "@nestjs/platform-express";
import { FileUploadService } from "src/services/fileupload.service";
import { Express } from 'express';


@Controller('/v1/api/fileUpload')
export class FileController {
    constructor(private fileUploadService: FileUploadService) {}

    @Post()
    @UseInterceptors(FileInterceptor('file'))
    async uploadFile(@UploadedFile() file: Express.Multer.File): Promise {
        const uploadedFile = await this.fileUploadService.uploadFile(file.buffer, file.originalname);
        console.log('File has been uploaded,', uploadedFile.fileName);        
    }

}

In this API, we are using FileInterceptor from NestJS which extracts file details from the request.

The API is straightforward. We pass the file metadata information to FileUploadService and that does the job of uploading the file to S3 as well as saving the data in the database.

4. FileInterceptor for File Upload

An interceptor is one of the fundamental concepts in NestJS. Interceptor is a class with annotation @Injectable() and implements NestInterceptor interface.

5. Create a FileUploadService

FileUploadService will perform two tasks. One is to upload the file to S3 and the second is to store the metadata in the database.

import { Injectable } from "@nestjs/common";
import { ConfigService } from "@nestjs/config";
import { FileEntity, Prisma } from "@prisma/client";
import { S3 } from "aws-sdk";
import { PrismaService } from "src/common/prisma.service";
import { v4 as uuid } from 'uuid';

@Injectable()
export class FileUploadService {
    constructor(private prismaService: PrismaService,
        private readonly configService: ConfigService,){}

    async uploadFile(dataBuffer: Buffer, fileName: string): Promise {
        const s3 = new S3();
        const uploadResult = await s3.upload({
            Bucket: this.configService.get('AWS_BUCKET_NAME'),
            Body: dataBuffer,
            Key: `${uuid()}-${fileName}`,
        }).promise();

        const fileStorageInDB = ({
            fileName: fileName,
            fileUrl: uploadResult.Location,
            key: uploadResult.Key,
        });

        const filestored = await this.prismaService.fileEntity.create({
            data: fileStorageInDB
        });

        return filestored;
    }
}

We have a method called uploadFile in this service class. It takes two parameters one for a data buffer and the other for a file name.

We create an S3 instance using S3() and use that to upload files to S3. It needs a bucket name, a data buffer in the body, and a unique key for storing the file.

We use the location as fileUrl from the uploaded file. And then save the metadata information in the database.

6. Final Demo

Let's run our application to see how it is working now.

npm run start - will start the application at default port 3000.

We will use postman to call our upload API.

Once we upload the file, we will see a console message of file uploaded.

And similarly, a database entry

Just to make sure that we have our files uploaded in S3, let's look at the bucket.

You will see two entries in the bucket.

Conclusion

In this post, we discussed how we can upload a file to AWS Simple Storage Service (S3) using NestJS application. We only covered the surface of this vast topic of S3. You can also manage to upload private files, encrypt, and decrypt those files. You can also subscribe to events of file upload. There are a lot of options when it comes to AWS S3.  We will cover more in the upcoming series of NestJS applications.