Edmar Diaz

Posted on Apr 7, 2020

FileUpload with NestJS using MinIO

#nestjs #minio #typescript #s3

Introduction

Hello guys, since we are in a pandemic crisis due to coronavirus. Most of us are working from home and today I've decided to make a simple guide about FileUpload with NestJs and MinIO. This is going to be my first post on Dev.to.

Before I start the guide let me first give you a background about the technology that we are going to use in this post.

What is MinIO?

Minio is an open source object storage server released under Apache License V2. It is compatible with Amazon S3 cloud storage service. Minio follows a minimalist design philosophy.

Minio is light enough to be bundled with the application stack. It sits on the side of NodeJS, Redis, MySQL and the likes. Unlike databases, Minio stores objects such as photos, videos, log files, backups, container / VM images and so on. Minio is best suited for storing blobs of information ranging from KBs to TBs each. In a simplistic sense, it is like a FTP server with a simple get / put API over HTTP.

What is NestJS?

Nest (NestJS) is a server-side (backend) application framework beautifully crafted to support developers productivity and make their lives happier.

It uses progressive JavaScript, is built with and fully supports TypeScript (yet still enables developers to code in pure JavaScript) and combines elements of OOP (Object Oriented Programming), FP (Functional Programming), and FRP (Functional Reactive Programming).

It is heavily inspired by common libraries and frameworks such as Angular and Java Spring Boot which improve developer productivity and experience.

Let's Get Started

Before starting make sure you have the following installed and configured:

NestJS - https://docs.nestjs.com/first-steps
MinIO - (I have my MinIO on docker) https://docs.min.io/

Create a new project on nest-cli:

 nest new minio-fileupload

You will be asked to choose between yarn or npm as package manager. Feel free to choose what suits you.

We will be using nestjs-minio-client as our nestjs minio client:

via NPM:

npm install nestjs-minio-client --save

via YARN:

yarn add nestjs-minio-client

After installing the nestjs-minio-client, lets generate our module, service on nest-cli:

nest g module minio-client
nest g service minio-client

After generating module and service let's create a config file:

touch src/minio-client/config.ts

Inside the config.ts (this is my MinIO configuration. You must use yours):

export const config = {
  MINIO_ENDPOINT: 'localhost',
  MINIO_PORT: 9001,
  MINIO_ACCESSKEY: 'AKIAIOSFODNN7EXAMPLE',
  MINIO_SECRETKEY: 'wJalrXUtnFEMIK7MDENGbPxRfiCYEXAMPLEKEY',
  MINIO_BUCKET: 'test'
}

Let us now register and configure our MinioModule and export our MinioService on our minio-client.module.ts file:

import { Module } from '@nestjs/common';
import { MinioClientService } from './minio-client.service';
import { MinioModule } from 'nestjs-minio-client';
import { config } from './config'
@Module({
  imports: [
    MinioModule.register({
      endPoint: config.MINIO_ENDPOINT,
      port: config.MINIO_PORT,
      useSSL: false,
      accessKey: config.MINIO_ACCESSKEY,
      secretKey: config.MINIO_SECRETKEY,
    })
  ],
  providers: [MinioClientService],
  exports: [MinioClientService]
})
export class MinioClientModule {}

Now that we register our MinioModule let's start creating our file models:

touch src/minio-client/file.model.ts

Here is the code for our file.model.ts:

export interface BufferedFile {
  fieldname: string;
  originalname: string;
  encoding: string;
  mimetype: AppMimeType;
  size: number;
  buffer: Buffer | string;
}

export interface StoredFile extends HasFile, StoredFileMetadata {}

export interface HasFile {
  file: Buffer | string;
}
export interface StoredFileMetadata {
  id: string;
  name: string;
  encoding: string;
  mimetype: AppMimeType;
  size: number;
  updatedAt: Date;
  fileSrc?: string;
}

export type AppMimeType =
  | 'image/png'
  | 'image/jpeg';

Nest step, inside our minio-client.service.ts:

import { Injectable, Logger, HttpException, HttpStatus } from '@nestjs/common';
import { MinioService } from 'nestjs-minio-client';
import { Stream } from 'stream';
import { config } from './config'
import { BufferedFile } from './file.model';
import * as crypto from 'crypto'

@Injectable()
export class MinioClientService {
    private readonly logger: Logger;
    private readonly baseBucket = config.MINIO_BUCKET

  public get client() {
    return this.minio.client;
  }

  constructor(
    private readonly minio: MinioService,
  ) {
    this.logger = new Logger('MinioStorageService');
  }

  public async upload(file: BufferedFile, baseBucket: string = this.baseBucket) {
    if(!(file.mimetype.includes('jpeg') || file.mimetype.includes('png'))) {
      throw new HttpException('Error uploading file', HttpStatus.BAD_REQUEST)
    }
    let temp_filename = Date.now().toString()
    let hashedFileName = crypto.createHash('md5').update(temp_filename).digest("hex");
    let ext = file.originalname.substring(file.originalname.lastIndexOf('.'), file.originalname.length);
    const metaData = {
      'Content-Type': file.mimetype,
      'X-Amz-Meta-Testing': 1234,
    };
    let filename = hashedFileName + ext
    const fileName: string = `${filename}`;
    const fileBuffer = file.buffer;
    this.client.putObject(baseBucket,fileName,fileBuffer,metaData, function(err, res) {
      if(err) throw new HttpException('Error uploading file', HttpStatus.BAD_REQUEST)
    })

    return {
      url: `${config.MINIO_ENDPOINT}:${config.MINIO_PORT}/${config.MINIO_BUCKET}/${filename}` 
    }
  }

  async delete(objetName: string, baseBucket: string = this.baseBucket) {
    this.client.removeObject(baseBucket, objetName, function(err, res) {
      if(err) throw new HttpException("Oops Something wrong happend", HttpStatus.BAD_REQUEST)
    })
  }
}

If you notice on my metaData object i have X-Amz-Meta-Testing this is just a sample if you want to add your custom metadata.

At this point, our minio-client module is now ready, to test it lets make a new module with service and controller:

nest g module file-upload
nest g service file-upload
nest g controller file-upload

Now let's open our src/file-upload/file-upload.module.ts and import our MinioClientModule:

import { Module } from '@nestjs/common';
import { FileUploadService } from './file-upload.service';
import { FileUploadController } from './file-upload.controller';
import { MinioClientModule } from 'src/minio-client/minio-client.module';

@Module({
  imports: [
    MinioClientModule
  ],
  providers: [FileUploadService],
  controllers: [FileUploadController]
})
export class FileUploadModule {}

Now we will create our first route on our file-upload.controller.ts:

import { Controller, Post, UseInterceptors, UploadedFile } from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express'
import { FileUploadService } from './file-upload.service';
import { BufferedFile } from 'src/minio-client/file.model';

@Controller('file-upload')
export class FileUploadController {
  constructor(
    private fileUploadService: FileUploadService
  ) {}

  @Post('single')
  @UseInterceptors(FileInterceptor('image'))
  async uploadSingle(
    @UploadedFile() image: BufferedFile
  ) {
    console.log(image)
  }
}

In this code, we created the HTTP POST Method with and endpoint /file-upload/single first lets try to console.log the image to see if we get the BufferFile. Here is the screenshot:

Via Insomnia / Postman (Multipart Form):

Buffered Image File

Our next step is to use our MinioClientService to upload it on our MinIO S3 Storage, let's open our file-upload.service.ts:

import { Injectable } from '@nestjs/common';
import { MinioClientService } from 'src/minio-client/minio-client.service';
import { BufferedFile } from 'src/minio-client/file.model';

@Injectable()
export class FileUploadService {
  constructor(
    private minioClientService: MinioClientService
  ) {}

  async uploadSingle(image: BufferedFile) {

    let uploaded_image = await this.minioClientService.upload(image)

    return {
      image_url: uploaded_image.url,
      message: "Successfully uploaded to MinIO S3"
    }
  }
}

The next step is to use our FileUploadService on FileUploadController, let's go back to our file-upload.controller.ts:

import { Controller, Post, UseInterceptors, UploadedFile } from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express'
import { FileUploadService } from './file-upload.service';
import { BufferedFile } from 'src/minio-client/file.model';

@Controller('file-upload')
export class FileUploadController {
  constructor(
    private fileUploadService: FileUploadService
  ) {}

  @Post('single')
  @UseInterceptors(FileInterceptor('image'))
  async uploadSingle(
    @UploadedFile() image: BufferedFile
  ) {
    return await this.fileUploadService.uploadSingle(image)
  }
}

Now try sending another POST request on Postman or Insomnia. And you should see the follwing:

Congrats, at this point we successfully uploaded our image on our MinIO storage. But wait!!! There's more.. How about multiple image field??? Is it possible on nestjs?? Well, my answer is YES!!! xD and heres how

First we need to create a new endpoint on our file-upload.controller.ts:

import { Controller, Post, UseInterceptors, UploadedFile, UploadedFiles } from '@nestjs/common';
import { FileInterceptor, FileFieldsInterceptor } from '@nestjs/platform-express'
import { FileUploadService } from './file-upload.service';
import { BufferedFile } from 'src/minio-client/file.model';

@Controller('file-upload')
export class FileUploadController {
  constructor(
    private fileUploadService: FileUploadService
  ) {}

  @Post('single')
  @UseInterceptors(FileInterceptor('image'))
  async uploadSingle(
    @UploadedFile() image: BufferedFile
  ) {
    return await this.fileUploadService.uploadSingle(image)
  }

  @Post('many')
  @UseInterceptors(FileFieldsInterceptor([
    { name: 'image1', maxCount: 1 },
    { name: 'image2', maxCount: 1 },
  ]))
  async uploadMany(
    @UploadedFiles() files: BufferedFile,
  ) {
    return this.fileUploadService.uploadMany(files)
  }
}

Then we add the uploadMany method on our FileUploadService:

import { Injectable } from '@nestjs/common';
import { MinioClientService } from 'src/minio-client/minio-client.service';
import { BufferedFile } from 'src/minio-client/file.model';

@Injectable()
export class FileUploadService {
  constructor(
    private minioClientService: MinioClientService
  ) {}

  async uploadSingle(image: BufferedFile) {

    let uploaded_image = await this.minioClientService.upload(image)

    return {
      image_url: uploaded_image.url,
      message: "Successfully uploaded to MinIO S3"
    }
  }

  async uploadMany(files: BufferedFile) {

    let image1 = files['image1'][0]
    let uploaded_image1 = await this.minioClientService.upload(image1)

    let image2 = files['image2'][0]
    let uploaded_image2 = await this.minioClientService.upload(image2)

    return {
      image1_url: uploaded_image1.url,
      image2_url: uploaded_image2.url,
      message: 'Successfully uploaded mutiple image on MinioS3'
    }
  }
}