Intro
In this tutorial I will explain how to upload images to a S3-compatible object storage (AWS S3, DigitalOcean Spaces, Linode Object Storage, etc.) bucket using NestJS and GraphQL for both Apollo and Mercurius drivers.
Common
Most of the set-up as well as the entirety of the upload process is the same for both drivers.
Set Up
Start by installing the class-transformer and class-validator packages for dto validation (see more in the docs), sharp for image optimization, and the S3 client packages.
Note: I assume you have UUID installed, however I still added it to the following command
$ yarn add class-transformer class-validator sharp @aws-sdk/client-s3 @aws-sdk/signature-v4-crt uuid
$ yarn add -D @types/sharp @types/uuid
Configuration
Assuming you are using nestjs ConfigModule start by creating an interfaces folder.
Bucket Data:
// bucked-data.interface.ts
export interface IBucketData {
// name of the bucket
name: string;
// folder of the service
folder: string;
// uuid for the app
appUuid: string;
// the bucket url to create the key
url: string;
}
Upload Middleware Options:
// uploader-middleware-options.interface.ts
export interface IUploaderMiddlewareOptions {
maxFieldSize?: number;
maxFileSize?: number;
maxFiles?: number;
}
Uploader Options:
// uploader-options.interface.ts
import type { S3ClientConfig } from '@aws-sdk/client-s3';
import type { IBucketData } from './bucket-data.interface';
import type { IUploaderMiddlewareOptions } from './uploader-middleware-options.interface';
export interface IUploaderOptions {
clientConfig: S3ClientConfig;
bucketData: IBucketData;
middleware: IUploaderMiddlewareOptions;
}
On the config.ts function add the following:
import { IConfig } from './interfaces/config.interface';
export function config(): IConfig {
const bucketBase = `${process.env.BUCKET_REGION}.${process.env.BUCKET_HOST}.com`;
return {
// ...
uploader: {
clientConfig: {
forcePathStyle: false,
region: process.env.BUCKET_REGION,
endpoint: `https://${bucketBase}`,
credentials: {
accessKeyId: process.env.BUCKET_ACCESS_KEY,
secretAccessKey: process.env.BUCKET_SECRET_KEY,
},
},
bucketData: {
name: process.env.BUCKET_NAME,
folder: process.env.FILE_FOLDER,
appUuid: process.env.SERVICE_ID,
url: `https://${process.env.BUCKET_NAME}.${bucketBase}/`,
},
middleware: {
maxFileSize: parseInt(process.env.MAX_FILE_SIZE, 10),
maxFiles: parseInt(process.env.MAX_FILES, 10),
},
},
// ...
};
}
Module Creation
The uploader module set up changes depending if it is a library on a nestjs (or nx) mono-repo, or a monolith NestJS GraphQL API.
Monolith API
Create the uploader module and service:
$ nest g mo uploader
$ nest g s uploader
On the uploader.service.ts file add the S3 Client, the bucket data and the logger:
import { S3Client, S3ClientConfig } from '@aws-sdk/client-s3';
import { Injectable, Logger, LoggerService } from '@nestjs/common';
@Injectable()
export class UploaderService {
private readonly client: S3Client;
private readonly bucketData: IBucketData;
private readonly loggerService: LoggerService;
constructor(
private readonly configService: ConfigService,
) {
this.client = new S3Client(
this.configService.get<S3ClientConfig>('uploader.clientConfig'),
);
this.bucketData = this.configService.get<IBucketData>('uploader.bucketData');
this.loggerService = new Logger(UploaderService.name);
}
// ...
}
On the uploader.module.ts file add the global decorator:
import { Global, Module } from '@nestjs/common';
import { UploaderService } from './uploader.service';
@Global()
@Module({
providers: [UploaderService],
exports: [UploaderService],
})
export class UploaderModule {}
Library
Create the uploader library:
-
NestJS mono-repo:
$ nest g lib uploader -
NX mono-repo:
$ npx nx @nrwl/nest:library uploader --global --service --strict --no-interactive
To be able to create a DynamicModule, we need to create a library options, start by creating an interfaces folder and move the bucket-data.interface.ts, and add an options.interface.ts:
import type { S3ClientConfig } from '@aws-sdk/client-s3';
import type { IBucketData } from './bucket-data.interface';
export interface IOptions {
clientConfig: S3ClientConfig;
bucketData: IBucketData;
}
Create a new constants folder and add the options constant:
// options.constant.ts
export const UPLOADER_OPTIONS = 'UPLOADER_OPTIONS';
Unlike a normal module the class parameters come from the options passed to the module and not from the ConfigService:
import { S3Client, S3ClientConfig } from '@aws-sdk/client-s3';
import { Injectable, Logger, LoggerService } from '@nestjs/common';
import { UPLOADER_OPTIONS } from './constants/options.constant';
import { IBucketData } from './interfaces/bucket-data.interface';
import { IOptions } from './interfaces/options.interface';
@Injectable()
export class UploaderService {
private readonly client: S3Client;
private readonly bucketData: IBucketData;
private readonly loggerService: LoggerService;
constructor(@Inject(UPLOADER_OPTIONS) options: IOptions) {
this.client = new S3Client(options.clientConfig);
this.bucketData = options.bucketData;
this.loggerService = new Logger(UploaderService.name);
}
// ...
}
Turn the module into a dynamic global module:
import { DynamicModule, Global, Module } from '@nestjs/common';
import { UPLOADER_OPTIONS } from './constants/options.constant';
import { IOptions } from './interfaces/options.interface';
import { UploaderService } from './uploader.service';
@Global()
@Module({})
export class UploaderModule {
public static forRoot(options: IOptions): DynamicModule {
return {
global: true,
module: UploaderModule,
providers: [
{
provide: UPLOADER_OPTIONS,
useValue: options,
},
UploaderService,
],
exports: [UploaderService],
};
}
}
Image Uploading
Upload Scalar DTO
The graphql-upload package Upload scalar will be process into the following DTO, so add it into a dtos folder:
import { IsMimeType, IsString } from 'class-validator';
import { ReadStream } from 'fs';
export abstract class FileUploadDto {
@IsString()
public filename!: string;
@IsString()
@IsMimeType()
public mimetype!: string;
@IsString()
public encoding!: string;
public createReadStream: () => ReadStream;
}
File Validation
Since we want to upload only images the first thing we need to check is the mimetype:
//...
@Injectable()
export class UploaderService {
// ...
private static validateImage(mimetype: string): string | false {
const val = mimetype.split('/');
if (val[0] !== 'image') return false;
return val[1] ?? false;
}
// ...
}
Stream Processing
As you saw the Upload scalar returns a read stream that we need to transform into a buffer:
//...
import { Readable } from 'stream';
@Injectable()
export class UploaderService {
// ...
private static async streamToBuffer(stream: Readable): Promise<Buffer> {
const buffer: Uint8Array[] = [];
return new Promise((resolve, reject) =>
stream
.on('error', (error) => reject(error))
.on('data', (data) => buffer.push(data))
.on('end', () => resolve(Buffer.concat(buffer))),
);
}
// ...
}
Image Compression and Conversion
Before uploading the image we need to optimize it by compressing the image. Start by creating the constants necessary for compression, therefore on a constants folder add a file named uploader.constant.ts:
// The max width an image can have
// reducing the pixels will reduce the size of the image
export const MAX_WIDTH = 2160;
// an array with the percentage of quality ranging from 90 to 10%
export const QUALITY_ARRAY = [
90, 80, 75, 70, 65, 60, 55, 50, 45, 40, 35, 30, 25, 20, 15, 10,
];
// Max image size, I recommend 250kb
export const IMAGE_SIZE = 256000;
I also recommend converting all images into a lighter format such as jpeg or webp, in this tutorial I will use jpeg:
// ...
import sharp from 'sharp';
import {
IMAGE_SIZE,
MAX_WIDTH,
QUALITY_ARRAY,
} from './constants/uploader.constant';
@Injectable()
export class UploaderService {
// ...
private static async compressImage(
buffer: Buffer,
ratio?: number,
): Promise<Buffer> {
let compressBuffer: sharp.Sharp | Buffer = sharp(buffer).jpeg({
mozjpeg: true,
chromaSubsampling: '4:4:4',
});
if (ratio) {
compressBuffer.resize({
width: MAX_WIDTH,
height: Math.round(MAX_WIDTH * ratio),
fit: 'cover',
});
}
compressBuffer = await compressBuffer.toBuffer();
if (compressBuffer.length > IMAGE_SIZE) {
for (let i = 0; i < QUALITY_ARRAY.length; i++) {
const quality = QUALITY_ARRAY[i];
const smallerBuffer = await sharp(compressBuffer)
.jpeg({
quality,
chromaSubsampling: '4:4:4',
})
.toBuffer();
if (smallerBuffer.length <= IMAGE_SIZE || quality === 10) {
compressBuffer = smallerBuffer;
break;
}
}
}
return compressBuffer;
}
// ...
}
The ratio although not mandatory is a nice to have, so optionally you can create an enum with all common image ratios:
// ratio.enum.ts
export enum RatioEnum {
SQUARE = 1, // 192 x 192
MODERN = 9 / 16, // 1920 x 1080
MODERN_PORTRAIT = 16 / 9 // 1080 x 1920
OLD = 3 / 4, // 1400 x 1050
OLD_PORTRAIT = 4 / 3 // 1050 x 1400
BANNER = 8 / 47, // 1128 x 192
ULTRA_WIDE = 9 / 21, // 2560 x 1080
SUPER_WIDE = 9 / 32, // 3840 x 1080
}
This will add consistent with all image sizes on your application.
File upload
With the file buffer created, we just need to add a key to our file, and do a PutObjectCommand to our bucket:
import {
// ...
PutObjectCommand,
// ...
} from '@aws-sdk/client-s3';
import {
// ...
InternalServerErrorException,
// ...
} from '@nestjs/common';
import { v4 as uuidV4, v5 as uuidV5 } from 'uuid';
// ...
@Injectable()
export class UploaderService {
// ...
private async uploadFile(
userId: number,
fileBuffer: Buffer,
fileExt: string,
): Promise<string> {
const key =
this.bucketData.folder +
'/' +
uuidV5(userId.toString(), this.bucketData.appUuid) +
'/' +
uuidV4() +
fileExt;
try {
await this.client.send(
new PutObjectCommand({
Bucket: this.bucketData.name,
Body: fileBuffer,
Key: key,
ACL: 'public-read',
}),
);
} catch (error) {
this.loggerService.error(error);
throw new InternalServerErrorException('Error uploading file');
}
return this.bucketData.url + key;
}
// ...
}
I highly recommend using UUIDs both for the user ID and filename therefore if there is ever a breach on your bucket you are the only one that know which person owns which images.
I assumed that users are saved in a SQL database, if the user ID is not an int but a UUID or a MongoID, just change it to a string and it should work as expected.
Image upload
Putting all the previous private method together, we are able to create a public method that only accepts and uploads optimized images to our bucket:
// ...
import {
BadRequestException,
// ...
InternalServerErrorException,
// ...
} from '@nestjs/common';
import { RatioEnum } from './enums/ratio.enum';
// ...
@Injectable()
export class UploaderService {
// ...
/**
* Upload Image
*
* Converts an image to jpeg and uploads it to the bucket
*/
public async uploadImage(
userId: number,
file: Promise<FileUploadDto>,
ratio?: RatioEnum,
): Promise<string> {
const { mimetype, createReadStream } = await file;
const imageType = UploaderService.validateImage(mimetype);
if (!imageType) {
throw new BadRequestException('Please upload a valid image');
}
try {
return await this.uploadFile(
userId,
await UploaderService.compressImage(
await UploaderService.streamToBuffer(createReadStream()),
ratio,
),
'.jpg',
);
} catch (error) {
this.loggerService.error(error);
throw new InternalServerErrorException('Error uploading image');
}
}
// ...
}
File deletion
Files should be deleted asynchronously as it is often not a main event and can be done in the background:
import {
DeleteObjectCommand,
// ...
} from '@aws-sdk/client-s3';
// ...
@Injectable()
export class UploaderService {
// ...
/**
* Delete File
*
* Takes a file url and deletes the file from the bucket
*/
public deleteFile(url: string): void {
const keyArr = url.split('.com/');
if (keyArr.length !== 2 || !this.bucketData.url.includes(keyArr[0])) {
this.loggerService.error('Invalid url to delete file');
}
this.client
.send(
new DeleteObjectCommand({
Bucket: this.bucketData.name,
Key: keyArr[1],
}),
)
.then(() => this.loggerService.log('File deleted successfully'))
.catch((error) => this.loggerService.error(error));
}
// ...
}
Driver specific
Apollo Driver
Using the latest version of graphql-upload with NestJS can be quite tricky since it uses mjs, this means we need to use dynamic imports, alternative you can use graphql-upload-minimal, or graphql-upload version 13.
In order to cover both approaches I will divide this section in 2 parts, one for version 16 and one for version 13.
Version 16
Using version 16 is not the best approach for production projects as it forces us to use a lot of experimental features through our app.
Start by installing graphql-upload:
$ yarn add graphql-upload
Middleware Set up
On your main file dynamically import the graphql-upload/graphqlUploadExpress.mjs:
import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';
async function bootstrap() {
const app = await NestFactory.create(AppModule);
const configService = app.get(ConfigService);
// ...
const { default: graphqlUploadExpress } = await import(
'graphql-upload/graphqlUploadExpress.mjs'
);
app.use(graphqlUploadExpress(configService.get<IUploaderMiddlewareOptions>('uploader.middleware')));
// ...
app.useGlobalPipes(new ValidationPipe());
await app.listen(configService.get<number>('port'));
}
bootstrap();
Asynchronous Scalar
The scalar now will be a function that you add to your Field and Args decorators:
// upload-scalar.util.ts
import { GraphQLScalarType } from 'graphql/type';
let GraphQLUpload: GraphQLScalarType;
import('graphql-upload/GraphQLUpload.mjs').then(({ default: Upload }) => {
GraphQLUpload = Upload;
});
export const uploadScalar = () => GraphQLUpload;
In the end a DTO (ArgsType) would look something like this:
import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import { FileUploadDto } from './file-upload.dto';
import { uploadScalar } from '../utils/upload-scalar.util';
@ArgsType()
export abstract class PictureDto {
@Field(uploadScalar)
@ValidatePromise()
@Type(() => FileUploadDto)
public picture: Promise<FileUploadDto>;
}
Testing
Using mjs will break your tests, so to be able to use mjs on your app, add the --experimental-vm-modules flag to jest:
{
"...": "...",
"scripts": {
"...": "...",
"test": "yarn node --experimental-vm-modules $(yarn bin jest)",
"test:watch": "yarn node --experimental-vm-modules $(yarn bin jest) --watch",
"test:cov": "yarn node --experimental-vm-modules $(yarn bin jest) --coverage",
"test:e2e": "yarn node --experimental-vm-modules $(yarn bin jest) --config ./test/jest-e2e.json",
"...": "..."
},
"...": "..."
}
Version 13
Using version 13 is what I recommend, start by installing the package:
$ yarn add graphql-upload@13
$ yarn add -D @types/graphql-upload
Middleware Set up
With version 13 we can just do a normal import:
import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import { graphqlUploadExpress } from 'graphql-upload';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';
async function bootstrap() {
const app = await NestFactory.create(AppModule);
const configService = app.get(ConfigService);
// ...
app.use(graphqlUploadExpress(configService.get<IUploaderMiddlewareOptions>('uploader.middleware')));
// ...
app.useGlobalPipes(new ValidationPipe());
await app.listen(configService.get<number>('port'));
}
bootstrap();
DTOs
The GraphQLUpload scalar will just be an export from graphql-upload:
import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import { GraphQLUpload } from 'graphql-upload';
import { FileUploadDto } from './file-upload.dto';
@ArgsType()
export abstract class PictureDto {
@Field(() => GraphQLUpload)
@ValidatePromise()
@Type(() => FileUploadDto)
public picture: Promise<FileUploadDto>;
}
Mercurius
Mercurius has its own adaptation of graphql-upload, start by installing the mercurius-upload package:
$ yarn add mercurius-upload
Middleware Set up
As any fastify middleware you just need to import the default value of the library:
Note: do not forget to set "esModuleInterop": true on your tsconfig.json file
import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import {
FastifyAdapter,
NestFastifyApplication,
} from '@nestjs/platform-fastify';
import mercuriusUpload from 'mercurius-upload';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';
async function bootstrap() {
const app = await NestFactory.create<NestFastifyApplication>(
AppModule,
new FastifyAdapter(),
);
const configService = app.get(ConfigService);
// ...
app.register(
mercuriusUpload,
configService.get<IUploaderMiddlewareOptions>('uploader.middleware'),
);
// ...
app.useGlobalPipes(new ValidationPipe());
await app.listen(
configService.get<number>('port'),
'0.0.0.0',
);
}
bootstrap();
DTOs
mercurius-upload is dependent on graphql-upload version 15 so you need to import default the scalar from graphql-upload/GraphQLUpload.js:
import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import GraphQLUpload from 'graphql-upload/GraphQLUpload.js';
import { FileUploadDto } from './file-upload.dto';
@ArgsType()
export abstract class PictureDto {
@Field(() => GraphQLUpload)
@ValidatePromise()
@Type(() => FileUploadDto)
public picture: Promise<FileUploadDto>;
}
Conclusion
A complete version of this code can be found in this repository.
About the Author
Hey there my name is Afonso Barracha, I am a Econometrician made back-end developer that has a passion for GraphQL.
I try to do post once a week here on Dev about Back-End APIs and related topics.
If you do not want to lose any of my posts follow me here on dev, or on LinkedIn.
Top comments (3)
github.com/tugascript/nestjs-graph... code is not complete. main.ts, app.ts, uploader resolver files are missing.
Yeah I wanted the code to be platform agnostic, so I only added a plug and play library that you can add to any project, microservice or monolith alike, that works both for mercurius and apollo.
So there has been an update to the @types/graphql-upload so you can now use version 15 and not version 13. The way you import the middleware and the Scalar type is a bit different but it is easy to transition to.