Intro
In this tutorial I will explain how to upload images to a S3-compatible object storage (AWS S3, DigitalOcean Spaces, Linode Object Storage, etc.) bucket using NestJS and GraphQL for both Apollo and Mercurius drivers.
Common
Most of the set-up as well as the entirety of the upload process is the same for both drivers.
Set Up
Start by installing the class-transformer and class-validator packages for dto validation (see more in the docs), sharp for image optimization, and the S3 client packages.
Note: I assume you have UUID installed, however I still added it to the following command
$ yarn add class-transformer class-validator sharp @aws-sdk/client-s3 @aws-sdk/signature-v4-crt uuid
$ yarn add -D @types/sharp @types/uuid
Configuration
Assuming you are using nestjs ConfigModule start by creating an interfaces folder.
Bucket Data:
// bucked-data.interface.ts
export interface IBucketData {
// name of the bucket
name: string;
// folder of the service
folder: string;
// uuid for the app
appUuid: string;
// the bucket url to create the key
url: string;
}
Upload Middleware Options:
// uploader-middleware-options.interface.ts
export interface IUploaderMiddlewareOptions {
maxFieldSize?: number;
maxFileSize?: number;
maxFiles?: number;
}
Uploader Options:
// uploader-options.interface.ts
import type { S3ClientConfig } from '@aws-sdk/client-s3';
import type { IBucketData } from './bucket-data.interface';
import type { IUploaderMiddlewareOptions } from './uploader-middleware-options.interface';
export interface IUploaderOptions {
clientConfig: S3ClientConfig;
bucketData: IBucketData;
middleware: IUploaderMiddlewareOptions;
}
On the config.ts function add the following:
import { IConfig } from './interfaces/config.interface';
export function config(): IConfig {
const bucketBase = `${process.env.BUCKET_REGION}.${process.env.BUCKET_HOST}.com`;
return {
// ...
uploader: {
clientConfig: {
forcePathStyle: false,
region: process.env.BUCKET_REGION,
endpoint: `https://${bucketBase}`,
credentials: {
accessKeyId: process.env.BUCKET_ACCESS_KEY,
secretAccessKey: process.env.BUCKET_SECRET_KEY,
},
},
bucketData: {
name: process.env.BUCKET_NAME,
folder: process.env.FILE_FOLDER,
appUuid: process.env.SERVICE_ID,
url: `https://${process.env.BUCKET_NAME}.${bucketBase}/`,
},
middleware: {
maxFileSize: parseInt(process.env.MAX_FILE_SIZE, 10),
maxFiles: parseInt(process.env.MAX_FILES, 10),
},
},
// ...
};
}
Module Creation
The uploader module set up changes depending if it is a library on a nestjs (or nx) mono-repo, or a monolith NestJS GraphQL API.
Monolith API
Create the uploader module and service:
$ nest g mo uploader
$ nest g s uploader
On the uploader.service.ts file add the S3 Client, the bucket data and the logger:
import { S3Client, S3ClientConfig } from '@aws-sdk/client-s3';
import { Injectable, Logger, LoggerService } from '@nestjs/common';
@Injectable()
export class UploaderService {
private readonly client: S3Client;
private readonly bucketData: IBucketData;
private readonly loggerService: LoggerService;
constructor(
private readonly configService: ConfigService,
) {
this.client = new S3Client(
this.configService.get<S3ClientConfig>('uploader.clientConfig'),
);
this.bucketData = this.configService.get<IBucketData>('uploader.bucketData');
this.loggerService = new Logger(UploaderService.name);
}
// ...
}
On the uploader.module.ts file add the global decorator:
import { Global, Module } from '@nestjs/common';
import { UploaderService } from './uploader.service';
@Global()
@Module({
providers: [UploaderService],
exports: [UploaderService],
})
export class UploaderModule {}
Library
Create the uploader library:
-
NestJS mono-repo:
$ nest g lib uploader -
NX mono-repo:
$ npx nx @nrwl/nest:library uploader --global --service --strict --no-interactive
To be able to create a DynamicModule, we need to create a library options, start by creating an interfaces folder and move the bucket-data.interface.ts, and add an options.interface.ts:
import type { S3ClientConfig } from '@aws-sdk/client-s3';
import type { IBucketData } from './bucket-data.interface';
export interface IOptions {
clientConfig: S3ClientConfig;
bucketData: IBucketData;
}
Create a new constants folder and add the options constant:
// options.constant.ts
export const UPLOADER_OPTIONS = 'UPLOADER_OPTIONS';
Unlike a normal module the class parameters come from the options passed to the module and not from the ConfigService:
import { S3Client, S3ClientConfig } from '@aws-sdk/client-s3';
import { Injectable, Logger, LoggerService } from '@nestjs/common';
import { UPLOADER_OPTIONS } from './constants/options.constant';
import { IBucketData } from './interfaces/bucket-data.interface';
import { IOptions } from './interfaces/options.interface';
@Injectable()
export class UploaderService {
private readonly client: S3Client;
private readonly bucketData: IBucketData;
private readonly loggerService: LoggerService;
constructor(@Inject(UPLOADER_OPTIONS) options: IOptions) {
this.client = new S3Client(options.clientConfig);
this.bucketData = options.bucketData;
this.loggerService = new Logger(UploaderService.name);
}
// ...
}
Turn the module into a dynamic global module:
import { DynamicModule, Global, Module } from '@nestjs/common';
import { UPLOADER_OPTIONS } from './constants/options.constant';
import { IOptions } from './interfaces/options.interface';
import { UploaderService } from './uploader.service';
@Global()
@Module({})
export class UploaderModule {
public static forRoot(options: IOptions): DynamicModule {
return {
global: true,
module: UploaderModule,
providers: [
{
provide: UPLOADER_OPTIONS,
useValue: options,
},
UploaderService,
],
exports: [UploaderService],
};
}
}
Image Uploading
Upload Scalar DTO
The graphql-upload package Upload scalar will be process into the following DTO, so add it into a dtos folder:
import { IsMimeType, IsString } from 'class-validator';
import { ReadStream } from 'fs';
export abstract class FileUploadDto {
@IsString()
public filename!: string;
@IsString()
@IsMimeType()
public mimetype!: string;
@IsString()
public encoding!: string;
public createReadStream: () => ReadStream;
}
File Validation
Since we want to upload only images the first thing we need to check is the mimetype:
//...
@Injectable()
export class UploaderService {
// ...
private static validateImage(mimetype: string): string | false {
const val = mimetype.split('/');
if (val[0] !== 'image') return false;
return val[1] ?? false;
}
// ...
}
Stream Processing
As you saw the Upload scalar returns a read stream that we need to transform into a buffer:
//...
import { Readable } from 'stream';
@Injectable()
export class UploaderService {
// ...
private static async streamToBuffer(stream: Readable): Promise<Buffer> {
const buffer: Uint8Array[] = [];
return new Promise((resolve, reject) =>
stream
.on('error', (error) => reject(error))
.on('data', (data) => buffer.push(data))
.on('end', () => resolve(Buffer.concat(buffer))),
);
}
// ...
}
Image Compression and Conversion
Before uploading the image we need to optimize it by compressing the image. Start by creating the constants necessary for compression, therefore on a constants folder add a file named uploader.constant.ts:
// The max width an image can have
// reducing the pixels will reduce the size of the image
export const MAX_WIDTH = 2160;
// an array with the percentage of quality ranging from 90 to 10%
export const QUALITY_ARRAY = [
90, 80, 75, 70, 65, 60, 55, 50, 45, 40, 35, 30, 25, 20, 15, 10,
];
// Max image size, I recommend 250kb
export const IMAGE_SIZE = 256000;
I also recommend converting all images into a lighter format such as jpeg or webp, in this tutorial I will use jpeg:
// ...
import sharp from 'sharp';
import {
IMAGE_SIZE,
MAX_WIDTH,
QUALITY_ARRAY,
} from './constants/uploader.constant';
@Injectable()
export class UploaderService {
// ...
private static async compressImage(
buffer: Buffer,
ratio?: number,
): Promise<Buffer> {
let compressBuffer: sharp.Sharp | Buffer = sharp(buffer).jpeg({
mozjpeg: true,
chromaSubsampling: '4:4:4',
});
if (ratio) {
compressBuffer.resize({
width: MAX_WIDTH,
height: Math.round(MAX_WIDTH * ratio),
fit: 'cover',
});
}
compressBuffer = await compressBuffer.toBuffer();
if (compressBuffer.length > IMAGE_SIZE) {
for (let i = 0; i < QUALITY_ARRAY.length; i++) {
const quality = QUALITY_ARRAY[i];
const smallerBuffer = await sharp(compressBuffer)
.jpeg({
quality,
chromaSubsampling: '4:4:4',
})
.toBuffer();
if (smallerBuffer.length <= IMAGE_SIZE || quality === 10) {
compressBuffer = smallerBuffer;
break;
}
}
}
return compressBuffer;
}
// ...
}
The ratio although not mandatory is a nice to have, so optionally you can create an enum with all common image ratios:
// ratio.enum.ts
export enum RatioEnum {
SQUARE = 1, // 192 x 192
MODERN = 9 / 16, // 1920 x 1080
MODERN_PORTRAIT = 16 / 9 // 1080 x 1920
OLD = 3 / 4, // 1400 x 1050
OLD_PORTRAIT = 4 / 3 // 1050 x 1400
BANNER = 8 / 47, // 1128 x 192
ULTRA_WIDE = 9 / 21, // 2560 x 1080
SUPER_WIDE = 9 / 32, // 3840 x 1080
}
This will add consistent with all image sizes on your application.
File upload
With the file buffer created, we just need to add a key to our file, and do a PutObjectCommand to our bucket:
import {
// ...
PutObjectCommand,
// ...
} from '@aws-sdk/client-s3';
import {
// ...
InternalServerErrorException,
// ...
} from '@nestjs/common';
import { v4 as uuidV4, v5 as uuidV5 } from 'uuid';
// ...
@Injectable()
export class UploaderService {
// ...
private async uploadFile(
userId: number,
fileBuffer: Buffer,
fileExt: string,
): Promise<string> {
const key =
this.bucketData.folder +
'/' +
uuidV5(userId.toString(), this.bucketData.appUuid) +
'/' +
uuidV4() +
fileExt;
try {
await this.client.send(
new PutObjectCommand({
Bucket: this.bucketData.name,
Body: fileBuffer,
Key: key,
ACL: 'public-read',
}),
);
} catch (error) {
this.loggerService.error(error);
throw new InternalServerErrorException('Error uploading file');
}
return this.bucketData.url + key;
}
// ...
}
I highly recommend using UUIDs both for the user ID and filename therefore if there is ever a breach on your bucket you are the only one that know which person owns which images.
I assumed that users are saved in a SQL database, if the user ID is not an int but a UUID or a MongoID, just change it to a string and it should work as expected.
Image upload
Putting all the previous private method together, we are able to create a public method that only accepts and uploads optimized images to our bucket:
// ...
import {
BadRequestException,
// ...
InternalServerErrorException,
// ...
} from '@nestjs/common';
import { RatioEnum } from './enums/ratio.enum';
// ...
@Injectable()
export class UploaderService {
// ...
/**
* Upload Image
*
* Converts an image to jpeg and uploads it to the bucket
*/
public async uploadImage(
userId: number,
file: Promise<FileUploadDto>,
ratio?: RatioEnum,
): Promise<string> {
const { mimetype, createReadStream } = await file;
const imageType = UploaderService.validateImage(mimetype);
if (!imageType) {
throw new BadRequestException('Please upload a valid image');
}
try {
return await this.uploadFile(
userId,
await UploaderService.compressImage(
await UploaderService.streamToBuffer(createReadStream()),
ratio,
),
'.jpg',
);
} catch (error) {
this.loggerService.error(error);
throw new InternalServerErrorException('Error uploading image');
}
}
// ...
}
File deletion
Files should be deleted asynchronously as it is often not a main event and can be done in the background:
import {
DeleteObjectCommand,
// ...
} from '@aws-sdk/client-s3';
// ...
@Injectable()
export class UploaderService {
// ...
/**
* Delete File
*
* Takes a file url and deletes the file from the bucket
*/
public deleteFile(url: string): void {
const keyArr = url.split('.com/');
if (keyArr.length !== 2 || !this.bucketData.url.includes(keyArr[0])) {
this.loggerService.error('Invalid url to delete file');
}
this.client
.send(
new DeleteObjectCommand({
Bucket: this.bucketData.name,
Key: keyArr[1],
}),
)
.then(() => this.loggerService.log('File deleted successfully'))
.catch((error) => this.loggerService.error(error));
}
// ...
}
Driver specific
Apollo Driver
Using the latest version of graphql-upload with NestJS can be quite tricky since it uses mjs, this means we need to use dynamic imports, alternative you can use graphql-upload-minimal, or graphql-upload version 13.
In order to cover both approaches I will divide this section in 2 parts, one for version 16 and one for version 13.
Version 16
Using version 16 is not the best approach for production projects as it forces us to use a lot of experimental features through our app.
Start by installing graphql-upload:
$ yarn add graphql-upload
Middleware Set up
On your main file dynamically import the graphql-upload/graphqlUploadExpress.mjs:
import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';
async function bootstrap() {
const app = await NestFactory.create(AppModule);
const configService = app.get(ConfigService);
// ...
const { default: graphqlUploadExpress } = await import(
'graphql-upload/graphqlUploadExpress.mjs'
);
app.use(graphqlUploadExpress(configService.get<IUploaderMiddlewareOptions>('uploader.middleware')));
// ...
app.useGlobalPipes(new ValidationPipe());
await app.listen(configService.get<number>('port'));
}
bootstrap();
Asynchronous Scalar
The scalar now will be a function that you add to your Field and Args decorators:
// upload-scalar.util.ts
import { GraphQLScalarType } from 'graphql/type';
let GraphQLUpload: GraphQLScalarType;
import('graphql-upload/GraphQLUpload.mjs').then(({ default: Upload }) => {
GraphQLUpload = Upload;
});
export const uploadScalar = () => GraphQLUpload;
In the end a DTO (ArgsType) would look something like this:
import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import { FileUploadDto } from './file-upload.dto';
import { uploadScalar } from '../utils/upload-scalar.util';
@ArgsType()
export abstract class PictureDto {
@Field(uploadScalar)
@ValidatePromise()
@Type(() => FileUploadDto)
public picture: Promise<FileUploadDto>;
}
Testing
Using mjs will break your tests, so to be able to use mjs on your app, add the --experimental-vm-modules flag to jest:
{
"...": "...",
"scripts": {
"...": "...",
"test": "yarn node --experimental-vm-modules $(yarn bin jest)",
"test:watch": "yarn node --experimental-vm-modules $(yarn bin jest) --watch",
"test:cov": "yarn node --experimental-vm-modules $(yarn bin jest) --coverage",
"test:e2e": "yarn node --experimental-vm-modules $(yarn bin jest) --config ./test/jest-e2e.json",
"...": "..."
},
"...": "..."
}
Version 13
Using version 13 is what I recommend, start by installing the package:
$ yarn add graphql-upload@13
$ yarn add -D @types/graphql-upload
Middleware Set up
With version 13 we can just do a normal import:
import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import { graphqlUploadExpress } from 'graphql-upload';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';
async function bootstrap() {
const app = await NestFactory.create(AppModule);
const configService = app.get(ConfigService);
// ...
app.use(graphqlUploadExpress(configService.get<IUploaderMiddlewareOptions>('uploader.middleware')));
// ...
app.useGlobalPipes(new ValidationPipe());
await app.listen(configService.get<number>('port'));
}
bootstrap();
DTOs
The GraphQLUpload scalar will just be an export from graphql-upload:
import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import { GraphQLUpload } from 'graphql-upload';
import { FileUploadDto } from './file-upload.dto';
@ArgsType()
export abstract class PictureDto {
@Field(() => GraphQLUpload)
@ValidatePromise()
@Type(() => FileUploadDto)
public picture: Promise<FileUploadDto>;
}
Mercurius
Mercurius has its own adaptation of graphql-upload, start by installing the mercurius-upload package:
$ yarn add mercurius-upload
Middleware Set up
As any fastify middleware you just need to import the default value of the library:
Note: do not forget to set "esModuleInterop": true on your tsconfig.json file
import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import {
FastifyAdapter,
NestFastifyApplication,
} from '@nestjs/platform-fastify';
import mercuriusUpload from 'mercurius-upload';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';
async function bootstrap() {
const app = await NestFactory.create<NestFastifyApplication>(
AppModule,
new FastifyAdapter(),
);
const configService = app.get(ConfigService);
// ...
app.register(
mercuriusUpload,
configService.get<IUploaderMiddlewareOptions>('uploader.middleware'),
);
// ...
app.useGlobalPipes(new ValidationPipe());
await app.listen(
configService.get<number>('port'),
'0.0.0.0',
);
}
bootstrap();
DTOs
mercurius-upload is dependent on graphql-upload version 15 so you need to import default the scalar from graphql-upload/GraphQLUpload.js:
import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import GraphQLUpload from 'graphql-upload/GraphQLUpload.js';
import { FileUploadDto } from './file-upload.dto';
@ArgsType()
export abstract class PictureDto {
@Field(() => GraphQLUpload)
@ValidatePromise()
@Type(() => FileUploadDto)
public picture: Promise<FileUploadDto>;
}
Conclusion
A complete version of this code can be found in this repository.
About the Author
Hey there my name is Afonso Barracha, I am a Econometrician made back-end developer that has a passion for GraphQL.
I try to do post once a week here on Dev about Back-End APIs and related topics.
If you do not want to lose any of my posts follow me here on dev, or on LinkedIn.
Oldest comments (3)
So there has been an update to the @types/graphql-upload so you can now use version 15 and not version 13. The way you import the middleware and the Scalar type is a bit different but it is easy to transition to.
github.com/tugascript/nestjs-graph... code is not complete. main.ts, app.ts, uploader resolver files are missing.
Yeah I wanted the code to be platform agnostic, so I only added a plug and play library that you can add to any project, microservice or monolith alike, that works both for mercurius and apollo.