Solving Performance Issues with Redis and Bull

Introduction

In a fintech environment, system performance is crucial to ensuring that financial transactions are quick and secure. The ability to process a large volume of simultaneous requests without compromising user experience or overloading infrastructure is one of the main challenges. To achieve this, tools like Redis and Bull have played a fundamental role in our fintech.

In this article, I will share how we used Redis and Bull to solve performance problems, improve transaction response times, and ensure that our financial system remains scalable and efficient. I will also include a practical guide on how to implement Redis and Bull in NestJS.

The Problem: Slowness and System Overload

With the growth of our user base and the increase in daily transactions, we began to notice a drop in system performance, especially during peak hours. Operations such as payment processing, report generation, and notifications were starting to slow down, and server overload was becoming a real threat.

Additionally, the need to process each transaction synchronously was beginning to create bottlenecks, increasing response times for end users. We needed a way to optimize the processing of these tasks to avoid service degradation.

Solution 1: Optimization with Redis

To enhance performance and reduce latency in read and write operations, we implemented Redis as a caching solution. Redis is an in-memory database that allows for extremely fast access and writes, making it ideal for storing temporary or frequently accessed data.

Using Redis in Our Infrastructure

• Session and Authentication Cache: Instead of making constant database queries to check active sessions, we started storing authentication tokens in Redis. This significantly reduced latency in user authentications and transaction processing.

• Temporary Transaction Cache: During payment processing, transaction data is temporarily stored in Redis before being written to the main database. This allowed us to process transactions much faster while the system handled critical operations asynchronously.

• Reduction of Database Queries: We stored frequently accessed data in Redis, such as configuration information, payment states, and temporary logs. This reduced the load on the relational database and improved response times for end users.

Results

The implementation of Redis significantly reduced latency in several critical parts of our system, especially in operations that rely on quick and repeated queries. The improvement was noticeable on both the front-end, with faster response times, and the back-end, which handled the increased requests better.

Solution 2: Asynchronous Processing with Bull

Even with optimization through Redis, we found that some operations needed to be decoupled from the main flow, particularly tasks that required more processing time, such as transaction notifications, report generation, and integration with external systems.

This is where we implemented Bull, a queue management library based on Redis, which allowed us to process tasks asynchronously without blocking the main execution flow.

Using Bull for Queue Management

• Asynchronous Transaction Processing: Instead of processing all transactions synchronously, we moved parts of the process to a task queue with Bull. This allowed our application to continue responding quickly to users while longer tasks were processed in the background.

• Real-Time Notifications: One of the first applications of Bull was in the queue for sending notifications. Since sending notifications via email or push can be time-consuming and involves calls to external APIs, we moved this process to a queue, ensuring that users received their updates without impacting the overall platform performance.

• Retry and Failure Management: Bull also allowed us to manage task failures efficiently. When a task failed, it was automatically reprocessed after a configured time, without the need for manual intervention, ensuring greater system resilience.

Results

By introducing Bull, we were able to distribute the workload more efficiently and avoid blocking the main application flow. This resulted in a significant improvement in response times, even during peak usage times. Furthermore, we were able to scale more predictably, as tasks were processed according to the system's capacity.

Practical Implementation: Integrating Redis and Bull in NestJS

Below, I present a step-by-step guide on how to integrate Redis and Bull into a NestJS application, based on the lessons learned from our practical experience.

1. Installing Dependencies

First, install the Bull and Redis packages in your NestJS project:

npm install @nestjs/bull bull redis
npm install @types/redis --save-dev

2. Configuring the Bull Module

Create a module to configure Bull with Redis. In queue.module.ts, register Bull:

import { BullModule } from '@nestjs/bull';
import { Module } from '@nestjs/common';
import { QueueProcessor } from './queue.processor';
import { QueueService } from './queue.service';

@Module({
  imports: [
    BullModule.forRoot({
      redis: {
        host: 'localhost',
        port: 6379,
      },
    }),
    BullModule.registerQueue({
      name: 'email', // Name of the queue
    }),
  ],
  providers: [QueueProcessor, QueueService],
  exports: [QueueService],
})
export class QueueModule {}

3. Creating the Queue Processor

Create queue.processor.ts to process tasks in the queue:

import { Processor, Process } from '@nestjs/bull';
import { Job } from 'bull';

@Processor('email') // Name of the queue we will process
export class QueueProcessor {
  @Process() // Here we handle the processing of each job
  async handleEmailJob(job: Job) {
    console.log(`Processing job #${job.id} with data:`, job.data);

    // Simulate sending email
    const { to, subject, text } = job.data;
    await this.sendEmail(to, subject, text);

    console.log('Email sent successfully');
  }

  private async sendEmail(to: string, subject: string, text: string) {
    // Simulate sending email (place your logic here)
    return new Promise((resolve) => {
      setTimeout(() => {
        console.log(`Email sent to: ${to}, with subject: ${subject}`);
        resolve(true);
      }, 3000);
    });
  }
}

4. Service for Adding Tasks to the Queue

Create a service to add new tasks to the queue in queue.service.ts:

import { Injectable } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bull';
import { Queue } from 'bull';

@Injectable()
export class QueueService {
  constructor(@InjectQueue('email') private emailQueue: Queue) {}

  async addEmailJob(to: string, subject: string, text: string) {
    await this.emailQueue.add({
      to,
      subject,
      text,
    });
  }
}

5. Using the Service in the Controller

Create a controller to expose this functionality. In email.controller.ts:

import { Controller, Post, Body } from '@nestjs/common';
import { QueueService } from './queue.service';

@Controller('email')
export class EmailController {
  constructor(private readonly queueService: QueueService) {}

  @Post('send')
  async sendEmail(
    @Body('to') to: string,
    @Body('subject') subject: string,
    @Body('text') text: string,
  ) {
    await this.queueService.addEmailJob(to, subject, text);
    return { message: 'Email added to the queue!' };
  }
}

Testing the Implementation

1. Start Redis Locally: Ensure that Redis is running locally. You can use Docker to easily start Redis:

   docker run -p 6379:6379 redis

1. Run the Application: With Redis running, execute the NestJS application with the command:

   npm run start:dev

1. Send a Request: Now you can test the API using a tool like Postman or cURL, sending a POST request to http://localhost:3000/email/send with the following body:

   {
     "to": "user@example.com",
     "subject": "Email Test",
     "text": "This is a test email."
   }

If everything is configured correctly, you will see console logs showing that the job was added to the queue and that the processor handled it.

General Benefits of the Implementation

This is a practical example of how to integrate Redis and Bull in NestJS to manage asynchronous tasks effectively. We use Redis as a foundation for Bull, which is responsible for efficiently managing task queues.

Conclusion

In this example, we demonstrated how to integrate Redis and Bull into a NestJS project to create a robust queue system where lengthy tasks are processed asynchronously. This approach is ideal for systems that need to manage large volumes of transactions or other operations requiring high performance and resilience.