Joan Roucoux

Posted on Nov 8

Building a Scalable URL Shortener with Node.js (Part 1/2)

#node #redis #mongodb #zookeeper

Introduction

I'm sure you're familiar with URL shortener tools like TinyURL and Bitly, as they are widely used online. It simply takes a long URL and creates a shorter, unique alias that redirects to the original link.

For example, a URL like https://www.example.com/movies/avengers-infinity-war/casting/chris-hemsworth could be shortened to something like https://short.url/abc123, making it much easier to read and share. And then, when the user accesses the shortened link, the browser requests the server, which looks up the alias in its database and redirects the user to the original URL.

This might look simple when you first think about it, but it's actually tricky to build which is why they are commonly used in system design interviews. So creating one yourself is a great way to practice and see the challenges involved (handling high traffic, concurrency and race conditions in key generation, data storage and scaling...).

In the first part of this tutorial, we will focus on developing the backend of our URL shortener application using the following stack:

Node.js: A JavaScript runtime to power the server instances.
MongoDB: A NoSQL database for storing original URLs.
Redis: In-memory data store for caching frequently accessed URLs.
Apache ZooKeeper: A centralized service to generate unique IDs and prevent race conditions between instances.
Nginx: A load balancer and reverse proxy to distribute traffic across server instances.
Docker: A containerization tool to manage all the services.

Here's the high-level architecture of the application:

So let's jump into it 🚀

Initializing the Project

First, let's see what our file tree will look like:

url-shortener-demo-app
├─ .env
├─ docker-compose.yml
├─ client - Part 2 of this tutorial
├─ nginx
│  ├─ Dockerfile
│  └─ nginx.conf
└─ server
   ├─ src
   │  ├─ config
   │  │  ├─ mongoose.ts
   │  │  ├─ redis.ts
   │  │  └─ zookeeper.ts
   │  ├─ controllers
   │  │  └─ urlsController.ts
   │  ├─ models
   │  │  └─ Url.ts
   │  ├─ repositories
   │  │  └─ urlsRepository.ts
   │  ├─ routes
   │  │  └─ urlsRoutes.ts
   │  ├─ services
   │  │  └─ urlsService.ts
   │  ├─ utils
   │  │  └─ index.ts
   │  └─ index.ts
   ├─ .dockerignore
   ├─ Dockerfile
   ├─ package-lock.json
   ├─ package.json
   └─ tsconfig.json

Our project app url-shortener-demo-app will include a client folder for the React app that we will build in the second part of this tutorial, a nginx folder for Nginx configuration and a server folder for managing URL-related operations. It's a good thing to separate concepts like routers, controllers and services to enhance code organization and improve maintainability.

We will first start with the server code. Create a new directory for the project and initialize a new application:

mkdir url-shortener-demo-app
cd url-shortener-demo-app
mkdir server
cd server
npm init -y

Then, install all the required dependencies:

npm install fastify @fastify/cors mongoose ioredis zookeeper
npm install --save-dev typescript @types/node ts-node

The packages you installed above include:

fastify: A simple web framework for handling routes and middleware.
@fastify/cors: Middleware for enabling Cross-Origin Resource Sharing (CORS).
zookeeper: A client for interacting with Apache ZooKeeper.
mongoose: An Object Data Modeling (ODM) library for MongoDB.
ioredis: A client for interacting with Redis.
And some devDependencies for Typescript support.

Next, open the package.json file and a in the scripts section a new script to run the application:

...
"scripts": {
  "start": "ts-node src/index.ts"
},
...

And finally, create a tsconfig.json file for Typescript configuration by running:

tsc --init

Great, we can now move on and add the configuration files for MongoDB, Redis and ZooKeeper.

Setting Up MongoDB

Using mongoose in our application makes it easier to create and manage data within MongoDB, providing a more convenient and structured approach.

In src/config/mongoose.ts, add the following code to configure our MongoDB connection:

import { connect } from 'mongoose';

const {
  MONGODB_USER,
  MONGODB_PASSWORD,
  MONGODB_DATABASE,
  MONGODB_HOST,
  MONGODB_DOCKER_PORT,
} = process.env;

const MONGO_URI = `mongodb://${MONGODB_USER}:${MONGODB_PASSWORD}@${MONGODB_HOST}:${MONGODB_DOCKER_PORT}/${MONGODB_DATABASE}?authSource=admin`;

// Connect to MongoDB
export const connectToMongoDB = async (): Promise<void> => {
  try {
    await connect(MONGO_URI);
    console.log('Successfully connected to MongoDB');
  } catch (error) {
    console.error('Error connecting to MongoDB:', error);
    throw error;
  }
};

We're loading some environment variables from the .env file in the root directory, so be sure to create it and add the following keys:

MONGODB_USER=user
MONGODB_PASSWORD=pass
MONGODB_DATABASE=urls
MONGODB_HOST=mongo
MONGODB_LOCAL_PORT=27017
MONGODB_DOCKER_PORT=27017

Then, create the URL data model, which will define the structure of our documents in the database. This model will include the following attributes:

originalUrl: The original URL that users want to shorten.
shortenUrlKey: A unique identifier generated for the shortened URL.
createdAt: A timestamp indicating when the URL was created.
expiresAt: A timestamp indicating when the shortened URL will expire.

In src/models/Url.ts, add the following code:

import { Document, Schema, model } from 'mongoose';

export interface IUrl extends Document {
  originalUrl: string;
  shortenUrlKey: string;
  createdAt: Date;
  expiresAt: Date;
}

const schema = new Schema<IUrl>({
  originalUrl: {
    type: String,
    required: true,
    unique: true,
  },
  shortenUrlKey: {
    type: String,
    required: true,
    unique: true,
  },
  createdAt: {
    type: Date,
    default: new Date(),
  },
  expiresAt: {
    type: Date,
    default: new Date(new Date().setMinutes(new Date().getMinutes() + 10)), // default is 10 minutes, for demonstration only
  },
});

export default model<IUrl>('url', schema);

Setting Up Redis

In the same way that we use mongoose to interact with our MongoDB database, we will use ioredis client to interact with Redis and manage frequently accessed URLs by leveraging caching mechanisms for faster retrieval.

We will override some of the initial Redis methods to:

Add a new entry in Redis cache with set().
Retrieve a Redis cache entry with get().
Extend TTL of an existing entry with extendTTL().

In src/config/redis.ts, add the following code:

import { Redis } from 'ioredis';

const { REDIS_HOST, REDIS_DOCKER_PORT } = process.env;

export enum RedisExpirationMode {
  EX = 'EX', // Expire in seconds
}

let client: Redis | null;

// Get Redis client
const getRedisClient = (): Redis => {
  if (!client) {
    const config = {
      host: REDIS_HOST,
      port: Number(REDIS_DOCKER_PORT),
      maxRetriesPerRequest: null,
    };

    client = new Redis(config);
  }

  return client;
};

// Connect to Redis
export const connectToRedis = async (): Promise<void> => {
  const client = getRedisClient();

  client
    .on('connect', () => {
      console.log('Successfully connected to Redis');
    })
    .on('error', (error) => {
      console.error('Error on Redis:', error.message);
    });
};

// Set a key/value pair
export const set = async (
  key: string,
  value: string,
  expirationMode: RedisExpirationMode,
  seconds: number
): Promise<void> => {
  try {
    await getRedisClient().set(key, value, expirationMode, seconds);
    console.info(`Key ${key} created in Redis cache`);
  } catch (error) {
    console.error(`Failed to create key in Redis cache: ${error}`);
  }
};

// Get a value from a key
export const get = async (key: string): Promise<string | null> => {
  try {
    const value = await getRedisClient().get(key);
    console.info(`Value with key ${key} retrieved from Redis cache`);
    return value;
  } catch (error) {
    console.error(
      `Failed to retrieve value with key ${key} in Redis cache: ${error}`
    );
    return null;
  }
};

// Extend TTL of a key
export const extendTTL = async (
  key: string,
  additionalTimeInSeconds: number
) => {
  // Get the current TTL of the key
  const currentTTL = await getRedisClient().ttl(key);

  if (currentTTL > 0) {
    // Calculate the new TTL
    const newTTL = currentTTL + additionalTimeInSeconds;

    // Set the new TTL
    await getRedisClient().expire(key, newTTL);
    console.info(`TTL for key ${key} extended to ${newTTL} in Redis cache`);
  } else {
    console.error(`Failed to extend TTL of key ${key} in Redis cache`);
  }
};

Finally, add the following keys to the .env file to configure Redis service:

REDIS_HOST=redis
REDIS_LOCAL_PORT=6379
REDIS_DOCKER_PORT=6379

Setting Up ZooKeeper

Apache ZooKeeper will help us avoid race conditions by making sure that only one node generates a token at a time, maintaining data integrity.

One approach could be to assign each server registered to the ZooKeeper service a specific token range and generate a token within that range. However, I chose a simpler solution: checking if a token already exists under the /tokens path. If the token is not found, it will attempt to generate a new token repeatedly until it finds one that is available.

For instance, if /tokens/existingToken already exists, it will try again and register /tokens/newToken if available. In our example, we will use a token that is 6 characters long, which gives us around 69 billion possibilities (64^6). This should provide a comfortable buffer before we encounter any collisions in our demo app.

First, add a method to generate a base64 token in src/utils/index.ts:

import { randomBytes } from 'crypto';

export const generateBase64Token = (length: number): string => {
  const buffer = randomBytes(Math.ceil((length * 3) / 4)); // Generate enough random bytes
  return buffer
    .toString('base64') // Convert to Base64
    .replace(/\+/g, '-') // URL-safe: replace + with -
    .replace(/\//g, '_') // URL-safe: replace / with _
    .replace(/=+$/, '') // Remove padding
    .slice(0, length); // Ensure fixed length
};

Next, in src/config/zookeeper.ts, add the following code to:

Connect the client to ZooKeeper.
Create the /tokens node if it doesn't exist.
Generate the unique token using the generateBase64Token() method we just created.

import ZooKeeper from 'zookeeper';
import { generateBase64Token } from '../utils';

const { ZOOKEEPER_HOST, ZOOKEEPER_DOCKER_PORT } = process.env;

const host = `${ZOOKEEPER_HOST}:${ZOOKEEPER_DOCKER_PORT}`;

let client: ZooKeeper | null;

const TOKENS_NODE_PATH = '/tokens';

const MAX_RETRIES = 3;
const MAX_TOKEN_SIZE = 6;

// Get ZooKeeper client
const getZookeeperClient = (): ZooKeeper => {
  if (!client) {
    const config = {
      connect: host,
      timeout: 5000,
      debug_level: ZooKeeper.constants.ZOO_LOG_LEVEL_WARN,
      host_order_deterministic: false,
    };

    client = new ZooKeeper(config);
  }

  return client;
};

// Connect to ZooKeeper
export const connectToZookeeper = async (): Promise<void> => {
  const client = getZookeeperClient();

  await new Promise<void>((resolve, reject) => {
    client.connect(client.config, async (error) => {
      if (error) {
        console.error('Error connecting to ZooKeeper:', error);
        reject();
      }
      console.log('Successfully connected to ZooKeeper');
      await createTokensNode();
      resolve();
    });
  });
};

// Create '/tokens' node if it doesn't exist
const createTokensNode = async (): Promise<void> => {
  const client = getZookeeperClient();
  const doesTokensNodeExist = await client.pathExists(TOKENS_NODE_PATH, false);

  // If it does, do nothing
  if (doesTokensNodeExist) {
    console.info(`Tokens node ${TOKENS_NODE_PATH} already exists`);
    return;
  }

  // If it doesn't exist, create the root path
  await new Promise<void>((resolve, reject) => {
    client.mkdirp(TOKENS_NODE_PATH, (error) => {
      if (error) {
        console.error(`Failed to create tokens node: ${error}`);
        reject();
      }
      console.info(`Tokens node ${TOKENS_NODE_PATH} created`);
      resolve();
    });
  });
};

// Create a node
const createNode = async (path: string, data: Buffer): Promise<void> => {
  try {
    await getZookeeperClient().create(
      path,
      data,
      ZooKeeper.constants.ZOO_EPHEMERAL
    );
    console.info(`Node ${path} created`);
  } catch (error) {
    console.error(`Failed to create node: ${error}`);
    throw error;
  }
};

// Generate a unique token with retries for collision detection
export const generateUniqueToken = async (retryCount = 0): Promise<string> => {
  const client = getZookeeperClient();
  const token = generateBase64Token(MAX_TOKEN_SIZE);
  const uniqueTokenPath = `${TOKENS_NODE_PATH}/${token}`;

  // Create a child node with the generated token
  try {
    // Check if the unique token node already exists
    const doesUniqueTokenNodeExist = await client.pathExists(
      uniqueTokenPath,
      false
    );

    // If it does, retry
    if (doesUniqueTokenNodeExist) {
      if (retryCount < MAX_RETRIES) {
        console.log(
          `Token collision detected for path: ${uniqueTokenPath}. Retrying... Attempt ${
            retryCount + 1
          } of ${MAX_RETRIES}`
        );
        return await generateUniqueToken(retryCount + 1);
      } else {
        throw new Error(
          `Failed to generate a unique token after ${MAX_RETRIES} attempts due to collisions.`
        );
      }
    }

    // If it doesn't exist, create the node
    await createNode(uniqueTokenPath, Buffer.from(token));

    return token; // Return the unique token on success
  } catch (error) {
    console.error(`Error generating the unique token node: ${error}`);
    throw error;
  }
};

And finally, add the following keys to the .env file to configure the service:

ZOOKEEPER_HOST=zookeeper
ZOOKEEPER_LOCAL_PORT=2181
ZOOKEEPER_DOCKER_PORT=2181

Great, we have all of our configuration files now ready! We can move on and start implementing the actual URL logic.

Implementing the URL Shortening Logic

In this section, we will walk through the process of implementing the logic needed to shorten a URL and manage the redirection when users access the shortened link.

Implementing URL repository

The URL repository will handle all database operations for managing shortened URLs:

Add a new URL to the database with create().
Retrieve all saved URLs with findAll().
Fetch a specific URL based on given parameters with findOne().

Add the following code in src/repositories/urlRepository.ts:

import Url, { IUrl } from '../models/Url';

interface ICreateParams {
  shortenUrlKey: string;
  originalUrl: string;
}

interface IFindOneParams {
  shortenUrlKey?: string;
  originalUrl?: string;
}

// Create a shortened URL
const create = async (params: ICreateParams): Promise<IUrl> => {
  console.log(`Creating URL with params: ${JSON.stringify(params)}`);
  const result: IUrl = await Url.create({ ...params });
  console.log(`Created URL: ${JSON.stringify(result)}`);
  return result;
};

// Find all URLs
const findAll = async (): Promise<IUrl[]> => {
  console.log('Finding all URLs');
  const result: IUrl[] = await Url.find();
  console.log(`Found URLs: ${result?.length || 0}`);
  return result;
};

// Find a specific URL
const findOne = async (params: IFindOneParams): Promise<IUrl | null> => {
  console.log(`Finding one URL with params: ${JSON.stringify(params)}`);
  const result: IUrl | null = await Url.findOne({ ...params });
  console.log(`Found URL: ${JSON.stringify(result)}`);
  return result;
};

export { create, findAll, findOne };

Implementing URL validation

Before shortening a URL, it's essential to verify that the input provided by the user is valid. We will use a simple Regex found online for it (you can find plenty of other patterns as well depending on your needs).

Add the following method to src/utils/index.ts:

...
export const isValidUrl = (value: string): boolean => {
  const pattern: RegExp = new RegExp(
    '^https?:\\/\\/' + // Protocol (http or https)
      '(?:www\\.)?' + // Optional www.
      '[-a-zA-Z0-9@:%._\\+~#=]{1,256}' + // Domain name characters
      '\\.[a-zA-Z0-9()]{1,6}\\b' + // Top-level domain
      '(?:[-a-zA-Z0-9()@:%_\\+.~#?&\\/=]*)$', // Optional query string
    'i' // Case-insensitive flag
  );

  return pattern.test(value);
};

Implementing Service Logic

The service layer is responsible for orchestrating the business logic of the URL shortener application, acting as an intermediary between the controller and the repository we just created.

As mentioned in the Redis section, we will leverage Redis for caching frequently accessed URLs to improve performance by reducing database queries.

In src/services/urlService.ts, add the following code:

import { generateUniqueToken } from '../config/zookeeper';
import { get, set, extendTTL, RedisExpirationMode } from '../config/redis';
import { IUrl } from '../models/Url';
import { isValidUrl } from '../utils';
import { create, findAll, findOne } from '../repositories/urlsRepository';

const ONE_MINUTE_IN_SECONDS = 60;

// Get all shortened URLs
export const getAllUrls = async (): Promise<IUrl[]> => await findAll();

// Get a specific shortened URL by its key
export const getUrlByShortenUrlKey = async (
  shortenUrlKey: string
): Promise<string | null> => {
  // Try to get the original URL from Redis cache
  const cachedOriginalUrl = await get(shortenUrlKey);
  if (cachedOriginalUrl) {
    // Extend TTL
    await extendTTL(shortenUrlKey, ONE_MINUTE_IN_SECONDS);

    return cachedOriginalUrl; // Return the cached original URL
  }

  // If not in cache, retrieve from database
  const savedUrl = await findOne({ shortenUrlKey });
  if (savedUrl) {
    // Cache the original URL created by its shorten URL key
    await set(
      savedUrl.shortenUrlKey,
      savedUrl.originalUrl,
      RedisExpirationMode.EX,
      ONE_MINUTE_IN_SECONDS
    );

    return savedUrl.originalUrl; // Return the saved original URL
  }

  return null; // Return null if nothing found
};

// Create a new shortened URL
export const createShortenedUrl = async (
  originalUrl: string
): Promise<string | null> => {
  // Check if URL is valid
  if (!isValidUrl(originalUrl)) {
    return null;
  }

  // Retrieve from database
  const savedUrl = await findOne({ originalUrl });
  if (savedUrl) {
    return savedUrl.shortenUrlKey; // Return the saved shortened URL key
  }

  // If not in database, generate a new shortened URL key and save it
  const shortenUrlKey = await generateUniqueToken();
  if (shortenUrlKey) {
    const newUrl = await create({
      originalUrl,
      shortenUrlKey,
    });

    // Cache the original URL created by its shorten URL key
    await set(
      newUrl.shortenUrlKey,
      newUrl.originalUrl,
      RedisExpirationMode.EX,
      ONE_MINUTE_IN_SECONDS
    );

    return newUrl.shortenUrlKey; // Return shortened URL key
  }

  return null; // Return null if token generation failed
};

Implementing Controller Logic

Next, let's implement the URL controller methods to manage the HTTP status codes and return appropriate messages for our operations. As you might have seen, I chose to return a 200 status code along with the original URL (and not a 301 redirect) in the getUrl() method to prevent any CORS issues between the client and the requested URLs later on.

In src/controllers/urlController.ts, add the following code:

import { FastifyReply, FastifyRequest } from 'fastify';
import {
  createShortenedUrl,
  getAllUrls,
  getUrlByShortenUrlKey,
} from '../services/urlsService';

// Get all shortened URLs
export const getUrls = async (
  _request: FastifyRequest,
  reply: FastifyReply
): Promise<void> => {
  try {
    const urls = await getAllUrls();

    return reply.code(200).send(urls);
  } catch (error) {
    return reply
      .code(500)
      .send('Failed to retrieve the list of URLs. Please try again later');
  }
};

// Get a specific URL by its key
export const getUrl = async (
  request: FastifyRequest<{
    Params: {
      shortenUrlKey: string;
    };
  }>,
  reply: FastifyReply
): Promise<void> => {
  try {
    const { shortenUrlKey } = request.params;
    const originalUrl = await getUrlByShortenUrlKey(shortenUrlKey);

    if (!originalUrl) {
      return reply
        .code(404)
        .send('The requested shortened URL could not be found');
    }

    return reply.code(200).send(originalUrl);
  } catch (error) {
    return reply.code(500).send('Unable to retrieve the specified URL');
  }
};

// Create a new shortened URL
export const postUrl = async (
  request: FastifyRequest<{
    Body: {
      originalUrl: string;
    };
  }>,
  reply: FastifyReply
): Promise<void> => {
  try {
    const { originalUrl } = request.body;
    const shortenUrlKey = await createShortenedUrl(originalUrl);

    if (!shortenUrlKey) {
      return reply.code(400).send('The provided URL is invalid');
    }

    return reply.code(201).send(shortenUrlKey);
  } catch (error) {
    return reply.code(500).send('Failed to create a shortened URL');
  }
};

Implementing Routing Logic

Now, let's register the routes under the /urls prefix.

In src/routes/urls.ts, add the following code:

import { FastifyInstance } from 'fastify';
import { postUrl, getUrls, getUrl } from '../controllers/urlsController';

export const urlsRoutes = async (fastify: FastifyInstance) => {
  fastify.register(
    async (router: FastifyInstance) => {
      // Get all shortened URLs
      router.get('/', getUrls);

      // Get a specific URL by its key
      router.get('/:shortenUrlKey', getUrl);

      // Create a new shortened URL
      router.post('/', postUrl);
    },
    { prefix: '/urls' }
  );
};

Setting Up The Server

And finally, let's create a src/index.ts to set up the Fastify server and:

Configure CORS.
Register the URL router under the /api prefix.
Connect MongoDB, Redis, and ZooKeeper.
Start the server.

import Fastify, { FastifyInstance } from 'fastify';
import fastifyCors from '@fastify/cors';
import { connectToMongoDB } from './config/mongoose';
import { connectToRedis } from './config/redis';
import { connectToZookeeper } from './config/zookeeper';
import { urlsRoutes } from './routes/urlsRoutes';

// Fastify server instance
const fastify = Fastify();

// Configure server
fastify
  .register(fastifyCors) // Register CORS
  .register(
    async (fastify: FastifyInstance) => {
      fastify.register(urlsRoutes); // Register URL routes
    },
    { prefix: '/api' }
  );

// Start the server
const start = async () => {
  try {
    // Connect to MongoDB, Redis and ZooKeeper
    await connectToMongoDB();
    await connectToRedis();
    await connectToZookeeper();

    // Start Fastify server
    await fastify.listen({
      port: Number(process.env.NODE_SERVER_LOCAL_PORT),
      host: process.env.NODE_SERVER_HOST,
    });
    console.log('Server is now listening');
  } catch (error) {
    console.error('Failed to start server:', error);
    process.exit(1);
  }
};

start();

Don't forget to add the following keys to the .env file:

NODE_SERVER_HOST=0.0.0.0
NODE_SERVER_LOCAL_PORT=3000

Awesome, our server is now ready! We can move on and configure Nginx.

Setting Up Nginx

As mentioned in the introduction, we will use Nginx as a load balancer and reverse proxy to distribute traffic across server instances and improve response times. We will use the default round-robin algorithm, which is ideal for distributing requests evenly and making our application more resilient.

In a nginx/nginx.conf file, add the following configuration:

Create an upstream node_servers block to define the group of servers listening on local port 3000. You will see later in the docker-compose setup that we did not define a specific port for each server instance, allowing Docker to dynamically assign ports and manage load balancing.
Add a server block which listens on port 80 for incoming requests.
Include a location block to forward requests made on the /api/ path to the node_servers group using proxy_pass. And add proxy_set_header directives to ensure that client request details are forwarded to the servers.

upstream node_servers {
  server server:$NODE_SERVER_LOCAL_PORT;
}

server {
  listen $NGINX_DOCKER_PORT;

  # Serve backend
  location /api/ {
    proxy_pass http://node_servers/api/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;
  }
}

Also add the following keys to the .env file to configure the Nginx service:

NGINX_HOST=localhost
NGINX_LOCAL_PORT=80
NGINX_DOCKER_PORT=80

Containerization with Docker

With everything set up, we will now use Docker to containerize all of our services.

Let's first dockerize the server application. Create a Dockerfile in the /server folder that sets up the Node environment, installs dependencies, and starts the app like below:

# Use an official Node runtime as the base image
FROM node:22.11.0

# Set the working directory
WORKDIR /usr/src/server

# Copy package.json and package-lock.json to the container
COPY package*.json ./

# Install application dependencies
RUN npm install

# Copy the rest of the application code
COPY . .

# Run the application
CMD [ "npm", "run", "start" ]

Also add a .dockerignore file with node_modules inside because some libraries like zookeeper can cause some issues when compiled on different OS.

Next, create a Dockerfile in the nginx folder to configure Nginx by using the official Nginx image, copying our custom nginx.conf file to the container, and running the service:

# Use an official Nginx as the base image
FROM nginx:stable-alpine

# Copy nginx.conf to the container
COPY nginx.conf /etc/nginx/templates/default.conf.template

# Run the server
CMD ["nginx", "-g", "daemon off;"]

Finally, create a docker-compose.yml file at the root of your project setting up the services (MongoDB, Redis and ZooKeeper), while building the server and Nginx from their respective Dockerfiles:

services:
  mongo:
    image: mongo:latest
    environment:
      - MONGO_INITDB_ROOT_USERNAME=${MONGODB_USER}
      - MONGO_INITDB_ROOT_PASSWORD=${MONGODB_PASSWORD}
    ports:
      - ${MONGODB_LOCAL_PORT}:${MONGODB_DOCKER_PORT}
    volumes:
      - ./mongodb:/data/db

  redis:
    image: redis:latest
    ports:
      - ${REDIS_LOCAL_PORT}:${REDIS_DOCKER_PORT}

  zookeeper:
    image: zookeeper:latest
    ports:
      - ${ZOOKEEPER_LOCAL_PORT}:${ZOOKEEPER_DOCKER_PORT}

  server:
    depends_on:
      - mongo
      - redis
      - zookeeper
    environment:
      - MONGODB_USER=${MONGODB_USER}
      - MONGODB_PASSWORD=${MONGODB_PASSWORD}
      - MONGODB_DATABASE=${MONGODB_DATABASE}
      - MONGODB_HOST=${MONGODB_HOST}
      - MONGODB_DOCKER_PORT=${MONGODB_DOCKER_PORT}
      - REDIS_HOST=${REDIS_HOST}
      - REDIS_DOCKER_PORT=${REDIS_DOCKER_PORT}
      - ZOOKEEPER_HOST=${ZOOKEEPER_HOST}
      - ZOOKEEPER_DOCKER_PORT=${ZOOKEEPER_DOCKER_PORT}
      - NODE_SERVER_HOST=${NODE_SERVER_HOST}
      - NODE_SERVER_LOCAL_PORT=${NODE_SERVER_LOCAL_PORT}
    build:
      context: ./server
      dockerfile: Dockerfile
    volumes:
      - ./server:/usr/src/server
      - /usr/src/server/node_modules
    deploy:
      mode: replicated
      replicas: 3

  nginx:
    depends_on:
      - server
    environment:
      - NODE_SERVER_LOCAL_PORT=${NODE_SERVER_LOCAL_PORT}
      - NGINX_DOCKER_PORT=${NGINX_DOCKER_PORT}
    build:
      context: ./nginx
      dockerfile: Dockerfile
    ports:
      - ${NGINX_LOCAL_PORT}:${NGINX_DOCKER_PORT}

As you can see, we are passing down all the environment variables defined in the .env file, keeping all configurations centralized in one place, which makes adjustments simple.

Testing and Deployment

Run docker compose up -d to start all containers in detached mode. Once they're running, you can use cURL or any other tool to test the API routes:

Save a new URL (you can also test an incorrect URL format to check if the service returns a 400 status code):

curl --location 'http://localhost/api/urls' \
--header 'Content-Type: application/json' \
--data '{
    "originalUrl": "URL_HERE"
}'

Get all URLs:

curl --location 'http://localhost/api/urls'

Retrieve the original URL:

curl --location 'http://localhost/api/urls/SHORTENED_TOKEN_HERE'

Or use Postman which is more friendly:

You can also check the logs in Docker to monitor the activity across the containers (I'm using Docker Desktop here):

Conclusion

You have reached the end of the first part of this tutorial! I hope you enjoyed 😄

We learned how to build a scalable URL shortener application from scratch using Node.js, Redis, MongoDB, Apache ZooKeeper, Nginx and Docker. You can find the complete code for this project here.

In the second part, we will focus on developing the frontend of our application using React and RTK Query, allowing users to interact with our servers through a minimal UI.

And if you're interested in going further, check out my other repository here. In this version, I've added extra features like a visit counter and a purge system to clean all expired URLs, all managed through a task queue service.

DEV Community