Building a URL Shortener with Rate Limiting and Redis in Go

URL shorteners like Bitly or TinyURL are incredibly popular tools, but have you ever wondered what goes into building one? In this blog, we’ll dive into how I built a custom URL shortener in Go with rate limiting and a Redis database for data storage and IP tracking. We’ll cover the core features, the tech stack, and the challenges I encountered along the way.

Project Overview

This URL shortener application accepts long URLs from users, generates shorter, unique links, and allows users to customize the shortened alias if desired. The server redirects any visitor who uses the shortened URL to the original long URL.

Here's a quick overview of the primary components:

• WebServer: Handles routing and requests using the Fiber framework.
• Rate Limiter: Manages user requests to prevent abuse, limiting the number of requests an IP can make in a given timeframe.
• URL Validator: Ensures URLs are in a valid format and ready for storage.
• URL Generator: Generates unique short links for each long URL or uses custom aliases provided by users.
• Redis Database: Stores URL mappings and IP rate limits.

With these features in mind, let’s break down the implementation.

Tech Stack

• Go: Fast and efficient, Go is ideal for building a high-performance URL shortener.
• Fiber: A web framework in Go, chosen for its lightweight performance and simplicity.
• Redis: An in-memory database used to store URL mappings and IP quotas, giving us speed and persistence.
• Docker: Containers make setting up Redis easy and ensure the application is portable and scalable.

Project Structure

The core files and folders are organized as follows:

.
├── main.go               # Entry point for the application
├── routes/
│   ├── shorten.go        # Handles URL shortening and redirects
├── database/
│   ├── redis.go          # Database connection logic
├── helpers/
│   ├── helper.go         # Utility functions for URL validation
├── .env                  # Environment variables
├── docker-compose.yml    # Docker setup for Redis

Setting Up the Application

1. Main Server Logic (main.go)

Our main application file sets up the routes for shortening and resolving URLs. Here’s the code snippet:

package main

import (
    "log"
    "os"

    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/logger"
    "github.com/joho/godotenv"

    "github.com/ravikisha/url-shortener/routes"
)

func setupRoutes(app *fiber.App) {
    app.Get("/:url", routes.ResolveURL)
    app.Post("/api/v1", routes.ShortenURL)
}

func main() {
    err := godotenv.Load()
    if err != nil {
        log.Fatal("Error loading .env file")
    }

    app := fiber.New()
    app.Use(logger.New())

    setupRoutes(app)

    log.Fatal(app.Listen(os.Getenv("APP_PORT")))
}

2. Rate Limiting and URL Shortening (shorten.go)

To prevent abuse, we use Redis to keep track of each IP address and limit the number of allowed requests. Here’s the flow:

1. Check Rate Limits: When a request is made, the rate limiter checks Redis to see how many requests the IP has made.
2. Process URL: If the rate limit is not exceeded, the server validates the URL and shortens it.
3. Generate or Use Alias: If the user provides a custom alias, we store it. Otherwise, we generate a unique ID.

package routes

import (
    "os"
    "strconv"
    "time"

    "github.com/asaskevich/govalidator"
    "github.com/go-redis/redis/v8"
    "github.com/gofiber/fiber/v2"

    "github.com/ravikisha/url-shortener/database"
    "github.com/ravikisha/url-shortener/helpers"
)

// Define structs for the request and response data
type request struct {
    URL         string        `json:"url"`
    CustomShort string        `json:"short"`
    Expiry      time.Duration `json:"expiry"`
}

type response struct {
    URL             string        `json:"url"`
    CustomShort     string        `json:"short"`
    Expiry          time.Duration `json:"expiry"`
    XRateRemaining  int           `json:"x-rate-remaining"`
    XRateLimitReset time.Duration `json:"x-rate-limit-reset"`
}

3. Redis Database Setup (redis.go)

In redis.go, we define a helper function to connect to Redis. This connection is used across different components to store short URLs and enforce rate limits. Here's a simple example of how Redis is configured:

package database

import (
    "context"
    "github.com/go-redis/redis/v8"
)

var Ctx = context.Background()

func NewClient(dbNum int) *redis.Client {
    rdb := redis.NewClient(&redis.Options{
        Addr:     "localhost:6379",
        Password: "",
        DB:       dbNum,
    })
    return rdb
}

Docker Setup

To simplify setting up Redis, I used Docker. This makes the application portable and easy to deploy.

version: '3'
services:
  redis:
    image: "redis:alpine"
    ports:
      - "6379:6379"

Running the Application

1. Clone the repository from GitHub: URLShortener

2. Run the Docker container for Redis:
   

   docker-compose up -d
   

3. Set environment variables in .env:
   

   DB_ADDR="localhost:6379"
   DB_PASSWORD=""
   APP_PORT=":6767"
   DOMAIN="localhost:6767"
   APP_QUOTA=10
   

4. Run the Go application:
   

   go run main.go
   

Now, the application is live, and you can start shortening URLs!

Testing the URL Shortener

Shortening a URL

Send a POST request to /api/v1 with the following JSON payload:

{
    "url": "https://example.com",
    "short": "exmpl",
    "expiry": 24
}

Accessing a Shortened URL

Use the generated short URL, like http://localhost:6767/exmpl, to be redirected to https://example.com.

Key Learnings

1. Using Redis for Rate Limiting: Redis is incredibly fast, and using it for both URL storage and rate limiting was efficient and effective.
2. Building a REST API with Fiber: Fiber's simplicity and performance are well-suited for building APIs in Go.
3. Error Handling and Validation: Ensuring robust error handling and URL validation was essential to providing a user-friendly experience.

Future Improvements

There are a few features and optimizations I’d like to add in the future:

• Admin Dashboard: A UI to track URL usage and monitor statistics.
• Detailed Analytics: Track click counts, referrers, and user demographics.
• Scalability: Deploy the app on a cloud provider and use distributed Redis for handling more extensive data.

Conclusion

Building this URL shortener was a rewarding experience and a great way to explore Go, Fiber, and Redis. This project provides a solid foundation, whether you're learning about backend development or exploring Go’s potential in web services.

If you'd like to see the code in action, check out the GitHub repository here. Let me know your thoughts or if you have suggestions for improving the project!