Concurrency patterns in Go; worker pools and fan-out/fan-in

Go is known for its exceptional concurrency model, but many developers focus only on goroutines and channels. However, concurrency patterns like worker pools and fan-out/fan-in provide real efficiency.

This article will get into these advanced concepts, helping you maximize throughput in your Go applications.

Why Concurrency Matters

Concurrency allows programs to perform tasks efficiently, especially when dealing with tasks like I/O operations, web requests, or background processing. In Go, goroutines provide a lightweight way to manage thousands of concurrent tasks, but without structure, you can run into bottlenecks. That’s where worker pools and fan-out/fan-in patterns come in.

Worker Pools

Worker pools allow you to limit the number of goroutines by assigning tasks to fixed "workers." This prevents oversubscription, reduces resource consumption, and makes task execution manageable.

package main

import (
    "fmt"
    "sync"
    "time"
)

func worker(id int, jobs <-chan int, results chan<- int, wg *sync.WaitGroup) {
    defer wg.Done()
    for j := range jobs {
        fmt.Printf("Worker %d started job %d\n", id, j)
        time.Sleep(time.Second) // Simulate work
        fmt.Printf("Worker %d finished job %d\n", id, j)
        results <- j * 2
    }
}

func main() {
    jobs := make(chan int, 100)
    results := make(chan int, 100)
    var wg sync.WaitGroup

    // Start 3 workers
    for w := 1; w <= 3; w++ {
        wg.Add(1)
        go worker(w, jobs, results, &wg)
    }

    // Send jobs
    for j := 1; j <= 5; j++ {
        jobs <- j
    }
    close(jobs)

    // Wait for workers to finish
    wg.Wait()
    close(results)

    for result := range results {
        fmt.Println("Result:", result)
    }
}

In this example:

• We have three workers that process jobs concurrently.
• Each job is passed to the workers via channels, and results are gathered for processing.

Fan-Out/Fan-In Pattern

The fan-out/fan-in pattern allows multiple goroutines to process the same task, while fan-in gathers the results back into a single output. This is useful for dividing tasks and then aggregating results.

The fan-in mechanism here will collect results after each task is processed by the workers.

package main

import (
    "fmt"
    "sync"
    "time"
)

// Task struct represents the data being processed
type Task struct {
    id int
}

// processTask simulates work by returning a processed result
func processTask(task Task) string {
    time.Sleep(time.Second) // Simulate work
    return fmt.Sprintf("Processed task %d", task.id)
}

// worker processes tasks and sends results to the results channel
func worker(tasks <-chan Task, results chan<- string, wg *sync.WaitGroup) {
    defer wg.Done()
    for task := range tasks {
        results <- processTask(task)
    }
}

func main() {
    tasks := make(chan Task, 5)
    results := make(chan string, 5)
    var wg sync.WaitGroup

    // Fan-out: Start 3 workers
    for i := 0; i < 3; i++ {
        wg.Add(1)
        go worker(tasks, results, &wg)
    }

    // Send tasks to the tasks channel
    go func() {
        for i := 1; i <= 10; i++ {
            tasks <- Task{id: i}
        }
        close(tasks)
    }()

    // Fan-in: Close results channel once all workers complete
    go func() {
        wg.Wait()
        close(results)
    }()

    // Collect results from the results channel
    for result := range results {
        fmt.Println(result)
    }
}

In this code:

• Fan-Out: Multiple goroutines (workers) handle tasks concurrently, each calling processTask.
• Fan-In: The results from each worker are collected back through the results channel and processed.

Concurrency patterns like worker pools and fan-out/fan-in are excellent for optimizing web servers, batch processing, and other I/O-bound applications, making sure resources are efficiently managed.

Next Steps to increase your knowledge:

• Experiment with applying these patterns to more complex concurrency challenges.
• Build out a web service using a worker pool to manage incoming requests.

The key to success in Go’s concurrency is structure. Mastering these concurrency patterns will level up your Go skills and help you write highly performant applications.

Stay tuned for more insights into Go in the next post!

You can support me by buying me a book :)