Golang, concurrency - don't over-complicate life.

Web development is what I normally specialize in but now and then I have to drop to Golang for some CLI tool or specialized microservice.

I have never had the time to spend learning Golang properly, however having quite a bit of programming experience; Golang is a fairly easy language to pick up in a short space of time. Although to be honest, it's misleading in its simplicity - because as you dive deeper, you realize it's not so easy to master.

One of the things I realized over time, is that I have been overcomplicating concurrency in for loops. While looping I usually want to process the items using goroutines.

Goroutines work fine however you cannot control how many goroutines you spawn in a loop, and further - if you don't use something like wait groups, the main function can exit, while these still run - which often, is not desirable.

I found a very cool and powerful package that is great and has many use cases, however for simple loops it's just overkill: https://github.com/alitto/pond

The general goal of Golang and, why I prefer it over other languages - is its minimalism and compact language design.

You can often do tasks natively in the language without external packages. Far too often in Python/PHP/JavaScript, we tend to pull in a package and just implement - which is not a bad thing, however, doing so can at times add more overhead and negatively impact performance as well.

Coming back to concurrency in for loops; while the pond library is great and you definitely should check it out - there is a native way in Golang to achieve controlled concurrency in for loops:

package main

import (
    "fmt"
    "sync"
)

func doSomethingWithUrl(url string, wg *sync.WaitGroup) {
    fmt.Println("Some implementation here...")
    defer wg.Done()
}
func main() {
    // Main will not wait for goroutines to finish, so use a waitGroup to stop the main from exiting.
    var wg sync.WaitGroup

    urls := []string{"https://example.com", "https://example2.com", "https://example3.com", "https://example4.com"}

    // How many tasks to do concurrently
    numWorkers := 2

    for i, url := range urls {

        // Every time we reach a goroutine count of numWorkers - pause the loop for those to finish.
        if i > 0 && i%numWorkers == 0 {
            wg.Wait()
        }

        wg.Add(1)
        go doSomethingWithUrl(url, &wg)
    }

    wg.Wait()
}

As you can see - it's slightly more verbose than using the pond library, however, we avoid an extra third-party dependency and, avoid features like channels that can add unnecessary overhead to our code.

Image credit: https://www.pexels.com/photo/close-up-shot-of-green-plant-8412830/