Resilient Systems using Go: Semaphores

Previously, we talked about retry mechanism and circuit-breaker, two resiliency techniques, and what their packages may look like. In this final chapter of the resilience series, we will take a look at semaphores and convert the abstract information to a working package.

Introduction

Let's say we have to create a search page for our cyber security application. The search page shows all the addresses and headers of all the possible malicious emails against a suspicious sender email and subject. To get the email information, the system would have to interact with the email system API, which has a very high threshold for accepting requests. So the search would lead to a search throughout your whole organisation against the keywords mentioned, do a malicious check on them, and return the results.
Now, these can be big or small emails that are to be processed for malicious threats. You would like to process as many mailboxes as possible at a time, but you also don't want the system to slow down by having too many concurrent tasks on a service with unlimited bandwidth.

Semaphores

What we would like to have is a mechanism that restricts the number of concurrent requests we can perform with the resources. The number would align with what the system can handle without interrupting the performance of other processes.
We can achieve this by using semaphores.
Semaphore is a mechanism to put an upper-bound on the number of requests that one can perform at a time. If the semaphore is running under capacity, it can accept further requests. Whenever a request is completed, the semaphore package can notify our system that it is available to take in more requests. If it is already at full capacity, it will return an error.

Designing the Semaphore Package

Package Structure

So a basic functioning package implementing Semaphore would require

• Weight : Maximum number of requests that can run concurrently
• Count : Count of requests under progress
• Notifier: A notification function that will tell the system whenever it is available to take more requests.
• mutex : a mutex would be used to access and update the count variable. Two requests may try to update the count variable at the same time, creating a race condition.

This is what the structure may look like

type Semp struct {
    weight uint32
    count  uint32
    mu     *sync.Mutex
    notify NotifyFunc
}

Functionality

Functionality is pretty straightforward. There are two important methods

Acquire

The system will call the acquire method whenever it wants the request to be processed. If the semaphore is at its full capacity, an error should be returned. Otherwise, the counter should increment.

func (s *Semp) Acquire(i int) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    if s.count+uint32(i) > s.weight {
        return ErrCannotAcquire
    }

    s.count += uint32(i)
    return nil
}

Release

Here we just need to decrement the counter and call the notify function passed by the user

func (s *Semp) Release(i int) error {

    s.mu.Lock()
    defer s.mu.Unlock()
    s.count -= uint32(i)
    if s.count <= 0 {
        s.count = 0
    }
    go s.notify()
    return nil
}

And that's it. That's how you implement the basic functionality of semaphore. The code can be found here. It goes without saying that all the resiliency mechanisms should be context aware. One should be able to cancel any ongoing request if certain criterias can't be met.
What other scenarios do you think we can use semaphores for ?