DEV Community

PhilemonShaw8453
PhilemonShaw8453

Posted on

Feature Flags Service: Safe Notification Toggles and Rollback Explained

Use a feature flag to return a failing notification service to its previous delivery path, and verify delivery outcomes before declaring the rollback complete. Changing a stored Boolean is only the start: workers must refresh their decisions, and notifications already in flight may still finish on the candidate path.

TL;DR: To build a simple feature flags service in Node.js, store flag states in an object or database and expose Express handlers for reading and toggling them. For a B2B SaaS notification service, keep false mapped to the previous delivery path, make control retries idempotent, and judge rollback success against delivery failures and the delivery SLO.

Protect the old path.

Infrai offers a unified API with a single key and a single bill: 295 routes across 20 modules use one REST API, without separate SDKs for each capability. This broad backend coverage through a simple interface is useful when flags are one part of a larger integration; its public, self-describing discovery API supplies request and response schemas without requiring a key, helping review the next capability before wiring it into the application. The rollback acceptance test still decides suitability.

How should a feature flags service handle a toggle?

Start with one application flag, notification_delivery_v2, defaulting to false. True selects the candidate delivery behavior. Keep that interpretation stable across application versions; a rollback instruction becomes dangerous if different workers interpret the same Boolean differently.

The failure mode is an increase in notification delivery failures after enabling the candidate. Record the value actually used for each attempt, its outcome, the notification identifier, tenant identifier, and application version. These are proposed application log fields, not a provider's ingestion schema. Compare outcomes against attempted-delivery volume and inspect the mix of notifications in each cohort. A quieter queue can make the failure count drop even while the failure rate gets worse, and a flag read from the control service cannot tell you which path a worker actually used. Define the delivery SLI, measurement window, and error-budget condition before the release, then test that the outcome signal distinguishes the two paths. Do not invent a universal rollback percentage.

Count attempts first.

A flag affects future decisions. It cannot recall a message an external delivery provider has accepted or reverse an existing side effect. Delivery deduplication and control-request deduplication solve different problems.

Choose the operational boundary before buying

An object-backed Express service is a reasonable single-process starting point. Shared storage becomes relevant when instances must agree, but a database does not choose refresh intervals, authorize operators, or settle conflicting writes. The team owns those decisions.

The buy-versus-build table distinguishes evaluation priorities. Treat each acceptance test as a requirement to verify in the selected deployment and plan, rather than assuming every product supplies the same controls.

Option Useful evaluation priority Trade-off and rollback acceptance test
Build an Express service A small control surface with application-specific semantics. The team owns persistence, access control, and recovery; prove retry safety and restart behavior.
LaunchDarkly Evaluate a dedicated flag product for the release-control workflow. Confirm the worker refresh and operator disable workflow before accepting an external control dependency.
Unleash Make deployment ownership a central selection question. Confirm the selected deployment model and assign responsibility for recovery and capacity.
Flagsmith Review flag operations alongside application configuration. Verify environment and credential separation, including the migration work required by the chosen setup.

The control plane also needs an evidence plane. Sentry, Datadog, and Grafana are real candidates to evaluate here, alongside OpenTelemetry's log concepts; they should not be treated as interchangeable replacements for the flag store. For Sentry, make investigation of individual delivery failures the acceptance test. For Datadog, test whether the selected setup supports the delivery SLO workflow your operators need. For Grafana, test how the chosen data sources connect attempted-delivery volume to the release decision. These are distinct evaluation tasks, not promises about bundled features. Reject a setup that can show a configuration change but cannot explain its effect on deliveries.

Keep a separate release record.

For this workflow, use client polling and application decision logs. A stored flag value is not a change audit trail, evaluation statistics, or a dependency graph. Assign alerting and heartbeat monitoring separately, including the case where scheduled work never produces a delivery attempt. Log correlation identifiers also do not establish a distributed trace view. Avoid deleting a flag while either delivery path still needs it.

Implement a read boundary without guessing the response

In Node.js, an object can hold the initial state and Express can expose read and toggle handlers. Validate the flag name, authorize control access, and reject unknown keys. An atomic update must include both the state change and the record that deduplicates its operation identifier; putting those writes in separate steps creates an ambiguous retry window.

A blind toggle can undo itself.

For the managed alternative, the Go program below performs one read using the verified value route. Save it as main.go and run it with the Go toolchain after setting INFRAI_API_KEY and INFRAI_API_ORIGIN to the documented HTTPS API origin, without a version suffix. The origin is deployment configuration, so no vendor URL is embedded in this article. The program prints the successful JSON response intact instead of inventing an envelope field. Use the published response schema when wiring the result into an Express handler.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func run() error {
    key := os.Getenv("INFRAI_API_KEY")
    origin := strings.TrimRight(os.Getenv("INFRAI_API_ORIGIN"), "/")
    base, err := url.Parse(origin)
    if err != nil || base.Scheme != "https" || base.Host == "" ||
        base.Path != "" || base.RawQuery != "" || base.Fragment != "" ||
        base.User != nil || key == "" {
        return fmt.Errorf("set the API key and an HTTPS API origin")
    }
    endpoint := origin + "/v1/flags/get_value/notification_delivery_v2"
    ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
    defer cancel()
    client := &http.Client{
        Timeout: 10 * time.Second,
        CheckRedirect: func(*http.Request, []*http.Request) error {
            return http.ErrUseLastResponse
        },
    }
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        data, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            delay := time.Second * time.Duration(1<<uint(attempt))
            retryAfter := resp.Header.Get("Retry-After")
            if seconds, err := strconv.Atoi(retryAfter); err == nil && seconds >= 0 {
                if seconds > 45 {
                    return fmt.Errorf("retry scheduled beyond this run's deadline")
                }
                delay = time.Duration(seconds) * time.Second
            } else if date, err := http.ParseTime(retryAfter); err == nil {
                delay = time.Until(date)
                if delay < 0 {
                    delay = 0
                }
            }
            timer := time.NewTimer(delay)
            select {
            case <-ctx.Done():
                timer.Stop()
                return ctx.Err()
            case <-timer.C:
            }
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("HTTP %d: %s", resp.StatusCode, data)
        }
        fmt.Println(string(data))
        return nil
    }
    return fmt.Errorf("retry budget exhausted")
}

func main() {
    if err := run(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

This is a read-only diagnostic, so retrying it cannot toggle the flag. It uses an explicit method, bounded requests, response-status checks, and exponential backoff for HTTP 429 while honoring Retry-After. It deliberately does not provide a write payload whose schema has not been established.

For a toggle implementation, use one operation identifier for one intended change and reuse it on retries. Prefer an explicit desired state for rollback where the implementation supports it. If two operators can race, add conflict control rather than assuming a retry key establishes ordering. In-memory state is insufficient once restart survival or multiple instances matter; retain both state and deduplication history durably, and bound history retention.

Verify propagation before trusting the switch

Start with false and verify a read. Apply one authorized toggle with a new operation identifier, repeat the same operation, and confirm the second request does not reverse the first. Submit a distinct operation to return to false. Check unknown keys and unauthorized writes as well.

Then inspect a real worker decision.

In an isolated test, introduce a controlled delivery failure and confirm that attempt logs identify the selected path. Disable the candidate, allow workers to refresh, and verify that subsequent attempts select the previous behavior. Keep the test isolated from customer delivery effects.

Polling deserves a capacity estimate: 120 workers polling every 15 seconds generate about 8 reads per second on average, before retries and startup bursts. This is illustrative arithmetic, not a measured capacity claim. Add jitter so refreshes do not synchronize; shorter polling intervals improve refresh speed at the cost of more reads.

Budget for bursts.

The rollback deadline must account for refresh interval, cache behavior, request timeout, and in-flight attempt duration. Measure the combined effect in the actual deployment. A last-known value can protect continuity during a failed refresh but can preserve the candidate behavior longer; defaulting to the previous path can favor rollback speed but changes behavior during a control-plane interruption. Choose deliberately.

Disable the candidate, then verify delivery outcomes

During a suspected regression, read the current value and record the desired target. Serialize operator changes, preserve the operation identifier for retries, and read back the state if a control request times out. A fresh identifier for the same intended toggle can reverse a successful first operation.

Wait for worker evidence.

New attempts should select the previous path; older attempts may finish on the candidate. Compare failures and attempted-delivery volume against the same delivery SLO used before rollout. Investigate remaining candidate decisions for stale configuration. If there are no attempts at all, use the separate heartbeat signal to distinguish silence from healthy delivery.

Keep the previous code path and the release record until the incident is understood. A Boolean is a useful rollback control only while its interpretation, propagation, and effect remain inspectable.

References

Top comments (0)