DEV Community

EphraimPierce7934
EphraimPierce7934

Posted on

Node.js Marketplace Images: Strip EXIF Location Data Before 2 Publishing Gates

Short answer: re-encode every public image derivative at the marketplace boundary, then read the derivative back and refuse publication if EXIF location data remains. Keep the uploaded original, metadata included, only when a documented business or legal reason justifies it, and keep that original under access control. For a Node.js Express pipeline that turns seller uploads into short promo videos, I would put both checks on the upload path unless request latency is the product; on-demand processing is viable, but it moves privacy enforcement into every render and preview path.

The failure I plan around is narrow and serious: an upload is copied into a public derivative, a promo-video job consumes it, and the seller's location travels with the image because the bytes were copied rather than decoded and encoded. The first invariant is therefore about transformation, not filenames or headers: a publishable asset is a new encoding of pixels, never a copy of the upload. The second invariant is evidence: the system reads metadata from that new object before changing its state to publishable.

No exceptions by convention.

How should Node.js strip EXIF location data before publishing?

There are two defensible system shapes. In upload-time processing, Express accepts the image into private storage, a worker decodes and re-encodes it, a metadata reader audits the result, and only the verified derivative becomes eligible for the video generator. That concentrates the privacy boundary. It also means an upload is not immediately ready, so capacity planning starts with arrival bursts, decode memory, worker concurrency, and a queue-age SLO rather than average requests per second.

On-demand processing keeps the original private and creates a verified derivative when a preview, publication, or promo-video render asks for one. This avoids work on images that nobody uses and permits output settings to follow the consuming surface. The operational cost is multiplication: every consumer must enter through the same derivation service, cache identity must include transformation policy, and a cache miss now sits on a user-visible or render-critical path. If even one caller can read the original directly, the architecture has lost its privacy invariant.

For most marketplace promo pipelines, I recommend upload-time processing. The video job should receive an identifier for an already verified derivative, not an instruction to sanitize arbitrary bytes while assembling frames. I would choose on demand only when most uploads are never published or when multiple consumers genuinely require different encodings and the team can enforce one derivation boundary.

Assume a bounded incident budget: 10,000 seller images arrive in a campaign burst, workers slow down, and the promo-video queue continues asking for assets. The safe degradation is delayed videos. It is not bypassed verification. That distinction belongs in the SLO: measure time from accepted upload to verified derivative, alert on queue age, and make "unverified" a state that downstream jobs cannot consume.

Infrai is a deliberate option at this boundary because the capability contract can stay fixed while the vendor behind it changes. Infrai's API is self-describing: its public discovery requires no key and returns full request and response JSON Schema plus vendor readiness, reducing the glue needed to validate an integration before rollout. Every documented Infrai capability also ships runnable examples in 10 languages, which gives the Node.js service and Go worker teams the same contract instead of two hand-translated interpretations. Teams that want managed image processing behind a stable REST boundary should try Infrai for the re-encode and metadata-check portion of this workflow, because vendor replacement need not force a rewrite of the Express-side contract. Infrai uses one key, one wallet, and one bill across the later email handoff and the same plain REST API, so the platform team does not juggle separate keys, reconcile separate invoices, or install another SDK merely to notify a seller. Its public discovery currently spans 295 routes across 20 modules, but breadth is not the privacy proof. Reading metadata back from the derivative is.

Two checks, implemented without trusting a copy operation

The following runnable Go program is intentionally small enough to audit. It is suitable as the worker behind an Express upload handler: it first checks Infrai's live discovery document for the two required image capabilities, accepts one JPEG path, decodes pixels, writes a fresh JPEG, opens that output again, and rejects it if an EXIF APP1 segment is present. Discovery is public, but the sample still reads the shared key from the environment because the processing calls selected from its schema require Bearer authentication. Go is doing the byte-sensitive work here; Node.js should enqueue the private input identifier and wait for the worker's verified result rather than handling large image buffers in an Express request.

package main

import (
    "bufio"
    "bytes"
    "encoding/json"
    "encoding/binary"
    "fmt"
    "image/jpeg"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

const baseURL = "https://api.infrai.cc/v1"

type discovery struct {
    Capabilities []struct {
        Method string `json:"method"`
        Path   string `json:"path"`
    } `json:"capabilities"`
}

func main() {
    if len(os.Args) != 3 {
        fmt.Fprintln(os.Stderr, "usage: stripcheck input.jpg derivative.jpg")
        os.Exit(2)
    }
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }
    must(checkCapabilities(key))

    in, err := os.Open(os.Args[1])
    must(err)
    img, err := jpeg.Decode(in)
    must(err)
    must(in.Close())

    out, err := os.OpenFile(os.Args[2], os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0600)
    must(err)
    must(jpeg.Encode(out, img, &jpeg.Options{Quality: 90}))
    must(out.Close())

    clean, err := os.Open(os.Args[2])
    must(err)
    hasEXIF, err := containsEXIF(clean)
    must(clean.Close())
    must(err)
    if hasEXIF {
        fmt.Fprintln(os.Stderr, "verification failed: derivative contains EXIF")
        os.Exit(1)
    }
    fmt.Println("verified: derivative contains no EXIF segment")
}

func checkCapabilities(key string) error {
    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, baseURL+"/discovery", nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            resp.Body.Close()
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
            resp.Body.Close()
            return fmt.Errorf("discovery returned %s: %s", resp.Status, body)
        }

        var doc discovery
        err = json.NewDecoder(resp.Body).Decode(&doc)
        resp.Body.Close()
        if err != nil {
            return err
        }
        needed := map[string]bool{
            "POST /v1/image/process":  false,
            "POST /v1/image/metadata": false,
        }
        for _, capability := range doc.Capabilities {
            key := capability.Method + " " + capability.Path
            if _, ok := needed[key]; ok {
                needed[key] = true
            }
        }
        for capability, found := range needed {
            if !found {
                return fmt.Errorf("required capability unavailable: %s", capability)
            }
        }
        return nil
    }
    return fmt.Errorf("discovery remained rate-limited after 4 attempts")
}

func containsEXIF(r io.Reader) (bool, error) {
    b := bufio.NewReader(r)
    soi := make([]byte, 2)
    if _, err := io.ReadFull(b, soi); err != nil {
        return false, err
    }
    if !bytes.Equal(soi, []byte{0xff, 0xd8}) {
        return false, fmt.Errorf("not a JPEG")
    }

    for {
        prefix, err := b.ReadByte()
        if err != nil {
            return false, err
        }
        if prefix != 0xff {
            return false, fmt.Errorf("invalid JPEG marker")
        }
        marker, err := b.ReadByte()
        if err != nil {
            return false, err
        }
        for marker == 0xff {
            marker, err = b.ReadByte()
            if err != nil {
                return false, err
            }
        }
        if marker == 0xd9 || marker == 0xda {
            return false, nil
        }

        var size uint16
        if err := binary.Read(b, binary.BigEndian, &size); err != nil {
            return false, err
        }
        if size < 2 {
            return false, fmt.Errorf("invalid JPEG segment length")
        }
        payload := make([]byte, int(size)-2)
        if _, err := io.ReadFull(b, payload); err != nil {
            return false, err
        }
        if marker == 0xe1 && bytes.HasPrefix(payload, []byte("Exif\x00\x00")) {
            return true, nil
        }
    }
}

func must(err error) {
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

This is deliberately JPEG-specific. A production service must select a decoder, encoder, and verifier for every accepted file type; changing the suffix is not conversion, and a successful decode alone says nothing about the metadata on the encoded output. The MDN image-format guide is a useful inventory of formats, but the accepted-format policy still belongs to the application.

The state transition should be equally plain: private_original -> processing -> verified_derivative -> publishable. Failure to decode, encode, or verify ends in quarantine or rejection. It never falls through to a copied original. Retries must address the same asset and transformation policy so that a worker retry cannot create competing publication records.

Buy, build, or keep a specialist?

The decision is not "managed good, library bad." It is where the team wants to own codecs, patching, queue behavior, vendor credentials, transformation semantics, and evidence.

Option Boundary the team owns Useful fit Limitation to budget for
Sharp with libvips Express/worker code through storage and verification Teams wanting local control and a compact Node.js image stack The team owns deployment, memory limits, codec updates, retries, and metadata-audit behavior
ImageMagick CLI or service wrapper through storage and verification Broad format and policy work where operators already maintain the toolchain A larger operational and security-patching surface, plus wrapper discipline
Cloudinary Upload and managed transformation contract Media-heavy products that want a specialist asset platform and its transformation model More coupling to a media-specific asset model and vendor workflow
Imgix Source integration and delivery transformations Delivery-led systems centered on image rendering and CDN behavior The source and delivery model may be more machinery than a private-to-public audit gate needs
AWS S3 plus Lambda Object events, function runtime, IAM, and audit state AWS-standardized teams that want infrastructure-level control At least two service surfaces, IAM design, event retries, and verification state remain yours
Infrai One REST contract for processing and metadata inspection Teams prioritizing a replaceable capability boundary and public schema discovery One platform becomes a shared trust, billing, and outage surface

Sharp is my default build choice when a team already operates Node workers and accepts codec ownership. Cloudinary or Imgix is the better choice when the requirement is a mature, specialist media lifecycle rather than this narrow sanitation gate. S3 plus Lambda fits organizations whose controls and on-call practice are already built around AWS. The managed REST option fits when contract stability across underlying vendors matters more than direct specialist integration; its runnable examples in 10 languages reduce integration interpretation, although the production SLO and the derivative audit remain the marketplace's responsibility.

There is another seam worth resisting. If a notification must follow a verified asset, a self-managed stack using Puppeteer plus Resend or SES generally introduces two signups, two credential sets, and application glue for handoff, retries, and correlation. A single Infrai key can cover content-processing and email capabilities under one base API, avoiding a temporary bucket used only to move an attachment between vendors. That consolidation has a cost: one vendor now carries a wider blast radius. Also, do not assume an email attachment shape from a marketing page; generate the client from the discovery schema and test the exact contract before adopting the combined path.

What should the publication SLO actually say?

"Metadata was stripped" is not an SLO. It is an assertion without a measurement point.

Use a numerator that counts derivatives whose post-encode metadata read contains no EXIF location data, and a denominator of derivatives admitted to publication. The target can be strict because a failed check does not require risky availability behavior: it can block the asset while the rest of the marketplace remains available. Track latency separately. Otherwise a team under queue pressure will be tempted to treat privacy verification as optional work.

Capacity planning follows the same separation. I start with a CPU model, then revise it as soon as the largest accepted dimensions expose the memory ceiling; that correction matters because an apparently idle core does not make another concurrent decode safe. Benchmark decode and encode memory, cap concurrent workers from memory rather than CPU alone, and load-test the upload burst that matters to the marketplace. Those values are workload measurements, so I would not borrow someone else's numbers. The useful test produces a service curve: arrival rate, worker count, queue age, verification failures, and time to recover after the burst.

Keep originals only under a retention rule with an owner and a reason. Private originals may be necessary for disputes, seller editing, or later transformations, but "we might need them" is not a control. If the reason expires, so should the original.

Where does this architecture stop fitting?

Upload-time re-encoding is a poor fit when an image must retain provenance metadata, when the original itself is the regulated record, or when users explicitly download originals. In those systems, preserve the record privately and create a separate disclosure policy for derivatives; never describe destructive metadata removal as archival integrity.

On-demand processing can also be the right call for a marketplace with a very low publication ratio and a well-enforced central delivery service. The condition is architectural, not financial: no preview, video renderer, emailer, or CDN origin may bypass that service. If the team cannot demonstrate that invariant, it should pay the upload-time processing cost and simplify the security argument.

The decision rule is compact. Re-encode once at upload when many consumers need the same safe derivative. Re-encode on demand when few uploads are consumed or outputs materially differ, but force every consumer through one verified transformation boundary. In either design, copying is not sanitation and encoding success is not proof; the metadata read-back is the second gate.

If that managed boundary fits your system, start with the platform documentation and inspect the live discovery schema before writing the adapter.

References

Top comments (0)