Which Agent Step Deserves a Cache? A Glossary, a Decision Tree, and Four Worked Leaves

A nightly agent job runs 40 steps. Last sprint someone added a response cache to save calls on a free model endpoint. Tonight the summary cites a documentation paragraph that was deleted six hours ago, and nobody notices until a human reads the output.

Caching is the cheapest optimization in an agent pipeline and the easiest way to ship stale facts. The problem is not "should we cache" — it is which step deserves a cache, and with what invalidation contract. This post gives you a glossary, a four-question decision tree, and a worked example at each leaf, plus a replay test you can run before trusting any of it.

Glossary: seven terms worth agreeing on first

1. Step — one model call plus its deterministic wrapper (prompt assembly, parsing, validation). Caching decisions belong to the step, not the pipeline.
2. Cache key — the exact bytes that identify a reusable result. If the key omits an input, you get a silent wrong answer, not a miss.
3. Normalization — the rewrite applied to a prompt before hashing (whitespace collapse, Unicode NFKC, lowercasing). It raises hit rate and raises collision risk in the same move.
4. Staleness class — a label for how bad a stale answer is: cosmetic, misleading, or unsafe. Only cosmetic is a safe default for TTL caching.
5. TTL — the wall-clock bound on reuse. A TTL is a promise you make to readers about how old your facts can be.
6. Input digest — a hash of the files or records the step reads, not just the prompt text. Prompt-only keys miss content changes.
7. Hit budget — the fraction of steps you expect to serve from cache. Write it down; an unmeasured cache is a guess with extra failure modes.

The decision tree

Ask four questions per step, in order. Stop at the first leaf.

Q1. Does the step read external mutable state (live DB, clock, web, file tree)?
    yes -> Q2
    no  -> LEAF A (exact memo, no TTL)

Q2. Is a stale answer ever unsafe (side effects, money, access, published text)?
    yes -> LEAF C (no cache)
    no  -> Q3

Q3. Do prompts vary only by whitespace/casing/Unicode form across runs?
    yes -> LEAF D (normalized key + namespace)
    no  -> LEAF B (TTL cache, TTL from staleness class)

Four leaves. Each one below is a runnable decision, not a vibe.

Leaf A — exact memo, no TTL

Pure steps: schema validation of a tool call, prompt linting, deterministic reformatting, unit conversion. The output cannot change unless the input changes, so a TTL is pointless — the input digest is the invalidation.

from dataclasses import dataclass
from hashlib import sha256
import json

@dataclass(frozen=True)
class Step:
    name: str
    prompt: str
    input_digest: str          # hash of files/records the step reads
    reads_mutable_state: bool
    stale_is_unsafe: bool
    normalizable: bool = False

def cache_key(step: Step, model_tag: str) -> str:
    payload = {
        "model": model_tag,
        "prompt": step.prompt,
        "input": step.input_digest,
    }
    blob = json.dumps(payload, sort_keys=True).encode("utf-8")
    return sha256(blob).hexdigest()[:16]

Note model_tag is in the key. Swapping the model behind the same prompt must not reuse the old answer — that is the single most common cache bug in free-tier pipelines.

Leaf B — TTL cache, TTL set by staleness class

Steps that read mutable state but tolerate staleness: doc-snippet retrieval, changelog summaries, format detection. Here the cache lives for a bounded time and the bound is chosen from the staleness class, not from "how much money would we save".

This is where placement matters. A cache needs to survive between runs but not between deploys, and it needs to be reachable from wherever the step executes. Running the cache on the same box as the worker removes a network hop and one credential.

MonkeyCode offers free model access and a free server option (both operator-supplied availability claims — confirm current limits on the project page), which makes Leaf B practical for a hobby-scale job: model calls on the free endpoint, cache and state on the free server.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

import time

TTL_BY_CLASS = {"cosmetic": 6 * 3600, "misleading": 15 * 60}

def should_reuse(step: Step, cached_at: float, staleness_class: str) -> bool:
    ttl = TTL_BY_CLASS.get(staleness_class)
    if ttl is None:
        return False           # 'unsafe' has no TTL — see Leaf C
    return (time.time() - cached_at) < ttl

One rule: store the staleness class next to the value, not in your head. When the class changes after a code review, you want the old entries refused automatically.

Leaf C — no cache at all

Anything with a side effect or an accuracy obligation: sending mail, writing to a repo, granting access, producing the final user-facing answer. Recompute. If a free-tier call budget forces the issue, shrink the step instead of caching it — cache the retrieval that feeds the step, not the conclusion.

def admit(step: Step) -> str:
    if not step.reads_mutable_state:
        return "leaf_a_exact_memo"
    if step.stale_is_unsafe:
        return "leaf_c_no_cache"
    if step.normalizable:
        return "leaf_d_normalized"
    return "leaf_b_ttl"

That function is the whole artifact. Pipe every step definition through it in a test and assert the mapping matches your written design; the test is what stops someone from adding a cache to the final-answer step during a Friday refactor.

Leaf D — normalized key with a namespace

Prompts that differ only in whitespace or Unicode form deserve a normalized key — but the namespace must record the normalization version, or an edit to your normalizer silently reuses entries built under the old rules.

import re, unicodedata

def normalize(text: str, version: str = "v1") -> str:
    t = unicodedata.normalize("NFKC", text).strip().lower()
    t = re.sub(r"\s+", " ", t)
    return f"{version}:{t}"

Test the collision behavior explicitly:

assert normalize("Summarize  this.") == normalize("summarize this.\n")
assert normalize("a") != normalize("a", version="v2")

If the second assertion ever fails, every cached entry is now ambiguous across normalizer versions.

The replay test: three checks before you enable anything

1. Hit-rate check. Replay recorded step inputs with the cache on. Record actual hits; compare to the hit budget you wrote down. A hit rate far above budget usually means your key is missing an input.
2. Invalidation check. Mutate the source document that feeds one Leaf-B step. The next run must miss. If it hits, your TTL is carrying all the safety and your digest is decorative.
3. Kill-switch check. Flip an environment flag that disables the cache and confirm the pipeline still produces identical output. A cache you cannot turn off in one move is an outage waiting for a bad deploy.

Decision table

Limitations and who should not use this

• Latency-sensitive paths. If a step runs once per request and the cache lives on a separate free server, the round trip can cost more than the recompute. Co-locate or skip.
• Steps with hidden inputs. If your prompt assembly pulls in conversation history or a user profile, the digest must include it. Steps you cannot fully describe should stay at Leaf C.
• Workloads with no repetition. Low-overlap traffic produces misses plus a cache to maintain. Measure first; if your replay shows a single-digit hit rate, delete the layer.
• Anything with a compliance clock. If readers are entitled to fresh data, a TTL cache is a policy change, not a performance tweak. Get it reviewed as one.

The cache is not the interesting part. The interesting part is the sentence you can write afterwards: this step may serve a result up to N minutes old, and here is the test that proves it. Steps that cannot survive that sentence belong at Leaf C.

If you want to try the shape — free model calls plus a free server to host the cache and the replay harness — MonkeyCode's project page is the place to check current terms, then run the three checks above against your own recorded steps before enabling anything in production.