Most multi-agent frameworks are benchmarked on 70B+ cloud models. Because frontier models have massive parameter capacity, they silently mask architectural flaws—recovering from ambiguous agent descriptions, ignoring bloated system prompts, and guessing missing tool schemas.
In SpaghettiLens-AI (v0.5.0), an open-source neuro-symbolic engine I built to refactor legacy Perl/PHP monoliths into Python/FastAPI microservices, we took the opposite approach. We used a compact 4B parameter model (qwen3-4b-instruct-2507) running locally on a Snapdragon NPU (within a strict 4096-token sliding context window) as a high-pressure "wind tunnel" to harden a 3-tier smolagents CodeAgent hierarchy (L0 Supervisor -> L1 Team Leads -> L2 Specialists).
Failure Modes Exposed on 4B Edge Hardware (and How We Solved Them)
1. Multi-Tier Blindness (smolagents PR #3115)
-
The Problem: In a 3-level hierarchy, the root
L0 Supervisorcould see theL1 Team Leads, but had zero visibility into the specialized tools nested beneath them. - The Fix: We patched capability propagation upstream so parent agents recursively append a summary of nested sub-agents to their descriptions, giving the root supervisor deterministic routing visibility across all 3 tiers.
2. Verbose Prompt Hallucinations
-
The Problem: Default
managed_agentsystem prompts ask sub-agents to write multi-section prose reports. Under a 4096-token ceiling, this verbosity caused the 4B model to hallucinate theoretical summaries instead of emitting executable<code>...</code>blocks. -
The Fix: We engineered compact, code-first Jinja2 prompt overrides and strict per-tier constraints that force every agent to invoke its assigned tool in Step 1 and pass the output directly to
final_answer.
3. Silent Token-Ceiling Loops (smolagents PR #3110)
-
The Problem: When extracting large static AST/SQL dependency graphs, outputs occasionally hit
max_completion_tokensmid-JSON. Because the closing}was missing,parse_json_blobraised a misleading "output does not contain any JSON blob" error, tricking the agent into regenerating the exact same oversized response in an infinite loop. - The Fix: We added explicit truncated JSON detection so the parser immediately surfaces a token-limit truncation error, allowing the agent to compact its output on retry.
Academic Alignment: Deterministic "Expert Blocks"
Our architecture aligns with recent research on Expert Block Frameworks for Small LLMs (Naphade et al., arXiv:2509.15561).
Naphade et al. demonstrate that locally-run small LLMs can match 100B+ frontier models (within ~0.9% of GPT-4) when paired with a deterministic block that transforms raw, noisy trajectories into structured context before LLM reasoning.
SpaghettiLens-AI extends this exact principle to multi-tier software modernization on edge NPUs:
- Deterministic Context Summarizers: Instead of feeding raw, noisy legacy monoliths into a 4096-token window, our static extractor acts as a deterministic Expert Block—compressing raw ASTs and SQL DDL schemas into a structured JSON dependency graph.
-
Deterministic Verification Blocks: We replace open-ended LLM self-critique with hard symbolic verification (AST parsing, parameterized SQL enforcement, and
decimal.Decimalledger checks), enabling a 4B parameter model to achieve deterministic, single-pass refactoring (iterations_used: 1) on consumer hardware.
I am the creator of SpaghettiLens-AI and an Open-Source ML Engineer focused on edge hardware acceleration and neuro-symbolic systems. If you are building with smolagents or local NPUs, let's connect!
Top comments (5)
Using a 4B model on a local NPU as a wind tunnel is a cheap way to measure orchestration overhead before paying for a frontier model. The failure mode is that the small model passes the routing test while the larger one changes how the same tool call behaves, so the benchmark has to compare tool traces, not just answers.
Spot on, Sammi. I have only run this 3-tier hierarchy locally on my Snapdragon NPU (qwen3-4b-instruct-2507) so far because I don't have the local VRAM to host a 72B model. My focus was getting the 4B model to route deterministically across the 3 tiers (L0 -> L1 -> L2) and pass our 4 symbolic verification gates (AST syntax, SQL bind-parameter enforcement against injection, strict Decimal currency math, and sandbox equivalence).
Next step is hooking our evaluation runner (benchmarks/run_eval.py) up to a serverless 72B endpoint (Qwen2.5-72B-Instruct) so we can diff the 4B NPU ActionStep tool traces head-to-head against the 72B traces. Appreciate the sharp feedback—just followed you back!
You completely called it! Your feedback on how different model scales fundamentally alter the tool-chaining behavior (even when they arrive at the same final answer) was the exact catalyst for our new evaluation suite.
We just published Part 2 which implements the tool-trace diffing you suggested. We built a static AST call-inventory extractor to prove that while the 4B model strictly iterates tool-by-tool (to protect its context window), a frontier model like Gemini will chain the same tools directly together in a single Python block and burn 7,500+ tokens doing it.
Thanks again for the sharp insight—it made the framework significantly more rigorous! dev.to/gcamiolo-dev/part-2-4-gate-...
"Frontier models silently mask architectural flaws" is the whole essay in one line, and it matches what we learned the hard way. A 70B will paper over a sloppy sub-agent description or a missing tool schema that a 4B faithfully chokes on — which makes the small model a better regression test for your orchestration layer than the one you actually ship. The multi-tier blindness bug (supervisor can see L1 but is blind to tools nested under it) is exactly the kind of thing that only surfaces under a tight context ceiling, where there's no slack for the model to guess.
Question on the capability-propagation fix: how do you keep the recursively-appended sub-agent summaries from eating the context budget they're meant to protect? In a 3+ tier hierarchy we've hit the point where the routing metadata itself threatens to blow the window. Do you cap propagation depth, or re-summarize the summaries at each tier so the root gets a compressed view rather than the full nested tree?
"Frontier models silently mask architectural flaws" is exactly it! When you have the slack, the model just guesses its way out of bad routing.
To answer your question on the capability-propagation fix: you are spot on that a naive recursive append will blow up the context window in a 3+ tier setup.
In spagnettilens (specifically in core/smol_hierarchy.py), we don't just blindly append the raw descriptions of the nested sub-agents. Instead, we use a deterministic summarization approach. When a Level 1 Team Lead propagates its capabilities up to the Level 0 Supervisor, it dynamically compresses the description of its L2 Specialists into a strict, single-line signature (e.g., Tool: [name] - [1-sentence summary]).
We essentially built a "routing metadata pruner" that caps the depth and strips out the verbose prompt instructions, ensuring the L0 Supervisor only gets the compressed API signatures of the nested tree. This keeps the metadata footprint predictable and prevents the context budget from exploding.
Also, I just published Part 2 which dives into how this hardware constraint forced us to build static call-inventory extractors to prove the routing actually works. Would love your thoughts on it! dev.to/gcamiolo-dev/part-2-4-gate-...