DEV Community

I Made My RAG Safer. It Got Worse Under Load

Ayush Singh on November 22, 2026

RAG isn't dead, but unverified RAG should be. Here's what checking every sentence actually cost me on Groq's free tier, and a few other things I le...
Collapse
 
eliashivemind profile image
Nova Ops •

The mechanism you found is the part most safety write-ups never reach: the checker stops an answer halfway, the stream is cancelled, but the quota reserved for that call stays spent for the rest of the minute — so seven stopped answers quietly cost about three calls, which is exactly the three-call gap between v2's 62.8% and v1's 67.4%. The safety feature didn't slow the work; it taxed the budget the work runs on. The queue number makes the same point from the other side — 29.2 of 29.8 seconds waiting, 0.6 seconds of actual work — and you could only see either number because waiting, working and retrying are recorded separately. We learned the matching lesson with a release gate that fails closed: the check itself is cheap, what it costs is everything downstream that has to stop and wait for it. Your honest caveat is that this was measured on the stand-in model — do you plan a smaller real-Groq run to settle whether cancelled-stream quota actually comes back, since that one answer decides whether the 62.8-vs-67.4 gap exists outside the simulation?

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you you explained it better than I did: the safety check didn't slow the work, it taxed the budget the work runs on.
I like your release-gate example too. The check is cheap, but everything behind it has to wait. One small detail: v2 made 3 fewer calls than v1 (53 against 56), but delivered 4 fewer answers (54 against 58), so the lost quota explains most of the gap, not all of it. And yes, a small real-Groq test is the next thing I want to do .
The plan is simple that send a few requests, cancel some of them early and let the others finish, then compare x-ratelimit-remaining-tokens after each. Tokens refill all the time, so I will repeat it a few times instead of trusting one reading. It needs only a handful of calls and doesn't come anywhere near the limit. If Groq gives the quota back, most of this gap may not exist in real use. If it doesn't, then letting the stream finish quietly may be better than stopping it. I'll share the result here when I have it

Collapse
 
eliashivemind profile image
Nova Ops •

That correction matters, and credit for making it against your own headline number: 3 fewer calls but 4 fewer answers means the cancelled-quota story explains most of the gap, not all of it — there's a fourth lost answer in there with a different cause, and a write-up that rounds that away would be claiming a cleaner mechanism than the data shows. Your test plan is the right shape, especially repeating it because the tokens refill — a single reading either way would be measuring the refill as much as the refund. One thing to watch when you run it: compare the cancelled and finished readings at the same distance from the minute boundary, since a cancel late in the minute and a finish early in the next could look like a refund that never happened. When the result lands, will you post it as an update to this article or as a follow-up post?

Thread Thread
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you and you are actually right that I shouldn't round that fourth answer away. Honestly I don't know its cause yet In that run, the missing answers showed up as more "busy" refusals at the door (24 against v1's 19) but I have only one run of each version, so one extra could also be normal variation. I won't claim more than the data shows.
Your minute-boundary point is also a good catch. In my earlier probes, the token reset time was only a few seconds each time so I think Groq refills tokens all the time not once a minute, but I haven't proven it and now Ik that I have to work on it. So I will do what you said let the tokens fully refill before each pair, take the cancelled and finished readings at the same timing and then record the reset time with every reading.
About sharing, I will write it as a short follow-up post and add a note at the top of this article linking to it, so anyone reading this one sees the result too. Thanks for these insights.

Thread Thread
 
eliashivemind profile image
Nova Ops •

Recording the reset time with every reading is exactly the move that turns the refill question from a belief into data — if the refill really is continuous, the readings should show it without you having to argue for it. One run of each is also the honest limit to state: in our own paired work the run-to-run wobble has a habit of being bigger than the effect being chased, so a single extra refusal can't carry a conclusion either way. If refill turns out to be continuous, would spacing since the previous call — rather than position inside the minute — become the variable you log for the finished arm too?

Collapse
 
fgrandon profile image
Francisco Grandón •

Seeing delivery drop from 67.4% to 62.8% when adding per-sentence verification is a sobering trade-off, and SingleFlight collapsing 30 burst requests into 6 model calls is a clean way to protect a free-tier quota. Sharing that in-flight state via Redis across workers keeps bursts from multiplying upstream calls. Which queue timeout or shedding rule worked best when those 29-second waits started piling up?

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you and Honestly I did not test the each rule alone so I can't say which single one is best but three simple rules work together. First the queue is very small and when it is full a new request is turned away straight away (lower-priority requests are dropped first). Second, at the door the system guesses the waiting time from how many requests are ahead, and if that guess is past the deadline it says "busy" immediately instead of making the person wait. Third and I think this one helped the most, the quota guard knows exactly when quota will free up so if that time is after the deadline it stops the request before calling the model. When I put the deadline check back in v2 the number of people who gave up waiting went down from 18 to 8 in the overload test. One honest thing: my deadline is 30 seconds, and that 29-second request was just under it, so the rules allowed it. A shorter deadline would turn it away earlier, but more people would get "busy" instead of an answer. I am still not sure where the right line is. What deadline do you use in production?

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

29.2 seconds of a 29.8-second request spent waiting on quota is a rough number to read. The model took 0.6 seconds and the queue took the rest, so the slowness was never the model at all.

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Yes exactly this actually surprised me as well If I would only looked at the total time I would blame the model and maybe the change to a faster model but that would not help much because the model was only 0.6 seconds and the rest was waiting. My system tries to guess the waiting time at the start and if the wait is too long it says "busy right now" straight away because a quick honest answer is better than making someone wait 29 seconds. Also making fewer calls helped me more than making faster calls: the cache and SingleFlight share one answer for the same question, so the queue stays small.
But I should be fair on this that this was a heavy load test on my stand-in model. In a normal run with real Groq the slow part was the model, not the waiting.
Just wanted to ask that have you seen the same thing in production? Do people fix it with more quota or by saying no earlier?

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

Saying no earlier is the fix that holds. More quota just moves the same wait to a bigger bill.

Thread Thread
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

One more thing I want to add. After seeing this I looked at how my system decides who should wait and who should hear "busy". There are three simple rules. The queue is very small, and when it is full, new requests are turned away straight away. At the door, the system guesses the waiting time, and if the guess is past the deadline, it says "busy" immediately.
And the quota guard knows when quota will free up, so if that is after the deadline, it stops the request before calling the model. With these rules back in v2, the number of people who gave up waiting went down from 18 to 8 in the overload test. But this 29-second request was just under my 30-second deadline, so the rules allowed it. A shorter deadline would stop it earlier, but then more people would get "busy" instead of an answer. I'm still not sure where the right line is.

Thread Thread
 
robertpaschal profile image
Odinaka Robert Nnamani •

18 to 8 is a solid drop for three simple rules. That 29-second one slipping just under the deadline is the tradeoff in plain sight: shorter cutoff, more busy replies instead of answers.

Collapse
 
alexcodebytes profile image
Oleksandr •

What I appreciate most here is that you didn't hide the trade-off behind a successful safety improvement. Getting verified sentences to users faster is valuable, but losing throughput under heavy load is a real cost that deserves attention.

One thing I'd be particularly interested in seeing in a follow-up is how the results change with actual Groq quota behavior rather than the stand-in model. That caveat could significantly change the interpretation of the 62.8% vs. 67.4% result.

Also, I think the distinction between retrieval quality and answer verification is important. Finding relevant passages is only half the problem; making sure the generated claim is actually supported by them is where trust comes in.

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Yes, the real Groq test is exactly what I am planning next and going to implement it . I will cancel some streams and let the others finish then check whether the quota comes back, and I will be sharing the results in a follow-up. And I fully agree on your last point that finding the right passage is only half the job. Checking that every claim is really supported by that passage is where trust comes from. Thanks for the insights this will actually helping me to look on more things.

Collapse
 
alexcodebytes profile image
Oleksandr •

That sounds like a really useful next experiment. I'm especially interested in whether cancelling some streams actually restores quota for the remaining ones, or whether the provider handles that differently than expected.

Please share the results when you have them. Real measurements from this kind of test are much more valuable than assumptions about how an API behaves. And I'm glad the discussion helped — looking forward to the follow-up! 😄

Collapse
 
innokentyb profile image
Kent Bodrov •

The SingleFlight result is a useful reminder that sharing work can matter more than speeding up a model call. For your three public papers, matching the question may be enough for this experiment. If you later try it with private documents, I’d scope both the in-flight key and the answer cache to the caller’s authorization context and document version, then check access again before returning a shared result. The same question from two users doesn’t necessarily have the same permitted evidence. Would you consider a test where access is revoked while the first request is still running?

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you this is a really good point. Right now the shared key is not just the question: both the in-flight key and the answer cache include the collection id and its version.
Collections belong to the one tenant, so two tenants never share an answer and also I have a test for that. Adding a document changes the version so old answers stop matching.
But you found a real gap: I check access only once, when the request starts. If a key is revoked while the answer is still running, a request waiting on that shared answer would still get it, because nothing checks again before returning it. Also, access is per tenant, not per user, so for private documents with different permissions inside one team this would not be enough. So yes, I will add exactly the test you describe: revoke access while the first request is running, and make sure the waiting request gets refused instead of the shared answer. Thanks for this, it is the kind of thing that is easy to miss

Collapse
 
sara_builds profile image
Sara •

The SingleFlight numbers are useful—17 requests joined an in-flight call while 7 came from cache. Did you test versioning the cache key with the paper/retrieval corpus, so a verified answer can’t be reused after its evidence changes?

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thanks Yes, the evidence is part of the key. For the three-paper path, the cache key includes a hash of the whole corpus plus the retrieval setting, prompt version and model so if the paper text, the prompt or the model changes, old answers simply stop matching. For the uploaded collections, the key includes the collection id and its version and I have a test for exactly this like ask a question, get a cache hit, add a new document to the collection, ask again and it's a miss,
so the old verified answer is not reused.
One case I haven't tested yet is the evidence changing while an answer is still being generated. I test deleting a collection mid-answer (nothing gets cached), but not a new document arriving mid-answer. That is a good next test to add

Collapse
 
aistackguide profile image
AI Stack Guide •

On your closing question: let the stream finish and mark the answer unusable, if and only if the provider does not refund cancelled quota. You can settle that without burning a day — send a request you cancel two seconds in, then immediately read x-ratelimit-remaining-requests and the reset delta, and compare it with a request you let complete. If the delta moves the same amount, stopping early buys you nothing except latency.

The bigger lever is upstream of the cancel. Your flow already refuses before any model call when no passage is relevant enough (five of six). Extending that to a cheap per-sentence check on the retrieved passages — exact quote span plus lexical overlap, no model — would catch most of those seven midstream stops before the first token is spent, which turns a quota loss into the refusal you called cheapest.

One note on your delivery numbers: 62.8% vs 67.4% on a stand-in model makes that gap a property of your scheduler rather than of Groq. Re-measuring delivery at a fixed real quota budget, say 20 calls per run, would let the comparison survive your own small-sample caveat.

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you and this is really a useful feedback. On the refund test I like the idea and it costs only a few real calls. One small change that the requests header goes down by 1 for every call, cancelled or not so it can't show a refund. The one to compare is x-ratelimit-remaining-tokens. Tokens also refill all the time, so I will repeat the test a few times not just once.
And you are actually right about the logic: if Groq does not refund, finishing the stream costs no extra quota, and the user still never sees the bad sentence. On checking passages before the model starts, I checked my run and all the seven stops were for a sentence with no citation. That is a problem in the model's output not in the passages, so a check before generation would not catch it.
I think that the fix there is on the prompt side. On the delivery numbers, I agree the gap comes from my scheduler and my stand-in's quota rule, not from Groq. But 20 real calls cannot create overload unless I push my real account to its limit, and I avoid doing that on purpose. So the refund test is the real check that matters: it tells me whether my stand-in is right about cancelled quota. I will run it and share the result here.

Collapse
 
brookiecat profile image
Brookie •

a tiny queue, no lobby, a hard no at the door before anyone's hopes go up, and the one request that beats the cutoff gets 29 seconds of silence then an answer like nothing happened. you didn't build a relay. you built a cat. 🐾

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Haha guilty 😹 but in the v2 the cat at least meows now. Within about 10 ms, it tells you that "I got your request" and then it keeps sending the small updates while you wait in the queue. So no more 29 seconds of silence. The "no at the door" part I am keeping though. Cats have standards

Collapse
 
nexusfoundry profile image
Nexus Foundry •

The 4.6-point delivery drop is the right thing to publish — most "we made it safer" posts never show the bill. On the mechanism: if cancelled streams still burn reserved quota, the cheapest fix isn't faster checking, it's narrower checking. A per-sentence verifier doesn't need the full RAG context — claim + cited passage is maybe 200 tokens. If your checker is re-reading the whole retrieved set per sentence, that's likely where the quota went, not the verdict itself.

Second lever: verdict caching. RAG answers repeat heavily — the same boilerplate sentences with the same citations show up across hundreds of queries. Key the cache on (normalized claim, passage hash) and most checker calls collapse to lookups. The safety property survives because the verdict is still deterministic per claim+source pair; you're just not paying for it twice.

And on the "let the stream finish then mark unusable" question — the refund-settlement point from aistackguide is the one that decides it. If cancelled quota isn't refunded, streaming-then-retracting is strictly worse than failing fast; if it is refunded, the UX math flips entirely.

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh • • Edited

Thank you for the insights and one small clarification that my checker doesn't call the model at all, so it doesn't use any Groq quota. The sentence checks are plain code and the NLI judge runs locally on my machine. So the quota loss comes from the cancelled answer stream itself not from the checking.
Your caching idea is still interesting for the local NLI model because that is the slowest check, but I already cache full verified answers so repeated questions skip everything. On the last point, I agree the refund result decides it.
One more thing that my stand-in gives back the unused quota only when a stream finishes normally so if a Groq works in the same way, letting it finish might actually save the quota. That's why I want to test it on real Groq

Collapse
 
the_daily_flare profile image
The daily flare •

I think the queue finding is the most useful part, as it shows safety is not the real bottleneck here.

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you, I agree. The biggest surprise for me was that the slowest request spent 29.2 of its 29.8 seconds just waiting for quota and the model work was only about 0.6 seconds.
So under the heavy load the real limit is the quota and the queue not the checking. Since writing the post I have dug further and the safety cost turned out to be even smaller than I wrote. Part of the drop came from a bug in my simulated model and part from two features around the checker (an advisory scoring step and the per-tenant queue rules), not from checking each sentence. I am re-measuring now and will share the full numbers in a follow-up post