DEV Community

Ayush Singh
Ayush Singh

Posted on

I Made My RAG Safer. It Got Worse Under Load

Free-tier limits and queue times exposed

RAG isn't dead, but unverified RAG should be. Here's what checking every sentence actually cost me on Groq's free tier, and a few other things I learned along the way.

I rebuilt my RAG system so that no sentence reaches the user until it has been checked against the passage it cites. The first sentence started showing up sooner, and nothing unverified was ever shown.

Then I ran it under heavy overload. It delivered 62.8% of requests. The old version delivered 67.4%.

Honestly this one hurt a bit, because my own safety feature was the reason. I will explain it below.
That run also changed how I read the "RAG is dead" posts. Yes, context windows are huge now, and coding agents find things with grep instead of vector databases. But none of that answers the question that actually cost me those requests: who checks the answer?

TL;DR

  • Checking every sentence made answers appear sooner, but under heavy load it cost throughput: 62.8% delivered against 67.4%.
  • When 30 requests came in 2 seconds, the system made only 6 model calls. Same questions share one answer.
  • The cheapest correct answer is a refusal that never calls the model.
  • My slowest request spent 29.2 of its 29.8 seconds just waiting in a queue.
  • A small "is this sentence supported?" judge model would have blocked about 1 in 5 good sentences.

What I built

The Relay answers questions only from three research papers: Attention Is All You Need, Moshi and H-Net. Every sentence must cite the passage it came from. If the papers don't say something, it has to say so instead of guessing. And it has to survive Groq's free tier while requests come in bursts.

The flow is: queue → cache → retrieval (BM25 plus embeddings) → evidence check → model call → per-sentence verification. No LangChain, no vector database. The papers make only about a thousand chunks, so plain NumPy search takes under a millisecond.
You can try it without any API key. It ships with a stand-in model that I calibrated from real Groq measurements and most of my load tests ran against that.

1. "RAG is dead" is really about retrieval. My problem was trust.

The good "RAG is dead" arguments aren't saying what the headlines say. This AI Engineer talk points out that grep and BM25 are retrieval and the debate mixes up RAG with vector search. This post by Akita argues that agents are moving to big context and simple text search.

I agree with both. But my requirements were:

  • Answer only from these papers.
  • Show where each claim came from.
  • Say no when the papers don't cover it.

A bigger context window doesn't give you any of these. A model with all three papers in front of it can still write a sentence the papers don't support. On a free tier, sending all three papers with every question would also blow past the per-minute limit straight away. (To be fair, I didn't test a long-context version, so I'm not claiming it gives worse answers.)

So retrieval isn't dead. The mandatory vector database might be. Answering without checking should be.

2. When everyone asks at once, ask the model once

Bursts are when things break: something goes wrong, and suddenly everyone asks the same thing at the same moment.

So I added SingleFlight. If the same question is already being answered, a new request doesn't make its own model call. It just waits for that answer and gets the same verified sentences. Once the answer is done, repeats come straight from the cache.

In my burst test, 30 requests arrived within 2 seconds, but there were only 6 different questions. The result:

What happened Requests
Made a model call 6
Joined a call already in progress 17
Served from the cache 7
Delivered 30 of 30

That's 30 answers from 6 model calls, one per distinct question. On a free tier, this is the difference between serving everyone and getting rate-limited in the first few seconds.
It also works with more than one server. With Redis shared between workers, my tests send 10 identical requests to two workers at the same time, and they make exactly one model call in total.


3. The best "no" never calls the model

I wrote 21 test questions. Six of them the papers can't answer: blockchain, photosynthesis, the TCP handshake, Newton's laws, MySQL vs MongoDB, and one trap.

Across three retrieval settings, the system said no correctly on all 17 graded cases. The 18th hit a network error, and I left it in the results as an error rather than quietly re-running it.

The interesting part is how. With reranking on, five of the six were refused before any model call, because no passage was relevant enough to answer from. No quota spent, just an honest "the papers don't cover this."


4. Three papers can still fool a naive retriever

The bibliography trap. I asked how WavLM is pretrained. The Moshi paper uses WavLM and cites it, so the keywords match strongly, but it never explains WavLM's pretraining. A naive system finds those passages and the model fills the gap from memory. My widest retrieval setting fell for it.

Shared words. "Transformer", "attention" and "chunk" appear in all three papers, so they can't be trusted to tell the papers apart.

Broken PDF text. The attention formula came out garbled when the PDF was converted to text. One answer picked up a stray 0 from it, the checker caught it, and the user got a safe refusal. That was technically a wrong refusal, but it was wrong in the safe direction, and I'll take that trade every time.


5. Two reasoning models thought until they ran out of room

I compared three Groq models on the same six questions. On three of them, both gpt-oss-20b and gpt-oss-120b used their entire output limit on thinking and never wrote an answer. qwen3.8-27b was never cut off, and every answer passed the citation checks. I picked qwen.

Reasoning models aren't bad. But test the model inside your own limits, not on a leaderboard.


6. Read the rate-limit headers carefully

Groq's x-ratelimit-limit-requests: 1000 header looked per-minute to me at first. But its reset time grew by exactly 86.4 seconds per request, and 86,400 seconds in a day divided by 1,000 is 86.4. So it behaves like a daily limit. I haven't proven it, because that would mean watching it refill for hours (Groq's [rate limits page] has the official numbers.)

My quota guard therefore checks the budget before every call, and there are no hidden retries: every retry is counted. In a test where 30% of calls failed on purpose, a retry limit cut extra attempts (1.27 per request against 1.41), but with the limit on, 2 requests failed, and in the run without it, none did. I still keep the limit, because in a real outage, extra retries only hit a provider that is already struggling.


7. Your slow request is probably a queue

Under overload, my slowest request took 29.8 seconds. Of that, 29.2 seconds was waiting for quota. The actual work, retrieval plus the model, took about 0.6 seconds.

We usually blame the model when things are slow. Under load, my problem was the queue. I could only see that because every request records its waiting, working and retrying time separately.


8. Safety has a cost, and mine showed up under overload

v1 waited for the full answer, checked it, then sent it all at once. v2 checks each sentence as it streams. If a sentence fails, the answer stops there, and the bad sentence is never shown.

In normal bursts, this helped: the first verified sentence arrived in a median of 324.5 ms, where v1 showed nothing until the whole answer was ready (614.2 ms median).

Under heavy overload, though, v2 delivered 62.8% against v1's 67.4%.

The reason: when the checker stops an answer halfway, the model stream is cancelled, but the quota reserved for that call stays used for the rest of the minute. Seven answers were stopped early in that run. That held-up quota was worth about three calls, and v2 made exactly three fewer calls than v1.

I fixed the part on my side, and delivery went up from 55.8% to 62.8%. The rest is quota held by the provider. One important caveat: I measured this on my stand-in model. I don't know yet whether Groq gives that quota back on a cancelled stream. If it does, most of this gap may not exist in real use.

So the trade-off is clear: users see verified sentences sooner and never see a wrong one, and under heavy load, slightly fewer people get an answer.


9. Test your judge model before trusting it

The obvious next step was a small AI model that judges whether a passage really supports a sentence. I scored 224 real sentences with one. At the usual 0.5 threshold, it would have blocked about 1 in 5 good sentences, and still missed more than a third of the bad ones.

So it only records a score and blocks nothing. A judge model has its own error rate, so measure it before you let it decide anything. (The labels I compared against were drafted with AI help, and I haven't reviewed them myself yet, so treat these rates as rough.)


What I'm not claiming

  • Answer correctness isn't final. Strict word matching marked 19 of 45 answers correct, but most of the "wrong" ones were flagged only because one exact phrase was missing, and several look like simple wording differences. I haven't graded them by hand yet. Citation checks are the firmer number: 44 of 45 passed.
  • Most load numbers come from the stand-in, not real Groq. I kept real calls few on purpose, to protect the free quota.
  • The samples are small, so small differences are within noise.
  • The judge-model analysis isn't public yet. The raw data behind every other number is in the repo linked below.

So, is RAG dead?

The version where you embed everything, grab the top five chunks and trust whatever the model says, yes, that should go.

What should stay: find the right passages, cite them, check every claim before showing it, say no when the evidence isn't there, and plan for your limits. Whether you find those passages with vectors, BM25, grep or a huge context window matters much less than whether anything checks the answer.

Over to you all: when a sentence fails the check halfway through a stream, what would you do? Stop and lose the quota like I do, or let the model finish and throw the rest away? I'm really curious what you'd pick, and why.

Top comments (27)

Collapse
 
eliashivemind profile image
Nova Ops •

The mechanism you found is the part most safety write-ups never reach: the checker stops an answer halfway, the stream is cancelled, but the quota reserved for that call stays spent for the rest of the minute — so seven stopped answers quietly cost about three calls, which is exactly the three-call gap between v2's 62.8% and v1's 67.4%. The safety feature didn't slow the work; it taxed the budget the work runs on. The queue number makes the same point from the other side — 29.2 of 29.8 seconds waiting, 0.6 seconds of actual work — and you could only see either number because waiting, working and retrying are recorded separately. We learned the matching lesson with a release gate that fails closed: the check itself is cheap, what it costs is everything downstream that has to stop and wait for it. Your honest caveat is that this was measured on the stand-in model — do you plan a smaller real-Groq run to settle whether cancelled-stream quota actually comes back, since that one answer decides whether the 62.8-vs-67.4 gap exists outside the simulation?

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you you explained it better than I did: the safety check didn't slow the work, it taxed the budget the work runs on.
I like your release-gate example too. The check is cheap, but everything behind it has to wait. One small detail: v2 made 3 fewer calls than v1 (53 against 56), but delivered 4 fewer answers (54 against 58), so the lost quota explains most of the gap, not all of it. And yes, a small real-Groq test is the next thing I want to do .
The plan is simple that send a few requests, cancel some of them early and let the others finish, then compare x-ratelimit-remaining-tokens after each. Tokens refill all the time, so I will repeat it a few times instead of trusting one reading. It needs only a handful of calls and doesn't come anywhere near the limit. If Groq gives the quota back, most of this gap may not exist in real use. If it doesn't, then letting the stream finish quietly may be better than stopping it. I'll share the result here when I have it

Collapse
 
eliashivemind profile image
Nova Ops •

That correction matters, and credit for making it against your own headline number: 3 fewer calls but 4 fewer answers means the cancelled-quota story explains most of the gap, not all of it — there's a fourth lost answer in there with a different cause, and a write-up that rounds that away would be claiming a cleaner mechanism than the data shows. Your test plan is the right shape, especially repeating it because the tokens refill — a single reading either way would be measuring the refill as much as the refund. One thing to watch when you run it: compare the cancelled and finished readings at the same distance from the minute boundary, since a cancel late in the minute and a finish early in the next could look like a refund that never happened. When the result lands, will you post it as an update to this article or as a follow-up post?

Thread Thread
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you and you are actually right that I shouldn't round that fourth answer away. Honestly I don't know its cause yet In that run, the missing answers showed up as more "busy" refusals at the door (24 against v1's 19) but I have only one run of each version, so one extra could also be normal variation. I won't claim more than the data shows.
Your minute-boundary point is also a good catch. In my earlier probes, the token reset time was only a few seconds each time so I think Groq refills tokens all the time not once a minute, but I haven't proven it and now Ik that I have to work on it. So I will do what you said let the tokens fully refill before each pair, take the cancelled and finished readings at the same timing and then record the reset time with every reading.
About sharing, I will write it as a short follow-up post and add a note at the top of this article linking to it, so anyone reading this one sees the result too. Thanks for these insights.

Thread Thread
 
eliashivemind profile image
Nova Ops •

Recording the reset time with every reading is exactly the move that turns the refill question from a belief into data — if the refill really is continuous, the readings should show it without you having to argue for it. One run of each is also the honest limit to state: in our own paired work the run-to-run wobble has a habit of being bigger than the effect being chased, so a single extra refusal can't carry a conclusion either way. If refill turns out to be continuous, would spacing since the previous call — rather than position inside the minute — become the variable you log for the finished arm too?

Collapse
 
fgrandon profile image
Francisco Grandón •

Seeing delivery drop from 67.4% to 62.8% when adding per-sentence verification is a sobering trade-off, and SingleFlight collapsing 30 burst requests into 6 model calls is a clean way to protect a free-tier quota. Sharing that in-flight state via Redis across workers keeps bursts from multiplying upstream calls. Which queue timeout or shedding rule worked best when those 29-second waits started piling up?

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you and Honestly I did not test the each rule alone so I can't say which single one is best but three simple rules work together. First the queue is very small and when it is full a new request is turned away straight away (lower-priority requests are dropped first). Second, at the door the system guesses the waiting time from how many requests are ahead, and if that guess is past the deadline it says "busy" immediately instead of making the person wait. Third and I think this one helped the most, the quota guard knows exactly when quota will free up so if that time is after the deadline it stops the request before calling the model. When I put the deadline check back in v2 the number of people who gave up waiting went down from 18 to 8 in the overload test. One honest thing: my deadline is 30 seconds, and that 29-second request was just under it, so the rules allowed it. A shorter deadline would turn it away earlier, but more people would get "busy" instead of an answer. I am still not sure where the right line is. What deadline do you use in production?

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

29.2 seconds of a 29.8-second request spent waiting on quota is a rough number to read. The model took 0.6 seconds and the queue took the rest, so the slowness was never the model at all.

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Yes exactly this actually surprised me as well If I would only looked at the total time I would blame the model and maybe the change to a faster model but that would not help much because the model was only 0.6 seconds and the rest was waiting. My system tries to guess the waiting time at the start and if the wait is too long it says "busy right now" straight away because a quick honest answer is better than making someone wait 29 seconds. Also making fewer calls helped me more than making faster calls: the cache and SingleFlight share one answer for the same question, so the queue stays small.
But I should be fair on this that this was a heavy load test on my stand-in model. In a normal run with real Groq the slow part was the model, not the waiting.
Just wanted to ask that have you seen the same thing in production? Do people fix it with more quota or by saying no earlier?

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

Saying no earlier is the fix that holds. More quota just moves the same wait to a bigger bill.

Thread Thread
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

One more thing I want to add. After seeing this I looked at how my system decides who should wait and who should hear "busy". There are three simple rules. The queue is very small, and when it is full, new requests are turned away straight away. At the door, the system guesses the waiting time, and if the guess is past the deadline, it says "busy" immediately.
And the quota guard knows when quota will free up, so if that is after the deadline, it stops the request before calling the model. With these rules back in v2, the number of people who gave up waiting went down from 18 to 8 in the overload test. But this 29-second request was just under my 30-second deadline, so the rules allowed it. A shorter deadline would stop it earlier, but then more people would get "busy" instead of an answer. I'm still not sure where the right line is.

Thread Thread
 
robertpaschal profile image
Odinaka Robert Nnamani •

18 to 8 is a solid drop for three simple rules. That 29-second one slipping just under the deadline is the tradeoff in plain sight: shorter cutoff, more busy replies instead of answers.

Collapse
 
alexcodebytes profile image
Oleksandr •

What I appreciate most here is that you didn't hide the trade-off behind a successful safety improvement. Getting verified sentences to users faster is valuable, but losing throughput under heavy load is a real cost that deserves attention.

One thing I'd be particularly interested in seeing in a follow-up is how the results change with actual Groq quota behavior rather than the stand-in model. That caveat could significantly change the interpretation of the 62.8% vs. 67.4% result.

Also, I think the distinction between retrieval quality and answer verification is important. Finding relevant passages is only half the problem; making sure the generated claim is actually supported by them is where trust comes in.

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Yes, the real Groq test is exactly what I am planning next and going to implement it . I will cancel some streams and let the others finish then check whether the quota comes back, and I will be sharing the results in a follow-up. And I fully agree on your last point that finding the right passage is only half the job. Checking that every claim is really supported by that passage is where trust comes from. Thanks for the insights this will actually helping me to look on more things.

Collapse
 
alexcodebytes profile image
Oleksandr •

That sounds like a really useful next experiment. I'm especially interested in whether cancelling some streams actually restores quota for the remaining ones, or whether the provider handles that differently than expected.

Please share the results when you have them. Real measurements from this kind of test are much more valuable than assumptions about how an API behaves. And I'm glad the discussion helped — looking forward to the follow-up! 😄

Collapse
 
innokentyb profile image
Kent Bodrov •

The SingleFlight result is a useful reminder that sharing work can matter more than speeding up a model call. For your three public papers, matching the question may be enough for this experiment. If you later try it with private documents, I’d scope both the in-flight key and the answer cache to the caller’s authorization context and document version, then check access again before returning a shared result. The same question from two users doesn’t necessarily have the same permitted evidence. Would you consider a test where access is revoked while the first request is still running?

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you this is a really good point. Right now the shared key is not just the question: both the in-flight key and the answer cache include the collection id and its version.
Collections belong to the one tenant, so two tenants never share an answer and also I have a test for that. Adding a document changes the version so old answers stop matching.
But you found a real gap: I check access only once, when the request starts. If a key is revoked while the answer is still running, a request waiting on that shared answer would still get it, because nothing checks again before returning it. Also, access is per tenant, not per user, so for private documents with different permissions inside one team this would not be enough. So yes, I will add exactly the test you describe: revoke access while the first request is running, and make sure the waiting request gets refused instead of the shared answer. Thanks for this, it is the kind of thing that is easy to miss

Collapse
 
sara_builds profile image
Sara •

The SingleFlight numbers are useful—17 requests joined an in-flight call while 7 came from cache. Did you test versioning the cache key with the paper/retrieval corpus, so a verified answer can’t be reused after its evidence changes?

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thanks Yes, the evidence is part of the key. For the three-paper path, the cache key includes a hash of the whole corpus plus the retrieval setting, prompt version and model so if the paper text, the prompt or the model changes, old answers simply stop matching. For the uploaded collections, the key includes the collection id and its version and I have a test for exactly this like ask a question, get a cache hit, add a new document to the collection, ask again and it's a miss,
so the old verified answer is not reused.
One case I haven't tested yet is the evidence changing while an answer is still being generated. I test deleting a collection mid-answer (nothing gets cached), but not a new document arriving mid-answer. That is a good next test to add

Collapse
 
aistackguide profile image
AI Stack Guide •

On your closing question: let the stream finish and mark the answer unusable, if and only if the provider does not refund cancelled quota. You can settle that without burning a day — send a request you cancel two seconds in, then immediately read x-ratelimit-remaining-requests and the reset delta, and compare it with a request you let complete. If the delta moves the same amount, stopping early buys you nothing except latency.

The bigger lever is upstream of the cancel. Your flow already refuses before any model call when no passage is relevant enough (five of six). Extending that to a cheap per-sentence check on the retrieved passages — exact quote span plus lexical overlap, no model — would catch most of those seven midstream stops before the first token is spent, which turns a quota loss into the refusal you called cheapest.

One note on your delivery numbers: 62.8% vs 67.4% on a stand-in model makes that gap a property of your scheduler rather than of Groq. Re-measuring delivery at a fixed real quota budget, say 20 calls per run, would let the comparison survive your own small-sample caveat.

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you and this is really a useful feedback. On the refund test I like the idea and it costs only a few real calls. One small change that the requests header goes down by 1 for every call, cancelled or not so it can't show a refund. The one to compare is x-ratelimit-remaining-tokens. Tokens also refill all the time, so I will repeat the test a few times not just once.
And you are actually right about the logic: if Groq does not refund, finishing the stream costs no extra quota, and the user still never sees the bad sentence. On checking passages before the model starts, I checked my run and all the seven stops were for a sentence with no citation. That is a problem in the model's output not in the passages, so a check before generation would not catch it.
I think that the fix there is on the prompt side. On the delivery numbers, I agree the gap comes from my scheduler and my stand-in's quota rule, not from Groq. But 20 real calls cannot create overload unless I push my real account to its limit, and I avoid doing that on purpose. So the refund test is the real check that matters: it tells me whether my stand-in is right about cancelled quota. I will run it and share the result here.

Collapse
 
brookiecat profile image
Brookie •

a tiny queue, no lobby, a hard no at the door before anyone's hopes go up, and the one request that beats the cutoff gets 29 seconds of silence then an answer like nothing happened. you didn't build a relay. you built a cat. 🐾

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Haha guilty 😹 but in the v2 the cat at least meows now. Within about 10 ms, it tells you that "I got your request" and then it keeps sending the small updates while you wait in the queue. So no more 29 seconds of silence. The "no at the door" part I am keeping though. Cats have standards

Collapse
 
nexusfoundry profile image
Nexus Foundry •

The 4.6-point delivery drop is the right thing to publish — most "we made it safer" posts never show the bill. On the mechanism: if cancelled streams still burn reserved quota, the cheapest fix isn't faster checking, it's narrower checking. A per-sentence verifier doesn't need the full RAG context — claim + cited passage is maybe 200 tokens. If your checker is re-reading the whole retrieved set per sentence, that's likely where the quota went, not the verdict itself.

Second lever: verdict caching. RAG answers repeat heavily — the same boilerplate sentences with the same citations show up across hundreds of queries. Key the cache on (normalized claim, passage hash) and most checker calls collapse to lookups. The safety property survives because the verdict is still deterministic per claim+source pair; you're just not paying for it twice.

And on the "let the stream finish then mark unusable" question — the refund-settlement point from aistackguide is the one that decides it. If cancelled quota isn't refunded, streaming-then-retracting is strictly worse than failing fast; if it is refunded, the UX math flips entirely.

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh • • Edited

Thank you for the insights and one small clarification that my checker doesn't call the model at all, so it doesn't use any Groq quota. The sentence checks are plain code and the NLI judge runs locally on my machine. So the quota loss comes from the cancelled answer stream itself not from the checking.
Your caching idea is still interesting for the local NLI model because that is the slowest check, but I already cache full verified answers so repeated questions skip everything. On the last point, I agree the refund result decides it.
One more thing that my stand-in gives back the unused quota only when a stream finishes normally so if a Groq works in the same way, letting it finish might actually save the quota. That's why I want to test it on real Groq

Collapse
 
the_daily_flare profile image
The daily flare •

I think the queue finding is the most useful part, as it shows safety is not the real bottleneck here.

Collapse
 
ayush_singh_9b0d83152be5b profile image
Ayush Singh •

Thank you, I agree. The biggest surprise for me was that the slowest request spent 29.2 of its 29.8 seconds just waiting for quota and the model work was only about 0.6 seconds.
So under the heavy load the real limit is the quota and the queue not the checking. Since writing the post I have dug further and the safety cost turned out to be even smaller than I wrote. Part of the drop came from a bug in my simulated model and part from two features around the checker (an advisory scoring step and the per-tenant queue rules), not from checking each sentence. I am re-measuring now and will share the full numbers in a follow-up post