If one provider's outage takes your feature down, you need a fallback path, and you have three realistic options. A hosted router such as OpenRouter is the least code but adds a third party to your request path. A self-hosted layer such as LiteLLM keeps keys and logs in your hands but is something you now operate. A 40-line wrapper around two SDKs is enough when you only ever call two models.
The symptom that sends people searching for this looks like 529 overloaded_error, 503 Service Unavailable, or a stream that opens and then dies halfway. I'll walk through what to fix first, then compare the three approaches. Model names below are placeholders; substitute whatever you run as of November 2026.
Why does retrying the same provider not fix a 529 or 503?
Retries handle blips. Overload and regional incidents last minutes to hours, and a retry loop with backoff just queues your users behind the problem. Worse, if every worker retries, you add load to a provider that is already shedding it.
The dead end I see most often: bumping max_retries in the SDK from 2 to 10. That turns a fast failure into a 90-second hang, and your own request timeouts then fire first. Retries are for 429s with a Retry-After header and for transient network errors. A different provider is the answer to "this one is down."
Takeaway: retry for blips, fail over for outages, and never use the same mechanism for both.
What should trigger a failover, and what should not?
Before choosing a tool, decide the policy, because every option below is only as good as the rules you feed it.
| Error class | Fail over? | Why |
|---|---|---|
| 5xx, 529, connection reset, timeout | Yes | Provider-side or network problem |
| 429 rate limit | Retry first, then fail over | Often your own quota; the backup may have headroom |
| 400 invalid request / context too long | No | The backup will likely reject it too, or silently truncate |
| 401/403 | No | Your key is wrong; failing over hides a config bug |
| Content-policy refusal | No | Not an outage; behavior differs by model |
The row people get wrong is context length. A fallback model with a smaller context window will fail your longest requests, which are usually your most important ones. Check the window of every model in the chain before you ship it.
Takeaway: a fallback chain that ignores 4xx semantics will mask your own bugs as provider flakiness.
How does LiteLLM handle fallbacks?
LiteLLM is an open-source Python library plus an optional proxy server that exposes an OpenAI-compatible API in front of many providers. Its Router takes a list of deployments and a fallback map.
import os
from litellm import Router
router = Router(
model_list=[
{
"model_name": "chat",
"litellm_params": {
"model": "anthropic/claude-sonnet-4-5", # placeholder
"api_key": os.environ["ANTHROPIC_API_KEY"],
},
},
{
"model_name": "chat-backup",
"litellm_params": {
"model": "openai/gpt-4o", # placeholder
"api_key": os.environ["OPENAI_API_KEY"],
},
},
],
fallbacks=[{"chat": ["chat-backup"]}],
num_retries=1,
timeout=20,
)
resp = router.completion(
model="chat",
messages=[{"role": "user", "content": "Summarize this ticket."}],
)
print(resp.choices[0].message.content)
What it does well: keys never leave your infrastructure, you can set per-model timeouts and cooldowns, and the same code path covers dozens of providers. What tripped me up: the fallback fires per request, so during a long outage every request pays the primary's timeout first unless you configure cooldowns for failing deployments. Also, running the proxy server means you own its uptime, its database if you enable spend tracking, and its upgrades. If you want a self-hosted gateway with fallbacks, budgets, and one OpenAI-compatible endpoint for every team, LiteLLM is the one that gives you that without sending traffic through a third party.
Takeaway: LiteLLM trades a service you rent for a service you run, and the timeout-before-fallback latency is the part to tune.
How does OpenRouter handle fallbacks?
OpenRouter is a hosted API that fronts many providers with one key and one billing relationship. You send an ordered list of models and it tries them in turn.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ["OPENROUTER_API_KEY"],
)
resp = client.chat.completions.create(
model="anthropic/claude-sonnet-4.5", # placeholder
messages=[{"role": "user", "content": "Summarize this ticket."}],
extra_body={
"models": [
"anthropic/claude-sonnet-4.5",
"openai/gpt-4o",
]
},
)
print(resp.choices[0].message.content)
This is the least code, and one account replaces several provider invoices. The drawbacks are real, though. Your prompts and your users' data pass through one more company, which matters if your contracts or compliance posture name specific subprocessors. The router itself becomes a single point of failure, so you have traded N providers' outages for N providers' outages plus one. And pricing includes a platform layer; check the current fee model on their site rather than trusting my memory. If you want one key and one invoice and don't want to run anything, OpenRouter is the lowest-effort way to get multi-provider fallback.
Takeaway: OpenRouter removes operational work and adds a dependency you cannot debug from the inside.
When is a hand-rolled fallback enough?
If you call exactly two models and the request shapes are simple, a wrapper is shorter than the config for either tool. The key is classifying errors, not looping blindly.
import os
import anthropic
import openai
primary = anthropic.Anthropic(max_retries=1, timeout=20)
backup = openai.OpenAI(max_retries=1, timeout=20)
def ask(prompt: str) -> str:
try:
msg = primary.messages.create(
model="claude-sonnet-4-5", # placeholder
max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
return msg.content[0].text
except (anthropic.APIConnectionError, anthropic.APITimeoutError,
anthropic.InternalServerError):
pass # 5xx, 529, network, timeout: fail over
# 4xx (BadRequestError, AuthenticationError) propagate on purpose
r = backup.chat.completions.create(
model="gpt-4o", # placeholder
messages=[{"role": "user", "content": prompt}],
)
return r.choices[0].message.content
The limitation: it does not scale. The third provider, tool calling with different schemas, and streaming mid-failure each add code, and that is the point where one of the two tools above starts paying for itself. Also note that a stream that dies halfway cannot be transparently resumed on another model; decide up front whether you discard the partial output and restart.
Takeaway: two models and no tool calls means write the wrapper; anything more and you are rebuilding a gateway.
Comparison
| LiteLLM (self-hosted) | OpenRouter | Hand-rolled | |
|---|---|---|---|
| Keys and prompts stay with you | Yes | No | Yes |
| Ops burden | Medium (proxy, DB optional) | None | Low, grows with providers |
| Added failure point | Your own proxy | Their service | None |
| Providers supported | Many | Many | Whatever you code |
| Best for | Platform teams, budgets | Prototypes, small teams | Two-model apps |
FAQ
Should I retry or fail over when my LLM API returns 529 overloaded?
Retry once with a short backoff, then fail over. Repeated retries during an overload extend the outage for your users and add load to the provider.
Does failing over to a different model change my output quality?
Yes. Prompts tuned for one model often behave differently on another, so run your evaluation set against the backup model before you rely on it in production.
Can I resume a streamed LLM response on another provider after it fails midway?
Not transparently. The backup model has no knowledge of the partial output, so you either restart the request or pass the partial text as context and accept a visible seam.
Bottom line
If you call two models and want to keep control, write the wrapper and classify errors carefully. If you are a platform team that needs shared keys, budgets, and many providers behind one endpoint, run LiteLLM. If you are a small team that wants one key and zero operations, accept the extra dependency and use OpenRouter. Whichever you choose, test the fallback by blocking the primary's hostname in staging, because an untested failover path is the one that fails during the incident.
Top comments (3)
The piece this usually misses is what happens to the secondary the moment the primary trips. Every client fails over at the same instant, so the backup gets its normal load plus the whole primary's traffic in one burst - and 503/529 both mean "retry later", so the burst arrives with retries attached.
Two things that helped in a similar setup: treat Retry-After as authoritative when it is present and use jittered backoff otherwise, and reserve headroom on the secondary instead of treating it as free capacity. A fallback you can oversubscribe is a shared-fate design, not a resilience one.
Worth instrumenting failover rate per provider separately - it predicts the next incident better than error rate does.
Worth classifying 429 separately from 529 and 503, since a rate limit on your own key may clear with backoff while an overloaded provider is better skipped right away.
Right, and the distinction has a billing consequence too: a 429 is your budget, a 529 or 503 is their capacity. So a 429 should reduce concurrency and honour Retry-After on the same key, while 529/503 should fail over immediately - retrying those just adds load to something that is already shedding.
The number that makes a 429 diagnosable is your own in-flight count at the moment it arrives. Without it you cannot separate "the provider is tight" from "my own agents are the tightness", and those need opposite responses.