I had a container sitting in docker ps marked (unhealthy) for the better part of a minute while every request I sent through it kept coming back 200. That shouldn't be possible if a health check is supposed to mean anything, so I stopped and built a small stack to find out exactly what Docker's HEALTHCHECK actually controls, and what it doesn't.
The honest answer turned out to be: almost nothing, on its own. I had three things pointed at the same container — Docker's own health status, a restart: always policy, and nginx sitting in front as a reverse proxy — and none of the three reacted to the other two the way I assumed they would.
The setup
Three identical Python backends behind an nginx reverse proxy, all on plain Docker Compose, bridge networking, nothing exotic:
services:
back1:
image: swarmtest:v1
restart: always
healthcheck:
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health', timeout=1)"]
interval: 2s
timeout: 1s
retries: 2
start_period: 2s
# back2, back3: identical, no restart policy
proxy:
image: nginx:1.27-alpine
ports: ["18090:80"]
Each backend is a tiny HTTP server with four routes: / returns a version string, /health reports the health flag, /break flips the health flag without touching anything else, and /crash flips the health flag and makes / itself start returning 500. That split matters, because it let me test two different failures separately: a health probe lying, and a health probe telling the truth.
nginx sat in front with a plain upstream block:
upstream backend {
server back1:8080 max_fails=2 fail_timeout=5s;
server back2:8080 max_fails=2 fail_timeout=5s;
server back3:8080 max_fails=2 fail_timeout=5s;
}
Finding 1: an unhealthy container keeps serving exactly as before
I hit back1's /break endpoint, which only flips the health flag — the real / route still answers correctly. Docker noticed within a few seconds:
NAMES STATUS
swarmtest-back1-1 Up About a minute (unhealthy)
Then I fired 160 real requests through nginx over the next 44 seconds while that status sat there unchanged.
| Result | |
|---|---|
Docker health flips to unhealthy
|
3.64s after the break |
Requests served successfully (200) |
160 / 160 |
| Requests that failed | 0 |
Nothing downstream of Docker's health state even looked at it. nginx has no concept of a Docker health check — it only knows about actual TCP connections and actual HTTP responses. A container can be "unhealthy" for as long as you like and keep taking its full share of traffic, provided the thing it's actually serving still works.
Finding 2: restart: always doesn't touch it either
I'd assumed a restart policy was the backstop here — if nginx doesn't care, surely Docker itself will restart something it's just told you is broken. It doesn't. I added restart: always to back1, broke it for real this time with /crash (so / now also returns 500), and polled docker inspect every 5 seconds for 35 seconds:
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
RestartCount=0 Status=running Health=unhealthy StartedAt=2026-11-20T05:25:00.616251195Z
Same RestartCount, same StartedAt, every single poll. A restart policy reacts to the container's main process exiting with a non-zero code (or just exiting, depending on the policy). It has no wiring to the health check at all. The process inside my container never exited — it just kept answering 500 — so from the restart policy's point of view, nothing had gone wrong.
That's the real limitation, and it's worth being specific about it: in plain Docker Engine and Compose, HEALTHCHECK is a reporting mechanism, full stop. Nothing acts on it unless you orchestrate with something that does — Swarm will reschedule a service's task, and Kubernetes has its own liveness/readiness machinery — or you write the thing that does.
Finding 3: the actual cause of the 500s, and why the obvious fix line didn't fix it
Having established that nothing automatic happens, I wanted to know what did happen to real traffic when a backend genuinely started failing. I crashed back1 the same way and sent 300 requests over 61 seconds.
| Count | |
|---|---|
Successful (200) |
200 |
Failed (500, reached the client) |
100 |
A clean third of requests failed, matching the round-robin share, and the rate never improved across the full minute — the first failure landed at 0.41s, the last at 60.77s. max_fails=2 and fail_timeout=5s were both set on the upstream, so I expected nginx to pull back1 out of rotation after two bad responses. It never did.
The config I'd written was:
proxy_next_upstream error timeout http_502 http_503 http_504;
That line looked complete to me — it's the set of codes I see in most nginx reverse-proxy examples. But nginx's own docs are specific about this: http_500 is only counted as a failed attempt if you list it explicitly, same as 502, 503, 504 and 429. My backend was returning plain 500, which wasn't in that list, so as far as max_fails was concerned, every one of those requests was a perfectly normal, successful round trip to back1. The failing status code quietly passed straight through to the client.
I added http_500 to the list and reloaded. Same crash, same 300 requests, same 61-second window:
Before (http_500 missing) |
After (http_500 added) |
|
|---|---|---|
| Failed requests reaching the client | 100 / 300 | 0 / 300 |
back1 present in any response |
yes, continuously | no, after the first couple of attempts |
With http_500 in the list, nginx retried the failing attempt against another backend inside the same client request, and back1 was excluded from the pool for the fail_timeout window after its second failure. The client never saw a single error.
This is the finding I'd act on fastest: if your application returns a bare 500 on its own errors — which is the default in most frameworks when something throws — and your proxy_next_upstream list was copied from a tutorial that only has 502 503 504, your failover silently does nothing for the exact case it's most likely to matter for.
Finding 4: the fix hides the failure from more than just the client
Masking the error at the proxy is good for whoever's using the app, but it's worth being honest about the trade-off. Any monitoring that counts response codes at the client-facing edge will now show a clean 200 for a request that actually failed once internally and was quietly retried. If you're alerting on nginx access-log status codes rather than on upstream-level metrics, a backend can be failing every single request routed to it and your dashboards won't move. I didn't instrument latency closely enough in this run to put a number on the added cost of that silent retry, but it's an extra full request-response cycle per failure, and that's not nothing under load.
Finding 5: what it actually costs to fix it yourself
Since nothing reacts to the health flag automatically, I wrote the small watcher a lot of teams end up writing: poll docker inspect, and after three consecutive unhealthy polls, issue a docker restart. Here's a real run, crash included:
t=2.11 health=healthy streak=0
CRASHED at relative t≈2s
t=6.22 health=unhealthy streak=1
t=7.26 health=unhealthy streak=2
t=8.28 health=unhealthy streak=3
t=8.28 ACTING: restarting swarmtest-back1-1
t=18.57 restart command returned
Two numbers stood out. Docker's own health check took about 4 seconds to report the problem at all, because interval: 2s and retries: 2 mean it needs two full cycles of bad probes before flipping status — that's a real, configurable cost, and it's one most people leave at whatever the tutorial they copied used. My watcher added its own 1-second poll and a 3-strikes rule on top, so the whole detection side cost about 6.3 seconds.
The bigger number was the restart itself: 10.3 seconds from "restart issued" to the command returning. My app doesn't handle SIGTERM, so Docker waited out its default 10-second grace period before killing it outright. The detection logic I wrote was the fast part. The actual recovery was dominated entirely by a stop timeout nobody had configured on purpose.
After the restart finished, I checked docker inspect out of habit, and RestartCount still read 0. It turns out that counter only moves when the restart policy fires it, not when something external calls docker restart, scripted or not. I'd been treating RestartCount as my go-to signal for "is this thing flapping" on real services. A watcher like the one above would quietly break that, so before I'd trust it anywhere that matters, I'd need to log the restarts myself rather than lean on Docker's own counter.
What I got wrong on the way
I spent a genuinely annoying ten minutes convinced my nginx config change wasn't taking effect, because docker exec ... cat /etc/nginx/nginx.conf kept showing the old file after I'd edited it on the host. The bind mount hadn't actually broken — my edit had. I'd used sed -i, which on this system replaces the file by writing a new temp file and renaming it over the old path. A bind mount of a single file is attached to the inode that existed when the container started, not to the path, so the rename left my running nginx looking at an orphaned, deleted-but-still-open copy of the old config forever. Switching back to a plain cat > file <<EOF, which truncates and rewrites the existing inode in place, fixed it immediately — once I recreated the container to re-establish the mount against the current file, since the old one was unrecoverable once detached.
Run it yourself
This is the core loop I used to measure availability through the proxy while breaking a backend on purpose:
import time, urllib.request, subprocess, json
def hit():
try:
r = urllib.request.urlopen("http://127.0.0.1:18090/", timeout=2)
return True, r.status, r.read().decode()
except Exception as e:
return False, getattr(e, 'code', None), str(e)
subprocess.run(["docker", "exec", "<your-backend-container>", "python3", "-c",
"import urllib.request; urllib.request.urlopen('http://localhost:8080/crash')"])
log = []
t0 = time.time()
for i in range(300):
ok, code, body = hit()
log.append({"t": round(time.time() - t0, 2), "ok": ok, "code": code})
time.sleep(0.2)
json.dump(log, open("log.json", "w"))
Point it at any Compose stack with a proxy in front of a few backends, and docker inspect --format '{{.State.Health.Status}}' <name> alongside it to correlate health state with what clients actually saw.
Where I landed
I'm adding http_500 to every proxy_next_upstream line I own, today, not as a nice-to-have. I'm also going to stop treating HEALTHCHECK as a safety net for anything outside Swarm or Kubernetes — in plain Docker and Compose it's a status light, and if I want it to do something, I have to wire that up myself and budget for the stop-grace-period cost when I do. The watcher pattern works, but it's worth building with the restart latency in mind rather than assuming "detected" means "fixed" a second later.
Top comments (16)
Useful experiment, and the proxy_next_upstream finding is the part people miss. A Docker healthcheck only changes the status flag, so nginx never learns about it unless you add active checks or tune max_fails and fail_timeout. In my experience the fix is to treat the health endpoint and the proxy's passive failure rules as two separate layers. I'd also add a readiness-style endpoint that starts failing before the container is truly broken, so traffic drains before errors show up.
agreed on the two layers. one catch with the active checks option though, in open source nginx the health_check directive is NGINX Plus only, so on stock nginx the passive max_fails/fail_timeout rules are the only layer that can react at all. that's partly why the missing http_500 hurt so much in my test.
the readiness idea runs into the same wall. nginx never calls your readiness endpoint, it only sees real requests failing. so to drain on plain nginx you either need something that watches readiness and rewrites the upstream (drop the server, nginx -s reload), or a proxy that does active checks natively, Traefik or HAProxy for example.
That's a useful distinction: on stock nginx the only signal is real traffic failing, so the passive rules have to carry the whole load. A small watcher that polls readiness and rewrites the upstream file, then runs nginx -s reload, is a workable stopgap, though it adds its own lag and a failure mode of its own. If draining matters a lot, HAProxy or Traefik with native active checks sounds like the cleaner route. Thanks for spelling out the Plus-only detail.
yeah, that's where I landed too. one thing that makes Traefik a nice fit for this exact setup: its Docker provider reads the container's health status and leaves unhealthy containers out of routing, so the HEALTHCHECK that nginx ignored in my test actually gets wired to something. haven't tried it against this setup yet though.
and if anyone does go the watcher route, I'd mark the server down in the upstream file rather than deleting the line, and only reload after nginx -t passes. that keeps a bad rewrite from taking the whole proxy with it.
Marking the server down instead of deleting the line is a good tip, because it keeps the upstream diff small and easy to revert. Gating the reload on nginx -t is a nice safeguard as well. If you try the Traefik route, it would be interesting to hear how quickly it drops a container after the health status flips.
Docker's
HEALTHCHECKonly updates daemon metadata that static nginx upstreams never poll, and while addinghttp_500toproxy_next_upstreamrecovers read traffic, it doubles upstream latency on failing requests and risks duplicate mutations on non-idempotent routes beforemax_failsevicts the peer. Retries hide failures; ejection stops them.mostly agree, and "retries hide failures, ejection stops them" is a good way to put it. one thing on the duplicate mutations though, nginx already guards that by default. since 1.9.13 it won't pass a POST, PATCH or LOCK to the next upstream once the request has been sent, unless you add non_idempotent to proxy_next_upstream. a 500 means it was sent, so a failing POST just goes back to the client as a 500.
which flips the risk around: no duplicates, but my test was all GETs, so the "client never saw an error" result only holds for reads. writes to that backend would still have failed until max_fails kicked it out. worth a line in the post.
The http_500 detail is the one that stings — 502/503/504 is the line every tutorial copies, and a plain 500 is the failure your app is most likely to actually return, so the failover silently does nothing for exactly the case it exists for. Finding 4 is the part I'd want a follow-up on though: once the proxy retries quietly, edge monitoring goes blind. Did you end up alerting on upstream metrics, or is that a blind spot you've decided to live with?
honestly, I didn't wire up alerting in that test, finding 4 is me naming the gap, not closing it. the cheapest fix I know of doesn't need a metrics stack though. add $upstream_status and $upstream_addr to the nginx log_format. when a request gets retried, $upstream_status shows every attempt, so you get "500, 200" with the two backend addresses next to it, while $status still says 200.
then alert on any line where $upstream_status contains a 5xx, grouped by $upstream_addr. one backend failing every request it gets shows up straight away, even though clients never see an error. I haven't tested that end to end yet, so treat it as the plan, not a result.
the proxy_next_upstream bit is the real fix in here. a healthcheck that only reports and never reroutes is just a label.
yeah, that's the whole post in one line. a status light nobody's wired to anything. the annoying part was that the proxy config looked fine, http_500 just wasn't in the list, and nothing complains about that until a backend starts throwing plain 500s.
http_500 missing from the list is the quiet failure. the config looks fine, nothing complains, then a backend starts throwing plain 500s and the healthcheck never noticed.
yep. and even with http_500 added, nginx only retries GETs and other idempotent requests by default, so a POST that hits the broken backend still gets the 500. the retry hides it for reads, not writes.
POSTs don't get the quiet retry, reads do. Writes take the 500 straight, so the list covers the easy half.
Three things pointed at the same container and none of them listening to the other two is a very common setup, and the missing http_500 in proxy_next_upstream is the kind of detail that only shows up when someone breaks it on purpose.
When Ascendra Ventures answered a tender for an airport's generative AI platform, the premise was that passenger-facing AI is an uptime business first. The platform design was treated as production infrastructure from the start: Kubernetes, geographic redundancy and automated failover, auto-scaling for passenger peaks, Prometheus and Grafana alerting, and a support spine where critical incidents are answered in 30 minutes and resolved in an hour. In a design like that, a health status that the router never reads would quietly undo most of the work.
Did you end up adding the proxy side to your alerting as well, so a container that is unhealthy but still taking traffic actually pages someone?
not yet, no. the test stopped at finding the gap. the plan is the cheap version: add $upstream_status and $upstream_addr to the nginx log_format, since a retried request logs both attempts ("500, 200") even when the client just sees a 200. then alert on any 5xx in $upstream_status, grouped by backend address, so one container failing everything it gets shows up even while the retries hide it from users. I haven't run that end to end yet though, so it's a plan, not a result.