I checked that image URL twice before I submitted it. Once from my own box, once through a resizer sitting in the Netherlands, because I had been burned before by "works here, not there" and I thought two vantage points made me careful. Both answered 200, image/png. I submitted the keyframe job, the API answered accepted, and six minutes forty-six seconds later it came back with:
media URL could not be downloaded (timeout or source server error)
The file was online. I could see it. The platform that could not download it was telling me about an image I had personally uploaded thirty minutes earlier.
So I did the natural thing: I retried. Same 6m46s, same line, same place. That's 13 minutes and 32 seconds of an afternoon spent queueing for a failure I had "proven" impossible.
What I tried before I stopped arguing
My first theory was caching — maybe the worker hit a stale edge. I appended a cache-buster to the URL. Nothing changed, which should have told me the URL was never the issue, but I kept going: re-uploaded under a different filename, checked the image dimensions were inside the documented bounds (1024×1024, well inside), re-read the mode/schema rules to make sure first_frame wasn't silently rejected as a string. Every one of those checks passed, and every one of them passed for the wrong reason.
For a while I was convinced the validator did a HEAD and the worker did a ranged GET, and that some header was missing. That was me inventing a mechanism to keep my original assumption alive. Dumb, but I've done it before and I'll do it again.
The one variable I had not touched was the host. I mirrored the exact same bytes to a domestic bucket, resubmitted with nothing else changed, and it rendered in 5m56s: 720×720, 5.2 seconds of video, 1.7 MiB. Same image. Same parameters. Same account.
I was wrong about what those two green checks proved. They proved my egress worked, twice, and nothing else. I re-ran the probes today, five days later, on a URL of the same kind, to make sure I wasn't remembering a fluke:
local GET -> 200 image/png 1,068,796 bytes 2.85s
DNS -> 199.232.161.55 / 2a04:4e42:8c::311
weserv GET -> 200 image/png 1,182,443 bytes 3.81s
Three green answers, and all three green answers are about my network path. The worker that fetches the URL minutes later lives somewhere else entirely. That is the whole bug, and it is not really a bug in the vendor's code — it's a bug in what I accepted as proof.
Accepted is not fetched
The submission endpoint did what it was built to do: it validated my JSON, wrote a record, and returned quickly. That response is a statement about the queue, not about my host. Between "accepted" and "dispatch" there is a gap of minutes and a change of network, and everything I had verified lived on the wrong side of that gap.
Two runs, 6m46s each, and the error string never changed. timeout or source server error names two causes and commits to neither. I asked for a root cause and never got one I could use — I don't know whether it's an egress policy on that region, a prefix that doesn't route, or something about the CDN's anycast edge being fine for TLS-and-HTML traffic and miserable for a 1 MB object pulled by a worker with a 10-second budget. I know the observation and the workaround, and I built the fix on exactly that much.
The fix is a hardcoded list of hosts I no longer hand to a third party's fetcher:
# generate_video.py
UNFETCHABLE_HOSTS = ("ucarecd.net", "ucarecdn.com") # backend can't fetch, mirror first
if any(h in url for h in UNFETCHABLE_HOSTS):
url = mirror_to_domestic(url) # same bytes, host the worker can actually reach
Crude. It also encodes a fact I could not get from documentation: which hosts a specific worker pool cannot reach. Note that the tuple is a list of hosts, not of reasons — I have no idea why, so I can't generalize it, and the next host that behaves like this will get past the check and cost me another 13 minutes.
The second guard came from the same afternoon. The response body reports seconds, size and quality — those are suggestions the backend accepted, not what it produced. So the script stopped trusting the response and started parsing the container: ftyp, moov, mvhd duration, tkhd dimensions, mdat length, hand-rolled because there's no ffprobe in this container. A 200 with a plausible JSON body and a truncated mdat would otherwise pass every check I cared to write.
My take, and I'll take the argument: any URL you hand to a third-party server-side fetch is untrusted until something in that vendor's region has fetched it, and "accepted" should start a timeout window, not end your verification. The corollary is worse and I'll say it plainly — my local curl is a test of my own egress, and I keep filing its results under a heading that says the file is fine.
I've only measured this once, two hosts, two runs, 13m32s on one Tuesday.
What I'd test first
- Time the terminal state, not the request. Submit a 1×1 PNG into your own pipeline and log
accepted_at,terminal_at,reason. If your gap is minutes, your detection lag is minutes and no amount of green curl output changes that. - Get one probe on the worker's side of the wall. A cheap box in the dispatch region, or ask support which region runs the worker. All three of my green checks were in the wrong place.
- Assert on the artifact. Parse duration and dimensions out of the media file itself; treat echoed parameters in the response body as a request echo, because that's all they are.
My take on the fix that matters: alerting on the 202 is a footgun, and a pipeline that can't tell you its own dispatch latency has no failure detection at all.
If you run an async job API on Python 3.11 + Celery that fetches user-supplied URLs: what's your measured accept→terminal-failure gap, and are you alerting on the status code or on the artifact? Mine was 6m46s twice, and I only found it because a video came back empty. If you think a retry-with-backoff policy covers this class of failure, I want the case where the retry happily re-picked the host your worker can't reach — comments are open.
Top comments (7)
This is a good reminder that a 200 from my own box only proves my egress, not the worker's. I would add your 1x1 PNG probe as a regular check, logging accepted_at against terminal_at so a minutes-long dispatch gap shows up before users notice it.
I ran that probe twice before writing it up: 6m46s to terminal, both times, and the third number that matters is the reason, because it never changed. Log the triple (accepted_at, terminal_at, reason) and set the alert from your slowest known-good render, not from the queue's p99 — mine was 5m56s on a host the worker could reach, so a 10-minute threshold flags the broken run on the first try instead of the second.
That 10-minute threshold makes sense — basing it on your slowest known-good render, not the queue's p99, is the part I would have missed. Logging the reason alongside the two timestamps is a good call too, since a stable reason is what turns a slow run into a diagnosable one.
The reason field paid for itself on the first two runs: both failures ended on the same line,
media URL could not be downloaded (timeout or source server error)at 6m46s each, while the good render off a domestic bucket finished in 5m56s. So the alert is two conditions — elapsed over 10m, or a terminal reason I haven't seen in the last 30 good runs — and it's the second one that fires on run #1 instead of run #3.That second condition is the clever part — an unseen terminal reason fires on run one instead of waiting for three slow failures. With both failures landing at exactly 6m46s against a 5m56s good run, that rule looks solid.
The line that stuck with me is 'accepted is not fetched' — two green checks that only ever proved my own egress is such a clean summary of the trap. Asserting on the artifact itself, parsing ftyp/moov/mvhd out of the file instead of trusting the echoed parameters, is the guard I'd steal first. With the hardcoded UNFETCHABLE_HOSTS list, do you ever re-probe a host to see if it recovers, or does a host stay on the blocklist once it fails?
Permanent until something in the worker's region fetches it, because a re-probe from my box is exactly the evidence that misled me in the first place. The unblock I'd accept is one job on that host with terminal_at - accepted_at under 60s; below that the mirror stays automatic and nobody has to remember the tuple.