Google Open-Sources XSE: 6 Million Queries a Second Without Python in the Path

On 12 November 2026 Google open-sourced XSE, the XLA Serving Engine that powers six million queries a second across Search, Ads and YouTube Shorts. The point of the project is simple and brutal: take the Python interpreter completely out of the serving path so the request never touches CPython at all.

That is the correct direction for any system that lives under a hard latency budget. It does not solve the problem when the input is live audio on a Windows desktop and the output is a private translation overlay that must stay invisible to screen share.

What XSE actually removes

XSE sits under the model server and replaces the Python runtime that normally owns the request loop. The published material says it handles six million queries a second in production. The engineering claim is that once the model graph is compiled, the interpreter is pure overhead—reference counting, GIL, object allocation, and the general mess of a dynamic language sitting on the hot path.

For a cloud serving fleet that is true. For a single Windows machine that needs to capture two audio streams, run a local or hybrid STT model, and paint a translation that the other side of the call never sees, the Python interpreter is not the dominant cost. The WASAPI shared-mode engine, the capture callback, and the time spent loading or warming a local model are.

The capture path that still owns the budget

A practical overlay for interviews or support calls still starts with dual-track WASAPI:

1. One IAudioClient in shared mode on the default render endpoint (or the endpoint the call app is using) with AUDCLNT_STREAMFLAGS_LOOPBACK. This is the remote party.
2. One IAudioClient in shared mode on the preferred capture endpoint. This is your microphone.

Both clients request a format the engine can actually mix (usually 48 kHz float32 or whatever GetMixFormat returns). Periodicity around 10 ms is a reasonable target; the engine will quantize to the mix format anyway.

The rule that matters is that IAudioCaptureClient::GetBuffer must return immediately. Copy the packet into a lock-free ring buffer and signal a worker. Any allocation, logging, or model call inside the callback adds jitter that shows up as gaps or increased end-to-end latency.

Align the two streams by device clock or a shared monotonic timestamp. Drift is inevitable; you correct it by dropping or duplicating a few frames on the slower side rather than resampling in the hot path unless you have measured that the quality difference is worth the cost.

From there the remote track goes to STT. The mic track can go to a separate stream, to a local model for your own speech, or simply be used for energy-based barge-in detection. The translation result is painted on a layered window excluded from screen capture with SetWindowDisplayAffinity(WDA_EXCLUDEFROMCAPTURE) on Windows 10+.

Local models versus cloud BYOK

Local models (faster-whisper, Whisper.cpp, Ollama with LFM3-audio or similar small audio checkpoints) remove the network hop and keep the audio on the machine. The tradeoff is load time, VRAM or RAM pressure, and accuracy on accented or noisy speech. Cloud endpoints (BYOK to whatever provider you already pay for) give better accuracy and diarization on clean audio, but they add RTT plus the model’s own time-to-final, and they stop working when the network is hostile or when you simply refuse to ship the audio off-box.

A practical hybrid keeps a small local model warm for the common case and falls back to a cloud endpoint only when the local confidence drops or when the user explicitly chooses the higher-quality path. Rate limits on the cloud side become a real problem once you are doing continuous streaming rather than occasional requests; the usual mitigations are token-bucket pacing, request coalescing, and having more than one provider key so a single 429 does not kill the overlay.

Latency budget that actually matters

A 150 ms cloud time-to-final is interesting only after you have already spent:

• 10–40 ms in the shared-mode WASAPI engine and buffering (measured, not guessed).
• Packet alignment and any small resampling.
• Network RTT + model compute for the cloud path.
• Translation and UI paint.

Local models trade the network hop for a longer first-token time and higher memory use. The right number is the one you measure on the machine you actually ship, with the same audio devices and the same process priority as production.

Practical notes from shipping this

Keep the capture callback empty. Use a single-producer single-consumer ring buffer (or two, one per track). Timestamp every packet with a shared QPC or device clock. Prefer shared mode so the rest of the machine keeps working; exclusive mode gives lower latency but breaks every other app that wants to play sound. Test with the actual call app (Teams, Zoom, Meet, Discord) because process-loopback capture behaves differently across them.

The XSE news is a reminder that removing interpreter overhead is valuable. On a desktop the interpreter is not the first thing that burns your latency budget. The capture path and the model load are.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.