llama.cpp 1.0.0 Stabilizes the Local Inference API — Dual-Track WASAPI Capture Still Owns the Latency Budget

On 13 November 2026 the ggml-org team released llama.cpp v1.0.0 and ggml v1.0.0. After 3.5 years the libraries now follow explicit semantic versioning: a MAJOR bump signals a breaking change to the public C API in ./include, MINOR covers backward-compatible features and new model support, PATCH is pure bug fixes. The release removes 35 deprecated functions (plus llama_set_warmup and the defrag threshold), promotes the NextN speculative API into llama.h, adds OCP FP8 KV caches, a host-memory MoE expert cache with configurable CPU share, and hardens llama-server against SSRF, CORS and DNS-rebinding. It also ships drama-3 TTS support and a /v1/audio/speech endpoint.

The stability promise matters if you embed local models for post-STT translation polish, short reply generation, or on-device transcription. It does not move the first millisecond of a live Windows overlay. That millisecond is still spent inside the audio engine.

The capture path is still the dominant term

A dual-track pipeline for a call or interview assistant separates the local microphone from the remote party:

1. Capture endpoint (USB headset, Bluetooth HFP, built-in, or exclusive-mode WASAPI).
2. Render loopback on the same or another endpoint so speaker labels stay intact.
3. Windows Audio Engine (audiodg.exe + audioeng.dll) in shared mode. Mixing, communications enhancements and vendor APOs routinely add 10–40+ ms and become non-deterministic under load or Bluetooth.
4. Your consumer thread: lock-free SPSC ring copy, float32→int16, 48 kHz→16 kHz resample, energy or Silero-style VAD.
5. Streaming STT (local GGUF or remote WebSocket) or direct local model call.
6. Overlay paint on a dedicated thread.

Even with FP8 KV and a warm context, the model’s contribution is downstream of steps 1–4. Bluetooth HFP alone can inject another 50–150 ms of buffering that no quantization scheme recovers.

A minimal shared-mode capture callback that stays under 1 ms and does no allocation:

// WASAPI shared-mode capture — callback must be allocation-free
static HRESULT STDMETHODCALLTYPE CaptureCallback(IAudioCaptureClient *pCapture) {
    UINT32 packetLength = 0;
    BYTE *pData = NULL;
    DWORD flags = 0;
    HRESULT hr = pCapture->GetBuffer(&pData, &packetLength, &flags, NULL, NULL);
    if (FAILED(hr) || !packetLength) return hr;

    // Writer side of a pre-allocated power-of-two SPSC ring (float32 frames)
    ring_push_float32(capture_ring, (const float *)pData, packetLength);
    pCapture->ReleaseBuffer(packetLength);
    return S_OK;
}

The consumer owns resampling (libsamplerate or a fixed polyphase), VAD, and the write to the model or WebSocket. Keep the audio thread pure; any lock or malloc here shows up as glitches or higher DPC latency.

Local models and the new speech endpoint

The drama-3 TTS addition and the /v1/audio/speech endpoint are useful for offline generation or for feeding synthetic reference audio into residual-echo experiments. For the live input path they are irrelevant. If you are running a local STT GGUF or a small translation model after the recognizer, the 1.0.0 API cleanup removes one class of future breakage, but the measured time-to-final on the model side is still measured after the audio has left the shared-mode engine.

Practical control surfaces that do help:

• Keep thinking/budget low for any online suggestion path; reserve higher compute for offline prep (job-description summary, question bank).
• Dual independent rings so mic and loopback never mix early. Mixing destroys speaker identity and forces the recognizer to handle two accents in one beam.
• Request the smallest period the driver will honor (often 10 ms default; some allow 5–8 ms). Below that the engine starts dropping or the APO stack becomes unstable.
• Overlay exclusion so screen-share does not capture the helper window — a Windows-specific problem that no model release solves.

Trade-offs that remain

FP8 KV caches shrink memory and can improve tokens/second on some backends, but the first token of a short rewrite is still gated by network (if remote) or by the CPU/GPU scheduling of the local process. The host-memory MoE cache is interesting for larger local models; for a 1–3 B parameter translation or suggestion model it is usually unnecessary. The real variable cost on a laptop during a call is the audio engine’s interaction with Bluetooth, spatial audio, or vendor “communications” enhancements — none of which appear in a llama.cpp changelog.

If you are shipping a desktop tool that shows text during a call, measure end-to-end from speech end (or VAD endpoint) to on-screen glyphs. The model’s TTFT is only one line item. The WASAPI period, the ring copy, the resample, and the paint are the others, and they do not get smaller because the inference library reached 1.0.0.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.