My API benchmark once reported 4,393,158 tokens per second. The useful result was finding out what that number actually measured.
The harness timed a streaming response and calculated throughput like this:
generation_time = total_request_time - time_to_first_token
tokens_per_second = output_tokens / generation_time
That calculation assumes the client observes tokens arriving over a meaningful interval after the first token. Some endpoints in the run instead held the response and delivered it in a burst. The first observed content arrived close to the end of the request. Subtracting those two timestamps left an interval close to zero.
Dividing the entire output by that tiny interval produced the spectacular number. It described the burst arriving at the client, not the model generating millions of tokens per second.
A streaming request is not proof of observable generation
Asking for streaming does not guarantee that the timing seen by the client exposes the underlying generation process. From that trace alone, I could not locate the buffering inside the serving path. I could only see that the arrivals did not support the throughput calculation I was making.
This is the distinction I needed in the results:
- Post-first-content throughput: output divided by the observed interval after first content, when the stream provides a useful interval.
- End-to-end throughput: output divided by the whole request duration, including waiting before content arrives.
The second number is useful for the caller, but it is a different metric. Quietly putting both in one ranking would hide the difference.
The guard I added
The benchmark now falls back to end-to-end timing when a streaming response has fewer than three data chunks or the calculated generation interval is below 0.2 seconds. It also withholds the first-token metric for responses classified as buffered. Empty output becomes an error rather than a row claiming zero tokens per second.
Here is the relevant logic, simplified from the harness:
generation_time = total_time - first_content_time
buffered = chunk_count < 3 or generation_time < 0.2
if buffered:
generation_time = total_time
throughput = output_tokens / generation_time
Those thresholds are conservative measurement guards, not a universal detector of server buffering. A short, fast response can also produce a small interval. The harness has a separate streaming capability probe that uses chunk count; its comments explicitly warn against treating a short duration alone as evidence that streaming is fake.
The fallback gives the run a usable wall-clock measurement. It does not recover the hidden generation speed, and I should not label it as though it did.
Reproduce the failure without calling a model
For a synthetic illustration, suppose 1,000 output tokens arrive after a ten-second wait, with only 0.001 seconds between the first and last observed content:
post-first-content calculation: 1,000 / 0.001 = 1,000,000 tokens/s
end-to-end calculation: 1,000 / 10 = 100 tokens/s
These are illustrative inputs, not the raw timings behind my 4,393,158 result. They show why the denominator matters: the same output can produce radically different rates depending on the interval being measured.
Before trusting an unusually fast row, I now want the total duration, first-content timing, output token count, chunk count, and the metric actually used beside it. If the trace only exposes a final burst, the honest result is end-to-end throughput with the generation rate left unknown.
Top comments (5)
Falling back to end-to-end timing was the sensible call, and your line about not labelling it as the hidden generation speed is the part most harnesses skip. We time each call at the client from request to complete answer, so a burst cannot turn into a rate, though it also means we say nothing about generation speed on its own. One cheap extra guard is to flag any row with a rate far beyond what the fastest hardware you know of could produce, and send it to a person with its trace instead of clamping it silently. Did the 0.2 second bound come from looking at your own traces, or from a rule of thumb?
The 0.2-second cutoff is a conservative rule of thumb in that code, not a boundary I calibrated across providers. It cannot establish buffering on its own: a genuinely fast, short response can fall below it too. I should have made that distinction more explicit in the post. Your trace-review suggestion is a better way to handle suspicious rows without silently turning a timing heuristic into a claim about what the server did.
Thanks for saying it plainly. One small thing that might help: store the raw first byte time and total time next to the label, so the 0.2 second cutoff can be re-tuned later without rerunning anything.
Keeping the measurements separate from the label would make that cutoff much easier to revisit. I would also distinguish the first received byte from the first content-bearing chunk, since an early header or empty event could make those different timestamps. Recording both alongside total duration would let a later analysis explain which interval it actually used. That is a useful refinement to the reporting format, rather than another reason to treat the cutoff as a universal boundary.
Good catch on the empty event, since it would stamp early and read as a fast first byte. Recording the status line time and the first chunk with real text as separate fields would show which one your cutoff actually used.