A few weeks ago I needed to pick a local inference server for a project. I'm on a Mac, so the candidates were the usual ones: Ollama, LM Studio, a couple of MLX servers. I did what everyone does and went looking for benchmarks.
I found plenty. I just couldn't reproduce any of them.
One post said server A was twice as fast as server B. On my machine they were almost the same. Another one gave a tok/s number for a model I had downloaded, and I was nowhere near it, in either direction. After a while I stopped reading and started measuring, and that turned out to be a lot more interesting than I expected.
It's not that the benchmarks are wrong
Most of those numbers were probably correct on the machine where they were taken. The problem is that a tok/s figure depends on so many things that it doesn't really travel.
The hardware is the obvious one. Memory bandwidth decides most of the speed, and "an Apple Silicon Mac" covers machines that are two times apart. Then there's the model file: "LFM3 1.7B" in MLX 8-bit, in a GGUF Q4_K_M and in Ollama's mxfp8 are three different things that happen to share a name.
The one that surprised me most was who holds the stopwatch. A lot of benchmarks use the numbers the server reports about itself, and every server counts in its own way. Some leave out the first token, some leave out prompt processing, some only tell you the generation speed. Put two of those next to each other and you're comparing different quantities.
Then reasoning models came along and made it worse. The same prompt can give you 30 tokens or 500 depending on whether the model thinks first, and servers don't agree on the default. I found out the hard way that Rapid-MLX turns thinking off unless you explicitly ask for it, while others leave it on. For a while I was comparing a model that reasoned with one that didn't, and wondering why the numbers looked strange. The Rapid-MLX maintainer explained it to me in an issue, which was very kind of him.
And finally, noise. On my laptop the exact same test moves 10 to 20% from one minute to the next. Background apps, temperature, whatever. A 5% difference from a single run means nothing.
What I ended up doing
The fix is boring: hold the stopwatch yourself, and use the same one for everything.
I send the same streaming request to each server and time it on the client with time.perf_counter(), ignoring whatever the server says about itself. What I care about is what my app will actually feel. I look at time to first token, generation speed, the overall effective speed (output tokens divided by total time, prompt processing included) and how much RAM the system is using.
The first run loads the model and warms everything up, so I throw it away. Then I take the median of at least three runs and keep the min and max around, because the spread tells you how much to trust the result. Reasoning gets set explicitly on every request, on or off, never left to the default. And each prompt starts with a unique ID so prompt caching doesn't quietly make the second run look amazing.
What it looked like on my Mac
Here's one example, just to show the kind of thing you get. M5 Pro with 24 GB, LFM3 1.7B at 8-bit, short prompt, 512 output tokens, reasoning off, median of 3 warm runs:
| Server | Weights | Italian prompt (tok/s) | English prompt (tok/s) |
|---|---|---|---|
| oMLX | MLX 8-bit | 129.1 | 117.2 |
| Rapid-MLX | MLX 8-bit | 126.8 | 103.2 |
| Ollama | mxfp8 | 113.9 | 104.4 |
I'd read this as "they're all close", not as a ranking. The first two are a tie given the noise, and the order even changes with the prompt language, which I didn't expect. With a different model, or on your machine, the picture may look completely different. That's kind of the point.
If you want to try it
I turned all of this into a small open-source app called TokenRace. If you have uv installed, this is all you need:
uvx tokenrace
You pick a server and a model, start the test and watch the numbers come in live. At the end there's a summary, a chart and a CSV you can keep. It all runs locally, nothing gets sent anywhere.
A single session from the app; numbers vary between runs, which is part of the point.
It already knows about 15 servers: Ollama, LM Studio, llama.cpp, vLLM, SGLang, Rapid-MLX, oMLX, mlx_lm.server, LocalAI, Jan, KoboldCpp, text-generation-webui, Lemonade, Docker Model Runner and Unsloth Studio. If yours isn't there but speaks the OpenAI API, you can just change the URL, or add it with a few lines in config/servers.toml.
I've only been able to test a few of these myself, on one Mac. So if you run it on an NVIDIA or AMD box, or on Linux, I'd really like to see your numbers. There's a Share your results section for that. Corrections on ports or reasoning parameters, and PRs for servers I've missed, are just as welcome.
Top comments (10)
Hash the rendered prompt, not the template text. The same entry in prompts.toml with different parameters is a different workload, so a template hash will call two different runs identical - which is the failure you are trying to prevent, moved one level up.
Per-row hashes also buy you a diff: comparing two CSVs then tells you which prompts changed, not only that the file did. That is worth more than the version string, since a file can be edited without anyone bumping it.
And nice work shipping it in a day.
The stopwatch is solved in this post; the denominator is not. Once you time on the client, tok/s is client seconds divided by output tokens - but those tokens come from the server, and they do not mean the same thing across providers.
On OpenAI, output_tokens includes reasoning tokens; Gemini reports thinking separately. So two servers producing identical visible text can report different denominators for the same wall-clock second, and the effective speed differs by exactly the thinking ratio. In one call I measured, reasoning was 86% of billed output. That is not a rounding error in a comparison, it inverts it.
So pick a denominator and name it: visible tokens for what the user feels, billed tokens for what you pay. Mixing them is the same mistake as trusting the server's stopwatch.
Fair point, and you're right that I glossed over it in the post.
What TokenRace does today: it takes completion_tokens from the server's usage (falling back to counting chunks when there's no usage), and shows reasoning and answer tokens as separate counts. But the tok/s figures use the total, so on a thinking model you're measuring "all generated tokens per second", not what the user actually reads.
Naming the denominator is a good rule. I think a "visible tok/s" next to the total, plus time to the first answer token (not the first reasoning token), would make the comparison much more honest. Adding it to the list, thanks for the push.
Matching tok/s across machines is harder than the hardware suggests. Prompt length moves the number a lot: a long context spends its time reading before the first output token, so two runs of the same model can differ widely. Compare runs at the same prompt size, or the comparison says little.
Completely agree, prompt length changes everything. That's why TokenRace uses fixed presets instead of free prompts: a short one, a long one with about 8,000 tokens of context, and a cache one that repeats the same prompt. It also reports TTFT and an estimated prompt speed separately, so you can see how much of the time goes into reading the context before the first token.
Good to hear. One thing that will bite when you implement it: the ratio only means something if both counts come from the same requests over the same prompt set. If visible tok/s comes from your fast prompt set and the ratio from a heavier one, the three columns stop being comparable - same model, two workloads, no way to tell which column moved why.
If the changelog credits the thread, pin the prompt set in that entry too. A number's provenance is the part that ages worst.
It's out: 1.0.2 has answer TTFT, visible tok/s and the reasoning/answer ratio, with this thread credited in the changelog: github.com/ernext/tokenrace/blob/m...
On provenance, agreed, and this one is covered by design: all three numbers come from the same request in each run, and the summary is grouped by server, model and preset, so a column can't come from a different workload. The prompts are fixed and versioned in the repo (config/prompts.toml), and the CSV records the preset and language of every row. You've got me thinking about adding a hash of the exact prompt to the CSV though, so a number can always be traced back even if the prompts change later.
Yes, and the ratio belongs in the table next to the two rates, not just the split. Reasoning tokens per visible token is what thinking costs you per readable word, and it moves by model and by effort setting - two models at identical visible tok/s can differ several times over in what one answer costs.
The real trap is comparing total tok/s across models: a model that thinks longer looks faster while producing less. Three columns - visible tok/s, total tok/s, ratio - make that impossible to hide, and also make low effort show up as the saving it actually is.
That's a really good point: total tok/s on its own actually rewards a model for thinking more. Three columns side by side (visible tok/s, total tok/s, reasoning tokens per visible token) make the trade-off obvious, and like you say, a lower effort setting finally shows up as the saving it is.
I'm adding all three in the next release, and I'll credit this thread in the changelog. Thanks, this is exactly the kind of feedback I was hoping for.
Link: github.com/ernext/tokenrace