I measured four GPUs serving the same 7B model on vLLM. Which one is cheapest per token has no single answer. It flips with how fast your answers must be.
The setup
Same model (Qwen2.5-7B-Instruct), same vLLM (0.27.1, default flags), same load on every card: prompts of about 512 tokens, 256-token answers, 1 to 256 requests at once, 60 seconds per level. Four rented cards on vast.ai:
- RTX 3090 at $0.259/hour
- RTX 5090 at $0.522/hour
- A100 80GB PCIe at $1.017/hour
- H100 80GB SXM at $3.557/hour
The cheapest card at each answer time
Answer time here is p95: 95% of requests finished within it. For each card I took the cheapest load level that still met the time.
- Under 2 s: only the H100 can do it. $0.441 per 1M output tokens.
- Under 3 s: H100 $0.162, RTX 5090 $0.382, A100 $0.803. The 3090 can't. The H100 is about 2.4x cheaper than the 5090.
- Under 4 s: RTX 5090 $0.120, H100 $0.162, A100 $0.243. Now the 5090 wins, by about 1.4x.
- Under 8 s: RTX 5090 $0.050, H100 $0.099, A100 $0.106, RTX 3090 $0.490. The 5090 is about 2x cheaper than the H100.
The H100 is the most expensive card per hour, and still the cheapest per token when answers must be fast. It's the only card that keeps answers under 3 s with 64 requests in flight (p95 2.83 s), so it gets to run full while staying fast. The 5090 has to stay at 4 requests at once to answer that fast, so it uses only a small part of what it can do (379 tokens/s, against 2,901 at 64 at once).
Same load, different answer
At 64 requests at once on every card:
- RTX 5090: $0.050 per 1M, answers in 5.4 s (p50)
- RTX 3090: $0.082, answers in 17.9 s
- A100: $0.106, answers in 6.0 s
- H100: $0.162, answers in 2.6 s
Compare at the same load and the H100 looks like the worst deal. Compare at the same answer time and it can be the best one. Most GPU comparisons only show one of these.
The price you pay changes it again
All of the above uses one vast.ai price per card. Cost per token scales with the hourly price, so a different price can flip the result.
At another cloud's listed rates ($0.78/hour for the 5090, $2.65/hour for an H100 SXM), the same measurements give:
- Under 8 s: 5090 $0.075, H100 $0.074. About the same.
- Under 3 s: H100 $0.121, 5090 $0.571. The H100 is 4.7x cheaper.
So "the 5090 is cheaper per token" can be true at one price and false at another. The tool below lets you type in your own price per card.
How it was counted
Cost = GPU price x the measured window. The window runs until the last in-flight request finishes, because that time is paid for too. Requests and tokens came from vLLM's own /metrics before and after each level, and a separate client-side meter counted the same things. They matched exactly at every level on all four cards. 0 errors.
Limits
- These costs assume the GPU is busy the whole time. Idle hours in a real day raise them, and only real traffic shows by how much.
- One run per level, four different hosts on different days, synthetic prompts, one model.
- The 5090 and 3090 were run up to 64 requests at once, the A100 and H100 up to 256. So the 5090's cost at long answer times may go lower than shown.
- Nothing is interpolated between load levels. Only measured points are used.
Try it
Pick your answer time and type your own GPU prices: https://qaisermehdi3-coder.github.io/qvunex/
Every report, the raw /metrics and the server logs are here: https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/corpus
If you have a GPU, one command runs the same test on it (about 10 minutes, synthetic prompts, nothing leaves your machine). Add your model and vLLM flags after the price to test your own setup:
curl -sL https://raw.githubusercontent.com/qaisermehdi3-coder/qvunex/main/planb.sh | bash -s -- YOUR_PRICE_PER_HOUR
Send me the result and your card can go in the public table with your name on it. If you'd rather not run anything, send your model, vLLM flags and typical prompt and answer length, and I'll run a short mini-report on a rented GPU for free. The full report on your own setup is paid work: qvunexaudit@gmail.com
Update, 8 Nov
I ran the RTX 5090 again on a second machine (Norway, different CPU), this time up to 256 requests at once.
- Serving speed matched the first 5090 within about 4% at every load.
- At 256 at once it cost $0.038 per 1M output tokens at the first host's price, with answers in 24.3 s (p95). So when answers can take about 25 s, the 5090 is the cheapest card measured, about 2x cheaper than the 3090 and A100 ($0.082).
- Start-up was very different: 249 s on this host against 142 s on the first. For a server that scales to zero, the host can double the paid start-up of every wake.
Details and raw files: https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/planb-5090-2026-11-08
Top comments (7)
The concurrency cliff on the 5090 matches what happens with agent traffic. Request arrival in agent workflows is naturally spiky because eight agents finish tool execution and hit the endpoint at the exact same moment.
On consumer memory buses, that arrival spike tanks time-to-first-token because full-prompt prefill starves ongoing decode iterations. Did you run default vLLM settings or test
--enable-chunked-prefill? Chunking incoming prompts prevents concurrent prompt arrivals from blowing past the three-second latency ceiling, which recovers some concurrency on smaller memory buses before the card drops off.Thanks, both good points. These were default flags, and in vLLM 0.27.1 chunked prefill is already on by default. The server log shows chunked_prefill=True, so these numbers are with it on.
I checked time to first token from vLLM's own histogram for each level. On the 5090 at 64 at once, mean TTFT was 0.47 s out of a 5.4 s answer (p50). So most of the time is the 256 output tokens, not prefill stalls. At 256 at once it rises to 1.7 s.
The load is steady, not spiky: each request is replaced as soon as it finishes, and all prompts are about 512 tokens. So it doesn't show the mixed long-prompt case you describe. That would be a good next test.
And the answer time in the post is end-to-end, for the full 256-token answer, not TTFT. If TTFT is what you care about, the ranking can change.
Your question turned into a feature, thanks. The tool now has a "first token, on average" mode, and every measured point shows its mean time to first token from vLLM's own histogram. Under 0.1 s the H100 is about 3.3x cheaper than the RTX 5090. Under 0.5 s the 5090 wins. qaisermehdi3-coder.github.io/qvunex/
the prefill side is worth separating from raw concurrency. on a 3090 or 5090 the memory bus is the ceiling, so one long prompt arriving mid-stream stalls decode for everyone already reading, and that shows up as a tail latency spike rather than a worse average. chunked prefill removes the stall but moves the cost to the new request's first token, so a time-to-first-token benchmark can look worse while the streams already running are healthier. if the run was default settings, that alone explains part of the cliff. it matters which of those two numbers the cheapest card is cheapest for.
Thanks, both good points. These were default flags, and in vLLM 0.27.1 chunked prefill is already on by default. The server log shows chunked_prefill=True, so these numbers are with it on.
I checked time to first token from vLLM's own histogram for each level. On the 5090 at 64 at once, mean TTFT was 0.47 s out of a 5.4 s answer (p50). So most of the time is the 256 output tokens, not prefill stalls. At 256 at once it rises to 1.7 s.
The load is steady, not spiky: each request is replaced as soon as it finishes, and all prompts are about 512 tokens. So it doesn't show the mixed long-prompt case you describe. That would be a good next test.
And the answer time in the post is end-to-end, for the full 256-token answer, not TTFT. If TTFT is what you care about, the ranking can change.
Your question turned into a feature, thanks. The tool now has a "first token, on average" mode, and every measured point shows its mean time to first token from vLLM's own histogram. Under 0.1 s the H100 is about 3.3x cheaper than the RTX 5090. Under 0.5 s the 5090 wins. qaisermehdi3-coder.github.io/qvunex/
All runs used default settings, and in vLLM 0.27.1 chunked prefill is already on by default.
To test your point, I ran the same RTX 5090 again with the same 256-token answers, but prompts of about 2,000 tokens instead of 512.
1 at once: first token 0.05 s → 0.14 s. Cost per output token +5%.
64 at once: first token 0.47 s → 1.47 s, but the whole answer took 5.4 s → 13.3 s (p50). Cost per output token went up 2.4x.
So with chunked prefill on, the first token did get slower, but most of the extra time came after it. The streams already running paid most of the cost.
One limit: every prompt in this run was the same length, so it doesn't test one long prompt landing among short ones. That mixed case is the one I want to measure next.
Full run, raw /metrics and logs:
github.com/qaisermehdi3-coder/qvun...