Eight gigabytes of RAM minus a 2 GB model file is not 6 GB of usable inference memory. Calculate the KV cache, then test the longest request and peak concurrency.
That subtraction cannot approve a device. The file is a storage artifact; inference also keeps weight pages resident, allocates KV cache and compute buffers, and competes with the operating system and any retrieval, audio, or display service. One successful short chat says little about two overlapping long ones.
You can obtain two useful numbers before touching a board: the exact bytes of a fixed model artifact, and a theoretical KV calculation from its architecture. Runtime workspace, system use, allocation policy, and recovery after a peak remain measurements for the target device. The 2 GB file and 8 GB device here are a hypothetical purchasing question, not a tested board result.
Why "8 minus 2" mixes incompatible accounts
"This model needs 4 GB" could refer to storage, resident weights, a process peak, GPU memory, or the entire device. Ask which one. The useful question is who occupies physical memory under the same model, configuration, and workload — not whether two numbers on different datasheets can be subtracted.
The layers below are conceptual, not proportional to measured capacity. Unallocated headroom is not another use of RAM. If the runtime allocates KV for a second session, that demand consumes some of the same headroom; nothing here establishes a fixed multiplier or a device result.
| Account | What can establish it | What it does not establish |
|---|---|---|
| Weights and model metadata | Exact artifact version, format, file bytes, loader output | The process or device peak during inference |
| KV cache | Layer count, KV heads, head dimensions, allocated tokens, K/V types | Actual allocator overhead or a runtime's reservation policy |
| Compute and graph buffers | Runtime memory breakdown or a target-device measurement | A universal multiplier of model parameters |
| Operating system and companion tasks | Available memory after the real image and services start | The full advertised RAM as exclusive model capacity |
| Concurrent requests and headroom | Overlapping request lengths, error records, the system peak | A guarantee that every cache is shared — or that every byte is duplicated |
These are diagnostic accounts, not necessarily additive measurements. A final capacity figure needs a consistent runtime configuration and observation window. If the only known value is the GGUF or other model-file size, you have screened storage, not signed off RAM.
A four-bit weight calculation is only a floor
Suppose a hypothetical model has three billion parameters and every parameter occupies exactly four bits. Its bare arithmetic weight floor is 3,000,000,000 × 4 ÷ 8 = 1,500,000,000 bytes, about 1.40 GiB. At eight bits the floor is about 2.79 GiB; at sixteen bits it is about 5.59 GiB. The calculation deliberately distinguishes decimal GB from binary GiB.
Real artifacts are not packed uniformly at that theoretical floor. Quantization blocks may carry scales or zero-points; some tensors may stay at higher precision; headers, alignment, embeddings, and output layers also consume space. Use the bytes of an artifact with a fixed hash, then inspect what the chosen runtime actually loads. "Q4" and "INT4" are not interchangeable byte-count promises across formats and backends.
More importantly, weight quantization does not shrink the operating system, the compute graph, or the KV cache by the same factor. The llama.cpp server reference exposes model-loading mode, context size, device offload, and K/V cache types as separate controls. If two measurements differ in several of those settings, the file format alone cannot explain the difference.
Memory mapping does not make RAM free
A runtime may map a model file rather than copying it into one obvious heap allocation. Pages accessed during inference can still become resident and may be shared between processes. A process's RSS can include shared pages, so RSS is not automatically the device's exclusive model cost. Linux documents RSS and proportional-set-size accounting in smaps_rollup. PSS helps describe shared mappings, but neither RSS nor PSS alone answers whether a separate GPU has enough memory or whether a device allocator can satisfy a large request.
For a target-device test, keep both process and system observations: available memory, any container limit, swap or OOM events, and device-memory figures where applicable. Record when each figure was captured. A healthy idle reading and an inference peak are different facts.
Where the 512 MiB KV estimate comes from
During autoregressive generation, the model reuses previously calculated keys and values instead of rebuilding them for every new token. The cache exists across attention layers and grows or is reserved according to the runtime's cache strategy.
KV bytes ≈ 2 × attention layers
× KV heads × dimensions per head
× allocated tokens per session
× bytes per K/V element
× simultaneous independent sessions
The leading two represents keys and values. Use the number of key-value heads, not the number of query heads printed elsewhere in a configuration. Grouped-query and multi-query attention may use fewer KV heads than query heads. If keys and values use different data types, calculate their terms separately and add them. A sliding-window layer, a hybrid architecture, or shared state across layers needs a model-specific calculation instead of this uniform expression.
A calculation, not a board measurement
Assume 32 standard attention layers, eight KV heads, 128 dimensions per head, and two bytes per K or V element. If a runtime allocates 4,096 tokens for one session:
2 × 32 × 8 × 128 × 4,096 × 2 = 536,870,912 bytes ≈ 512 MiB
Two independently allocated sessions at that same limit would have a 1,024 MiB theoretical KV baseline. Doubling the per-session allocation to 8,192 tokens doubles that term if the other assumptions remain fixed.
The 512 MiB figure is not a measurement of a commercial model or board. "4,096 tokens" here is an assumed allocation, not a number of visible characters. System instructions, retrieved passages, user input, and reserved output all count toward a request's token budget. Some runtimes reserve the entire configured context; others grow or reuse cache space, allocate blocks, or carry batching overhead. A 100-token chat cannot establish the peak for a 4,096-token configuration, let alone two overlapping long chats.
Two sessions do not always mean "multiply by two." Under the same hypothetical architecture, one session allocated 4,096 tokens and another allocated 2,048 tokens would have a 512 MiB plus 256 MiB, or 768 MiB, theoretical KV baseline if their caches are independently allocated at those lengths. A runtime reserving two full 4,096-token slots would not obey that 768 MiB figure. Record each slot's effective allocation and the runtime's reported allocation; "two users online" is not a memory measurement. Sharing the model's weights does not imply that two different conversation histories share their KV.
Cache quantization and offloading move the trade-off
Lower-precision K/V may reduce the nominal cache-data term, but scales, residual storage, conversion buffers, and latency can change the real saving. Offloading cache from a GPU to the CPU changes where memory is consumed and adds transfer work; it does not create free device capacity. The llama.cpp server options let an operator choose K and V cache types separately.
When comparing two runs, record the weight artifact, K and V types, context allocation, session count, offload setting, and runtime version together. A result labeled only "4-bit" is ambiguous: it might describe weights while the cache stays at sixteen bits.
Turn 4,096 tokens into an actual request
Allocated tokens are not visible characters. Suppose the longest business input takes 3,000 tokens, the answer allowance is 800, and instructions and templates consume another 200. That is already 4,000. A 4,096-token context leaves little room for a longer retrieved passage or tokenizer variation; an extra 150 tokens can cause truncation before an out-of-memory failure. Raising the context to 8,192 may avoid truncation but can make some runtimes reserve more KV memory.
That is why a 100-token demo cannot approve an 8 GB product. Separate a short request, the longest allowed single request, and requests that genuinely overlap. If the product searches local manuals or drives a display, leave those services enabled during the concurrent run. Multiplying a single-request peak by a thread count is not a concurrency test: weights may be shared, KV allocation may be reserved or reused, and a runtime may silently queue work.
Likewise, "two users online" does not mean two full contexts are active. Define a concurrency workload that another engineer can reproduce — for example, two longest inputs overlapping during prompt processing. If a product intentionally serializes long requests to save memory, document the wait time and timeout behavior as part of acceptance. A memory reduction that breaks response-time requirements is not a free win.
Keep the unmeasured cells blank
Fix the product workload before treating any estimate as a requirement: longest actual input, output-token allowance, concurrent users, companion retrieval or audio services, and acceptable failures and latency. Then fix the model artifact hash, tokenizer, architecture configuration, runtime and backend versions, context limit, and K/V type. Without those inputs, a range of candidates is reasonable; "8 GB is enough" is not.
A working budget should show four stages side by side: static estimate, loaded idle device, longest single request, and maximum intended concurrency. The static stage can contain measured artifact bytes and a calculated KV baseline. Mark compute workspace and system use "to measure" rather than inventing a safety percentage. Observe the loaded stage with all normal services running.
Copy this blank worksheet for each candidate configuration. Enter values, units, and capture times, and attach the model-file hash, runtime version, and raw logs. Leave cells blank until measured; a static calculation must not masquerade as a device observation. Record both the device peak and MemAvailable in the final row so process RSS is not mistaken for the whole-system answer.
| Observation (MiB; rows are not additive) | Static estimate | Loaded idle | Longest request | Intended concurrency |
|---|---|---|---|---|
| Model artifact | File bytes converted to MiB: ____ | Same artifact hash: ____ | Same | Same |
| Model-file mappings, if separately identifiable | Not inferred from file size | Mapping RSS/PSS: ____ | Mapping RSS/PSS: ____ | Mapping RSS/PSS: ____ |
| KV cache | Architecture formula: ____ | Allocated: ____ | Allocated: ____ | Allocated: ____ |
| Compute, graph, batch buffers, if reported | To measure | ____ | ____ | ____ |
| Inference process total RSS/PSS | Not the sum of rows above | ____ / ____ | ____ / ____ | ____ / ____ |
| System and companion services; container/device limits | To measure | State and limit: ____ | ____ | ____ |
| Device peak / simultaneous MemAvailable | Not inferred from file size | ____ / ____ | ____ / ____ | ____ / ____ |
The process-total RSS/PSS can already contain mappings, KV, and buffers attributed to that process. The detailed rows explain a peak; they are not separate charges to add together. File bytes and RSS/PSS use different accounting boundaries too. Keep effective input/output tokens, allocated context, truly overlapping sessions, container limits, swap/OOM, and failed requests on the same record. Capture a timestamp for each number so MemAvailable and the reported peak refer to the same moment.
The measurement sequence is short enough to repeat. Boot the target image with its normal services and record idle MemAvailable. Load the artifact identified by a hash and preserve the runtime's effective weight, context, KV, and workspace settings. Run the longest single request, then genuinely overlapping requests, then observe recovery over several rounds.
Useful Linux fields include /proc/meminfo MemAvailable and the inference process's smaps_rollup RSS/PSS; record container current usage and limits if containerized, and device-memory counters when the runtime provides them. If the runtime omits a field, write "not reported," not a number borrowed from the model file.
Suppose all you have today is an 8 GB device specification, a 2 GB model artifact, and the hypothetical 512 MiB KV calculation above. The static column can hold artifact bytes and a formula. The next three columns must remain blank. That record says storage has a candidate and cache has a theoretical starting point. It does not say roughly 5.5 GB remains: resident pages, buffers, services, container limits, and actual allocation policy have not been measured on one accounting basis.
Once the first target-device log arrives, an unexplained difference between theoretical KV, runtime-reported allocation, and the system peak is worth investigating. It may reflect a longer reserved context, batching buffers, another process, or a different memory device. Do not "fix" the worksheet by tuning a multiplier until numbers match. A missing or surprising field tells you which log to collect next.
Two comparable rows fix the same device, operating-system image, runtime version, and request set while changing one candidate configuration. Comparing a 3B and a 7B model requires reading each model's own layer and KV-head counts; the first model's cache-per-token constant does not transfer. Comparing two weight formats requires checking that the tokenizer, context, and K/V precision did not change in the background. There are no paired device runs behind this article, so it supplies a test design, not a ranking of models for 8 GB hardware.
Locate the failure before buying more memory
"The model won't run" can mean that input was rejected before inference, that loading or generation ran out of memory, or that only two overlapping requests fail. Start from an observable failure stage rather than assuming every failure calls for more RAM.
Preserve the last successful action before a failure. If loading completed but the first request failed, inspect input processing and its allocation. If short requests work but a long one is truncated, inspect the tokenizer and output allowance; more RAM may not address a context-policy error. If only concurrency causes OOM, inspect overlapping time ranges, per-session KV, and container or device limits. If the first round succeeds and later rounds deteriorate, inspect cache release and queued work. Change one variable and rerun the original load; otherwise a new success does not show what fixed the original problem.
Interpret counters against the memory topology. On unified-memory hardware, process RSS, a GPU API counter, and system availability may describe some of the same physical pages, so adding them can double-count. With discrete VRAM, system RAM can remain available after device memory is exhausted. A container can also hit its own limit while the host still reports headroom. One OOM line without the request shape and effective runtime settings does not prove that the product needs "at least 16 GB."
Even if quantizing weights or shortening context makes a configuration fit, either change can damage answers, time to first token, or the manual passage the product must retain. After capacity screening, judge the result with a task-relevant answer set and a separate TTFT, TPOT, memory, and power validation pass.
When can you call 8 GB enough?
Only for a fixed artifact, effective runtime configuration, and target workload: the longest request and intended concurrency must complete, and the device peak, failures, swap/OOM record, and recovery must meet the product's requirements. File bytes and theoretical KV can screen candidates; they cannot approve capacity.
If the product must search long manuals while serving a display and multiple users, and reducing context or concurrency would break that requirement, compare a larger-memory device or another deployment approach instead of adding a safety multiplier.
There are no target-board runs, paired model scales, or paired quantization formats behind this article. The three-billion-parameter, 512 MiB KV, and 2 GB/8 GB figures are worked examples, not performance results for any specific device. A useful purchasing conclusion today may be: "The theoretical account is complete; the whole-device concurrency peak still needs measurement." That is more informative than an unqualified "it runs."
Sources
- Hugging Face Transformers: how caching works
- Hugging Face Transformers: cache strategies
- llama.cpp server options
- Linux kernel documentation: /proc and smaps_rollup
Top comments (15)
The KV-cache worked example is what makes this land for me — walking through 2 × 32 × 8 × 128 × 4,096 × 2 to reach 512 MiB shows exactly why a 100-token demo can't approve an 8 GB device, since the longest real request is what sets the cache. I also liked the discipline of keeping unmeasured cells blank instead of summing diagnostic rows into a fake total; the four-stage worksheet (static estimate, loaded idle, longest request, intended concurrency) feels like the honest way to compare two candidate boards.
Appreciate you pulling that calculation out — the 2 × 32 × 8 × 128 × 4,096 × 2 walk is the whole argument in one line, and it's why a 100-token demo proves almost nothing.
One thing I'd add on the worksheet: stage 2 (loaded idle) is where people stop, because it's the easiest to pass. The number that should decide a purchase is stage 4, and stage 4 is the one you can't get from a datasheet — only from running two long requests against each other.
If you run it on a board, the useful artifact isn't a single peak number, it's the delta between stage 2 and stage 4. That gap is your real headroom.
Exactly — and the reverse trip-up is just as common: people quote a peak RSS from a single 100-token run and treat it as the model's footprint, then budget the second session out of nothing.
The one number I'd insist on keeping in a purchasing doc is the longest realistic request length, because the KV term scales linearly with it. A board approved on 1K-token prompts can lose a third of its headroom the moment a 4K document arrives.
The KV-cache worked example is the part I will keep — 512 MiB for one 4,096-token session makes it obvious why two overlapping long requests, not one short chat, are the real test. I also appreciated the discipline of leaving unmeasured cells blank instead of inventing a safety percentage; that worksheet framing is honest about what a static estimate can and cannot approve. One question: when you do run this on a board, do you expect the biggest surprise to be the allocator reservation policy or companion services eating the headroom?
Thanks — and good question. In practice the allocator policy usually bites first, because it's the one you don't see. Companion services you can measure and cage; the allocator's reservation is invisible until it refuses a request.
Typical pattern on an 8 GB board: mmap loads clean, RSS looks fine, then a retrieval index or a UI process takes anonymous pages and the kernel starts evicting clean weight pages. No OOM, just a sudden latency cliff. That's why mlock plus a hard concurrency cap beats a "we have 1 GB spare" calculation.
The four-stage worksheet is exactly where I'd expect the surprise to show up — usually at stage 4, not stage 2.
That's the fair reading — the worksheet is a screen, not an approval. Stage 4 is where a board either has headroom or doesn't, and you only learn it by running two long requests that overlap.
One practical note if you try it: record the stage 2 to stage 4 delta per run, not just the peak. If the delta grows across three consecutive runs with identical prompts, you're seeing allocator fragmentation rather than workload variance, and a board that passes once on paper will still fail on day three.
Excellent breakdown of a frequently overlooked aspect of edge AI deployment: model size is not the same as inference memory requirements.
The KV cache becomes a critical bottleneck as context length and concurrent sessions increase, even when quantized model weights comfortably fit within available RAM. I particularly appreciate the distinction between theoretical memory calculations and actual device-level measurements.
One additional consideration is memory fragmentation and allocation behavior under sustained workloads. A model that passes a peak-memory test once may still encounter instability during repeated inference cycles.
Ultimately, edge deployment should be benchmarked across memory footprint, TTFT, TPOT, throughput, and recovery under peak concurrency,not just whether the model loads successfully.
The real engineering question isn't whether the model fits, but whether it remains reliable under production workloads.
Agreed on fragmentation — that's the part a single peak-memory test cannot detect. One added wrinkle: fragmentation and cross-session variance compound. Two 4K-token sessions of the same length can produce noticeably different resident footprints depending on which one started first, because the allocator's free-list state carries over between requests.
That's a strong argument for running the concurrency test repeatedly rather than once. A board that survives one peak run has not been approved — it has only failed to fail yet.
Your point about benchmarking TTFT, TPOT, throughput, and recovery together is the right frame. Recovery is the one most reports leave out, and it's the one that decides whether a device is deployable or merely demoable.
Recovery is the metric that separates a demo from a deployment, and it's the one almost nobody reports. Two things I'd add to it: time-to-first-token after the first OOM event (not before), and whether the second peak run allocates at the same footprint as the first.
If the second run's baseline is higher, the device has leaked headroom and will fail at an unpredictable job — usually a long one, which is the worst time to discover it.
The mmap and resident distinction gets sharper once companion services spike memory. Under default mmap loading, Linux treats model weights as evictable file-backed page cache, so an idle or single-session run looks healthy in smaps_rollup. When a retrieval index or UI process allocates anonymous pages during peak concurrency, the kernel drops clean weight pages before triggering an OOM, turning the next token step into flash read thrashing. Pinning weights with mlock prevents that silent eviction and forces the true resident floor to fail fast in concurrency tests.
This is the sharpest addition in the thread. The mmap detail is easy to miss precisely because smaps_rollup looks healthy — file-backed pages count as reclaimable, so nothing alarms until eviction actually starts and the next token step turns into flash reads.
One nuance worth adding: mlock is right for the weights, but the sizing matters as much as the flag. Lock too much and the OOM killer arrives faster and less predictably; lock too little and you keep the same thrash window. The practical split is to lock the weight region only and leave KV cache and activations unlocked, so memory pressure surfaces as a clean allocation failure on a request instead of as token-by-token degradation.
Your framing — force the true resident floor to fail fast — is exactly what the worksheet is for. A loud failure at stage 4 is cheaper than a quiet one in production.
Locking only the weight region and leaving KV unlocked is the split I'd use too, and it has a side benefit worth naming: the failure becomes attributable. An allocation failure on a request points at the KV budget you sized, whereas token-by-token degradation points at nothing and gets blamed on the model.
The sizing rule I'd pair with it: lock the weights at their measured resident size, never at the file size. On a quantized artifact those differ by a measurable margin once the loader unpacks blocks.
How much does q8_0 vs f16 KV quantization shrink that 512 MiB in practice, and what's the quality cost?
Rough numbers — treat them as a starting point, not a guarantee:
Size: q8_0 KV is about 2x smaller than f16 for the same context and session count. So the 512 MiB example in the post drops to roughly 256-270 MiB for one 4,096-token session.
Quality: the cost is small but not zero. KV quantization is more forgiving than weight quantization because the error doesn't compound through the weights, but it shows up first in long-context retrieval and exact-value copying — invoice numbers, table cells, code identifiers. If your workload is structured extraction or RAG over long documents, test that exact task before committing, not a generic benchmark.
Compute: there's a latency cost too. The cache gets dequantized on every attention step, so on a bandwidth-bound board you may trade memory savings for slower decoding. Measure TTFT and TPOT before and after, not just peak RSS.
The honest use case on an 8 GB device: spend the savings on concurrency headroom, not on making a model that doesn't fit appear to fit.
Two additions that matter if you actually ship with q8_0 KV.
First, the savings are per session, so the payoff shows up at concurrency, not at context length. If your deployment is single-session, q8_0 buys you almost nothing worth the dequantization cost on a bandwidth-bound board.
Second, check the runtime actually implemented q8_0 rather than falling back to f16 KV on your model's head dimensions. A cache that silently stayed at f16 would look like "quantization didn't help" when it never applied.