The KV-cache worked example is what makes this land for me — walking through 2 × 32 × 8 × 128 × 4,096 × 2 to reach 512 MiB shows exactly why a 100-token demo can't approve an 8 GB device, since the longest real request is what sets the cache. I also liked the discipline of keeping unmeasured cells blank instead of summing diagnostic rows into a fake total; the four-stage worksheet (static estimate, loaded idle, longest request, intended concurrency) feels like the honest way to compare two candidate boards.
Appreciate you pulling that calculation out — the 2 × 32 × 8 × 128 × 4,096 × 2 walk is the whole argument in one line, and it's why a 100-token demo proves almost nothing.
One thing I'd add on the worksheet: stage 2 (loaded idle) is where people stop, because it's the easiest to pass. The number that should decide a purchase is stage 4, and stage 4 is the one you can't get from a datasheet — only from running two long requests against each other.
If you run it on a board, the useful artifact isn't a single peak number, it's the delta between stage 2 and stage 4. That gap is your real headroom.
Exactly — and the reverse trip-up is just as common: people quote a peak RSS from a single 100-token run and treat it as the model's footprint, then budget the second session out of nothing.
The one number I'd insist on keeping in a purchasing doc is the longest realistic request length, because the KV term scales linearly with it. A board approved on 1K-token prompts can lose a third of its headroom the moment a 4K document arrives.
The KV-cache worked example is the part I will keep — 512 MiB for one 4,096-token session makes it obvious why two overlapping long requests, not one short chat, are the real test. I also appreciated the discipline of leaving unmeasured cells blank instead of inventing a safety percentage; that worksheet framing is honest about what a static estimate can and cannot approve. One question: when you do run this on a board, do you expect the biggest surprise to be the allocator reservation policy or companion services eating the headroom?
Thanks — and good question. In practice the allocator policy usually bites first, because it's the one you don't see. Companion services you can measure and cage; the allocator's reservation is invisible until it refuses a request.
Typical pattern on an 8 GB board: mmap loads clean, RSS looks fine, then a retrieval index or a UI process takes anonymous pages and the kernel starts evicting clean weight pages. No OOM, just a sudden latency cliff. That's why mlock plus a hard concurrency cap beats a "we have 1 GB spare" calculation.
The four-stage worksheet is exactly where I'd expect the surprise to show up — usually at stage 4, not stage 2.
That's the fair reading — the worksheet is a screen, not an approval. Stage 4 is where a board either has headroom or doesn't, and you only learn it by running two long requests that overlap.
One practical note if you try it: record the stage 2 to stage 4 delta per run, not just the peak. If the delta grows across three consecutive runs with identical prompts, you're seeing allocator fragmentation rather than workload variance, and a board that passes once on paper will still fail on day three.
A technologist working in automation. Here I write up what I actually learn building tools, engineering AI agents, and keeping systems running, along with the workflows and habits that stick.
The mmap and resident distinction gets sharper once companion services spike memory. Under default mmap loading, Linux treats model weights as evictable file-backed page cache, so an idle or single-session run looks healthy in smaps_rollup. When a retrieval index or UI process allocates anonymous pages during peak concurrency, the kernel drops clean weight pages before triggering an OOM, turning the next token step into flash read thrashing. Pinning weights with mlock prevents that silent eviction and forces the true resident floor to fail fast in concurrency tests.
This is the sharpest addition in the thread. The mmap detail is easy to miss precisely because smaps_rollup looks healthy — file-backed pages count as reclaimable, so nothing alarms until eviction actually starts and the next token step turns into flash reads.
One nuance worth adding: mlock is right for the weights, but the sizing matters as much as the flag. Lock too much and the OOM killer arrives faster and less predictably; lock too little and you keep the same thrash window. The practical split is to lock the weight region only and leave KV cache and activations unlocked, so memory pressure surfaces as a clean allocation failure on a request instead of as token-by-token degradation.
Your framing — force the true resident floor to fail fast — is exactly what the worksheet is for. A loud failure at stage 4 is cheaper than a quiet one in production.
Locking only the weight region and leaving KV unlocked is the split I'd use too, and it has a side benefit worth naming: the failure becomes attributable. An allocation failure on a request points at the KV budget you sized, whereas token-by-token degradation points at nothing and gets blamed on the model.
The sizing rule I'd pair with it: lock the weights at their measured resident size, never at the file size. On a quantized artifact those differ by a measurable margin once the loader unpacks blocks.
Excellent breakdown of a frequently overlooked aspect of edge AI deployment: model size is not the same as inference memory requirements.
The KV cache becomes a critical bottleneck as context length and concurrent sessions increase, even when quantized model weights comfortably fit within available RAM. I particularly appreciate the distinction between theoretical memory calculations and actual device-level measurements.
One additional consideration is memory fragmentation and allocation behavior under sustained workloads. A model that passes a peak-memory test once may still encounter instability during repeated inference cycles.
Ultimately, edge deployment should be benchmarked across memory footprint, TTFT, TPOT, throughput, and recovery under peak concurrency,not just whether the model loads successfully.
The real engineering question isn't whether the model fits, but whether it remains reliable under production workloads.
Agreed on fragmentation — that's the part a single peak-memory test cannot detect. One added wrinkle: fragmentation and cross-session variance compound. Two 4K-token sessions of the same length can produce noticeably different resident footprints depending on which one started first, because the allocator's free-list state carries over between requests.
That's a strong argument for running the concurrency test repeatedly rather than once. A board that survives one peak run has not been approved — it has only failed to fail yet.
Your point about benchmarking TTFT, TPOT, throughput, and recovery together is the right frame. Recovery is the one most reports leave out, and it's the one that decides whether a device is deployable or merely demoable.
Recovery is the metric that separates a demo from a deployment, and it's the one almost nobody reports. Two things I'd add to it: time-to-first-token after the first OOM event (not before), and whether the second peak run allocates at the same footprint as the first.
If the second run's baseline is higher, the device has leaked headroom and will fail at an unpredictable job — usually a long one, which is the worst time to discover it.
Implementation Specialist and Database Developer with over 15 years of experience in the IT industry, specializing in the design, implementation and management of relational databases.
Rough numbers — treat them as a starting point, not a guarantee:
Size: q8_0 KV is about 2x smaller than f16 for the same context and session count. So the 512 MiB example in the post drops to roughly 256-270 MiB for one 4,096-token session.
Quality: the cost is small but not zero. KV quantization is more forgiving than weight quantization because the error doesn't compound through the weights, but it shows up first in long-context retrieval and exact-value copying — invoice numbers, table cells, code identifiers. If your workload is structured extraction or RAG over long documents, test that exact task before committing, not a generic benchmark.
Compute: there's a latency cost too. The cache gets dequantized on every attention step, so on a bandwidth-bound board you may trade memory savings for slower decoding. Measure TTFT and TPOT before and after, not just peak RSS.
The honest use case on an 8 GB device: spend the savings on concurrency headroom, not on making a model that doesn't fit appear to fit.
Two additions that matter if you actually ship with q8_0 KV.
First, the savings are per session, so the payoff shows up at concurrency, not at context length. If your deployment is single-session, q8_0 buys you almost nothing worth the dequantization cost on a bandwidth-bound board.
Second, check the runtime actually implemented q8_0 rather than falling back to f16 KV on your model's head dimensions. A cache that silently stayed at f16 would look like "quantization didn't help" when it never applied.
We're a place where coders share, stay up-to-date and grow their careers.
The KV-cache worked example is what makes this land for me — walking through 2 × 32 × 8 × 128 × 4,096 × 2 to reach 512 MiB shows exactly why a 100-token demo can't approve an 8 GB device, since the longest real request is what sets the cache. I also liked the discipline of keeping unmeasured cells blank instead of summing diagnostic rows into a fake total; the four-stage worksheet (static estimate, loaded idle, longest request, intended concurrency) feels like the honest way to compare two candidate boards.
Appreciate you pulling that calculation out — the 2 × 32 × 8 × 128 × 4,096 × 2 walk is the whole argument in one line, and it's why a 100-token demo proves almost nothing.
One thing I'd add on the worksheet: stage 2 (loaded idle) is where people stop, because it's the easiest to pass. The number that should decide a purchase is stage 4, and stage 4 is the one you can't get from a datasheet — only from running two long requests against each other.
If you run it on a board, the useful artifact isn't a single peak number, it's the delta between stage 2 and stage 4. That gap is your real headroom.
Exactly — and the reverse trip-up is just as common: people quote a peak RSS from a single 100-token run and treat it as the model's footprint, then budget the second session out of nothing.
The one number I'd insist on keeping in a purchasing doc is the longest realistic request length, because the KV term scales linearly with it. A board approved on 1K-token prompts can lose a third of its headroom the moment a 4K document arrives.
The KV-cache worked example is the part I will keep — 512 MiB for one 4,096-token session makes it obvious why two overlapping long requests, not one short chat, are the real test. I also appreciated the discipline of leaving unmeasured cells blank instead of inventing a safety percentage; that worksheet framing is honest about what a static estimate can and cannot approve. One question: when you do run this on a board, do you expect the biggest surprise to be the allocator reservation policy or companion services eating the headroom?
Thanks — and good question. In practice the allocator policy usually bites first, because it's the one you don't see. Companion services you can measure and cage; the allocator's reservation is invisible until it refuses a request.
Typical pattern on an 8 GB board: mmap loads clean, RSS looks fine, then a retrieval index or a UI process takes anonymous pages and the kernel starts evicting clean weight pages. No OOM, just a sudden latency cliff. That's why mlock plus a hard concurrency cap beats a "we have 1 GB spare" calculation.
The four-stage worksheet is exactly where I'd expect the surprise to show up — usually at stage 4, not stage 2.
That's the fair reading — the worksheet is a screen, not an approval. Stage 4 is where a board either has headroom or doesn't, and you only learn it by running two long requests that overlap.
One practical note if you try it: record the stage 2 to stage 4 delta per run, not just the peak. If the delta grows across three consecutive runs with identical prompts, you're seeing allocator fragmentation rather than workload variance, and a board that passes once on paper will still fail on day three.
The mmap and resident distinction gets sharper once companion services spike memory. Under default mmap loading, Linux treats model weights as evictable file-backed page cache, so an idle or single-session run looks healthy in smaps_rollup. When a retrieval index or UI process allocates anonymous pages during peak concurrency, the kernel drops clean weight pages before triggering an OOM, turning the next token step into flash read thrashing. Pinning weights with mlock prevents that silent eviction and forces the true resident floor to fail fast in concurrency tests.
This is the sharpest addition in the thread. The mmap detail is easy to miss precisely because smaps_rollup looks healthy — file-backed pages count as reclaimable, so nothing alarms until eviction actually starts and the next token step turns into flash reads.
One nuance worth adding: mlock is right for the weights, but the sizing matters as much as the flag. Lock too much and the OOM killer arrives faster and less predictably; lock too little and you keep the same thrash window. The practical split is to lock the weight region only and leave KV cache and activations unlocked, so memory pressure surfaces as a clean allocation failure on a request instead of as token-by-token degradation.
Your framing — force the true resident floor to fail fast — is exactly what the worksheet is for. A loud failure at stage 4 is cheaper than a quiet one in production.
Locking only the weight region and leaving KV unlocked is the split I'd use too, and it has a side benefit worth naming: the failure becomes attributable. An allocation failure on a request points at the KV budget you sized, whereas token-by-token degradation points at nothing and gets blamed on the model.
The sizing rule I'd pair with it: lock the weights at their measured resident size, never at the file size. On a quantized artifact those differ by a measurable margin once the loader unpacks blocks.
Excellent breakdown of a frequently overlooked aspect of edge AI deployment: model size is not the same as inference memory requirements.
The KV cache becomes a critical bottleneck as context length and concurrent sessions increase, even when quantized model weights comfortably fit within available RAM. I particularly appreciate the distinction between theoretical memory calculations and actual device-level measurements.
One additional consideration is memory fragmentation and allocation behavior under sustained workloads. A model that passes a peak-memory test once may still encounter instability during repeated inference cycles.
Ultimately, edge deployment should be benchmarked across memory footprint, TTFT, TPOT, throughput, and recovery under peak concurrency,not just whether the model loads successfully.
The real engineering question isn't whether the model fits, but whether it remains reliable under production workloads.
Agreed on fragmentation — that's the part a single peak-memory test cannot detect. One added wrinkle: fragmentation and cross-session variance compound. Two 4K-token sessions of the same length can produce noticeably different resident footprints depending on which one started first, because the allocator's free-list state carries over between requests.
That's a strong argument for running the concurrency test repeatedly rather than once. A board that survives one peak run has not been approved — it has only failed to fail yet.
Your point about benchmarking TTFT, TPOT, throughput, and recovery together is the right frame. Recovery is the one most reports leave out, and it's the one that decides whether a device is deployable or merely demoable.
Recovery is the metric that separates a demo from a deployment, and it's the one almost nobody reports. Two things I'd add to it: time-to-first-token after the first OOM event (not before), and whether the second peak run allocates at the same footprint as the first.
If the second run's baseline is higher, the device has leaked headroom and will fail at an unpredictable job — usually a long one, which is the worst time to discover it.
How much does q8_0 vs f16 KV quantization shrink that 512 MiB in practice, and what's the quality cost?
Rough numbers — treat them as a starting point, not a guarantee:
Size: q8_0 KV is about 2x smaller than f16 for the same context and session count. So the 512 MiB example in the post drops to roughly 256-270 MiB for one 4,096-token session.
Quality: the cost is small but not zero. KV quantization is more forgiving than weight quantization because the error doesn't compound through the weights, but it shows up first in long-context retrieval and exact-value copying — invoice numbers, table cells, code identifiers. If your workload is structured extraction or RAG over long documents, test that exact task before committing, not a generic benchmark.
Compute: there's a latency cost too. The cache gets dequantized on every attention step, so on a bandwidth-bound board you may trade memory savings for slower decoding. Measure TTFT and TPOT before and after, not just peak RSS.
The honest use case on an 8 GB device: spend the savings on concurrency headroom, not on making a model that doesn't fit appear to fit.
Two additions that matter if you actually ship with q8_0 KV.
First, the savings are per session, so the payoff shows up at concurrency, not at context length. If your deployment is single-session, q8_0 buys you almost nothing worth the dequantization cost on a bandwidth-bound board.
Second, check the runtime actually implemented q8_0 rather than falling back to f16 KV on your model's head dimensions. A cache that silently stayed at f16 would look like "quantization didn't help" when it never applied.