DEV Community

#llamacpp

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Gemma 4 on a Used 8 GB RTX 3050: 116 Tokens/s for One User, 618 for 64
xbill profile

Gemma 4 on a Used 8 GB RTX 3050: 116 Tokens/s for One User, 618 for 64

#gemma #llamacpp #cuda #localllm
Comments
13 min read
Running MoE LLMs Larger Than Your RAM: A Practical Guide to mmap, Expert Offloading and SSD Streaming with llama.cpp
eme_gug_0821b41b948be6516 profile

Running MoE LLMs Larger Than Your RAM: A Practical Guide to mmap, Expert Offloading and SSD Streaming with llama.cpp

#llm #llamacpp #moe #performance
Comments
8 min read
Choosing a Local LLM Through Fate (2026 Edition)
akari_iku profile

Choosing a Local LLM Through Fate (2026 Edition)

#ai #llm #localhackday #llamacpp
2
Comments
28 min read
llama-server's prompt cache reuses KV computed under a different LoRA scale
homelabpm profile

llama-server's prompt cache reuses KV computed under a different LoRA scale

#llamacpp #llm #selfhosted #lora
Comments
6 min read
Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)
kharlampiiev profile

Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)

#ai #llm #machinelearning #llamacpp
Comments
6 min read
Running 100B+ MoE Models on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp
eme_gug_0821b41b948be6516 profile

Running 100B+ MoE Models on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp

#llm #llamacpp #gpu #machinelearning
Comments
8 min read
Running 100B+ MoE LLMs on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp
eme_gug_0821b41b948be6516 profile

Running 100B+ MoE LLMs on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp

#llm #llamacpp #gpu #selfhosted
Comments
7 min read
When Chat Templates Go Wrong
pielounw profile

When Chat Templates Go Wrong

#llm #opensource #ai #llamacpp
Comments
7 min read
Deploying LLMs Locally: Ollama, llama.cpp, vLLM & Text Generation Inference
ipsphil_digital profile

Deploying LLMs Locally: Ollama, llama.cpp, vLLM & Text Generation Inference

#ollama #llamacpp #vllm #tutorial
Comments
12 min read
Deploying LLMs Locally: Complete Guide to Ollama, llama.cpp, vLLM & TGI
ipsphil_digital profile

Deploying LLMs Locally: Complete Guide to Ollama, llama.cpp, vLLM & TGI

#ollama #llamacpp #vllm #tutorial
Comments
9 min read
Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison
ipsphil_digital profile

Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison

#llm #ollama #llamacpp #tutorial
Comments
5 min read
llama.cpp vs Ollama in 2026: Which Runtime Should You Run?
rosgluk profile
Rost

llama.cpp vs Ollama in 2026: Which Runtime Should You Run?

#llamacpp #ollama #llm #selfhosting
Comments
18 min read
A 4 GB Laptop GPU vs a 6-Core CPU on Gemma 4, Re-Measured in ABBA Order: 4.1x
xbill profile

A 4 GB Laptop GPU vs a 6-Core CPU on Gemma 4, Re-Measured in ABBA Order: 4.1x

#gemma #llamacpp #cuda #benchmarking
10
Comments 2
11 min read
ROCm vs Vulkan for AMD Local LLM Hosting: 2026 Guide
rosgluk profile
Rost

ROCm vs Vulkan for AMD Local LLM Hosting: 2026 Guide

#llm #selfhosting #llamacpp #ollama
Comments 2
20 min read
KV Cache on 16 GB GPUs: Making Long Context Actually Fit
rosgluk profile
Rost

KV Cache on 16 GB GPUs: Making Long Context Actually Fit

#llm #llamacpp #vllm #ollama
Comments 1
22 min read
👋 Sign in for the ability to sort posts by relevant, latest, or top.