DEV Community

AI Tech News profile picture

AI Tech News

404 bio not found

Joined Joined on  github website twitter website
KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly
wolfsea2357 profile

KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly

#llm #vllm #performance #devops
Comments
7 min read

Want to connect with AI Tech News?

Create an account to connect with AI Tech News. You can also sign in below to proceed if you already have an account.

Create Account
Already have an account? Sign in
What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You
wolfsea2357 profile

What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You

#llm #vllm #performance #devops
Comments
6 min read
Five Ways Your LLM Serving Benchmark Is Lying to You (and How to Catch Each One)
wolfsea2357 profile

Five Ways Your LLM Serving Benchmark Is Lying to You (and How to Catch Each One)

#llm #vllm #performance #devops
Comments 1
15 min read
A 4B Model Just Beat 8B — We Tested 18 Small LLMs and the Results Are Wild
wolfsea2357 profile

A 4B Model Just Beat 8B — We Tested 18 Small LLMs and the Results Are Wild

#ai #machinelearning #opensource #benchmark
Comments
6 min read
MARL: Runtime Middleware That Reduces LLM Hallucination Without Fine-Tuning
wolfsea2357 profile

MARL: Runtime Middleware That Reduces LLM Hallucination Without Fine-Tuning

#ai #llm #python #opensource
Comments 2
4 min read
loading...