Updated on March 14, more configs tested
Today, tools like LM Studio make it easy to find, download, and run large language models on consumer-grade hardware. A typical quantized 7B model (a model with 7 billion parameters which are squeezed into 8 bits each or even smaller) would require 4-7GB of RAM/VRAM which is something an average laptop has.
LM Studio allows you to pick whether to run the model using CPU and RAM or using GPU and VRAM. It also shows the tok/s metric at the bottom of the chat dialog
I have used this 5.94GB version of fine-tuned Mistral 7B and did a quick test of both options (CPU vs GPU) and here're the results.
Tokens/second
| Spec | Result |
|---|---|
| Apple M1 Pro CPU | 14.8 tok/s |
| Apple M1 Pro GPU | 19.4 tok/s |
| AMD Ryzen 7 7840U CPU | 7.3 tok/s |
| AMD Radeon 780M iGPU | 5.0 tok/s |
| AMD Ryzen 5 7535HS CPU | 7.4 tok/s |
| GeForce RTX 4060 Mobile GPU | 37.9 tok/s |
| AMD Ryzen 7 7800x3d CPU | 9.7 tok/s |
| GeForce RTX 4080 GPU | 78.1 tok/s |
Hardware Specs
2021 M1 Mac Book Pro, 10-core CPU(8 performance and 2 efficiency), 16-core iGPU, 16GB of RAM
-
2023 AOKZEO A1 Pro gaming handheld, AMD Ryzen 7 7840U CPU (8 cores, 16 threads), 32 GB LPDDR5X RAM, Radeon 780M iGPU (using system RAM as VRAM), TDP at 30W
- 3D Mark TimeSpy GPU Score 3000
- 3D Mark TimeSpy CPU Score 7300
-
2023 MSI Bravo C7VF-039XRU laptop, AMD Ryzen 5 7535HS CPU (6 cores, 12 threads, 54W), 16GB DDR RAM, GeForce RTX 4060 (8GB VRAM, 105W)
- GPU was slightly undervalued/overlocked, 3D Mark TimeSpy GPU Score 11300
- 3D Mark TimeSpy CPU Score 7600
Desktop PC, AMD Ryzen 7 7800x3d (8 cores 16 threads, 78w during test), 6200 DDR5, GeForce RTX 4080 16GB VRAM (slightly overclocked, 228w during test)
Screenshots
Mac
AOKZOE
MSI
Desktop PC
P.S>
It just hit me that while an average persons types 30~40 words per minute, RTX 4060 at 38 tokens/second (roughly 30 words per second) achieves 1800 WPM.
P.P.S>
Thanks to Sergey Zinchenko added the 4th config (
7800x3d + GeForce RTX 4080)
Top comments (20)
This depends much on the settings. I tried the same model and example query "tell me about Mars". Having Ryzen 3900 PRO CPU (12 cores, 24 threads, I got it for less than half price of 3900x), AMD RX 6700 (without x) which I also got cheap. RAM is pretty cheap as well so 128GB is in range of most. Using kobald-cpp rocm. With (14 layers on gpu, 14 cpu threads) it gave 6 tokens per second. (28,14) gave 15 T/s. (30,24) gave 4.43 T/s. Finally 35 layers, 24 CPU threads consumed total 7.3GB on GPU giving 34.61 T/s.
I'm writing to show that results depends very much on the settings.
JIC, I tested pure cases, 100% CPU and 100% offloading to GPU
How did you get to use 100% of the CPU?, which config or settings did you have?
You can offload all layers to GPU (CUDA, ROCm) or use CPU implementation (ex. HIPS). Just run LM Studio for your first steps. Run kobaldcpp or kobapldcpp-ROCm as second. Then try to use python and transformers. From there you should know enough about the basics to choose your directions. And remember that offloading all to GPU still consumes CPU
This is a peak when using full ROCm (GPU) offloading. See CPU usage on the left (initial CPU load is to start the tools, LLM was used on the peak at the end - there is GPU usage but also CPU used)
And this is windows - ROCm still is very limited on other operating systems :/
Adding some info here:
Running on a Razer Blade 2021 with a Ryzen 5900HX, a GF 3070Ti and 16GB RAM, I got 41.75tok/s. I used the same test as you, asking about Mars on the same model.
Hope that adds information to this very interesting topic.
Thanks for the contribution! I assume you used 100% GPU off-loading , right? Just checking:)
Indeed, 100%GPU off-loading.
I also tested an Ryzen 7950X with 0% off loading, but there’s something odd. I set 32 threads but CPU use is not going beyond 60% and only gets 7tok/s. Any thoughts how about possible cause?
Just for fun, I’ll check with an Asus ROG Ally later (Z1 Extreme version).
Seems the threads param is ignored, I saw same behaviour when testing CPU inference
Just for fun, here are some additional results:
iPad Pro M1 256GB, using LLM Farm to load the model: 12.05tok/s
Asus ROG Ally Z1 Extreme (CPU): 5.25 tok/s using the 25W preset, 5.05tok/s using the 15W preset
Update:
Asked a friend with a M3 Pro 12core CPU 18GB. Running from CPU: 17.93tok/s, GPU: 21.1tok/s
The CPU result for ROG is close to the one from 7840U, after all they almost identical CPUs
The ROG Ally has a Ryzen Z1 Extreme which appears to be nearly identical to the 7840U, but from what I can discern, the NPU is disabled. So if / when LM Studio gets around to implementing support for that AI accelerator the 7840U should be faster at inferencing workloads.
AMD GPU seems to be an underdog in the ML world, when compared to Nvidia... I doubt that AMD's NPU will see better compatibility with ML stack than it's GPUs
Intel i7 14700k - 9.82 token/s with no GPU offloading(peaked at 35% CPU usage in LMStudio. Guessing issue with multithreading)
Zotac Trinity non-OC 4080 Super - 71.61 tokens/s max GPU offloading
All numbers measured on non-overclocked factory default setup
Thanks for sharing the numbers!
Indeed there’s something odd with the multithreading of the CPUs
Just a quick update: using a RTX 4070 Super gets 58.2tok/s
In these tests is the 7840U utilizing the integrated NPU to accelerate the workload?
The result for "780M iGPU" is indeed the result coming from the GPU integrated into 7840U APU
@maximsaplin GPU != NPU
They are distinct accelerators.
NPU is not mentioned anywhere