Harnessing Quantization for Large Language Models on Modest Hardware

So I've been playing around with different models on GCP... but all this time I've ran into the issue of, say for instance, Yi-34B being giganormous (by my standards!) at 65GB when downloaded from Huggingface. This means that I have to have at least 65GB of VRAM cache combined for it to run on my GPU! I also still need at least 65GB RAM as well, it loads both on the GPU and the CPU. ALSO, it was running super slow even with enough VRAM and physical RAM cache, with 10+seconds response time for anything other than a hello. Then, I discovered quantization. This blog provides a simplified overview of how quantization enables the use of large models on modest hardware and touches upon the nuanced decisions in hardware selection for optimal performance.

In the world of machine learning, especially when dealing with Large Language Models (LLMs) like Yi-34B, the quest for efficiency is as important as the quest for capability. One key technique enabling the operation of these colossal models on relatively modest hardware is quantization. But what is quantization, and how does it allow for this computational wizardry?

The Magic of Quantization:
Quantization, in essence, is about reducing the precision of the numbers that represent a model's parameters. Think of it as lowering the resolution of an image. In doing so, we compress the model, making it smaller and less demanding on resources, particularly RAM and VRAM. This process involves converting parameters from floating-point representations (which are bulky) to integers (which are more compact).

PyTorch's Approach to Quantization:
PyTorch, a leading framework in machine learning, offers various quantization strategies:

1. Dynamic Quantization: This method quantizes weights in a pre-trained model but leaves the activations in floating-point. The conversion happens dynamically at runtime.

2. Static Quantization: In contrast to dynamic quantization, static quantization also quantizes the activations but requires calibration with a representative dataset to determine the optimal scaling parameters.

3. Quantization-Aware Training: This approach simulates the effects of quantization during the training phase, allowing the model to adapt to the lower precision.

Each method balances the trade-off between model size, computational demand, and performance accuracy.

Old vs. New GPUs: A VRAM Dilemma:
A common misconception in hardware selection is that newer always means better. However, when it comes to running large models, the amount of VRAM (Video RAM) can be more critical than the GPU's generation. Older GPUs with more VRAM might outperform newer ones with less VRAM in specific scenarios. This is because more VRAM allows for larger models or larger batches of data to be loaded simultaneously, enhancing the efficiency of model training and inference.

Conclusion:
Quantization is a powerful tool in the arsenal of machine learning practitioners, enabling the deployment of advanced LLMs on less powerful hardware. As we advance, the interplay between model optimization techniques and hardware choices will continue to be a critical area of focus, ensuring that the boundaries of AI and machine learning can be pushed further, even within the constraints of existing technology.