Insight · October 30, 2026
Smaller numbers,
same model.
A frontier model stores billions of weights as high precision numbers. Quantization stores them at lower precision, so the same model runs on less memory and cheaper hardware.
01 · The idea
Same weights, stored with less precision.
Quantization stores a model’s weights at lower numeric precision. A frontier model keeps its billions of parameters as 16-bit numbers. Quantization rounds them to 8-bit or 4-bit integers, so the same model holds the same knowledge in a fraction of the memory and runs on cheaper hardware.
Think of a thermostat with a dial that turns smoothly versus one with ten fixed clicks. The clicks lose a little detail, but the room still gets warm, and the dial is far simpler to build. A weight stored as a high precision float becomes one of a small set of integer steps. The model loses a little detail per number and keeps its behavior.
02 · How it works
Fewer bits per weight, times a billion weights.
A weight is just a number. Stored at 16 bits it can take tens of thousands of values. Stored at 8 bits it can take 256. Stored at 4 bits it can take 16. Quantization picks a scale, maps each original weight to the nearest step on that smaller ladder, and keeps the scale so the number can be read back out at run time.
The saving is mechanical. Halve the bits per weight and you roughly halve the memory the weights occupy. The bitsandbytes library describes its 8-bit path as enabling inference with only half the required memory, and its 4-bit path as the method behind QLoRA. Less memory means the model fits on a smaller card, and a smaller card is cheaper to rent and faster to feed.
The hard part is the outliers. A few weights sit far from the rest, and rounding them carelessly breaks the model. Good methods handle those few at higher precision and quantize the rest aggressively. That split is the whole craft.
Most practical quantization touches the weights and leaves the activations, the values that flow through the model as it runs, at higher precision. You will see this written as W4A16: 4-bit weights, 16-bit activations. The weights are the heavy, fixed cost, billions of numbers sitting in memory, so they are where the saving lives. The activations are smaller and more sensitive, so most methods leave them alone.
The one line to keep
“Quantization does not make the model dumber. It makes the same model cheaper to run.”
03 · The methods
Two families, a handful of names worth knowing.
01
Post-training quantization
Take a finished model and round its weights down afterward, using a small sample of data to set the scale. No retraining. This is the fast, cheap, common path.
02
Quantization-aware training
Simulate low precision during training so the model learns to absorb the rounding. More accurate at the low end, far more expensive to run.
03
GPTQ
A one pass method that uses second-order information to quantize the weights of a finished model to 3 or 4 bits. It covers the OPT and BLOOM families in the original work.
04
AWQ
Activation-aware weight quantization. Not every weight matters equally, so it protects the small share that carries the most signal and quantizes the rest to 3 or 4 bits.
05
QLoRA
A 4-bit frozen base with a small set of trainable adapters on top. It made finetuning a large model on a single consumer-class GPU possible.
The first two are categories: quantize after training, or train with quantization in mind. The last three are specific published methods you will see named in model cards and serving stacks.
04 · What it is not
Not distillation. Not pruning.
These three get grouped together because they all make a model cheaper, and they are not the same move. Distillation trains a different, smaller model to copy a larger one. You end with a new model that has fewer weights.
Pruning removes weights the model barely uses, cutting away connections so fewer remain. You end with the same model, minus the parts that did little.
Quantization touches neither the count of the weights nor which model you are running. It keeps every weight and stores each one in less space. That is why the three stack. A model can be distilled, then pruned, then quantized, and the savings compound. It is also why a quantized model often pairs well with a small language model strategy rather than replacing it.
05 · One result that shows the point
A 65B model, finetuned on one GPU.
QLoRA, from a team at the University of Washington, is the clearest case. The authors report that it reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. The base model is frozen at 4 bits and only a small set of adapters trains on top.
The precision comes from a data type they call 4-bit NormalFloat, which the paper describes as information theoretically optimal for normally distributed weights, paired with double quantization of the scale constants and paged optimizers for the memory spikes. Their Guanaco models reached 99.3% of the performance level of ChatGPT on the Vicuna benchmark, with 24 hours of finetuning on a single GPU.
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
)
model = AutoModelForCausalLM.from_pretrained(
model_id, quantization_config=config
)Four bit load with bitsandbytes · the weights arrive quantized
06 · The tradeoff
How much does the model lose?
Less than most people expect, and it depends on how far you push. The GPTQ work, from Frantar and colleagues, quantizes the OPT and BLOOM families to 2, 3, or 4 bits in a single pass. On LLaMa-7B its reported WikiText2 perplexity moves from 5.68 at full precision to 6.09 at 4 bits, a small gap, and to 8.07 at 3 bits, a larger one. Lower perplexity is better, so the lesson is plain: 4 bits is close, 3 bits starts to cost.
Speed moves the other way. GPTQ reports a 1.9x to 3.25x generation speedup for OPT-175B with its optimized kernels. AWQ, a best paper at MLSys 2024 from the MIT Han Lab, reports running a 4-bit LLaMA-3-8B 2.7x faster than full precision on a single RTX 4090. Smaller weights move through memory faster, and memory bandwidth is what inference waits on.
The working rule operators land on: 8 bits is nearly free and safe by default. 4 bits is the sweet spot for fitting a large model on one card, with a small, measurable accuracy cost. Below 4 bits you are trading real quality for the last of the memory, and you measure before you ship.
07 · When to think twice
Cheaper is not always the right call.
If the model already fits comfortably and the bill is not the constraint, quantization adds a step and a risk for little gain. A model you run a handful of times does not need it.
On tasks at the edge of a model’s ability, where it is already just barely good enough, the small accuracy loss can be the difference that matters. Precise math, long chains of reasoning, and strict formatting are the usual places a quantized model slips first.
And the only honest way to know is to measure. A benchmark drop of a point can hide a failure on the one case your product depends on. Quantize, then run the model against the work it will actually do, before it reaches anyone.
Closing
The cheapest win in AI is the one already in the weights.
At Bttr. this sits inside product engineering, full-stack systems built for performance and longevity. Before you buy a bigger card, check whether the model you already have fits once it is stored with the precision it actually needs.
Written by · October 30, 2026
Donny Smith · ECD, Founder, Bttr.
Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.
LinkedInShare this perspective
More insights
Adjacent perspectives.
Bttr. Field Brief
The brief Bttr. writes for senior buyers.
Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.