What is model quantization and how does it reduce LLM cost?

TL;DR

• Model quantization compresses model numbers by mapping high precision weights and activations to lower precision, for example from $32$-bit floats to $8$-bit integers.
• That reduces memory, cache pressure, and bandwidth, giving lower inference cost and faster latency on many CPUs and specialized hardware.
• You can use uniform, symmetric, per-channel and mixed precision schemes depending on accuracy needs and hardware support. Key tradeoffs: lower precision cuts cost and latency but can drop accuracy and require calibration or fine tuning.

In this question, we will learn what model quantization is and how it reduces LLM cost.

We will cover the following:

• The intuition
• How it actually works (with a worked example)
• Common quantization schemes and a comparison
• Practical tips for using quantization in LLMs
• Tradeoffs and failure modes
• Interviewer-style questions

Direct answer: Model quantization reduces LLM cost by replacing high precision numbers with lower precision representations so the model uses less memory, requires less bandwidth, and performs cheaper arithmetic. With careful choice of scheme and calibration you can often quantize to $8$-bit or lower with small accuracy loss, cutting memory and often FLOPs-equivalent compute cost by multiple times.

The intuition (an analogy that makes it click)

Think of weights and activations as water stored in many glass bottles of different sizes and shapes. Floating point is like having custom bottles with fine graduations so you can measure precisely. Quantization is like standardizing to a set of small labeled jars. You lose some granularity, but you store more water in less space and pass jars faster through the door. For many tasks the lost granularity does not change the final mix much. We choose jar sizes and labels so the important differences remain visible.

How it actually works (the real mechanics, with one concrete worked example)

At the core quantization finds a mapping from full precision values $x$ to integers $x_q$ using a scale $s$ and zero point $z$. A common uniform affine mapping is:

$$x_q = \text{round}\!\left(\frac{x}{s}\right) + z$$

and we reconstruct (dequantize) as

$$x \approx s\,(x_q - z).$$

Here $s$ is a positive real scale and $z$ is an integer zero point. Both are chosen to minimize quantization error over a calibration dataset or a tensor's range.

Worked example: quantize a small weight vector from $32$-bit float to $8$-bit integer.

Original weights (float32):

Find range: $x_\text{min} = -0.87$, $x_\text{max} = 0.45$. For unsigned $8$-bit we often use symmetric mapping with zero point $z=0$ and scale

$$s = \frac{\max(|x_\text{min}|,|x_\text{max}|)}{2^{b-1}-1} = \frac{0.87}{127} \approx 0.00685,$$

with $b = 8$. Quantized integers:

Memory savings: original $4$ weights at $32$-bit each use $4 \times 32 = 128$ bits. Quantized to $8$-bit they use $4 \times 8 = 32$ bits, a $32/8 = 4$ times reduction. In large LLMs this scales to gigabytes saved.

Practical notes: you store the integer array plus per-tensor or per-channel scales and zero points. For inference you operate in integer or mixed integer-float arithmetic.

Types of quantization and a comparison

When quantizing LLMs we often use per-channel weight quantization and either symmetric or affine activation quantization. For very low-bit ($4$-bit or lower) we sometimes use mixed precision: keep layernorm and embedding layers at $16$-bit while quantizing matrix multiplications to lower bits.

Practical tips for LLMs

• Calibrate activations with representative data to pick scales. A small holdout of tokens is often enough.
• Start with per-channel weight $8$-bit and affine activation $8$-bit as a safe baseline. Measure perplexity or task accuracy.
• If quality drops, try mixed precision: keep layernorm, softmax, and small matrices in $16$-bit.
• Some hardware supports efficient integer GEMM for $8$-bit and $4$-bit. Test latency on your target hardware, not just theoretical FLOPs.

Tradeoffs and failure modes

Quantization reduces memory, memory bandwidth, and often energy. But it introduces rounding noise that can change model outputs, especially in deep networks with many matrix multiplies. Small models or sensitive layers can suffer noticeable accuracy loss when naively quantized.

Common failure modes:

• Activation outliers cause scale choice to blow up error.
• Layernorm and softmax are sensitive and may need higher precision.
• Per-channel scales reduce error but complicate kernel implementations.

Questions the interviewer might ask

Some follow-up questions you might get: How does per-channel quantization help?
Per-channel uses a separate scale for each output channel so values with different dynamic ranges do not share one scale. That reduces quantization bias for heavy-tail or heterogeneous channels.

What is the difference between symmetric and affine quantization?
Symmetric sets zero point $z=0$ and maps around zero. Affine allows a nonzero $z$ so the integer mapping can represent tensors whose range is not symmetric about zero.

When would you use $4$-bit quantization versus $8$-bit?
Use $4$-bit when memory or throughput constraints are severe and you can tolerate more accuracy loss or apply fine tuning or quantization-aware training. Start with $8$-bit for a safer tradeoff.

How do you handle activations at inference time?
You collect activation ranges via calibration, choose per-tensor scales, and either quantize activations on the fly or use mixed-precision kernels that keep activations in higher precision where needed.

Does quantization always reduce FLOPs?
Not necessarily. It reduces memory and can enable cheaper integer ops. But if hardware does not provide efficient low-bit matmul kernels you may still compute in float and only save memory, not FLOPs.

Some things to note:

• Measure on target hardware for real latency and cost numbers.
• Calibration data should be representative of inference inputs.

What the interviewer is really testing

They want to see that you understand both the math and the systems impact: how quantization maps values to integers, how scales and zero points matter, and how memory, bandwidth, and hardware support affect cost. They also want to hear about practical mitigations like per-channel scales, calibration, and mixed precision so you can deploy LLMs safely.