Topic 13.3
Quantisation
In one line
Quantisation stores each weight in fewer bits, so a model needs less memory and often runs faster, at the price of a small rounding error. The memory is just parameters times bits divided by 8.
Think of it like this
Writing prices. The exact value of a pen might be 12.4837 rupees. Nobody needs that: you write 12.48, then 12.5, then 12. Each step makes the number shorter to write and store, and each step loses a little detail. Quantisation does this to the billions of numbers inside a model, choosing how short to make them.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Weight
- One learned number inside the model. A model has billions of them.
- Bit
- One 0-or-1 digit. 16 bits store a number more precisely than 8 or 4 bits.
- Quantisation
- Storing numbers with fewer bits by rounding them to a coarse grid.
- Scale
- The size of one step on the grid. Real value = whole number x scale.
- int8
- A whole number from -128 to 127 stored in one byte. Symmetric schemes use -127 to 127.
- Rounding error
- The difference between the original number and the one you get back. At most half a step.
- Outlier
- A weight much larger than the rest. It forces a coarse grid on everyone.
- GGUF
- The file format the llama.cpp tool loads models from, including quantised ones.
Step by step
01Step 1: do the memory maths first
Parameters x bits / 8 gives bytes. The first example prints a table for three made-up model sizes at 16, 8 and 4 bits and checks each against a 24 GB card. Notice that the largest size does not fit even at 4 bits, while the middle size fits only after quantising.
def weights_gb(params_billion, bits):
return params_billion * bits / 8 # billions of parameters x bytes each = gigabytes
print(weights_gb(8, 16), weights_gb(8, 8), weights_gb(8, 4)) # 16.0 8.0 4.002Step 2: round to a grid by hand
Suppose the grid step is 0.1. The weight 0.82 becomes 8 steps and goes back as 0.8, an error of 0.02. The weight 0.07 becomes 1 step (0.1), an error of 0.03. That is quantisation: a whole number of steps, and one shared step size.
03Step 3: choose the step size from the data
The step is the largest absolute weight divided by 127, so the largest weight lands exactly on the end of the grid and nothing is clipped. In the example the largest weight is 1.91, the scale is 1.91 / 127, about 0.015, and every error is at most half of that.
04Step 4: write round-half-away yourself
Languages differ in how they round exact halves (2.5), so the example rounds half away from zero by hand: add 0.5 to the absolute value, drop the fraction, restore the sign. All three languages then print the same numbers. Real libraries pick one rule and apply it everywhere.
05Step 5: print the error, not just the result
The example prints the original, the stored whole number, the value you get back and the error for all eight weights, then the largest and average error, then the size. Eight 32-bit numbers take 32 bytes; eight int8 values plus one 32-bit scale take 12. With only eight weights the saved scale still weighs on the total (about 2.7 to 1). With millions of weights sharing a scale, the ratio gets close to 4 to 1, which is the real saving.
06Step 6: break it with an outlier, then verify on your task
Change one weight to 40.0 in the break-it. The scale jumps, the other seven weights collapse onto a few steps, and the error balloons. That is why real schemes work on small blocks. Finally, remember that low error on weights does not guarantee the same answers: only your eval set tells you that.
Try it yourself
- 1
Predict the table
Before running the first example, write down which rows will say
yes. Then changeGPU_GBto 48 and predict again. Which model size changes fromnotoyesand at which bit width? - 2
Make the grid coarser
In the second example change 127 to 7, which is about what a 4-bit signed grid offers. Compare the largest error and the mean error with the int8 run. How many times larger did the error become?
- 3
Predict one weight
Pick the weight 0.55. Calculate by hand
0.55 / scale, round it, multiply back and compute the error. Check against the printed line.
Code & diagrams
The model sizes are made-up round numbers. The 20% overhead is a lesson rule of thumb, not a measurement.
Expected output
8B at 16-bit: weights 16.0 GB, with overhead 19.2 GB, fits in 24 GB: yes
8B at 8-bit: weights 8.0 GB, with overhead 9.6 GB, fits in 24 GB: yes
8B at 4-bit: weights 4.0 GB, with overhead 4.8 GB, fits in 24 GB: yes
30B at 16-bit: weights 60.0 GB, with overhead 72.0 GB, fits in 24 GB: no
30B at 8-bit: weights 30.0 GB, with overhead 36.0 GB, fits in 24 GB: no
30B at 4-bit: weights 15.0 GB, with overhead 18.0 GB, fits in 24 GB: yes
70B at 16-bit: weights 140.0 GB, with overhead 168.0 GB, fits in 24 GB: no
70B at 8-bit: weights 70.0 GB, with overhead 84.0 GB, fits in 24 GB: no
70B at 4-bit: weights 35.0 GB, with overhead 42.0 GB, fits in 24 GB: noSymmetric quantisation: scale = max absolute weight / 127. Rounding half away from zero is written by hand.
Expected output
scale: 0.015039
0.82 -> 55 -> 0.8272 (error 0.0072)
-1.91 -> -127 -> -1.9100 (error 0.0000)
0.07 -> 5 -> 0.0752 (error 0.0052)
0.55 -> 37 -> 0.5565 (error 0.0065)
-0.33 -> -22 -> -0.3309 (error 0.0009)
1.24 -> 82 -> 1.2332 (error 0.0068)
-0.02 -> -1 -> -0.0150 (error 0.0050)
0.96 -> 64 -> 0.9625 (error 0.0025)
max error 0.0072, mean error 0.0042
size: 32 bytes as 32-bit numbers, 12 bytes as int8 plus one 32-bit scaleBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget to divide by 8
In the first example change params * bits // 8 to params * bits.
Break #2
Let one outlier set the scale
In the second example replace the weight -1.91 by 40.0.
Myth vs fact
Myth
A 4-bit model is four times worse than a 16-bit one.
Fact
The saving is 4 times in memory, not 4 times in quality. The quality loss is usually small for well-made 4-bit and 8-bit variants but varies by model and task, so measure it.
Myth
Quantised models are always faster.
Fact
They read less memory per token, which often helps, but speed also depends on the hardware, the software and how the numbers are unpacked. Measure on your setup.
Myth
Quantisation adds new knowledge or fixes mistakes.
Fact
It only stores the same numbers less precisely. At best it keeps the behaviour, at worst it degrades it.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Text generation is usually limited by memory bandwidth: every new token needs the weights read once. Halving the bytes per weight can come close to halving the time per token on bandwidth-limited hardware, which is why quantisation improves speed as well as fit.
- ▸
The cache that stores the attention keys and values grows with context length and with the number of conversations running at once, and it can be bigger than the weights for long contexts (Topic 13.4 computes it). Quantising weights does not shrink it.
- ▸
Evaluate quantised variants on the hardest slice of your eval set, not the average. Averages hide that a 4-bit model may fail exactly the long, multi-step or rare-language cases that matter most.
- ▸
Keep the original and the quantised model side by side in your harness and compare per case (Topic 13.8). A change in the answers on cases the original got right is a regression even if the average pass rate looks the same.
Remember this
- 1
Why this exists. A model is mostly weights, the learned numbers (Topic 0.1). Stored as 16-bit numbers, a model with 8 billion weights is 16 GB, which already fills a small graphics card. A bigger one does not fit on any single card you can easily buy. Memory decides which hardware you need, so it decides your cost (Topic 13.2). Speed also depends on it: producing each token means reading the weights from memory, so smaller weights can be read faster.
- 2
The memory maths. Memory in bytes = parameters x bits per weight / 8, because a byte is 8 bits. A model with 8 billion parameters is 8,000,000,000 x 16 / 8 = 16 GB at 16 bits, 8 GB at 8 bits and 4 GB at 4 bits. In practice you need extra for the working buffers and for the attention cache (the keys and values the model keeps for the conversation so far, which grows as the conversation grows; Topic 13.4 explains it), so the example adds a labelled 20% rule of thumb. The honest way to know is to load the model and look.
- 3
How rounding to a grid works. Take a list of weights. Find the largest absolute value. Divide it by 127 to get the scale: the size of one step on the grid. Each weight becomes
q = round(w / scale), a whole number from -127 to 127 that fits in one byte (an int8). To use the weight you multiply back:w' = q x scale. The difference betweenwandw'is the rounding error, and it can never be more than half a step,scale / 2. The example prints every error so you can see it. - 4
Why outliers hurt. One very large weight makes the scale large, so every step on the grid is coarse, and the small weights, which are the majority, all round to a handful of values or to zero. Real schemes therefore use a separate scale for each small block of weights, and some keep unusual weights in higher precision. The break-it for this topic puts one huge weight in the list so you can watch the others collapse.
- 5
Quality trade-offs. 8-bit is usually very close to the original. 4-bit often works well for chat and summarising but can lose accuracy on hard reasoning, maths, code or less common languages, and smaller models usually suffer more than larger ones. There is no universal rule: a quantised model is a different model, so run your own evals on it (Topic 13.1) before you trust it. A large model at 4 bits is often better than a small model at 16 bits that needs the same memory, but test that on your task as well.
- 6
Where you meet it. Model files and local tools label their variants by bit width, and the llama.cpp tool loads quantised models from
.gguffiles (Topic 13.4). Quantising after training, as in the example, is called post-training quantisation. Training the model to expect low precision is a separate technique. This topic covers the weights only: the attention cache can also be stored in fewer bits, which is a different choice with its own trade-offs.
Explain it without notes
How much memory does a model with 7 billion parameters need for its weights at 16, 8 and 4 bits?
How does int8 quantisation work, and what is the maximum rounding error?
Why can a single large weight ruin a quantised layer, and what do real systems do about it?
How would you decide whether a 4-bit version of a model is good enough?
Practice
Write weights_gb(params_billion, bits) that returns the gigabytes needed for the weights, and print it for a 7 billion parameter model at 16, 8 and 4 bits.
Write to_steps(ws, scale) that returns each weight divided by the scale and rounded half away from zero, as a list of whole numbers. Print the result for the weights 0.26, -0.26, 0.04 and 1.27 with scale 0.1.
Trade-offs
- ↔
Fewer bits save memory and often time, but add rounding error that can hurt hard or rare cases.
- ↔
Block-wise scales fight outliers but store more scales, so the saving is a little less than bits alone suggest.
- ↔
A larger model quantised hard can beat a smaller model at full precision in the same memory, but only an eval on your task can say.
Done when you can
Done when you can compute weight memory from parameters and bits.
Done when you can quantise and dequantise a list with a scale and print the error.
Done when you can explain the half-step error bound.
Done when you can explain why outliers hurt and how blocks help.
Done when you can name the evidence you need before shipping a quantised model.