Memory for weights at 16, 8 and 4 bits
The model sizes are made-up round numbers. The 20% overhead is a lesson rule of thumb, not a measurement.
Quantisation stores each weight in fewer bits, so a model needs less memory and often runs faster, at the price of a small rounding error. The memory is just parameters times bits divided by 8.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write weights_gb(params_billion, bits) that returns the gigabytes needed for the weights, and print it for a 7 billion parameter model at 16, 8 and 4 bits.
Write to_steps(ws, scale) that returns each weight divided by the scale and rounded half away from zero, as a list of whole numbers. Print the result for the weights 0.26, -0.26, 0.04 and 1.27 with scale 0.1.
Explain it without notes
How much memory does a model with 7 billion parameters need for its weights at 16, 8 and 4 bits?
How does int8 quantisation work, and what is the maximum rounding error?
Why can a single large weight ruin a quantised layer, and what do real systems do about it?
How would you decide whether a 4-bit version of a model is good enough?
Expected output
8B at 16-bit: weights 16.0 GB, with overhead 19.2 GB, fits in 24 GB: yes
8B at 8-bit: weights 8.0 GB, with overhead 9.6 GB, fits in 24 GB: yes
8B at 4-bit: weights 4.0 GB, with overhead 4.8 GB, fits in 24 GB: yes
30B at 16-bit: weights 60.0 GB, with overhead 72.0 GB, fits in 24 GB: no
30B at 8-bit: weights 30.0 GB, with overhead 36.0 GB, fits in 24 GB: no
30B at 4-bit: weights 15.0 GB, with overhead 18.0 GB, fits in 24 GB: yes
70B at 16-bit: weights 140.0 GB, with overhead 168.0 GB, fits in 24 GB: no
70B at 8-bit: weights 70.0 GB, with overhead 84.0 GB, fits in 24 GB: no
70B at 4-bit: weights 35.0 GB, with overhead 42.0 GB, fits in 24 GB: no