LoRA: how many numbers are trained
Matrix shapes are example numbers, not a real model. The formula is the point: r x (d_in + d_out) against d_in x d_out.
Fine-tuning continues a model's training on your own examples. It is good at teaching a format, a style or a narrow task, and poor at adding facts, which belong in retrieval. Try prompts and retrieval first.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write lora_params(d_in, d_out, r) and full_params(d_in, d_out), and print both for a 4096 x 4096 matrix with r = 8.
Write train_size(n, percent) that returns how many of n examples go to training using whole-number maths (n * percent // 100), and print it for 10 examples at 80 percent and for 7 examples at 80 percent.
Explain it without notes
When should you fine-tune rather than use retrieval?
Explain LoRA and why its adapter is small.
Why must the validation set be kept out of training, and how do you check?
In what order would you try to fix a feature that gives answers in the wrong format?
Expected output
attention query: full 16777216, LoRA 65536 (0.39 percent)
attention value: full 16777216, LoRA 65536 (0.39 percent)
feed-forward up: full 45088768, LoRA 120832 (0.27 percent)
per layer: full 78643200, LoRA 251904
all 32 layers: full 2516582400, LoRA 8060928 (0.32 percent)
adapter file at 16 bits: 16.1 MB
rank sweep for a 4096 x 4096 matrix:
r=1: 8192 parameters (0.049 percent)
r=4: 32768 parameters (0.195 percent)
r=8: 65536 parameters (0.391 percent)
r=16: 131072 parameters (0.781 percent)
r=64: 524288 parameters (3.125 percent)