Storage maths, shortened vectors and int8
The 8-number vectors are toys. In a model trained for shortening, a cut vector keeps most of its quality. In an ordinary model it can lose much more.
Every vector costs dimensions times bytes per number. A million 1536-dimension float32 vectors take about 6 GB. More dimensions can hold more meaning but cost memory and speed, and a few tricks (shorter vectors, smaller number types) cut the bill when the model allows.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write gigabytes(n_vectors, dims, bytes_per_number=4) and print the size for 250,000 vectors at 768 dimensions in float32 and int8, to two decimals.
Write to_int8(v) that scales a vector to unit length and returns floor(x * 127 + 0.5) for each number, and print the result for [3, 4].
Explain it without notes
How do you compute the memory a vector index needs, and what does it leave out?
What does a larger number of dimensions buy, and what does it cost?
When can you safely keep only the first N numbers of an embedding?
What is int8 or binary quantization, and how are they used safely in search?
Expected output
384 dims: float32 1.54 GB, int8 0.38 GB, binary 0.05 GB
768 dims: float32 3.07 GB, int8 0.77 GB, binary 0.10 GB
1536 dims: float32 6.14 GB, int8 1.54 GB, binary 0.19 GB
3072 dims: float32 12.29 GB, int8 3.07 GB, binary 0.38 GB
full 8 dims: cosine 0.9217
first 4 dims: cosine 0.9390
first 2 dims: cosine 0.9374
int8 values of A: 78 63 47 47 31 16 16 16
int8 cosine: 0.9219