Topic 2.5
Dimensions and Storage
In one line
Every vector costs dimensions times bytes per number. A million 1536-dimension float32 vectors take about 6 GB. More dimensions can hold more meaning but cost memory and speed, and a few tricks (shorter vectors, smaller number types) cut the bill when the model allows.
Think of it like this
A wardrobe. Every outfit you keep takes space on a hanger. A bigger, finer description of each outfit (more dimensions) lets you tell them apart better, but the wardrobe fills up sooner. With ten outfits you do not care. With a million, you do the arithmetic first.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Float32
- A number stored in 4 bytes (32 bits). The usual format for embeddings.
- Int8
- A whole number from -128 to 127 stored in 1 byte. Four times smaller than float32.
- Binary quantization
- Keeping only one bit per dimension (is the number above 0?). 32 times smaller than float32, but coarse.
- Quantization
- Storing numbers with fewer bits to save space, accepting a small loss of precision.
- Truncation (of a vector)
- Keeping only the first N numbers of a vector. Safe only for models trained for it.
- Matryoshka training
- Training a model so that the first numbers of its vector are already a good, shorter embedding, like nested dolls. Named in the sentence-transformers documentation.
Step by step
01Step 1: bytes per vector
384 dimensions at 4 bytes is 1,536 bytes per vector. 1536 dimensions is 6,144 bytes. 3072 is 12,288 bytes. A million vectors is that many megabytes.
float32 = N_VECTORS * dims * 4 # 4 bytes per number
int8 = N_VECTORS * dims # 1 byte per number
binary = N_VECTORS * (dims // 8) # 1 bit per number02Step 2: read the table
The example prints, for a million vectors, float32 / int8 / binary: 384 dims 1.54 / 0.38 / 0.05 GB; 768 dims 3.07 / 0.77 / 0.10 GB; 1536 dims 6.14 / 1.54 / 0.19 GB; 3072 dims 12.29 / 3.07 / 0.38 GB.
03Step 3: shorten a vector and renormalise
Cut the first 4 of 8 numbers and recompute the cosine. In the example the cosine is computed with a formula that divides by the lengths, so it is correct without a separate normalise step. In a store that uses a plain dot product you must normalise the cut vector yourself.
04Step 4: quantise to int8
Scale the unit vector so its numbers lie between -127 and 127, and round: floor(x * 127 + 0.5). The example's result for A is 78 63 47 47 31 16 16 16. The cosine between the two int8 vectors is 0.9219, compared with 0.9217 at full precision.
05Step 5: add what the index needs
The raw vectors are the floor, not the total. Indexes that make search fast (graphs, clusters: Phase 3 and Phase 5) add memory of their own, often a large fraction of the raw size, and you also store the text and metadata.
06Step 6: decide with a test, not a guess
Shorter or smaller vectors lose a little accuracy. Measure retrieval quality on your test questions (Phase 9) before and after, and accept the change only when the loss is small enough for the saving.
Try it yourself
- 1
Size your own corpus
Set
N_VECTORSto the number of chunks you expect (try 20,000 for a handbook, 2,000,000 for a company wiki). Which row of the table fits on a laptop? Which would you need a larger machine for? - 2
Add a dimension count
Add
256to the list of dimensions. What does the float32 column say? What would an index with twice the raw size cost? - 3
Cut harder
Change the loop to also try
keep = 1. What does the cosine become? Why does one number lose so much?
Code & diagrams
The 8-number vectors are toys. In a model trained for shortening, a cut vector keeps most of its quality. In an ordinary model it can lose much more.
Expected output
384 dims: float32 1.54 GB, int8 0.38 GB, binary 0.05 GB
768 dims: float32 3.07 GB, int8 0.77 GB, binary 0.10 GB
1536 dims: float32 6.14 GB, int8 1.54 GB, binary 0.19 GB
3072 dims: float32 12.29 GB, int8 3.07 GB, binary 0.38 GB
full 8 dims: cosine 0.9217
first 4 dims: cosine 0.9390
first 2 dims: cosine 0.9374
int8 values of A: 78 63 47 47 31 16 16 16
int8 cosine: 0.9219Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget the bytes per number
In the Python tab, change float32 = N_VECTORS * dims * 4 to float32 = N_VECTORS * dims (forget the 4 bytes per number). Run.
Myth vs fact
Myth
More dimensions always means better search.
Fact
More dimensions usually helps up to a point, but a well-trained smaller model can beat a badly trained larger one. Compare on your own questions.
Myth
You can cut any embedding to its first 256 numbers.
Fact
Only models trained for it keep their quality when cut. For others the first numbers are not special, and the scores change unpredictably.
Myth
Compression always hurts quality a lot.
Fact
int8 usually costs very little. Binary costs more, and is normally followed by a re-check of the best few results with the full vectors.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Memory, not CPU, is often the first limit of a vector store. A graph index typically needs the vectors in RAM for speed, so the table above is a hosting decision. Disk-based and quantised indexes (Phase 5) trade speed for size.
- ▸
Embedding cost has two parts: tokens sent to the model (a one-off per chunk, Topic 2.7) and the storage you carry forever. Doubling dimensions doubles the second.
- ▸
Re-normalise after every lossy step (cutting or quantising), or pick a store that does it. Lengths drift after rounding, and a drifting length silently turns a dot product into something that is no longer cosine (Topic 2.3).
Remember this
- 1
Why this exists. Vector search keeps every vector ready to compare. A demo has twelve passages and no memory problem. A real corpus has hundreds of thousands or millions of chunks, and the vectors, plus the index built on top of them (Phase 5), must usually sit in fast memory. Knowing the maths up front stops surprises in the hosting bill.
- 2
The storage formula. One vector takes
dimensions * bytes per number. The usual number type is float32, 4 bytes per number. A million vectors of 384 dimensions take1,000,000 * 384 * 4 = 1.536 billion bytes, about 1.54 GB. At 1536 dimensions it is about 6.14 GB, and at 3072 about 12.29 GB (the example prints the whole table). This is only the raw vectors: an index adds more. - 3
What more dimensions buy. More dimensions give the model more room to separate fine differences, so larger models usually retrieve better. The gain is not linear, and the cost is: double the dimensions, double the memory and roughly double the time to compare. Published sizes for some models: OpenAI's
text-embedding-3-smallhas 1536 dimensions by default andtext-embedding-3-largehas 3072 (from OpenAI's embeddings guide). The model in this course's lab has 384. - 4
Shorter vectors. Some models are trained so that the *first* numbers carry the most meaning, so you may keep only the first few hundred. OpenAI's text-embedding-3 models accept a
dimensionsparameter for this, and sentence-transformers has atruncate_dimoption. After cutting a vector yourself you must normalise it again (Topic 2.3), because it is no longer length 1. Only do this with models that document it. The example cuts a toy vector from 8 numbers to 4 and 2: the toy's cosine moves from 0.9217 to 0.9390 and 0.9374, a reminder that a model not trained for it can change scores unpredictably. - 5
Smaller number types. A float32 can be stored as an 8-bit integer (int8) with a scale, which is 4 times smaller, or as a single bit per dimension (binary), 32 times smaller. The example squeezes a unit vector into integers between -127 and 127: A becomes
78 63 47 47 31 16 16 16, and the cosine moves from 0.9217 to 0.9219, almost nothing. Binary is cruder. Sentence-transformers documents both as 'embedding quantization'. The usual practice is to search with the small vectors and re-check the best few with the full ones. - 6
The rule of thumb. Do the arithmetic for the corpus you really have, include the index overhead, and then decide. Under a million vectors, ordinary hardware is fine in float32. In the tens of millions, shorter dimensions, smaller types or disk-based indexes start to matter.
Explain it without notes
How do you compute the memory a vector index needs, and what does it leave out?
What does a larger number of dimensions buy, and what does it cost?
When can you safely keep only the first N numbers of an embedding?
What is int8 or binary quantization, and how are they used safely in search?
Practice
Write gigabytes(n_vectors, dims, bytes_per_number=4) and print the size for 250,000 vectors at 768 dimensions in float32 and int8, to two decimals.
Write to_int8(v) that scales a vector to unit length and returns floor(x * 127 + 0.5) for each number, and print the result for [3, 4].
Trade-offs
- ↔
Higher dimensions: finer meaning and usually better recall, at proportional cost in memory and speed.
- ↔
int8 and binary: big savings, with a small to moderate loss that you must measure and often repair with a re-score.
- ↔
Shortened vectors: cheaper and faster, only safe for models trained for it.
Done when you can
Done when you can compute the raw size of an index for any corpus and dimension count.
Done when you can name what the raw size leaves out.
Done when you know which models in your stack support shorter vectors, and have read their documentation.
Done when you can describe the search-small-then-re-score pattern.