Will it fit? Weights plus attention cache
The architecture numbers are an example, not a real model. Use the values from your model's config file.
llama.cpp, Ollama and vLLM run open-weights models on your own machines, and each exposes an OpenAI-compatible chat endpoint, so your client changes only its base URL, model name and key. Memory is the first thing to calculate.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_number) and print the result for 32 layers, 8 key-value heads, head size 128 and 2 bytes.
Write max_users(gpu_gb, weights_gb, buffers_gb, context, per_token_bytes) that returns how many whole conversations fit, never below 0, and print it for a 24 GB card, 8 GB of weights, 1 GB of buffers, 8,000 tokens and 131,072 bytes per token.
Explain it without notes
Why can a model whose weights fit in GPU memory still fail to run well?
What does it mean that Ollama, llama-server and vLLM expose OpenAI-compatible endpoints, and what must you still test?
How do you calculate the attention-cache memory for a conversation?
Expected output
cache per token: 131072 bytes
16-bit, 1 users: weights 16.0 + cache 1.0 + buffers 1.0 = 18.0 GB -> fits
16-bit, 8 users: weights 16.0 + cache 8.4 + buffers 1.0 = 25.4 GB -> does not fit
16-bit, 32 users: weights 16.0 + cache 33.6 + buffers 1.0 = 50.6 GB -> does not fit
8-bit, 1 users: weights 8.0 + cache 1.0 + buffers 1.0 = 10.0 GB -> fits
8-bit, 8 users: weights 8.0 + cache 8.4 + buffers 1.0 = 17.4 GB -> fits
8-bit, 32 users: weights 8.0 + cache 33.6 + buffers 1.0 = 42.6 GB -> does not fit
4-bit, 1 users: weights 4.0 + cache 1.0 + buffers 1.0 = 6.0 GB -> fits
4-bit, 8 users: weights 4.0 + cache 8.4 + buffers 1.0 = 13.4 GB -> fits
4-bit, 32 users: weights 4.0 + cache 33.6 + buffers 1.0 = 38.6 GB -> does not fit