One client, two servers
A fake server stands in for the network. The addresses use the reserved example.test domain and the local Ollama address from its docs.
llama.cpp, Ollama and vLLM run open-weights models on your own machines, and each exposes an OpenAI-compatible chat endpoint, so your client changes only its base URL, model name and key. Memory is the first thing to calculate.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_number) and print the result for 32 layers, 8 key-value heads, head size 128 and 2 bytes.
Write max_users(gpu_gb, weights_gb, buffers_gb, context, per_token_bytes) that returns how many whole conversations fit, never below 0, and print it for a 24 GB card, 8 GB of weights, 1 GB of buffers, 8,000 tokens and 131,072 bytes per token.
Explain it without notes
Why can a model whose weights fit in GPU memory still fail to run well?
What does it mean that Ollama, llama-server and vLLM expose OpenAI-compatible endpoints, and what must you still test?
How do you calculate the attention-cache memory for a conversation?
Expected output
POST https://api.example.test/v1/chat/completions
headers: Authorization, Content-Type
body: {"model":"hosted-model-name","messages":[{"role":"user","content":"Say hi"}]}
reply: hello from the hosted model
POST http://localhost:11434/v1/chat/completions
headers: Authorization, Content-Type
body: {"model":"local-model-name","messages":[{"role":"user","content":"Say hi"}]}
reply: hello from the local model
only the base URL, the model name and the key changed