PHASE 13Advanced ~23 min· topic 4 of 8

Topic 13.4

Running Models Locally

In one line

llama.cpp, Ollama and vLLM run open-weights models on your own machines, and each exposes an OpenAI-compatible chat endpoint, so your client changes only its base URL, model name and key. Memory is the first thing to calculate.

Think of it like this

Cooking at home instead of ordering delivery. At home you control the ingredients, nobody sees your order, and it works with no internet. But you need a kitchen with enough room for the pots, you must buy the equipment, and a big dinner party strains it. Running a model locally has the same trade: control and privacy against hardware limits and your own upkeep.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

llama.cpp: An engine that runs models from .gguf files on ordinary computers and GPUs, with a server program called llama-server.
Ollama: A local service that downloads models by name and serves them with a simple command line.
vLLM: A serving engine for GPU servers, designed to handle many requests at once.
VRAM: The memory on a graphics card. The model weights and the attention cache must fit in it for fast generation.
Attention cache (KV cache): Stored keys and values for every token so far, so the model does not recompute them for each new token.
Base URL: The start of every API address, such as http://localhost:11434/v1.
OpenAI-compatible: Accepts the same request and gives the same response shape as the chat-completions API.

Step by step

01Step 1: will it fit?

Before downloading anything, compute weights plus cache plus buffers. The first example does it for an example architecture on a 24 GB card with 1, 8 and 32 conversations of 8,000 tokens. At 16 bits, one user fits and eight do not. At 8 bits, eight fit. At 4 bits, 32 users do not fit because the cache alone is about 33.6 GB.

kv_cache.pywhole filepython

# bytes per token = 2 (key and value) x layers x kv_heads x head_dim x bytes per number
per_token = 2 * 32 * 8 * 128 * 2
print(per_token)                      # 131072 bytes
print(8 * 8000 * per_token / 1e9)     # 8 users x 8000 tokens = 8.388608 GB

02Step 2: choose a tool for the job

To try a model on a laptop, Ollama is the shortest path. To control every setting of one model file, use llama.cpp. To serve many users on GPU servers, look at vLLM. All three speak the same chat format, so you can change your mind later without rewriting your client.

Step 2: choose a tool for the jobdiagram

Rendering diagram…

03Step 3: start a server

The code blocks below show the documented commands. Ollama: ollama pull then ollama serve. llama.cpp: llama-server -m your-model.gguf with context size -c, GPU layers -ngl and a port. vLLM: vllm serve with a model identifier, which listens on port 8000 unless you change it. Names in angle brackets are placeholders for the model you chose.

04Step 4: call it like any other chat API

Send the same JSON you already know: a model and a messages list, to <base URL>/chat/completions. The second example uses a fake server to show the two requests side by side: same body shape, same reply parsing, different base URL, model name and key.

05Step 5: switch with configuration

Read the base URL, model name and key from environment variables (Topic 12.7), so the same code runs against a hosted API in production and a local server on a developer's machine. The Python block at the end shows the standard OpenAI SDK pointed at a local address.

06Step 6: evaluate the local model like any candidate

A local model is a candidate in Topic 13.1's scorecard: same eval set, same prompts. Add the cost of the hardware (Topic 13.2) and the time to run it. Do not assume a model that feels fine in a chat window passes your cases.

Try it yourself

1
Find the user limit
In the first example set CONTEXT = 32000. How many conversations fit at 8 bits on the 24 GB card now? Change the users list to find the exact limit.
2
Halve the cache
Set KV_HEADS = 4, which is what fewer key-value heads would give. Predict the cache per token before running. What does this say about why model designers reduce the number of key-value heads?
3
Add a third server
In the second example add a call to http://localhost:8080/v1 with the model name your-model. Which lines of the output changed, and which did not?

Code & diagrams

Will it fit? Weights plus attention cache New tab

The architecture numbers are an example, not a real model. Use the values from your model's config file.

Expected output

cache per token: 131072 bytes
16-bit, 1 users: weights 16.0 + cache 1.0 + buffers 1.0 = 18.0 GB -> fits
16-bit, 8 users: weights 16.0 + cache 8.4 + buffers 1.0 = 25.4 GB -> does not fit
16-bit, 32 users: weights 16.0 + cache 33.6 + buffers 1.0 = 50.6 GB -> does not fit
8-bit, 1 users: weights 8.0 + cache 1.0 + buffers 1.0 = 10.0 GB -> fits
8-bit, 8 users: weights 8.0 + cache 8.4 + buffers 1.0 = 17.4 GB -> fits
8-bit, 32 users: weights 8.0 + cache 33.6 + buffers 1.0 = 42.6 GB -> does not fit
4-bit, 1 users: weights 4.0 + cache 1.0 + buffers 1.0 = 6.0 GB -> fits
4-bit, 8 users: weights 4.0 + cache 8.4 + buffers 1.0 = 13.4 GB -> fits
4-bit, 32 users: weights 4.0 + cache 33.6 + buffers 1.0 = 38.6 GB -> does not fit

One client, two servers New tab

A fake server stands in for the network. The addresses use the reserved example.test domain and the local Ollama address from its docs.

Expected output

POST https://api.example.test/v1/chat/completions
  headers: Authorization, Content-Type
  body: {"model":"hosted-model-name","messages":[{"role":"user","content":"Say hi"}]}
  reply: hello from the hosted model
POST http://localhost:11434/v1/chat/completions
  headers: Authorization, Content-Type
  body: {"model":"local-model-name","messages":[{"role":"user","content":"Say hi"}]}
  reply: hello from the local model
only the base URL, the model name and the key changed

Ollama: pull a model, serve it, call itbash

Commands from the Ollama docs. Replace <model-name> with a model from its library. If the Ollama service already runs on your computer, you do not need ollama serve. Not run here.

ollama pull <model-name>
ollama serve

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "Hello"}]}'

llama.cpp: llama-serverbash

Flags from the llama.cpp server README: -m model file, -c context size, -ngl layers on the GPU, --port, --alias the name the server reports for the model (by default it is the path of the -m file). The server listens on 127.0.0.1 unless you pass --host. The default port differs between versions, so the command sets it explicitly. Not run here.

./llama-server -m models/your-model.gguf --alias your-model -c 4096 -ngl 99 --port 8080

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "your-model", "messages": [{"role": "user", "content": "Hello"}]}'

vLLM: vllm servebash

From the vLLM docs: the server implements /v1/chat/completions for models that have a chat template. By default it listens on http://localhost:8000, so the base URL is http://localhost:8000/v1 and the model name in requests is the identifier you passed to vllm serve. Add --api-key (or the VLLM_API_KEY environment variable) to require a key. Not run here.

vllm serve <model-id>

The OpenAI Python SDK pointed at a local serverpython

Not run here. The base URL is Ollama's from its docs; the key is required by the SDK but ignored by a local server.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed-locally")
reply = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Say hi"}],
)
print(reply.choices[0].message.content)

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Forget the attention cache

In the first example set CONTEXT = 0, as if the cache did not exist.

terminal

$ Run

── what you'll see ──

cache per token: 131072 bytes

16-bit, 1 users: weights 16.0 + cache 0.0 + buffers 1.0 = 17.0 GB -> fits

16-bit, 8 users: weights 16.0 + cache 0.0 + buffers 1.0 = 17.0 GB -> fits

16-bit, 32 users: weights 16.0 + cache 0.0 + buffers 1.0 = 17.0 GB -> fits

8-bit, 1 users: weights 8.0 + cache 0.0 + buffers 1.0 = 9.0 GB -> fits

8-bit, 8 users: weights 8.0 + cache 0.0 + buffers 1.0 = 9.0 GB -> fits

8-bit, 32 users: weights 8.0 + cache 0.0 + buffers 1.0 = 9.0 GB -> fits

4-bit, 1 users: weights 4.0 + cache 0.0 + buffers 1.0 = 5.0 GB -> fits

4-bit, 8 users: weights 4.0 + cache 0.0 + buffers 1.0 = 5.0 GB -> fits

4-bit, 32 users: weights 4.0 + cache 0.0 + buffers 1.0 = 5.0 GB -> fits

Myth vs fact

Myth

If the weights fit, the model will run.

Fact

The attention cache and buffers need memory too, and the cache grows with every conversation and token.

Myth

A local server is automatically private and secure.

Fact

Private from the provider, yes. But a server opened to a network without authentication is open to everyone on it. Add access control, TLS and logging.

Myth

OpenAI-compatible means identical.

Fact

The basic chat shape matches, but features such as tool calls, structured output, token usage fields and streaming details can differ. Test each one you use.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Serving engines gain throughput by batching many conversations into the same step; that is why a server built for many users can beat a single-user tool by a wide margin on the same card. Measure requests per second on your own traffic (Topic 13.2).
▸
Context length is a memory decision: the cache grows linearly with tokens and with concurrent users. Limiting maximum context per request is often the cheapest way to fit more users.
▸
Pin the exact model file or identifier and the serving software version in your deployment, and record both next to your eval results (Topic 11.8), so a result can be reproduced.
▸
Run the same eval harness against local and hosted candidates through one interface (Topic 2.8). The only difference should be configuration.

Remember this

1
Why this exists. Reasons to run locally are privacy (prompts never leave your machine), no per-token bill while developing, no rate limits, working offline, the freedom to test open-weights models and fine-tuned variants, and the option to serve at scale on your own servers (Topic 13.2). Reasons not to are hardware cost, slower or weaker models for the same memory, and the operations work.
2
Three tools, three jobs. llama.cpp is an engine written in C and C++ that runs models stored in .gguf files on ordinary computers and on GPUs, and includes a server program called llama-server. Ollama is a local service that downloads models by name and serves them, the easiest way to start. vLLM is a serving engine for GPU servers, built to handle many requests at the same time efficiently. This topic shows commands for each from their documentation and does not run them in the browser.
3
OpenAI-compatible servers. Their documentation says that Ollama (base URL http://localhost:11434/v1), llama-server (POST /v1/chat/completions) and vLLM (/v1/chat/completions) each offer the same request and response shape as the chat-completions API you used in Phase 2. So the client you wrote (Topic 2.8) works unchanged except for three values: the base URL, the model name and the key (local servers usually ignore the key). The example runs one client against a hosted-looking and a local-looking address to show exactly that. Ollama's own docs call its API a subset, so test any feature you rely on, such as tool calls, JSON mode or usage counts.
4
Memory first, then speed. The weights must fit in the graphics card's memory (VRAM), with room for the attention cache. If they do not, the tool may place some layers in ordinary memory and run on the processor, and generation becomes much slower. The cache holds a key and a value for every token of every conversation: bytes per token = 2 x layers x key-value heads x head size x bytes per number. The first example computes it for an example architecture and shows how 32 users at 8,000 tokens each can need more memory than the model itself.
5
Pick the quantisation to fit. Topic 13.3's maths tells you which bit width fits your card; the example table shows that the same 8 billion parameter model needs 16 GB of weights at 16 bits and only 4 GB at 4 bits. Always run your eval set on the exact file you will run, since quantised variants behave differently from the original.
6
A local server is not a production setup. Bind it to your own machine by default; if you open it to a network, put authentication, TLS, rate limits and monitoring in front of it (Phases 10 to 12), because a bare model server has none of them for you. Treat prompts and outputs on it with the same care as on any service.

Explain it without notes

Why can a model whose weights fit in GPU memory still fail to run well?

What does it mean that Ollama, llama-server and vLLM expose OpenAI-compatible endpoints, and what must you still test?

How do you calculate the attention-cache memory for a conversation?

Practice

Write kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_number) and print the result for 32 layers, 8 key-value heads, head size 128 and 2 bytes.

Write max_users(gpu_gb, weights_gb, buffers_gb, context, per_token_bytes) that returns how many whole conversations fit, never below 0, and print it for a 24 GB card, 8 GB of weights, 1 GB of buffers, 8,000 tokens and 131,072 bytes per token.

Trade-offs

↔
Ollama is the quickest to start but gives you less control than the lower-level tools.
↔
llama.cpp runs on modest hardware with fine control; vLLM targets high throughput on GPU servers; neither is the right answer for every workload.
↔
A larger context and more concurrent users both multiply cache memory, so you trade capacity against how much each user can send.

Done when you can

Done when you can compute weights, cache and buffers and decide if a model fits.
Done when you can name what llama.cpp, Ollama and vLLM are for.
Done when you can call a local server with the same client as a hosted one.
Done when you can list the features to test before relying on compatibility.
Done when you can explain why a local server needs access control before it is shared.

Command Palette

Words you'll meet

Step by step

01Step 1: will it fit?

02Step 2: choose a tool for the job

03Step 3: start a server

04Step 4: call it like any other chat API

05Step 5: switch with configuration

06Step 6: evaluate the local model like any candidate

Try it yourself

Find the user limit

Halve the cache

Add a third server

Code & diagrams

Break it on purpose

Forget the attention cache

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can