Topic 13.2
Open-Weights vs Hosted Models
In one line
A hosted model is rented by the token and run by someone else. An open-weights model can be downloaded and run on hardware you control, which buys control and privacy but adds fixed costs and operations work. The break-even point decides when it pays.
Think of it like this
Taking taxis versus owning a car. A taxi charges per trip, needs no garage and never needs servicing, and it is cheapest if you ride a little. A car has a big fixed cost (purchase, insurance, parking, repairs) and a small cost per trip, so it wins only if you drive enough, and then you also have to fix it when it breaks.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Hosted model
- A model run by a provider and rented by the token through an API.
- Open weights
- A model whose trained numbers can be downloaded and run by you, under its licence.
- Fixed cost
- A cost you pay whether you use it or not, like GPUs by the hour.
- Variable cost
- A cost that grows with use, like a price per token.
- Break-even
- The traffic level at which two options cost the same.
- Utilisation
- The share of time your GPUs are busy. Low utilisation raises the real price per request.
- Operations work
- Deploying, monitoring, scaling, upgrading and fixing the serving system.
Step by step
01Step 1: price a hosted request
A request has input tokens and output tokens, each with its own price per million. The example uses 800 input and 200 output tokens and made-up prices, so one million requests cost 700 dollars. The prices are variables: put in your provider's current numbers.
IN_PER_MTOK, OUT_PER_MTOK = 0.50, 1.50 # example prices, check your provider
per_million_requests = 800 * IN_PER_MTOK + 200 * OUT_PER_MTOK
print(per_million_requests) # 700.0 dollars per million requests02Step 2: price the fixed side
Two rented GPUs for a month of 730 hours at the example hourly price, plus a monthly amount for the engineering time to run them. Do not leave the second part at zero: someone gets paged when the server falls over.
03Step 3: find the crossing point
Divide the fixed cost by the hosted price per request. Below that many requests per month, paying per token is cheaper. Above it, the fixed cost has been spread thin enough to win. The example prints a table so you see the switch happen between 6 and 9 million requests.
04Step 4: check the GPUs can serve that traffic
Break-even at 6.3 million requests means nothing if your two GPUs can only serve 4 requests per second, about 10.5 million per month at 100% busy. The break-even sits at 60% of capacity, and real traffic has peaks, so you would need headroom, which means more GPUs and a higher fixed cost. Always print the break-even as a share of capacity.
05Step 5: add the things the calculator leaves out
Time to build and test the serving setup, evals on the open model (it may need more prompt work or fine-tuning to match), security patching, and the cost of being wrong. These make self-hosting a project, not a switch. Hosted prices also fall over time, which moves the break-even upwards.
06Step 6: decide with the non-cost reasons too
Some decisions are not about money: a rule that data never leaves your network, a need to pin an exact model version for a regulator, or a need to fine-tune. Those can be gates (Topic 13.1) that settle the question before the calculation starts.
Try it yourself
- 1
Halve the hosted price
Change both hosted prices to half. Predict whether the break-even goes up or down and by how much, then run. What does this say about how often to re-run the calculation?
- 2
Add a third GPU
Set
GPUS = 3andCAPACITY_PER_SEC = 6. Is the new break-even a smaller or larger share of capacity? When does adding capacity stop making sense? - 3
Shorter answers
Change
OUT_TOKENSto 50. How does the hosted price per million requests change, and why do output tokens usually dominate the bill?
Code & diagrams
Every price and capacity number is a made-up variable. Replace them with real quotes and your measured throughput.
Expected output
self-hosted fixed cost: 4420.00 dollars per month
hosted cost per million requests: 700.00 dollars
1000000 requests: hosted 700.00, self-hosted 4420.00 -> hosted is cheaper
3000000 requests: hosted 2100.00, self-hosted 4420.00 -> hosted is cheaper
6000000 requests: hosted 4200.00, self-hosted 4420.00 -> hosted is cheaper
9000000 requests: hosted 6300.00, self-hosted 4420.00 -> self-hosted is cheaper
break-even: 6314286 requests per month
that is 60.1 percent of the 10512000 requests the GPUs could serveBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Count operations as free
Set OPS_PER_MONTH = 0.00.
Myth vs fact
Myth
Open-weights models are free.
Fact
The download is free, the hardware, the electricity or rent, and the people are not. Some licences also restrict commercial use.
Myth
Self-hosting is always cheaper at scale.
Fact
Only above the break-even and only if the GPUs are well utilised. Hosted prices also fall, and the calculation must include operations work.
Myth
A hosted model cannot be private.
Fact
Providers publish data policies and some offer private or regional deployments. Whether that is enough is a question for your contract and your rules.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Real GPU throughput depends on prompt length, output length, batch size and the serving software, so measure requests per second on your own traffic before trusting any capacity number. Peak load, not average load, sets how many GPUs you need.
- ▸
Utilisation is the hidden multiplier. At 30% average utilisation the real price per request is more than three times the price at 100%. Autoscaling and scheduling batch jobs into quiet hours both raise it.
- ▸
Opening the model also opens version control: you can pin an exact build, replay old evals and roll back. With a hosted model, "the same model" may change underneath you, so keep recorded responses and eval history (Topic 9.6).
- ▸
The licence is part of the architecture. Check what it says about commercial use, about using outputs to train other models (Topic 13.6), and about redistribution, for the exact model and version you deploy.
Remember this
- 1
Why this exists. Teams hear "open models are free" or "APIs are too expensive" and decide by feeling. The truth is two cost shapes. A hosted model has no fixed cost and a price per token, so the bill grows with traffic. An open-weights model you run yourself has a mostly fixed cost (GPUs by the hour plus people) that is the same whether you serve one request or a million. The right choice depends on where your traffic sits relative to the crossing point.
- 2
What "open weights" means. The trained numbers (the weights, Topic 0.1) can be downloaded and run on your own machines. It does not automatically mean the training data or code is open, and it does not mean you can do anything you like: each model comes with a licence, which may limit commercial use, set conditions on size of company, or require a notice. Read the actual licence of the exact model, and never assume. Hosted models come with terms of service and a data policy instead.
- 3
Control, privacy and customisation. Running the model yourself means prompts never leave your network, you choose when the model changes (a hosted model can be updated or retired under you, Topic 3.8), you can fine-tune it (Topic 13.5) and you can pin an exact version for reproducible evals. Hosted providers offer privacy terms too, and some offer private deployments, so "privacy" is a question for the contract as well as the architecture.
- 4
What self-hosting really costs. GPUs rented or bought, to serve at your peak load not your average, plus the work: deploying, scaling, monitoring, upgrading the serving software, handling outages and security patches (Topic 13.4 covers the tools). Count this as engineer time and on-call, not as zero. Idle GPUs cost the same as busy ones, so a quiet night is waste, and utilisation (how busy the GPUs are on average) decides the real price per request.
- 5
The break-even calculation. Hosted cost per month = requests x (input tokens x input price + output tokens x output price) / 1,000,000. Self-hosted cost per month = GPU price x hours x number of GPUs + operations cost. Break-even requests per month = fixed cost / hosted cost per request. Below it, hosting is cheaper. Above it, running your own wins on paper, but only if the GPUs can actually serve that many requests (the example prints this as a share of capacity).
- 6
Mixed strategies are normal. Use a hosted model for hard, rare or spiky tasks, and a small open model on your own hardware for the high-volume, narrow ones (Topic 12.6 routes between them). Start hosted to learn what you need, and move a workload only when the measured traffic, the privacy rules or the licence terms make the numbers clear.
Explain it without notes
When does self-hosting an open-weights model beat a hosted API?
What does "open weights" mean, and what does it not mean?
What hidden costs does a self-hosting calculation usually miss?
Practice
Write hosted_cost(requests, in_tokens, out_tokens, price_in, price_out) with prices per million tokens, and print the cost of 2,000,000 requests of 500 input and 100 output tokens at prices 0.50 and 1.50.
Write utilisation_percent(requests_per_month, capacity_per_month) rounded to one decimal, and print it for 3,000,000 requests against 10,512,000 capacity.
Trade-offs
- ↔
Hosted: no fixed cost, no operations, instant access to the strongest models; you pay per token, depend on the provider and accept its data terms.
- ↔
Self-hosted: control, privacy and fixed cost at scale; you take on operations, capacity planning and usually a quality gap you must close with prompting or fine-tuning.
- ↔
Mixed: the best of both when the workload splits cleanly, at the price of two systems to run and evaluate.
Done when you can
Done when you can explain open weights and why the licence still matters.
Done when you can compute the hosted cost per request from token counts and prices.
Done when you can find the break-even and print it as a share of GPU capacity.
Done when you can list the operations costs a calculation must include.
Done when you can name non-cost reasons that decide the question.