Command Palette

Search for a command to run...

Hectal
Phase 13Advanced14 of 15 in LLM Engineering

Choosing and Adapting Models

Pick a model with evidence, not hype: scorecards, open vs hosted, quantisation, local servers, fine-tuning, distillation and honest reading of benchmarks.

Until now the model was a given. In a real project it is a decision, and a decision you will make again, because new models appear every few months and prices, limits and licences change. This phase teaches how to make that decision with your own data instead of a leaderboard or a friend's opinion: which requirements are hard gates, which are trade-offs, how to compare candidates fairly, and how to know when the answer has expired.

You will do the maths by hand with made-up, clearly labelled numbers: a weighted scorecard, the break-even point between a hosted API and your own GPUs, the memory a model needs at 16, 8 and 4 bits per weight, int8 quantisation with its rounding error printed, how big a LoRA adapter is compared with the full weights, a distillation data set built from a fake teacher, and an n-gram check for benchmark contamination. No model names, prices or scores are invented: the numbers are variables you replace with your own quotes and eval results.

The phase also covers the practical side: running a model on your own machine with llama.cpp, Ollama or vLLM (shown as verified commands, not run in the browser), why an OpenAI-compatible local server means your client code barely changes, when fine-tuning helps and when retrieval (Phase 8) is the right tool, and how to read a benchmark table critically. It ends with a playbook of selection recipes you can copy.

0/8 · 0%
8 topics ~3 h 22 code blocks & diagrams
Start with the first topic
1
13.1

Choosing a Model

There is no best model, only the best model for your task, budget, speed, rules and data. Filter by hard requirements first, then test the shortlist on your own evals and compare with a weighted scorecard.

25 min 4 code practice

2
13.2

Open-Weights vs Hosted Models

A hosted model is rented by the token and run by someone else. An open-weights model can be downloaded and run on hardware you control, which buys control and privacy but adds fixed costs and operations work. The break-even point decides when it pays.

21 min 1 code practice

3
13.3

Quantisation

Quantisation stores each weight in fewer bits, so a model needs less memory and often runs faster, at the price of a small rounding error. The memory is just parameters times bits divided by 8.

24 min 2 code practice

4
13.4

Running Models Locally

llama.cpp, Ollama and vLLM run open-weights models on your own machines, and each exposes an OpenAI-compatible chat endpoint, so your client changes only its base URL, model name and key. Memory is the first thing to calculate.

23 min 6 code practice

5
13.5

When Fine-Tuning Helps (and When It Doesn't)

Fine-tuning continues a model's training on your own examples. It is good at teaching a format, a style or a narrow task, and poor at adding facts, which belong in retrieval. Try prompts and retrieval first.

24 min 2 code practice

6
13.6

Distillation and Small Models

Distillation teaches a small model from the outputs of a large one. The large model labels many of your real inputs, you keep the good labels, and the small model is trained on them. It is cheap to run afterwards, but it inherits the teacher's mistakes and the terms of use apply.

22 min 1 code practice

7
13.7

Reading Benchmarks Critically

A benchmark is a fixed test with a scoring rule, summarised as one number. The number can be inflated by test questions that leaked into training, hidden by noise on small tests, and irrelevant to your task, so use it to build a shortlist and your own evals to decide.

25 min 2 code practice

8
13.8

Playbook: Model Selection Scorecard

Five recipes for choosing and changing models with evidence: gates, the cheapest model that clears your quality bar, a weighted scorecard, a swap guard that compares models case by case, and configuration that makes switching a one-line change.

23 min 4 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

A weighted scorecard

Write total_score(candidate, weights) where candidate is a dict with the keys quality, cost_score and speed_score (all already on a 0 to 100 scale, higher is better) and weights is a tuple of three numbers in that order. It returns the sum of weight times score.

Intermediate
2

Will the weights fit on the card?

Write weights_gb(params_billion, bits) returning the gigabytes needed for the weights: billions of parameters times bits divided by 8.

Intermediate
3

Quantise to int8 and back

Write round_half_away(x) that rounds to the nearest whole number with exact halves going away from zero (2.5 gives 3, -2.5 gives -3). Python's built-in round rounds halves to even, so do not use it.

Advanced
4

Break-even against a hosted API

Write break_even(fixed_per_month, in_tokens, out_tokens, price_in, price_out) where prices are dollars per million tokens. It returns the smallest whole number of requests per month at which the hosted cost reaches fixed_per_month (use math.ceil).

Advanced
5

How big is a LoRA adapter?

Write lora_params(d_in, d_out, r) returning the number of trained numbers for one matrix: two thin matrices of sizes d_in x r and r x d_out.

Advanced
6

Is the test set in the training text?

Write ngrams(text, n) returning the set of n-word runs of a text: lower-case it, turn every character that is not a letter or digit into a space, split into words, and join each run of n consecutive words with a single space. A text with fewer than n words gives an empty set.

Advanced