Command Palette

Search for a command to run...

Hectal
PHASE 0Beginner ~19 min· topic 3 of 7

Topic 0.3

When to Use an LLM (and When Not To)

In one line

Use plain code for anything exact (sums, dates, lookups, rules) and an LLM for fuzzy language work. The strongest designs combine both: code handles what it can, the model handles the rest, and code checks the model.

0/7 · 0%

Think of it like this

A shop has a calculator and a very well-read assistant. You'd never ask the assistant to add up the bill when the calculator is right there, and you'd never ask the calculator to write a friendly thank-you note. Good engineering gives each job to the right tool, and has the calculator double-check any number the assistant writes.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Deterministic
Gives the same output every time for the same input. Ordinary code is deterministic; LLM output usually is not.
Probabilistic
Based on likelihoods, so the output can vary and can be wrong.
Fallback
A second way to handle a case when the first way can't.
Validation
Checking in code that a value is allowed (a real date, a total that matches).
Long tail
The many rare, odd cases that don't fit the common patterns.

Step by step

01Step 1: list the jobs in the feature

Take a feature like "answer a customer's order question". Break it into jobs: understand the question (language: model), look up the order (exact: code), compute the refund (exact: code), write the reply (language: model). Most features are a mix.

02Step 2: let the model write, let code compute

In the first example the model writes a friendly reply that includes the total. Code computes the true total from the order lines and compares. One reply matches; the other is off by 10, exactly the kind of mistake a model makes when it does arithmetic in text. The check costs one line and prevents sending a wrong bill.

03Step 3: code first, model for the rest

The second example handles dates. Two formats are parsed by code. next friday isn't a format code knows, so it goes to the model, and the model's answer is checked by the same code. 2026-13-40 is rejected by code: month 13 doesn't exist.

Step 3: code first, model for the restdiagram
Rendering diagram…

04Step 4: see why the model must not decide alone

Asked about 2026-13-40, the fake model (like many real ones) returns a plausible date instead of saying the input is wrong. Models are trained to be helpful, so they tend to produce *an* answer. When "this input is invalid" is the right answer, code is better at saying so.

05Step 5: count the calls

Only one of the four inputs reached the model. In production, log this ratio. If most inputs need the model, fine; if code could handle them with one more rule, add the rule and save the cost and latency on every future request.

Try it yourself

  1. 1

    Catch a smaller mistake

    Change the second fake reply to Rs 1,159.01. Is it still caught? What tolerance would you use for a currency with paise, and why not compare floats with ==?

  2. 2

    Add a format to code

    Add a third format YYYY/MM/DD (/ with order ymd) to parse_with_code. Which order of formats is safe, given that 10/10/2026 and 2026/10/10 both use /? Add 2026/10/12 to the inputs and run.

  3. 3

    Make the model the last resort

    Add the input tomorrow and give the fake model an answer for it. Run, and check the model calls line.

Code & diagrams

Code checks the model's maths New tab

Two scripted replies from a model. Code computes the real total and catches the wrong one.

Sign in to run this example in your browser.

Expected output

Total computed by code: 1159.00
Model says 1159.00 -> OK
Model says 1169.00 -> MISMATCH, do not send
Code first, the model only for what code can't parse New tab

The fake model stands in for a real call. Notice what it says about the impossible date.

Sign in to run this example in your browser.

Expected output

2026-10-10   -> 2026-10-10 (code)
10/10/2026   -> 2026-10-10 (code)
next friday  -> 2026-10-16 (model, checked by code: ok)
2026-13-40   -> rejected (code: not a real date)
model calls: 1 of 4
Asked about 2026-13-40, the model says 2026-12-31: it invents a fix instead of saying no.

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Trust the model with the total

In the first example, replace the comparison with verdict = "OK" (so nothing is checked) and run.

terminal
$ Run
── what you'll see ──
Model says 1169.00 -> OK

Myth vs fact

Myth

Modern models are good at maths now, so checks aren't needed.

Fact

They are much better than before, but still not guaranteed, and a guarantee is what money needs. When code can compute it, compute it.

Myth

Using an LLM for everything makes a product smarter.

Fact

It makes it slower, costlier and less predictable where an exact rule existed. Smart products use models where language understanding is the real problem.

Myth

If the model can't parse something, it will say so.

Fact

Models lean towards giving an answer. Design the "invalid input" path in code.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

  • ▸

    Prefer having the model produce structure (fields, labels, IDs) and letting code produce facts (totals, dates, prices) and final text from templates. It shrinks the surface where a hallucination can reach the user.

  • ▸

    Measure the code/model split per input type in production logs. A new rule that removes 30% of model calls is often the cheapest latency and cost win available.

  • ▸

    For exactness the model can't give, give it a tool: a calculator, a database lookup, a date parser (Phase 7). The model decides *what* to compute; code computes it.

Remember this

  1. 1

    Why this decision comes first. An LLM call is slow (often a second or more), costs money on every call, can be rate-limited, and gives answers that vary and are sometimes wrong. Ordinary code is instant, free to run, exact and the same every time. Using a model where code would do makes a feature slower, costlier and less reliable at the same time.

  2. 2

    The "could plain code do it?" test. Before reaching for a model, ask: is there an exact rule? Adding prices, checking a date is valid, finding a customer by ID, sorting, converting units: code. Is the input open-ended human language? Understanding a complaint, summarising a document, pulling an address out of a messy email, answering a question in words: a model fits.

  3. 3

    What LLMs are good at: classifying text ("is this a refund request?"), extracting fields from messy text, summarising, rewriting for a different reader, translating, drafting, answering questions from documents you provide, explaining and writing code, and turning a request into a structured call to your own functions (tools, Phase 7).

  4. 4

    What they are bad at: exact arithmetic on long numbers, counting characters, keeping a promise that something is *always* true, facts they never saw or that changed after training, and anything where a wrong answer is dangerous and undetectable. The model's answer is a likely guess, so it must not be the only safeguard for money, health or security.

  5. 5

    Combine them. The best pattern is a pipeline: code first for the cases it handles exactly, the model only for what is left, and code checks the model's answer wherever a check is possible. The second example parses dates with code, sends only next friday to the model, checks the model's answer with the same code, and shows that the model would happily "fix" an impossible date instead of rejecting it.

  6. 6

    Cost and speed follow from this. Every case code handles is a model call you don't pay for. In real systems the split is often lopsided: 80–95% of inputs follow a few known formats. Routing those to code cuts cost and latency dramatically and leaves the model for the long tail (Topic 12.6 does the same with cheap and expensive models).

Explain it without notes

01

Give two jobs you would never give an LLM and two you would, and explain the difference.

02

Why is "code first, model as fallback, code checks the model" a good default pipeline?

03

Why might a model return a date for an impossible input like 2026-13-40?

Practice

01

Write is_valid_iso_date(text) that returns True only for YYYY-MM-DD with a month 1–12 and a day that exists in that month (ignore leap years: February has 28 days).

02

Write route(text) that returns "code" if the text is a plain number like 42 or 3.5, and "model" otherwise.

Trade-offs

  • ↔

    More code rules: cheaper and more predictable, but each rule is code to maintain.

  • ↔

    More model: handles messy inputs with no new code, but costs per call and needs checks.

Done when you can

  • Done when you can apply the "could plain code do it?" test to a feature.

  • Done when you can list what LLMs are good and bad at.

  • Done when you can design a code-first pipeline that sends only the rest to a model.

  • Done when you check any exact value a model produces.