PHASE 0Beginner ~18 min· topic 5 of 7

Topic 0.5

What an LLM Engineer Does

In one line

An LLM engineer builds the system around the model: input checks, retrieval of the right data, the prompt, the model call, output validation, tools, evaluation, security and monitoring. This topic maps that stack to the phases of the course.

Think of it like this

A TV news programme. The presenter (the model) speaks well, but behind them are researchers who find the facts, a producer who writes the script, a fact-checker, a director who watches the time, and someone who cuts the feed if something goes wrong. The audience sees only the presenter. The quality comes from the team.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Pipeline: A chain of steps where each step's output is the next step's input.
Retrieval: Finding the pieces of your own data that are relevant to a question, to put them in the prompt.
Evaluation (eval): Scoring a model feature on a fixed set of test cases to see if a change made it better or worse.
Guardrail: A check before or after the model call that blocks or fixes bad input or output.
Observability: Being able to see what your system did and why, from logs, traces and metrics.

Step by step

01Step 1: the whole request, end to end

The example runs one question through every layer in miniature, each labelled with the phase that teaches it. Read it top to bottom: it's the shape of almost every LLM feature you will build.

Step 1: the whole request, end to enddiagram

Rendering diagram…

02Step 2: grounding changes the answer

For "How long does a refund take?", retrieval finds the refund document, the prompt includes it, and the answer cites its source. For "Can I pay with gold coins?", nothing is found and the model is told to answer only from documents, so it says it doesn't know. That is the difference between a grounded answer and a guess (Phase 8).

03Step 3: every layer is plain code

Only one function in the pipeline is the model. The rest (guarding, retrieving, prompt building, validating, logging) is ordinary code you can test, review and reason about. That is good news: most of LLM engineering uses skills you already have.

04Step 4: the log line

The pipeline logs sizes, not contents: question length, number of documents, prompt length. Real systems log tokens, latency, cost, model and outcome per call, and keep user text out of logs unless redacted (Topic 11.2).

Try it yourself

1
Add a document
Add a payment document saying We accept cards and UPI. and ask Which payment methods do you accept?. The fake model has no scripted answer for it: what comes back, and what would you change so it answers from the new document?
2
Trip the input guard
Ask a question longer than 200 characters ("a" * 250 in Python, "a".repeat(250) in JavaScript, "a".repeat(250) in Java). What happens, and what should the user see instead of a crash?

Code & diagrams

One question through the whole LLM stack New tab

Each function is labelled with the phase that teaches it properly. Only fake_model stands in for a real call.

Expected output

Q: How long does a refund take?
A: Within 5 working days. | source: refund
Q: Can I pay with gold coins?
A: Not in my documents. | source: none
log:
  question=28 chars, docs=1, prompt=112 chars
  question=26 chars, docs=0, prompt=91 chars

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

A reply without the expected field

Make fake_model return {"text": "Within 5 working days."} for the refund case and run.

terminal

$ Run

── what you'll see ──

ValueError: reply has no answer field (Java: IllegalArgumentException, JavaScript: Uncaught Error, with the same message)

Myth vs fact

Myth

LLM engineering is mostly prompt writing.

Fact

Prompts matter, but most of the work is the code around them: data, validation, limits, evaluation, security and operations.

Myth

You need a machine-learning degree.

Fact

Building products on hosted models is software engineering. Knowing how models work (Phase 1) helps you predict their behaviour, but you rarely train one.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Keep the model call behind a narrow interface (complete(messages) -> reply) from day one. Swapping providers, adding a cache, recording responses for tests and metering cost all become wrappers instead of rewrites (Topic 2.8, Topic 0.7).
▸
Most quality problems in RAG systems are retrieval problems, not model problems: the right document never reached the prompt. Log what was retrieved for every answer.
▸
Version prompts like code and evaluate every change. A prompt edit is a deployment.

Remember this

1
Why this role exists. Model providers make the models. Product teams want features. The LLM engineer connects them: turns a vague wish ("let customers ask about orders") into a reliable system with a clear input, the right data, a tested prompt, a checked output, known costs and a plan for failure. It is software engineering with one new, unusual component.
2
The stack, from the user inwards. Check the input (length, abuse, personal data). Retrieve the right information (your documents, the user's order). Build the prompt (instructions, the data, the question). Call the model (with timeouts, retries and limits). Validate the output (shape and facts). Sometimes call tools and loop. Log and measure everything. Each layer is a phase of this course.
3
Evaluation is the test suite. You can't unit-test a model's wording, but you can keep a set of real cases with known good answers and score every change against them: a new prompt, a new model, a new retrieval setting (Phase 9). Teams that do this ship changes with confidence; teams that don't find out from users.
4
Security is everyone's job here. Anything that reaches the prompt (user text, a web page, an email, a document) can try to steer the model. The engineer designs so that a steered model can't do much harm: least privilege for tools, checks on output, no secrets in prompts (Phase 10).
5
How this course is arranged. Phases 1–2: how models work and how to call them. 3–5: prompts, structured output, streaming. 6: cost, latency and limits. 7–8: tools and retrieval (each also gets its own full course later: AI Agents and RAG and Vector Search). 9–11: evaluation, security, observability. 12–13: building the service and choosing models. 14: interviews and a capstone.
6
What you need first. The basics of one programming language: variables, functions, lists and dictionaries, loops, and reading an error. Hectal's Python course (/python) is enough, and its last phase introduces the same ideas from the Python side. Every example here is in Python, Java and JavaScript, so you can stay in the language you know.

Explain it without notes

Name the layers of a typical LLM feature, from the user's input to the answer.

Why is evaluation called the test suite of LLM applications?

Practice

Add a step to the pipeline that refuses questions containing an email address (@ followed by a . later is enough for this exercise) with the message Please don't share personal details..

Change retrieve so it returns the matching document keys too, and print them as the sources.

Trade-offs

↔
Every layer adds latency and code, and removes a class of failures. Start with input checks, validation and logging; add the rest as the feature grows.

Done when you can

Done when you can draw the LLM stack and name a phase for each layer.
Done when you can explain why most of the work is ordinary code around the model.
Done when you can explain what evaluations are for.
Done when you know which separate courses go deeper on agents and RAG.

Command Palette