Topic 0.1
What a Large Language Model Is
In one line
A large language model (LLM) is a program that reads text and predicts what text should come next, one small piece at a time. Chat assistants are an ordinary app wrapped around that one ability.
Think of it like this
The word suggestions above your phone keyboard. Type "see you" and it offers "soon", "tomorrow", "later". It has seen millions of messages, so it knows which words usually come next. A large language model is the same idea grown enormous: it has read a large part of the public internet and many books, and it can continue almost any text in a way that sounds right.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Large language model (LLM)
- A program trained on a huge amount of text that predicts what text should come next.
- Model
- A program whose behaviour comes from numbers learned from data, not from rules a person wrote.
- Weights (parameters)
- The learned numbers inside a model. Large models have billions of them.
- Training
- Showing a model lots of examples and adjusting its weights until its predictions improve.
- Inference
- Using a trained model to get an answer. Every API call you make is inference.
- Probability
- How likely something is, as a number from 0 (never) to 1 (certain).
- Prompt
- The text you send to the model: instructions, data and earlier messages.
- Hallucination
- When a model states something false in a fluent, confident way.
Step by step
01Step 1: text in, probabilities out
Give a model the text the cat sat on the. It does not return one word. It returns a score for every possible next piece, which turns into probabilities: mat 0.62, floor 0.21, sofa 0.17, and tiny numbers for everything else. The first example on this page fakes this with a lookup table so you can see the shape.
02Step 2: pick one, add it, repeat
To write a sentence, a program picks one of the likely words, adds it to the text, and asks again with the longer text. Each round adds one piece. A 300-word answer is hundreds of rounds. This loop is why answers appear word by word in chat apps and why longer answers take longer (Topic 6.3).
03Step 3: where the knowledge comes from
During training the model saw sentences like "The capital of France is Paris" many times, so after The capital of France is, the piece Paris gets a very high probability. That is how facts end up inside: as patterns in the weights, not as rows in a table. It also explains the failure: for The capital of Mars is, the model still produces likely-sounding words. Our toy says "never seen this"; a real model does not have that option unless you design for it.
04Step 4: the app around the model
A chat product is a loop in ordinary code: keep a list of messages, append the user's message, send the whole list, append the reply. The model sees only what you send in that one call.
history = []
while True:
history.append({"role": "user", "content": read_user_message()})
reply = model(history) # one call; the model sees only this list
history.append({"role": "assistant", "content": reply})
show(reply)05Step 5: what you control as the engineer
You can't change the weights of a hosted model. You control everything around it: what you send (the prompt, the history, your documents), how you call it (which model, how long the reply may be, how random), and what you do with the reply (check it, parse it, retry, show it). The whole course is about those three.
Try it yourself
- 1
Add a prompt
In the first example, add an entry for
"once upon a"withtime0.9 andmidnight0.1, add it to the list of prompts, and run. Predict the two new lines before you press Run. - 2
Make App B smarter
In the second example, change App B to send the last three messages instead of one (
history[-3:],history.slice(-3),subList(Math.max(0, size - 3), size)). Does it now know the name? Why is "send the last N" a real trade-off in products (cost vs memory)? - 3
Compare the languages
Switch the language tabs on one example and read the three versions side by side. Find the line in each that formats the probability to two decimals.
Code & diagrams
A toy lookup table stands in for the model. Switch between Python, Java and JavaScript: all three print the same lines.
Expected output
Input: the cat sat on the
mat 0.62
floor 0.21
sofa 0.17
Input: i would like a cup of
tea 0.55
coffee 0.40
water 0.05
Input: the capital of mars is
(the toy has never seen this text)The fake model can only know your name if the app sends the earlier message again.
Expected output
App A sends the whole conversation every time:
you: Hi, I'm Asha.
bot: Hello Asha! How can I help? (sent 1 message(s))
you: What is my name?
bot: Your name is Asha. (sent 3 message(s))
App B sends only the newest message:
you: Hi, I'm Asha.
bot: Hello Asha! How can I help? (sent 1 message(s))
you: What is my name?
bot: I don't know your name yet. (sent 1 message(s))Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget to re-send history
In the second example, set App A to send_history=False too (chat(turns, false) in Java and JavaScript) and run.
Myth vs fact
Myth
The model looks up answers in a database.
Fact
It predicts likely text from patterns in its weights. Nothing is looked up unless your app retrieves documents and puts them in the prompt (Phase 8).
Myth
Chatbots remember everything you told them.
Fact
The model is stateless. The app re-sends history, and some apps also store notes about you and add them to the prompt. Either way, it is your code doing the remembering.
Myth
A bigger model is always the right choice.
Fact
Bigger models are usually better but slower and more expensive per call. Many product tasks (classifying, extracting) work well on small, fast models.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Treat the model as a remote, non-deterministic, rate-limited dependency with a per-call price, like a payment API that sometimes answers wrong. Every production habit you have for flaky dependencies applies, plus a few new ones for wrong-but-valid answers.
- ▸
Hosted models change underneath you. Providers update or retire model versions, and the same prompt can behave differently after an update. Pin model versions where the provider allows it and keep an evaluation set to catch changes (Phase 9).
- ▸
Statelessness is a feature: any server can handle any request, so LLM backends scale horizontally. The cost is that you pay for the re-sent history on every call, which is why long chats get slower and more expensive (Topic 1.5, Topic 6.1).
Remember this
- 1
Why this matters to an engineer. An LLM is not a database of facts and not a person. It is a function: text goes in, and out comes a list of possible next pieces of text, each with a probability (how likely it is, from 0 to 1). Everything else (answering questions, writing code, chatting) is built by calling that function again and again. Once you see it as a function, you can reason about it like any other dependency: what it costs, how fast it is, how it fails.
- 2
"Model", "weights" and "parameters". A model is a program whose behaviour comes from numbers learned from data rather than rules a person wrote. Those learned numbers are called weights or parameters. A large model has billions of them. Nobody types them in: they are found by training, a long process that shows the model huge amounts of text and nudges the numbers until its predictions get better. After training, the weights are fixed; using the model is called inference.
- 3
"Large" and "language". *Language* because it works on text (and, in newer models, images and audio turned into the same kind of pieces). *Large* because of the number of parameters and the amount of training text. Size buys ability: bigger models follow instructions better and know more, but they are slower and cost more per call. Choosing a model is always a trade between quality, speed and cost (Phase 13).
- 4
A chat assistant is an app around the model. When you use a chat app, a normal program collects your message, adds earlier messages and some hidden instructions, sends all of it to the model, and shows you the reply. The model itself remembers nothing between calls: the app re-sends the conversation every time. The second example below shows what happens when an app forgets to.
- 5
What it is good at, and what it is not. LLMs are excellent at language work: summarising, rewriting, translating, pulling fields out of messy text, classifying, drafting, explaining code. They are unreliable at exact work: arithmetic on long numbers, quoting a fact they never saw, counting letters, or anything that needs a guarantee. Because the output is predicted, not looked up, a model can write a fluent, confident sentence that is false. This is called hallucination (Topic 1.9).
- 6
What "LLM engineering" means. Calling a model takes three lines of code. Making a feature that is correct, fast, affordable and safe for thousands of users takes engineering: clear prompts, structured output, retrieval of your own data, tools, tests, limits, monitoring and security. That is this course. Topic 0.4 lists exactly what breaks and which phase fixes it.
Explain it without notes
Explain what a large language model does, in two sentences, to someone who has never programmed.
Why does a chat app need to send the earlier messages again on every call?
What is the difference between training and inference, and which one does an LLM engineer usually do?
Why can an LLM state something false with confidence?
Practice
Extend the toy model so predict returns the single most likely word (or None if the text is unknown), and print it for the three prompts.
Write trim_history(history, keep) that returns the last keep messages but always keeps the first message if its role is system.
Trade-offs
- ↔
Bigger models: better answers, but slower and pricier per call. Smaller models: fast and cheap, good enough for many narrow tasks.
- ↔
Sending more history: better memory, but every call costs more tokens and takes longer.
- ↔
Hosted API: no servers to run, but your data leaves your system and the model can change. Self-hosted open model: control, but you run the hardware (Phase 13).
Done when you can
Done when you can explain an LLM as "text in, next-piece probabilities out, repeated".
Done when you can define model, weights, training, inference and hallucination in plain words.
Done when you can explain why the app, not the model, keeps the conversation.
Done when you can name three jobs LLMs do well and three they do badly.