Topic 9.6
Fakes and Recorded Responses
In one line
Most tests of an LLM feature should not call a real model. A fake model with scripted replies tests your code (parsing, retries, fallbacks, prompt assembly), and recorded real responses replayed by a request hash test it against realistic data, for free and in milliseconds.
Think of it like this
A driving school's simulator. Learners practise braking, junctions and emergencies on a screen, where nothing costs petrol and nobody gets hurt, and the simulator can create a skid whenever the teacher wants. The real car comes later, for the things only a real road shows. A fake model is the simulator, a recording is a tape of one real drive, and the live model is the road.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Fake
- A stand-in with the same interface as the real thing that returns scripted answers.
- Unit test
- A small test of one piece of your code that runs in milliseconds and must always pass.
- Record and replay
- Saving real responses once, then returning the saved ones in later runs.
- Cache key
- The label a saved reply is stored under, made from everything that affects the answer.
- Hash
- A function that turns text into a number, the same number every time for the same text.
- Stale recording
- A saved reply that no longer matches what the real system would say.
- Integration test
- A test of several parts working together, such as your code with recorded model replies.
Step by step
01Step 1: give the model a small interface
Your feature should receive the model as a parameter: triage(model, ticket). In production you pass the real client, in tests you pass a fake. This is the whole trick, and it costs nothing if you do it from the start (Topic 2.8).
02Step 2: script the replies, including the bad ones
The first example's FakeModel takes a list of replies. Five tests script a clean reply, chatter followed by a good reply, a category that is not allowed, two unusable replies, and a plain call that checks the recorded request.
m = FakeModel(["Sure! " + reply("bug", "high"), reply("bug", "high")])
check("chatter is retried once", triage(m, "t") == "bug/high" and len(m.requests) == 2)03Step 3: assert on what was sent
Because the fake records every request, a test can check the system prompt, the user text and temperature 0. This catches a refactor that quietly stops sending the system prompt, a bug no output check would see.
04Step 4: record the real replies once
The second example wraps a live model (a fake stand-in here, because the page cannot call a real one). In record mode every call goes to the live model and its reply is saved under a key. The key is a hash of the system prompt, the user text and the temperature, joined with a separator.
05Step 5: replay for free
In replay mode the live model is never called: 0 live calls, the same three answers. A test run that used to take seconds and cost tokens now takes milliseconds and costs nothing.
06Step 6: let a changed prompt fail loudly
Adding " Be brief." to the system prompt changes the key, so replay raises MissingRecording. This is a feature: the old recording describes the old prompt. Re-record against the real model, run the live eval (Topic 9.5), and commit the new recording together with the prompt change.
Try it yourself
- 1
Script a failure
In the first example add a test where the fake returns
[](a JSON list, not an object) twice. Predict the result, then run. Which exception does each language catch for it? - 2
Break a test on purpose
Change
temperaturefrom0to1insidetriage. Which test fails, and what does its name tell you? - 3
Change the key
In the second example, add the model name
"small-model"to the key text. Run the record and the replay. Why do the printed key numbers change, and do old recordings still work?
Code & diagrams
A scripted FakeModel returns replies in order and records requests. Five tests run in milliseconds with no key and no network.
Expected output
PASS a clean reply is returned
PASS chatter is retried once
PASS a category outside the list is rejected
PASS two bad replies fall back
PASS the request carries the prompt and temperature 0
5 passed, 0 failedThe live model is itself a fake here so the page can run it. tiny_hash is h = (h * 31 + code) % 1000000007, identical in all three languages.
Expected output
recording: 3 live calls, 3 replies saved
key 283656824: {"category": "billing", "priority": "high"}
key 472295308: {"category": "bug", "priority": "high"}
key 761913016: {"category": "other", "priority": "low"}
replay: 0 live calls, same answers: yes
after editing the prompt: no recording for key 40295764Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
A key that forgets part of the request
In the second Python example, change the key line in Recorder.complete to key = tiny_hash(system), so only the system prompt makes the key.
Myth vs fact
Myth
Testing with a fake means testing nothing real.
Fact
Fakes test your code, which is where most bugs are. They do not test model quality, and they are not meant to. Use them for unit tests and use evals for quality.
Myth
Recordings never go out of date.
Fact
A recording describes one prompt and one model version. When either changes, the key changes (or should), and you record again. Keep a nightly live run to notice when the real model drifts from the tape.
Myth
Mocking the HTTP library is the same as a fake model.
Fact
Mocking the HTTP layer works, but it ties tests to one SDK and to the exact wire format. A small complete(...) interface of your own is easier to fake and survives a change of provider.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep the model client behind one interface in your code. Then the real client, the fake, the recorder and a caching wrapper are all the same shape and can be stacked.
- ▸
Record rare real failures (a 429, a cut-off reply, a reply with chatter) and replay them in tests. Real failure shapes are often not what you would script by hand.
- ▸
Do not record secrets: strip authorization headers and personal data from what you save, and review recordings in pull requests.
- ▸
Run the same recorded set against the new code in CI on every pull request. If you also store the recorded reply's token counts, you can test your cost accounting without any live call.
Remember this
- 1
Why this exists. Real model calls are slow (seconds), cost money, need a key and a network, and give different answers each time. A test suite built on them is slow, expensive and flaky, so people stop running it. But most of the bugs in an LLM feature are in your code, not in the model: the JSON parser that crashes on chatter, the retry loop that retries forever, the prompt assembled in the wrong order. A fake model reaches those bugs in milliseconds.
- 2
A fake model has the same method as the real client (
complete(...)) but returns replies from a script, in order, and records every request it receives. You choose the replies, including the awkward ones: chatter around the JSON, an unknown category, an empty string, a cut-off reply, a timeout (by raising an error). Because the fake records requests, a test can also check what your code sent: the system prompt, the user text, temperature 0. - 3
What fakes catch, and what they cannot. They prove your code handles every reply shape you scripted. They cannot tell you whether the real model would produce those shapes, nor whether the prompt gives good answers. That is why fakes are for unit tests (must always pass), while quality is measured by evals with the real model (Topics 9.1 to 9.5).
- 4
Recorded responses. Run your feature once against the real model, save every request and its reply (record), and later feed the saved replies back (replay). Replay gives realistic data (real chatter, real wording, real token counts) with fake-like speed and cost, and it makes the test repeatable. Each recording is looked up by a key: a hash of everything that influences the answer (model name, system prompt, user text, temperature, tools). The second example builds this with a small hand-written hash.
- 5
The key decides what is safe. If the key leaves out something that changes the answer, replay returns a stale reply for a changed request, and the test passes for the wrong reason. If the key includes too much (for example a timestamp), nothing ever matches. When you edit the prompt, the key changes, replay fails with "no recording", and that is correct: it tells you the old recordings do not describe the new prompt, and you must record again and re-run the eval live.
- 6
Where each fits. Fake model: unit tests of parsing, retries and fallbacks, run on every commit. Recorded replay: integration tests and fast CI runs of the eval harness, re-recorded when the prompt or model changes. Live model: the real eval on a schedule and before release. Keep recordings in the repository, review them like code, and remove personal data before saving (Topic 10.5). Libraries that record HTTP traffic exist in most ecosystems; the hand-built version here shows what they do.
Explain it without notes
What bugs does a fake model catch, and what does it not catch?
Describe record and replay for model calls, and what goes into the key.
Why should a changed prompt make a replay fail?
Which kinds of test would you run on every commit, and which on a schedule?
Practice
Write ScriptedModel(replies) with a complete(text) method that returns the replies in order and raises IndexError with the message script exhausted when asked for more. Call it three times with two replies and print what happens.
Write make_key(system, user, temperature) that returns the tiny_hash of system + "|" + user + "|" + str(temperature) (hash: h = 7, then h = (h * 31 + ord(ch)) % 1000000007 for each character). Print the key for ("s", "u", 0) and show that changing the temperature to 1 changes it.
Trade-offs
- ↔
Fakes are fast and exact but only test your code; recordings are realistic but go stale; live calls are true but slow, costly and variable.
- ↔
A strict key avoids stale replies but forces more re-recording; a loose key reuses more and risks false passes.
- ↔
Committing recordings makes CI free and offline but adds files to review and data to clean.
Done when you can
Done when you can pass the model into your feature as a parameter and swap in a fake.
Done when you can script good and bad replies and assert on the recorded requests.
Done when you can build a key from a hash of the full request.
Done when you can explain why an edited prompt must miss the recording.
Done when you can say which tests run on every commit and which run on a schedule.