Topic 14.1
Concept Questions Interviewers Ask
In one line
A dozen concept questions come up in almost every LLM engineering interview: tokens, context, sampling, hallucination, RAG against fine-tuning, evaluation, injection, structured output, cost and failures. This topic gives a model answer to each and points to the topic that taught it.
Think of it like this
A driving test. The examiner does not want the history of the car. They want to see that you know what each control does, what you do when the road is wet, and what you check before you set off. A good interview answer works the same way: short, accurate, with one number or one failure mode that shows you have really done it.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Trade-off
- A choice where gaining one thing costs another, such as lower latency for lower quality. Good answers name both sides.
- Follow-up question
- The next question an interviewer asks to see how deep your answer goes.
- Failure mode
- A specific way something breaks, such as a reply cut off by a token limit.
- Evidence
- A number, a log line or a test result that supports your claim, instead of an opinion.
- Grounding
- Giving the model the source text to answer from, so it does not rely on memory.
- Logit
- The raw score a model gives each possible next token before it is turned into a probability.
- Softmax
- The step that turns a list of raw scores into probabilities that add up to 1. See Topic 1.6.
- RAG
- Retrieval-augmented generation: find the relevant text first, put it in the prompt, then ask the model to answer from it. See Topic 8.1.
- Fine-tuning
- Training an existing model a little more on your own examples, to change its style or format. See Topic 13.5.
- Prompt injection
- Text the model reads that contains instructions, which the model may follow instead of yours. See Topic 10.2.
Step by step
01Step 1: use a fixed shape for every answer
Say what the thing is, why it exists, how it works, what it costs or how it fails, and how you would check it. The diagram shows the order. The shape keeps you short and makes sure the failure mode and the check are never forgotten, which is where most weak answers fall down.
02Step 2: back one answer with a running example
The sampling example below turns four raw scores into probabilities at three temperatures. Run it. At temperature 0.2 the top token has almost all the probability, at 2.0 the choices are close to even. Now "what does temperature do?" has a picture and numbers behind it.
03Step 3: tie each answer to a topic
After each model answer below, the topic numbers show where you built it. If your answer is thin, open that topic, run its example and try the break-it exercise. Interview answers are strongest when you remember breaking the thing yourself.
04Step 4: know how to say "I do not know"
If you do not know a detail (a provider's exact limit, a library's name), say what you would check and where. "I would read the provider's rate-limit headers and design for them" is a strong answer. Inventing a number is the one thing that loses an offer.
05Step 5: answer the follow-up before it is asked
End answers with a limit: "this fails when the retrieved text is stale, so I would log the document date". Interviewers then ask about something you chose, not something you missed.
Try it yourself
- 1
Predict before running
Before you run the example, write down which token is most likely at temperature 0.2 and whether
fishwill ever be above 5% at any of the three temperatures. Run it and check. - 2
Move the top-p line
Change
0.9to0.6, then to0.99. How many tokens are kept each time? Explain in one sentence why top-p keeps a different number of tokens for different distributions, while top-k always keeps the same number. - 3
Say it out loud
Set a 90-second timer and answer "What does temperature do and when would you set it to 0?" aloud, using the five beats. Then read the model answer to question 3 and note what you left out.
Code & diagrams
Four raw scores, softmax at three temperatures, and the top-p cut. The same numbers back the sampling answer below.
Expected output
temperature 0.2: cat=0.993 dog=0.007 bird=0.001 fish=0.000
temperature 1.0: cat=0.609 dog=0.224 bird=0.136 fish=0.030
temperature 2.0: cat=0.434 dog=0.263 bird=0.205 fish=0.097
top-p 0.9 at temperature 1.0 keeps cat, dog, bird (together 0.970)Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget to divide by the temperature
In softmax, replace x / temperature with x (Python), so the temperature is ignored.
Myth vs fact
Myth
Temperature 0 makes the model fully deterministic.
Fact
It picks the top token at each step, which removes most variation, but servers can still return slightly different answers because of how they batch and compute numbers. Design tests that tolerate small differences.
Myth
A bigger context window means you can put everything in the prompt.
Fact
More tokens cost more and add latency, and models use facts in the middle of very long prompts less reliably. Retrieve what is relevant instead of sending everything.
Myth
Interviewers want the most advanced technique.
Fact
They want the simplest thing that works, a reason, and a way to check it. Starting with plain prompting, then adding retrieval or tuning only when a test shows the need, is the senior answer.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
When asked to compare approaches, give the order you would try them and the evidence that would make you move to the next one: prompt, then retrieval, then tuning, each gated by your eval set (Topics 8.1, 9.1, 13.5).
- ▸
Quote costs in the form tokens times price, with prices as example values, and say that you would read the provider's current price list. Never quote a model's price or context size from memory in an interview.
- ▸
Bring one story of something that broke: a reply cut off by
max_tokens, a retry storm, a prompt change that silently dropped quality. One concrete incident is worth ten definitions.
Remember this
- 1
Why this exists. Interviewers for LLM roles do not have time to read your code, so they use a small set of questions that expose real experience. Each question hides a habit: do you measure instead of guess, do you design for failure, do you know what is cheap and what is expensive? People who only watched demos give long, vague answers. People who built things give short answers with a trade-off in them.
- 2
A shape that works. Answer in five beats: what it is in one sentence; why it is needed (the problem); how it works in two or three sentences; what it costs or how it fails; how you would check it. Add a number when you can (a token count, a latency, a price per million tokens written as an example). Stop after about 90 seconds and let the interviewer ask for more.
- 3
Where each answer lives. Tokens and context: Phase 1. Calling, errors and timeouts: Phase 2. Prompts: Phase 3. Structured output: Phase 4. Streaming: Phase 5. Cost, latency and limits: Phase 6. Tools: Phase 7. Retrieval: Phase 8. Evaluation: Phase 9. Safety: Phase 10. Operations: Phases 11 and 12. Choosing and adapting models: Phase 13. The questions below name the exact topics, so use them as a map for revision.
- 4
Questions that sound new but are not. "How would you reduce hallucinations in a medical assistant?" is the hallucination question with a higher cost of failure. "How would you make this cheaper?" is the token-maths question. "What if the provider is down?" is the retry-and-fallback question. Learn the dozen core answers well and you can rebuild most variations on the spot.
- 5
What interviewers listen for. Three things: that you separate what the model does from what your code does (the model proposes, your code checks and acts), that you reach for measurement (an eval set, a log, a metric) before opinion, and that you are honest about limits ("I would test that" is a good answer, a confident guess is not).
- 6
How to practise. Say each answer out loud with a timer, then compare with the model answer below. Run the sampling example in this topic and change one number at a time, because a question like "what does temperature do?" is much easier to answer when you have watched the probabilities move.
Explain it without notes
1. What is a token, and why does it matter for cost, speed and limits?
2. What is the context window, and what do you do when a conversation outgrows it?
3. Why does the same prompt give different answers, and what do temperature, top-k and top-p do?
4. Why do models hallucinate, and how do you reduce it?
5. When would you use prompting, RAG or fine-tuning?
6. Walk through a RAG pipeline and say where it usually fails.
7. How do you evaluate an LLM feature before and after launch?
8. What is prompt injection, and how do you defend against it?
9. How do you get reliable structured output from a model?
10. How do you reduce the cost and latency of an LLM feature?
11. How do you handle rate limits, timeouts and provider outages?
12. How does tool calling work, and what makes it dangerous?
Practice
Write request_cost(tokens_in, tokens_out, price_in_per_mtok, price_out_per_mtok) that returns the cost in dollars, with prices per million tokens. Print the cost of one request with 2,000 input and 300 output tokens at example prices of 0.50 and 1.50, and of 100,000 such requests.
Write top_k(probs, k) that keeps the k most likely tokens of a dictionary of probabilities and renormalises them so they add up to 1. Show it on four tokens with k = 2.
Trade-offs
- ↔
Short answers show clarity but risk missing the failure mode; the five-beat shape keeps both.
- ↔
Quoting numbers shows experience but invents risk if they are wrong; use clearly labelled example values.
- ↔
Going deep on one topic impresses, but a panel also wants breadth; know all twelve at a one-minute level first.
Done when you can
Done when you can answer all twelve questions in under 90 seconds each.
Done when every answer names a failure mode and a way to check it.
Done when you can explain temperature, top-k and top-p using the probabilities from the example.
Done when you can point to the topic where you built each idea.
Done when you can say "I would measure that" and name what you would measure.