Command Palette

Search for a command to run...

Hectal
Phase 6Intermediate7 of 15 in LLM Engineering

Cost, Latency and Limits

What every model call costs and how long it takes, then caches, rate limits, retries with jitter, fallbacks, circuit breakers and concurrency, built by hand first.

A model call costs money by the token, takes seconds, and can be refused when you send too much too fast. This phase makes those three facts concrete. You will work out the cost of a request, a user and a month with plain arithmetic, see where the seconds go (waiting, reading your prompt, writing the answer), and put guards in place so one bug or one heavy user cannot empty the budget.

Then it builds, step by step and with a simulated clock (a counter in milliseconds, so every example is instant and gives the same output every time), the tools that keep a product running when the provider is slow or refusing: caches that skip repeated calls, a token bucket that respects rate limits, retries with backoff and jitter (the full story of why 100 clients retrying together make things worse), fallback chains, a circuit breaker, and a concurrency limit. Each is built by hand before any library is shown.

Prices in every example are variables with example numbers. Read your provider's price page and keep real prices in configuration. The two real-model labs show the rate-limit headers a live service sends back. The phase ends with a playbook that joins limiter, retry, fallback, cache and cost meter into one resilient client in Python, Java and JavaScript.

0/11 · 0%
11 topics ~4 h 20 code blocks & diagrams
Start with the first topic
1
6.1

Token Maths and Cost

A model call is billed by tokens: input tokens times the input price, plus output tokens times the output price. A few lines of arithmetic tell you the cost per request, per user and per month, and show why long chat histories get expensive fast.

20 min 1 code practice

2
6.2

Budgets and Guards

Guards stop one bug or one heavy user from emptying the budget: a cap on tokens per request, a daily limit per user, a global spend ceiling with an alert, and a kill switch you can flip without a deploy.

20 min 1 code practice

3
6.3

Where Latency Comes From

A model call's time is network, queue, reading your prompt (prefill) and writing the answer one token at a time (decode). Knowing the four parts tells you what to shorten, and percentiles (p50, p95) tell you how slow your slowest users really are.

20 min 1 code practice

4
6.4

Exact-Match Caching

An exact-match cache stores the answer to a request and returns it when the same request comes again, skipping the model call completely. It is free, instant and safe only when the answer should not change, so it needs a careful key, an expiry time and clear rules for what never to cache.

21 min 1 code practice

5
6.5

Semantic Caching

A semantic cache returns a saved answer when a new question means almost the same as an old one, found by comparing embeddings with a similarity threshold. It saves more calls than exact caching, but a threshold set too low returns the wrong answer to a question that only looks similar.

20 min 1 code practice

6
6.6

Provider Prompt Caching

Many providers remember the work they did on the start of your prompt and reuse it when a later request begins with the same text, which makes repeated long prompts cheaper and faster to start. You benefit by putting the stable text first and the changing text last.

21 min 1 code practice

7
6.7

Rate Limits

Providers cap how many requests and tokens you may send per minute and answer with HTTP 429 when you go over. Headers tell you how much quota is left and when it resets, and a token bucket lets you stay under the limit before the server has to refuse you.

22 min 3 code practice

8
6.8

Retries with Backoff and Jitter

Many model errors (429, 503, timeouts) go away if you try again later, but a naive retry loop can turn a small hiccup into an outage. This topic builds the full retry from first principles: why immediate retries pile up, exponential backoff, random jitter, Retry-After, retrying only what is retryable, and idempotency.

28 min 4 code practice

9
6.9

Fallbacks, Deadlines and Circuit Breakers

When your main model is slow or down, a fallback chain tries the next model within a deadline, a circuit breaker stops you calling a provider that keeps failing, and graceful degradation gives users something useful instead of an error.

22 min 2 code practice

10
6.10

Concurrency and Batching

Running many model calls at once finishes a big job much sooner, but too many at once hits rate limits. A concurrency limit keeps a fixed number in flight, results are stored by position so order survives, and batch APIs handle offline work more cheaply.

21 min 2 code practice

11
6.11

Playbook: Resilient, Affordable Calls

The phase's recipes in one place: a client that wraps any model with a cache, a rate limiter, retries with jitter, a fallback chain and a cost guard, and a per-feature report that turns your call log into costs, p95 latency and alerts.

26 min 3 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Price a request and a chat

Write request_cost(tokens_in, tokens_out, price_in_per_mtok, price_out_per_mtok) that returns the cost in dollars. Prices are dollars per million tokens.

Beginner
2

Cache answers with a key and an expiry

Write make_key(model, prompt, temperature) that returns "<model>|<temperature to one decimal>|<prompt with runs of spaces collapsed and the ends trimmed>". For example make_key("m", " Hi there ", 0) is "m|0.0|Hi there".

Beginner
3

Full jitter with Retry-After

Write retry_delay_ms(attempt, base_ms, cap_ms, rand, retry_after_s=None). attempt counts from 1 (the number of the attempt that just failed) and rand is a number from 0 up to but not including 1.

Intermediate
4

Write the retry loop

Write call_with_retry(send, max_attempts, sleep). send(attempt) returns (status, text, retry_after_s); attempt counts from 1. sleep(ms) is a function you call to wait (the tests record it instead of waiting).

Intermediate
5

Build a token bucket

Write TokenBucket(capacity, refill_per_sec) with try_take(now_ms). It starts full. It returns 0 when a request is allowed (and uses one token), otherwise the number of milliseconds to wait until one token is back, rounded up.

Intermediate
6

Write a circuit breaker

Write CircuitBreaker(threshold, open_ms) with the attribute state (starts as "closed"), allow(now) and record(ok, now).

Advanced