PHASE 2Beginner ~21 min· topic 7 of 9

Topic 2.7

Timeouts and Cancellation

In one line

A model call can take seconds or hang for minutes. Set a connect timeout, a read timeout and a total deadline on every call, pass the remaining time down, and cancel calls whose answer nobody needs any more.

Think of it like this

Waiting for a friend outside a cinema. You agree to wait 10 minutes. Without a limit you might stand there all evening while the film starts without you. And if the friend texts "I'm not coming", you stop waiting at once instead of finishing your ten minutes. A timeout is the agreed limit; cancellation is stopping when the answer no longer matters.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Timeout: A limit on how long to wait. When it passes, the call is given up.
Connect timeout: How long to wait for the connection to the server to be made.
Read timeout: How long to wait for the next data from the server.
Total deadline: A limit on the whole call from start to finish.
Cancellation: Stopping a call because its result is no longer needed.
Latency: How long a call takes from request to reply.
Percentile (p99): The time that 99 out of 100 calls beat. A better guide for limits than the average.

Step by step

01Step 1: see what no limit costs

The no limits at all scenario waits 4,000 ms for a reply that had stalled at 600 ms. The user stared at a spinner for 3.4 extra seconds. With a real hang, the wait would be unlimited.

02Step 2: add a connect timeout

slow connect needs 3,000 ms to connect and the limit is 1,000 ms, so the call stops at 1,000 ms. Failing quickly lets you retry or fall back while the user is still there.

Step 2: add a connect timeoutdiagram

Rendering diagram…

03Step 3: add a read timeout

stalled reply receives two pieces, then silence. With a 2,000 ms read limit the call stops at 600 + 2,000 = 2,600 ms after two pieces, instead of waiting for the third at 4,000 ms.

04Step 4: add a total deadline

slow trickle never stalls but never ends. Every gap is 1,500 ms, under the read limit. Only the 8,000 ms total deadline stops it, after five pieces.

05Step 5: add cancellation

In user cancels the cancel time is 600 ms, so the call stops there with one piece received. The cancel check is just another limit in the same comparison, which is how real clients implement it too.

06Step 6: see a real streaming call

The lab below streams a reply. Look at the time to first token and the total time: they are different clocks, and a read timeout between pieces is much shorter than a total deadline.

Try it yourself

1
Predict the six lines
Before running, predict the result of each scenario. Then change the read limit from 2000 to 3000 and re-predict stalled reply.
2
Find the trickle limit
Change the total deadline from 8000 to 20000. How many pieces does slow trickle receive now, and does it finish OK?
3
Budget three steps
A request has 20 seconds. Retrieval took 3 s and the first model attempt failed after 9 s. What timeout would you give a retry so the user is never kept waiting past 20 s?

Code & diagrams

Connect, read and total limits, and a cancel, on a simulated clock New tab

Time is a number in milliseconds, so there is no real waiting and the output never changes.

Expected output

healthy           -> ok at 900 ms with 3 chunk(s)
slow connect      -> connect timeout at 1000 ms
stalled reply     -> read timeout at 2600 ms after 2 chunk(s)
slow trickle      -> deadline exceeded at 8000 ms after 5 chunk(s)
no limits at all  -> ok at 4000 ms with 3 chunk(s)
user cancels      -> cancelled at 600 ms after 1 chunk(s)

The same limits in real clientspython

Not run here: they call a real server. httpx and asyncio need Python 3.11+ for asyncio.timeout; AbortSignal.any needs a recent Node (20.3+) or browser.

# Timeouts and cancellation with httpx (asyncio). The same ideas exist in every client library.
import asyncio
import os

import httpx

# Four separate limits, in seconds. "read" applies to each wait for data, not to the whole call.
timeout = httpx.Timeout(connect=5.0, read=30.0, write=10.0, pool=5.0)


async def ask(client, body, headers):
    # asyncio.timeout is a TOTAL deadline for everything inside it (Python 3.11+)
    async with asyncio.timeout(60):
        r = await client.post("https://api.groq.com/openai/v1/chat/completions", json=body, headers=headers)
        return r.json()


async def main():
    async with httpx.AsyncClient(timeout=timeout) as client:
        task = asyncio.create_task(ask(client, {"model": "openai/gpt-oss-20b", "messages": [{"role": "user", "content": "Hi"}]}, {"Authorization": "Bearer " + os.environ["GROQ_API_KEY"]}))
        await asyncio.sleep(0.5)
        task.cancel()  # the user closed the tab: stop waiting and free the connection
        try:
            await task
        except asyncio.CancelledError:
            print("cancelled")


asyncio.run(main())

Time to first token vs total timereal model

Try: Press Send with streaming on. Note the time to first token and the total time. Then set max tokens to 600 and send again: which time changed most, and which of your three timeouts would each one count against?

System

user

Modelsmall, fast, reasons firstTemperature · 1.0Max tokens (1–1024)Reasoning effort Stream (SSE) JSON mode

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Only a read timeout

In the Python example set the total deadline of the slow trickle scenario to NO_LIMIT (the fifth argument of its call).

terminal

$ Run

── what you'll see ──

healthy -> ok at 900 ms with 3 chunk(s)

slow connect -> connect timeout at 1000 ms

stalled reply -> read timeout at 2600 ms after 2 chunk(s)

slow trickle -> ok at 15000 ms with 10 chunk(s)

no limits at all -> ok at 4000 ms with 3 chunk(s)

user cancels -> cancelled at 600 ms after 1 chunk(s)

Myth vs fact

Myth

The library has a sensible default timeout.

Fact

Many libraries wait without limit, and SDK defaults are often minutes. Set your own.

Myth

A longer timeout is safer.

Fact

It only delays the failure, ties up resources and keeps users waiting. Pick a limit from real latency numbers.

Myth

Cancelling on the client stops the cost.

Fact

Often it does, because the provider stops when the connection closes, but not always for work already queued. Check your provider's behaviour and track tokens used.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Measure p50, p95 and p99 latency per model and per prompt size (Topic 6.3, Topic 11.4). Set timeouts from p99 plus margin and alert when p95 drifts up.
▸
Use deadlines, not durations, across service boundaries: pass an absolute "must finish by" time so every hop computes its own remaining budget.
▸
On cancel, release everything: close the stream, stop reading, free the semaphore slot and record how many tokens were used. Leaked connections and unreleased slots cause slow, mysterious outages.

Remember this

1
Why this exists. Model calls are slow and variable: a reply may take 2 seconds, 30 seconds, or never come if the server or network stalls. Many HTTP libraries (Java's HttpClient, fetch, Python requests) have no timeout by default, and SDKs often default to minutes, so a stuck call can hold a thread, a connection and a user's screen for ever. One hanging dependency can then use up all your workers and take your whole service down. Every network call needs a limit.
2
Three different clocks. The connect timeout limits how long to wait for the connection to be set up (usually short, a few seconds). The read timeout limits how long to wait for the next piece of data (for streaming, between pieces; for a normal call, until the reply starts). The total deadline limits the whole call from start to finish, however it is spread. Libraries differ in which they offer, so check what yours measures. The simulation shows how each one fires.
3
Why the total deadline matters. A server that trickles one piece every 1.5 seconds never trips a 2-second read timeout, yet the call can run for ever. Only a total deadline stops it. The slow trickle scenario hits the 8-second deadline after five pieces. A stalled reply, by contrast, is caught by the read timeout much sooner.
4
Choosing values. Look at real latency (Topic 6.3) and set the limit above the slow-but-normal case, for example the 99th percentile plus margin, not the average. Long answers from large or reasoning models legitimately take tens of seconds, so use longer limits there or stream (Phase 5) so that read timeouts work between pieces. Too short causes false failures; too long defeats the purpose.
5
Pass the deadline down. If a user request has 20 seconds in total, and your code makes a retrieval call, then a model call, then maybe a retry, each step should get the remaining time, not a fresh full timeout, or three 20-second steps take a minute. Compute remaining = deadline - now before each call and stop early when it is gone.
6
Cancellation. If the user closes the tab, presses stop or a faster result arrives, abort the call so you stop waiting and, importantly, stop paying: many providers stop generating when the connection closes. In Python asyncio you cancel the task, in JavaScript you call abort() on an AbortController, in Java you cancel the future. The user cancels scenario stops at 600 ms with only one piece received.

Explain it without notes

What are connect, read and total timeouts and why do you need all three?

How do you choose a timeout value for a model call?

Why should a user cancelling a request also cancel the model call?

Practice

Write remaining_ms(deadline_ms, now_ms) that returns the time left, never below zero, and step_timeout(remaining, cap) that returns the smaller of the two.

A request has a 20,000 ms deadline starting at time 0. Retrieval ends at 3,000 ms. Print the timeout in ms you would give the model call if you cap any single call at 15,000 ms.

Trade-offs

↔
Short timeouts fail fast and protect resources but cut off legitimate slow answers; long ones tolerate slow models but hold resources.
↔
Streaming allows tight read timeouts between pieces, at the cost of more complex code (Phase 5).

Done when you can

Done when every network call in your code has a connect limit, a read limit and a total deadline.
Done when you can explain which clock stops a stalled call and which stops a slow trickle.
Done when later steps get the remaining time, not a fresh timeout.
Done when closing a page or pressing stop cancels the model call.

Command Palette

Words you'll meet

Step by step

01Step 1: see what no limit costs

02Step 2: add a connect timeout

03Step 3: add a read timeout

04Step 4: add a total deadline

05Step 5: add cancellation

06Step 6: see a real streaming call

Try it yourself

Predict the six lines

Find the trickle limit

Budget three steps

Code & diagrams

Break it on purpose

Only a read timeout

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can