PHASE 15Intermediate ~30 min· topic 1 of 10

Topic 15.1

Calling Web APIs: HTTP with httpx and requests

In one line

A web API is a program on another computer that answers HTTP requests, usually with JSON. With httpx or requests you send a request (a method, a URL, headers, maybe a body) and read back a response (a status code, headers, a body); you must always set a timeout and check the status.

Think of it like this

Ordering at a restaurant counter. You walk up and say what you want (the request). The kitchen sends back a tray, and the tray comes with a number on a slip: "ready", "we are out of that", or "the kitchen is broken" (the response and its status code). Nobody sees the kitchen; you only speak through the counter. A web API is that counter for a program: another computer's programs are the kitchen, and HTTP is the language you order in. The computer that answers is called the server. Your program, which asks, is the client.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

API: A set of rules that lets one program ask another to do something. A web API is reached over the network.
HTTP: The rule book for how a client asks for something and a server answers.
Request: What the client sends: a method, a URL, headers and sometimes a body.
Response: What the server sends back: a status code, headers and a body.
Status code: A three-digit number saying how the request went. 200 is OK, 404 is not found, 503 is the server being unavailable.
Header: A name and value attached to a request or response, such as Content-Type or Authorization.
JSON: A text format for structured data that looks like Python dictionaries and lists.
Timeout: The longest you are willing to wait before giving up on a call.
Client: The side that asks. In httpx, an httpx.Client object that holds shared settings and open connections.
Server: The computer (or program) that waits for requests and sends answers back.
Endpoint: One address on an API that does one job, such as /users/7 or /search.
API key: A secret string that tells the service who you are. You send it in a header and never share it.
Query parameter: A name and value added to the end of a URL after a ?, such as ?q=python, to give the server details.
Mock transport: A stand-in for the network: a Python function that plays the server, so code can be run and tested with no real network.

Step by step

01A request and a response, drawn

Every web API call is one question and one answer. The client sends a request line (GET /users/7), some headers, and sometimes a body. The server sends back a status line, headers and a body.

A request and a response, drawndiagram

Rendering diagram…

02The server is a function (in the browser)

There is no real network in your browser, so we give httpx a mock transport: a function that receives the request and returns a response. Your client code does not know the difference, so everything you learn here works the same against a real server.

The function below is the whole "server". It looks at request.url.path and returns httpx.Response(status, json=...).

main.pywhole filepython

import httpx


def handler(request: httpx.Request) -> httpx.Response:
    if request.url.path == "/users/7":
        return httpx.Response(200, json={"id": 7, "name": "Asha"})
    return httpx.Response(404, json={"error": "not found"})


client = httpx.Client(base_url="https://api.example.com",
                      transport=httpx.MockTransport(handler))

03Make a GET request and read the response

client.get("/users/7") joins the base URL and the path, sends the request and returns a Response. Its useful parts are status_code, headers (case-insensitive, like a dictionary), text (the body as a string) and json() (the body parsed).

Query parameters (?q=python&limit=5) go in params={...}. httpx escapes them for you: the space in python apis becomes +. Never paste values into a URL with an f-string; special characters would break it.

main.pywhole filepython

r = client.get("/users/7")
print(r.status_code, r.json())

r = client.get("/search", params={"q": "python apis", "limit": 5})
print(r.url)   # https://api.example.com/search?q=python+apis&limit=5

04Send data with POST

client.post("/todos", json={"title": "learn httpx"}) converts the dictionary to JSON text, sets Content-Type: application/json and sends it as the body. Use data={...} for classic HTML-form encoding, content=... for raw bytes or text, and files=... for uploads.

A 201 Created reply is the usual success answer for a POST that made something new.

main.pywhole filepython

r = client.post("/todos", json={"title": "learn httpx"})
print(r.status_code)          # 201
print(r.request.content)      # b'{"title":"learn httpx"}'

05Timeouts, and what they raise

A timeout is a limit per stage. httpx.Timeout(5.0, connect=2.0) allows two seconds to open the connection and five for each read or write. When a limit is hit, httpx raises a subclass of httpx.TimeoutException (ConnectTimeout, ReadTimeout, ...). Network trouble that is not a timeout (no route to the host, DNS failure) raises httpx.ConnectError or another httpx.TransportError.

Catch the parent class when you want one handler for the whole family. httpx.HTTPError is the parent of both transport errors and status errors.

Timeouts, and what they raisediagram

Rendering diagram…

06Turn bad status codes into exceptions

response.raise_for_status() does nothing for 2xx and raises httpx.HTTPStatusError for 4xx and 5xx (and for an unfollowed 3xx redirect). The exception carries the whole response in e.response, so you can still read the error body the server sent (APIs often explain the problem in it).

A common pattern: call raise_for_status() right after every request, and catch httpx.HTTPStatusError only at the place that knows what to do about it.

main.pywhole filepython

try:
    client.get("/users/99").raise_for_status()
except httpx.HTTPStatusError as e:
    print(e.response.status_code, e.response.json())

07The same call with requests, against a real server

In your own terminal (after pip install requests) the code is almost identical. The key difference: with requests you must pass timeout= yourself. The block in the "code" section shows both libraries; it cannot run here because it needs the real internet.

Try it yourself

1
Add a route to the fake server
In the first example, add if request.url.path == "/health": return httpx.Response(200, json={"status": "ok"}) to the handler and call client.get("/health"). Predict the printed status before you run it.
2
Break the key
In the second example change the key in the client's headers to "Bearer wrong". Predict which line of output changes (the POST now returns 401) and what happens to body = json.loads(request.content) (it is never reached).
3
Raise on every error
Add .raise_for_status() after the client.post(...) call in the second example and run it with a wrong key. Notice that the program now stops with httpx.HTTPStatusError instead of quietly continuing with an error body.

Code & diagrams

GET, query parameters and raise_for_status New tab

The fake server is the handler function. Add another path to it and call it from the client.

Expected output

status: 200 OK
content-type: application/json
json: {'id': 7, 'name': 'Asha', 'plan': 'free'}
url: https://api.example.com/users/7
url: https://api.example.com/search?q=python+apis&limit=5
json: {'query': 'python apis', 'hits': 2}
status: 404 ok? False
error: Client error '404 Not Found' for url 'https://api.example.com/users/99'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

POST JSON, auth header and a timeout New tab

The API key here is made up. Real keys come from the environment (Topic 15.3), never from the source code.

Expected output

201 {'id': 1, 'title': 'learn httpx', 'done': False}
method: POST
sent content-type: application/json
sent body: {"title":"learn httpx"}
timeout setting: Timeout(connect=2.0, read=5.0, write=5.0, pool=5.0)
timeout caught: ReadTimeout - server took too long
without key: 401

The same calls against a real API (needs the internet)python

Not runnable here: the browser has no real network. Run it on your own computer after pip install httpx requests. The URL is a public example; any JSON API works the same way.

import httpx
import requests

# httpx
with httpx.Client(base_url="https://api.github.com", timeout=10.0) as client:
    r = client.get("/repos/python/cpython", headers={"Accept": "application/vnd.github+json"})
    r.raise_for_status()
    print(r.json()["full_name"])

# requests: the same idea, but timeout is NOT optional in practice
r = requests.get("https://api.github.com/repos/python/cpython", timeout=10)
r.raise_for_status()
print(r.json()["full_name"])

Status code familiesdiagram

Rendering diagram…

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Treat a 404 as success

Fetch a missing user and index into the reply as if the call had worked.

main.pywhole filepython

import httpx

client = httpx.Client(
    base_url="https://api.example.com",
    transport=httpx.MockTransport(lambda request: httpx.Response(404, json={"error": "not found"})),
)
user = client.get("/users/99").json()
print(user["name"])

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 8, in <module>

print(user["name"])

~~~~^^^^^^^^

KeyError: 'name'

Myth vs fact

Myth

A 404 or 500 response makes Python raise an exception by itself.

Fact

No. The request succeeded as far as the library is concerned. Only raise_for_status() (or your own check) turns a bad status into an exception. Network failures and timeouts do raise by themselves.

Myth

requests and httpx have a sensible default timeout, so I can ignore it.

Fact

requests has no default timeout at all and can hang forever. httpx defaults to 5 seconds, which may still be wrong for you (large language model replies can take much longer). Always choose a timeout on purpose.

Myth

A mock transport is only for tests.

Fact

It is a perfect learning and development tool as well: you control the server, so you can show every status code, slow reply or broken body, with no network or API key. Real test suites use the same trick.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Reuse one Client (or AsyncClient) per service for the life of the program. It keeps a connection pool: opening a new TCP connection (the reliable pipe between two computers) and a TLS handshake (the step that encrypts it) for every call costs several round trips, which dominates the time for small requests. Use it as a context manager (with httpx.Client() as client:) so connections are closed.
▸
Timeout has four separate limits. A single number is applied to all four. Streaming responses (such as tokens from an LLM, Topic 16.5) need a long read timeout because the gap between two chunks, not the total time, is what read limits.
▸
Do not log headers wholesale. Authorization headers and API keys end up in log files, which are shared far more widely than secrets should be. Log method, path and status instead.
▸
httpx.MockTransport is also how you test code that calls LLM APIs without paying for tokens: return the response shape a provider would return, and assert on what your code sent (Topic 16.11).
▸
A timeout limits one operation (connect, one read, one write, waiting for a pooled connection), not the whole request. A server that sends one byte every four seconds never trips a 5-second read timeout, so a call can last far longer than the number you set. For a hard deadline on the whole call, wrap it in asyncio.timeout (Topic 12.7) or enforce a deadline in your own code. requests behaves the same way: its timeout= is not a total-time limit either.
▸
Know the method semantics, because retry logic (Topic 15.2) depends on them. Safe methods (GET, HEAD, OPTIONS) should not change anything on the server. Idempotent methods (the safe ones plus PUT and DELETE) have the same end effect when repeated. POST is neither, and PATCH is not guaranteed to be idempotent. Also check Content-Type before calling .json(): an error page from a proxy is often HTML, and .json() then raises json.JSONDecodeError, which is not an httpx exception.

Remember this

1
HTTP is the rule book for the conversation. A request has four parts. The method says what you want to do: GET to read, POST to create, PUT or PATCH to change, DELETE to remove. The URL says where, and the part of the address that names one thing (such as /users/7) is called an endpoint. The headers are extra facts, such as the format you send and who you are. The optional body is the data you send. A response has a status code, headers and a body.
2
The status code is a three-digit number that says how it went. 2xx means success (200 OK, 201 Created). 3xx means look elsewhere. 4xx means *you* made a mistake (400 bad request, 401 not signed in, 403 not allowed, 404 not found, 429 too many requests). 5xx means the *server* has a problem (500, 502, 503, 504). That 4xx versus 5xx split decides whether trying again can ever help (Topic 15.2).
3
Most APIs today speak JSON: the text format whose shape matches Python dictionaries and lists (dictionaries are Topic 4.4; json is Topic 7.7). With httpx you pass json={...} and the library converts the dictionary to JSON text, adds the Content-Type: application/json header and sends it. response.json() turns the reply back into dictionaries and lists. This is why dictionaries are the shape of nearly every API message, including every message you send to an AI model. Most services also want to know who is calling. You get a secret password-like string called an API key when you sign up, and you send it in a header, usually Authorization: Bearer <key>. Treat it like a house key: anyone who has it can act as you (Topic 15.3).
4
httpx and requests are the two popular libraries. requests is the older, famous one and is simple. httpx has almost the same style, plus async support (Phase 12), HTTP/2 and a built-in way to plug in a fake server for tests, which is why this course uses it. Both are installed with pip install or uv add (Topics 6.5 and 6.6); neither comes with Python. The standard library has urllib.request, which works but is more awkward.
5
Always set a timeout. If a server never answers, a call with no timeout can wait forever and freeze your whole program. requests has *no* default timeout, so you must pass one every time. httpx defaults to five seconds, and you can set separate limits for connecting, reading and writing with httpx.Timeout. A timeout raises an exception (httpx.TimeoutException) that your code can catch and handle. Recall from Topic 7.1: try and except catch an error so the program can carry on.
6
Check the status, not just the body. A 404 response is not a Python error: the call worked, the server just said no. Call response.raise_for_status() to turn any 4xx or 5xx into an exception (httpx.HTTPStatusError), or test response.is_success yourself. (httpx does not follow redirects unless you pass follow_redirects=True, so a 301 or 302 also raises here, as a "Redirect response" error; requests follows them by default.) Reuse one httpx.Client for many calls instead of calling httpx.get each time: it keeps connections open (faster), and it lets you set a base URL, shared headers and a timeout in one place.

Explain it without notes

What are the parts of an HTTP request and an HTTP response?

What do the status code families 2xx, 4xx and 5xx mean, and why does the difference matter for client code?

Why must a client always set a timeout, and what does raise_for_status() do?

Why use an httpx.Client instead of calling httpx.get every time?

Which HTTP methods are safe and which are idempotent, and why does a client care?

How do the default timeouts and redirect behaviour of httpx differ from requests?

Practice

Using a MockTransport handler that returns {"id": 1, "name": "Asha"} for /users/1, write a function get_user(client, user_id) that returns the dictionary, and raises on any error status.

Write a function search(client, term, limit=5) that calls /search with query parameters and returns the list in the results key of the reply.

Write a handler that raises httpx.ConnectError for the path /down, and show how to catch it without catching 404s.

Trade-offs

↔
httpx versus requests. requests is everywhere and has the most tutorials. httpx has the same easy style plus async, HTTP/2 and mock transports. New projects, especially those that also call AI APIs concurrently, usually pick httpx; both are fine for plain scripts.
↔
One shared client versus a fresh client per call. A shared client is faster and keeps settings in one place, but you must close it and be careful with mutable state (such as changing headers) when several parts of the program share it.

Done when you can

Done when you can name the parts of an HTTP request and response.
Done when you can say what 200, 201, 401, 404, 429 and 503 mean.
Done when you can send a GET with query parameters and a POST with a JSON body using an httpx.Client.
Done when you always pass a timeout and can name the exceptions it raises.
Done when you use raise_for_status() and can explain why it is needed.
Done when you can write a MockTransport handler to run client code with no network.

Command Palette

Words you'll meet

Step by step

01A request and a response, drawn

02The server is a function (in the browser)

03Make a GET request and read the response

04Send data with POST

05Timeouts, and what they raise

06Turn bad status codes into exceptions

07The same call with requests, against a real server

Try it yourself

Add a route to the fake server

Break the key

Raise on every error

Code & diagrams

Break it on purpose

Treat a 404 as success

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can