PHASE 16Advanced ~27 min· topic 3 of 14

Topic 16.3

Calling an LLM API from Python

In one line

Calling a language model is sending a list of messages (system, user, assistant) as JSON over HTTP and reading text back. Different providers use slightly different shapes, so you hide the call behind a small function and test everything else with a fake client.

Think of it like this

Ordering at a counter by writing on a slip. The slip has a header ("you are a polite waiter"), then the conversation so far, then your newest request. The kitchen reads the whole slip every time, because it remembers nothing between orders. A model API works the same way: you send the whole conversation each call.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Message: One turn in a conversation: a dictionary with a role and some content.
Role: Who is speaking: system (standing instructions), user (the asker) or assistant (the model).
Stateless: Remembering nothing between calls. You must resend the context every time.
Request / response: What you send to the API, and what it sends back.
SDK: A Python library from the provider that wraps the HTTP API in functions and classes.
Environment variable: A named setting that lives outside your code, such as OPENAI_API_KEY.
Fake client: A stand-in object with the same methods as the real client that returns scripted replies. Used in examples and tests.
Prompt: The text (here, the list of messages) you send to the model to tell it what to do.
Token: A small chunk of text, often a short word or part of a word, that a model reads and writes. Topic 16.4 covers tokens in detail.

Step by step

01The request is plain data

Before any SDK, see the request for what it is: a dictionary that becomes JSON. Recall from Topic 7.7: json.dumps turns a dictionary into JSON text. Compare the two shapes. The only real difference is where the system text lives.

The "model" field names which model you want. A provider offers several: bigger ones answer better and cost more. The name your-model-name is a placeholder, because real names change often.

main.pywhole filepython

import json

system = "You are a concise tutor."
question = "What is a list comprehension?"

openai_style = {
    "model": "your-model-name",
    "messages": [
        {"role": "system", "content": system},
        {"role": "user", "content": question},
    ],
}
anthropic_style = {
    "model": "your-model-name",
    "max_tokens": 200,
    "system": system,
    "messages": [{"role": "user", "content": question}],
}

print("OpenAI-compatible request body:")
print(json.dumps(openai_style, indent=2))
print("Anthropic Messages request body:")
print(json.dumps(anthropic_style, indent=2))

02Real SDK code, for when you have a key

This block does not run here (no network, no key). It shows the real call shapes. Replace the model name with a current model from your provider's documentation.

real_openai.pywhole filepython

from openai import OpenAI

client = OpenAI()   # reads OPENAI_API_KEY from the environment

response = client.chat.completions.create(
    model="your-model-name",
    messages=[
        {"role": "system", "content": "You are a concise tutor."},
        {"role": "user", "content": "What is a list comprehension?"},
    ],
)
print(response.choices[0].message.content)

03The same call with the Anthropic Messages API

Note the separate system argument, the required max_tokens, and that the answer is a list of blocks.

real_anthropic.pywhole filepython

import anthropic

client = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY from the environment

response = client.messages.create(
    model="your-model-name",
    max_tokens=300,
    system="You are a concise tutor.",
    messages=[{"role": "user", "content": "What is a list comprehension?"}],
)
text = "".join(block.text for block in response.content if block.type == "text")
print(text)

04A fake client with the same interface

The examples in this phase use a fake because a real call needs a paid account, a secret key and the internet, and none of those exist inside this page. A fake also never changes its answer, so you can test against it. The second runnable example builds a fake with the same path as the real SDK: client.chat.completions.create(...). It uses SimpleNamespace, a small built-in class that makes an object with whatever attributes you give it. The fake validates roles, records every call and returns scripted text. Your ask function cannot tell the difference, which is the whole point.

05The history grows each turn

Look at the output of the example: the second call sends four messages (system, user 1, assistant 1, user 2). The model does not remember turn one. Your list does. This is why long chats get expensive and slow, and why Topic 16.4 teaches trimming.

The history grows each turndiagram

Rendering diagram…

06Where this shows up in real systems

The Hectal AI tutor works exactly this way: your code builds messages, calls a model, streams the answer back (shows it word by word as it is produced; Topic 16.5), enforces daily limits and falls back to another model when one is down. Every one of those is a section in this phase, because a real system needs all of them around the one API call.

Try it yourself

1
Add a third turn
In the second example add a third scripted reply and a third question. Predict how many messages the third call sends, then check calls.
2
Trigger the role check
Change a role to "robot" in the history and run. Your fake raises the same kind of error a real API returns as an HTTP 400.

Code & diagrams

Two request shapes side by side New tab

Compare where the system text sits in each body.

Expected output

OpenAI-compatible request body:
{
  "model": "your-model-name",
  "messages": [
    {
      "role": "system",
      "content": "You are a concise tutor."
    },
    {
      "role": "user",
      "content": "What is a list comprehension?"
    }
  ]
}
Anthropic Messages request body:
{
  "model": "your-model-name",
  "max_tokens": 200,
  "system": "You are a concise tutor.",
  "messages": [
    {
      "role": "user",
      "content": "What is a list comprehension?"
    }
  ]
}

A fake client and a chat loop New tab

The fake has the same method path as an OpenAI-style SDK. Change the scripted replies and run again.

Expected output

   system: Be concise.
     user: What is a list comprehension?
assistant: A list comprehension builds a list in one line.
     user: Can it filter?
assistant: Yes: add an if at the end.
calls made: 2
messages sent on call 2: 4

Streaming with the real OpenAI-style SDK (not runnable here)python

You build the same loop with generators in Topic 16.5.

stream = client.chat.completions.create(
    model="your-model-name",
    messages=[{"role": "user", "content": "Count to three."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="")

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

A message with no role

Build a message dictionary that forgets the role key.

main.pywhole filepython

messages = [{"content": "hello"}]
for m in messages:
    print(m["role"], m["content"])

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 3, in <module>

print(m["role"], m["content"])

~^^^^^^^^

KeyError: 'role'

Myth vs fact

Myth

The model remembers our earlier chat.

Fact

It remembers nothing between calls. The app resends the conversation each time, and "memory" features are your code choosing what to include.

Myth

You need the provider's SDK to call a model.

Fact

The API is HTTP with JSON. An SDK adds convenience (retries, typed objects, streaming helpers), and you can also use plain httpx.

Myth

Tests of AI code need a real model.

Fact

Tests should use a fake client with scripted replies. A real model is slow, costs money, and gives different text on different days.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Many providers and local servers expose an OpenAI-compatible endpoint, so you can often switch by changing the base_url and key while keeping the same code. Check each provider's documentation for the differences that remain (supported options, tool-call details).
▸
Treat the raw SDK response as an implementation detail. Convert it at the boundary into your own small type (for example a dataclass with text, stop_reason, tool_calls), so the rest of the program does not depend on one vendor's object shape.
▸
Log every request and response (with secrets removed) with a request ID. When an answer is wrong in production, the exact messages sent are the first thing you need (Topic 16.11).
▸
Always check why generation stopped. An OpenAI-style response carries choices[0].finish_reason ("stop" for a natural end, "length" when it hit the token cap, "tool_calls" when the model wants a tool run). The Anthropic response carries stop_reason ("end_turn", "max_tokens", "tool_use"). Treating a reply cut off by the cap as a complete answer is a classic production bug, and it silently breaks JSON output (Topic 16.6).
▸
In streaming loops, do not assume every chunk carries text. In OpenAI-style streams, delta.content can be None (for a role-only or tool-call chunk), and with usage reporting switched on the last chunk can have an empty choices list, so guard before indexing choices[0].

Remember this

1
A message is a dictionary with a role and content. The role system holds standing instructions ("be concise, answer in English"). user is the person (or your program) asking. assistant is what the model said earlier. The list of messages is the entire memory of the conversation: the model itself is stateless, so a "chat" is your code appending to a list and sending it again.
2
Two request shapes are common. OpenAI-compatible Chat Completions puts the system text inside the messages list as a system message, and many other providers and local servers copy this shape. The Anthropic Messages API takes the system text as a separate system argument, requires max_tokens (the longest answer you will accept, counted in tokens: small chunks of text, explained in Topic 16.4), and the messages list holds only user and assistant turns. The ideas are identical; only the field names move.
3
The response also differs. In the OpenAI-compatible shape the text is at response.choices[0].message.content. In the Anthropic shape response.content is a list of blocks, and text blocks have block.type == "text" and block.text. A reply can hold several blocks (text, then a tool request, Topic 16.10), so real code loops over blocks instead of assuming one.
4
API keys are secrets. Read them from environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY), never write them in code, notebooks or screenshots. SDKs read these variables by default when you create a client. Phase 15 covered configuration and secrets; the same rules apply with extra force because a leaked key is spendable money.
5
Hide the provider behind your own function. Write ask(client, history, question) that returns text. Your tests then pass a fake client with the same method names and scripted replies. No network, no cost, no flakiness. Switching providers later changes one function. This one habit is what makes AI code testable.
6
Calls can fail: network errors, timeouts, rate limits (a cap on how many calls you may make per minute; Topic 16.4), invalid requests (HTTP status 400; status codes are in Topic 15.1), and overloaded servers. Always set a timeout, and wrap the call so the rest of your program deals with your own error types, not each SDK's.

Explain it without notes

Why must you send the whole conversation on every call?

What are the main differences between the OpenAI-compatible and Anthropic Messages request and response shapes?

How do you make code that calls an LLM testable?

A model reply arrives but your JSON parser fails on it. Before blaming the prompt, what do you check in the response object?

Practice

Write build_messages(system, history, user) that returns the full list with the system message first and the new user message last.

Write openai_text(response) and anthropic_text(response) that pull the answer text out of each shape (as plain dictionaries).

Write load_key(env, name) that returns the key from a dictionary of settings, or raises RuntimeError with a helpful message.

Trade-offs

↔
SDK versus plain HTTP. An SDK gives typed objects, retries and streaming helpers, but ties you to its version and release pace. Plain httpx is a few more lines and fully under your control.
↔
One provider versus a portable layer. A thin adapter of your own makes switching and fallback easy. A big abstraction layer can hide useful provider-specific features and become another thing to maintain.

Done when you can

Done when you can write a messages list with system, user and assistant roles.
Done when you can explain why the full history is resent each call.
Done when you can name where the system text and the answer text live in both API shapes.
Done when you can write a fake client and use it to test a function that calls a model.
Done when you load keys from the environment and can explain why.
Done when you know a full Hectal course on LLM engineering is planned, going deeper into providers, options and production practice.

Command Palette

Words you'll meet

Step by step

01The request is plain data

02Real SDK code, for when you have a key

03The same call with the Anthropic Messages API

04A fake client with the same interface

05The history grows each turn

06Where this shows up in real systems

Try it yourself

Add a third turn

Trigger the role check

Code & diagrams

Break it on purpose

A message with no role

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can