PHASE 2Beginner ~21 min· topic 8 of 10

Topic 2.8

Calling Embedding Models

In one line

A hosted embedding model is one HTTP call: send a list of texts, get back a list of vectors plus a token count. Batch the texts, never trust the order of the rows, handle rate limits, and know the local alternatives (sentence-transformers, Transformers.js).

Think of it like this

A print shop with a counter. You hand over a stack of pages with a note saying which job each one belongs to, and later collect printed copies. Handing in one page at a time wastes trips. Handing in 500 at once may be refused. And the copies may come back in a different order, which is why each one carries a job number.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Endpoint: The URL a program sends a request to, such as /v1/embeddings.
Batch: Several inputs sent in one request.
Usage: The part of the response that says how many tokens were billed.
Rate limit: A cap on how many requests or tokens you may send per minute. Going over gives HTTP 429.
Backoff: Waiting longer after each failed attempt before trying again.
OpenAI-compatible: Using the same request and response shape as OpenAI's API, so one client works with several providers.
Bearer token: A secret key sent in the Authorization header to prove who you are.

Step by step

01Step 1: build the request body

A model name, a list of inputs and the format. In the example the body for the first batch is {"model":"fake-embed-1","input":["Can I cancel a ride?","Forgot password"],"encoding_format":"float"}.

request.pywhole filepython

request = json.dumps(
    {"model": MODEL, "input": batch, "encoding_format": "float"},
    separators=(",", ":"),
)

02Step 2: send it, read the response

The fake endpoint in the example stands in for the network. It returns the data rows in reverse order and a usage object. A real client sends the body with an HTTP library and a header with the key.

Step 2: send it, read the responsediagram

Rendering diagram…

03Step 3: sort by index

The first request comes back as rows 1, 0. Sorting by index puts them right. This is one line in the client, and it prevents the nastiest bug in this area: every vector attached to the wrong text, with nothing failing.

04Step 4: batch the whole corpus

Split the texts into groups of at most BATCH_SIZE. Five texts in groups of two are three requests: 2, 2, 1. Add up usage.total_tokens to count what was billed: 16 tokens here.

05Step 5: handle failure

A real call can fail. Retry 429 and 5xx errors with backoff (Topic 2.10), fail fast on 400 and 401, and log the request id the provider returns. Do not retry forever.

06Step 6: choose where it runs

Hosted API for strength and no hardware, local model for privacy and no per-token fee. Put the choice behind one Embedder interface (Topic 2.9) so the rest of the product never knows.

Try it yourself

1
Change the batch size
Set BATCH_SIZE to 3 and to 5. How many requests are made each time? What would a batch size of 1 cost on 10,000 texts if each request takes 100 ms?
2
Break the order on purpose
Remove the line that reverses the rows in the fake server and run. The output of the last section stays the same. Why does a correct client still need the sort?
3
Read the bill
Add a line that prints the tokens billed per batch. Which batch is the biggest? How would you estimate the cost of a corpus before sending it?

Code & diagrams

An embeddings request and response, by hand New tab

The server is a fake that returns three-number vectors. Real vectors have hundreds of numbers, but the request, the response, the reversed rows and the batching are exactly what you code against. Real Java projects use Jackson or Gson for JSON, and the course's Json helper keeps this runnable.

Expected output

request body: {"model":"fake-embed-1","input":["Can I cancel a ride?","Forgot password"],"encoding_format":"float"}
request 1: 2 in batch, rows came back in order 1,0
request 2: 2 in batch, rows came back in order 1,0
request 3: 1 in batch, rows came back in order 0
5 vectors of 3 numbers, 16 tokens billed
Can I cancel a ride? -> [0.50, 0.60, 0.50]
Forgot password -> [0.20, 0.10, 0.50]
Is a helmet included? -> [0.40, 0.00, 0.50]
Monthly pass price -> [0.30, 0.40, 0.50]
Lost items -> [0.20, 0.30, 0.50]

A model on your own machinepython

Not run here: it needs a package and a model download. The Python and JavaScript calls follow the sentence-transformers and Transformers.js documentation. The Java file is the plain-HTTP shape for a local server, since Java has no one standard library for this.

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
vectors = model.encode(["Can I cancel a ride?", "Forgot password"], normalize_embeddings=True)
print(vectors.shape)  # (2, 384)

A hosted OpenAI-style embeddings callpython

Not run here: it needs a key and the network. The request and response shape follow OpenAI's embeddings guide: model, input, optional dimensions (text-embedding-3 models), data rows with index and embedding, and usage. For another provider, change the URL, the model name and the key variable, and check its limits and prices.

import os
import httpx

response = httpx.post(
    "https://api.openai.com/v1/embeddings",
    headers={"Authorization": "Bearer " + os.environ["OPENAI_API_KEY"]},
    json={"model": "text-embedding-3-small", "input": ["Can I cancel a ride?", "Forgot password"], "dimensions": 256},
    timeout=30,
)
response.raise_for_status()
rows = sorted(response.json()["data"], key=lambda row: row["index"])
vectors = [row["embedding"] for row in rows]

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Trust the order of the rows

In the Python tab, change rows = sorted(response["data"], key=lambda row: row["index"]) to rows = response["data"] (trust the order of the rows). Run.

terminal

$ Run

── what you'll see ──

request body: {"model":"fake-embed-1","input":["Can I cancel a ride?","Forgot password"],"encoding_format":"float"}

request 1: 2 in batch, rows came back in order 1,0

request 2: 2 in batch, rows came back in order 1,0

request 3: 1 in batch, rows came back in order 0

5 vectors of 3 numbers, 16 tokens billed

Can I cancel a ride? -> [0.20, 0.10, 0.50]

Forgot password -> [0.50, 0.60, 0.50]

Is a helmet included? -> [0.30, 0.40, 0.50]

Monthly pass price -> [0.40, 0.00, 0.50]

Lost items -> [0.20, 0.30, 0.50]

Myth vs fact

Myth

The vectors come back in the order I sent the texts.

Fact

Only by luck. The response carries an index on every row for exactly this reason: use it.

Myth

One request per text is fine.

Fact

It multiplies the overhead and the chance of hitting a rate limit. Batch within the provider's limits.

Myth

Every provider's embedding API is the same.

Fact

Many copy OpenAI's shape, but model names, limits, optional parameters and prices differ. Read the documentation of the one you use.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Log the model name, the number of inputs, the tokens used and the latency of every batch. Embedding jobs are where bills and slowdowns hide, and the usage field makes the cost measurable per job.
▸
Make embedding jobs idempotent and resumable: if a job over a million chunks dies at 70 percent, you want to restart from the cache (Topic 2.9) instead of paying again for the first 700,000.
▸
A single too-long input can fail a whole batch with a 400 error. Count tokens first (Topic 2.6) and handle over-long chunks before they reach the API, not in the error handler.

Remember this

1
Why this exists. Topic 2.7 chose a model. Now your program has to *use* it, thousands of times, from three languages and under the provider's limits. Most hosted providers copy the same request shape, so one client works for many of them. Knowing that shape also tells you what to log, what to retry and what to count for the bill.
2
The request. An OpenAI-style POST /v1/embeddings call has a JSON body with model (which model), input (one string, or a list of strings) and optionally encoding_format (such as "float") and dimensions (the number of numbers you want, supported by OpenAI's text-embedding-3 models). The key goes in an Authorization: Bearer header and comes from an environment variable, never from the code. Many other providers accept the same shape; check their documentation for names and limits.
3
The response. A JSON object with object: "list", a data array, the model, and usage with prompt_tokens and total_tokens. Each element of data is { "object": "embedding", "index": 0, "embedding": [numbers] }. Use index to match each vector with its input: do not assume the rows come back in order. The example makes its fake server return them reversed on purpose, and the client sorts by index.
4
Batching. One request can carry many inputs. That saves the fixed cost of each call (connection, headers, waiting), so embedding 500 texts takes one call instead of 500. Providers cap how many inputs and how many tokens one request may carry, and each single input has the model's length limit (for OpenAI's embedding models the guide lists 8192 tokens). Read your provider's limits, set a batch size below them, and split bigger jobs. In the example, five texts with a batch size of two take three requests.
5
Rate limits and errors. A provider limits requests and tokens per minute. When you exceed it, the call fails with HTTP status 429. The right reaction is to wait and retry, with growing waits (backoff), not to hammer. Network errors and 5xx server errors deserve the same treatment, while 4xx errors such as a wrong key or a too-long input will never succeed on retry (LLM Engineering Phase 6 builds the full retry). Topic 2.10 has a small version.
6
Running locally. You do not need an API. The Python library sentence-transformers loads an open model with SentenceTransformer(...) and encodes lists of texts with .encode(...). In JavaScript, Transformers.js (@huggingface/transformers) runs a model with pipeline("feature-extraction", ...), in Node or in the browser. This course's search lab does the latter, with the 8-bit version of all-MiniLM-L6-v2 in a Web Worker, one text at a time because batching in padded groups changes the quantised model's numbers slightly. Local use costs no tokens but needs CPU or GPU.

Explain it without notes

What does a request to an OpenAI-style embeddings endpoint contain, and what comes back?

Why must a client sort the response rows by index?

How do batching and rate limits interact, and how should a client react to a 429?

When would you run an embedding model locally instead of calling an API?

Practice

Write make_batches(items, size) that splits a list into lists of at most size items, and print the batches for [1, 2, 3, 4, 5] with size 2.

Write restore_order(rows) that takes response rows like {"index": 1, "embedding": [0.1]} in any order and returns the embeddings sorted by index. Print the result for rows with indexes 1, 2, 0.

Trade-offs

↔
Larger batches: fewer requests and less overhead, but a bigger blast radius when one input fails and a higher chance of hitting token limits.
↔
Hosted API: strong models and no hardware, but a per-token cost, rate limits and your text leaving your system.
↔
Local model: private and free per call, but you run and update it.

Done when you can

Done when you can write an embeddings request and read its response by hand.
Done when your client sorts rows by index.
Done when you know your provider's batch and input limits and have set a batch size below them.
Done when you retry 429 and 5xx with backoff and do not retry other 4xx errors.
Done when you can run a model locally in at least one language.

Command Palette

Words you'll meet

Step by step

01Step 1: build the request body

02Step 2: send it, read the response

03Step 3: sort by index

04Step 4: batch the whole corpus

05Step 5: handle failure

06Step 6: choose where it runs

Try it yourself

Change the batch size

Break the order on purpose

Read the bill

Code & diagrams

Break it on purpose

Trust the order of the rows

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can