PHASE 4Intermediate ~21 min· topic 8 of 10

Topic 4.8

Ingestion as a Pipeline

In one line

Ingestion is a pipeline of stages. Embedding calls go out in batches, failures are retried with waits, one failed batch must not stop the rest, and changing the chunker or model means a planned backfill.

Think of it like this

Moving house with a van. You pack boxes (a stage), load them in groups that fit the van (batches), and drive to the new house (another stage). If one trip fails because of a flat tyre you try again after fixing it, but the boxes already delivered stay delivered. If one box is lost, you write it down and carry on with the rest. A pipeline is that way of working for chunks.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Pipeline: A fixed series of stages where each stage's output feeds the next.
Batch: A group of items sent in one request.
Retry: Trying a failed operation again.
Exponential backoff: Waiting longer after each failure, for example 100 ms, 200 ms, 400 ms.
Partial failure: Some items succeed and some fail in the same run.
Dead-letter list: A record of items that could not be processed, kept for a later retry or for a person to look at.
Backfill: Rebuilding stored records so they match a new recipe or version.
Queue: A line of work items waiting between two stages.

Step by step

01Step 1: draw the stages

Write down each stage's input and output. If two stages share hidden state, split them or make the state explicit.

Step 1: draw the stagesdiagram

Rendering diagram…

02Step 2: send chunks in batches

Slice the list into groups of BATCH. The last group may be smaller. Number the batches so logs can point at one.

03Step 3: retry a failed batch

Wrap the call. On failure, wait BACKOFF_MS * 2 ** (attempt - 1) and try again, up to TRIES. Count the waiting on a simulated clock so tests do not sleep.

retry.pywhole filepython

for attempt in range(1, TRIES + 1):
    try:
        vectors = embed_batch(number, texts)
        break
    except RuntimeError:
        if attempt == TRIES:
            failed.extend(texts)      # give up on this batch only
        else:
            clock_ms += BACKOFF_MS * 2 ** (attempt - 1)

04Step 4: record failures and carry on

A failed batch is data, not a reason to stop. Put its chunk ids in a list that the report prints and the next run retries.

05Step 5: stamp every record with the recipe

Add chunker, context_format and embedding_model to the metadata. A record without a stamp cannot be judged stale or fresh.

06Step 6: backfill without downtime

Build the new records in a new collection (Topic 5.6), compare quality on your questions, then switch the application over and remove the old collection.

Try it yourself

1
Change the batch size
Set BATCH to 5. How many batches are there now, and which chunks are in the failing batch? Predict the to retry next run line.
2
Give up sooner or later
Set TRIES to 5, then to 1. Write down the simulated waiting time each time. What is the cost of trying harder on a batch that never works?
3
Make the service recover
Change FAILURES to {2: 1, 3: 2}. With three tries, batch 3 now succeeds on its last attempt. Check the stored count and the waiting time.

Code & diagrams

Batches, retries, a batch that never works, and a stale check New tab

The embedding service is a fake that fails on a schedule, and time is a counter, so the run is instant and identical in every language. Real clients also need jitter (random spread) in their waits.

Expected output

batch 1 (4 chunks): ok on attempt 1
batch 2 (4 chunks): ok on attempt 2
batch 3 (2 chunks): gave up after 3 attempts
stored 8 of 10, to retry next run: c8, c9
time spent waiting: 400 ms (simulated)
backfill: 8 of 8 stored chunks were made by an older chunker

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Retry after a success

In the Python tab, remove the line break that follows the ok on attempt print. Run.

terminal

$ Run

── what you'll see ──

batch 1 (4 chunks): ok on attempt 1

batch 1 (4 chunks): ok on attempt 2

batch 1 (4 chunks): ok on attempt 3

batch 2 (4 chunks): ok on attempt 2

batch 2 (4 chunks): ok on attempt 3

batch 3 (2 chunks): gave up after 3 attempts

stored 8 of 10, to retry next run: c8, c9

time spent waiting: 400 ms (simulated)

backfill: 8 of 8 stored chunks were made by an older chunker

Myth vs fact

Myth

If a call fails, retry forever until it works.

Fact

A batch that never works (too large, malformed, a rejected input) will block everything. Limit the tries and move on with a record of the failure.

Myth

Bigger batches are always faster.

Fact

Batches are limited by the provider, and one failure in a huge batch wastes more work. Moderate batches with retries are more robust.

Myth

You can mix old and new vectors in one index while migrating.

Fact

Vectors from different models are not comparable. Build a second index and switch over.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Idempotency makes retries safe: because the sync decides what to write by content id, repeating a batch writes the same records. Without it, a retry after a timeout (when the first call actually succeeded) creates duplicates.
▸
Recall from LLM Engineering Topic 6.8: retry only errors that can succeed later (timeouts, rate limits, server errors). A batch rejected as too large or malformed will fail the same way every time, so split it and try the halves instead of waiting and repeating.
▸
Log per batch: chunk count, tokens, attempts, latency and the provider's error text. Failures cluster (one oversize chunk, one rate-limit burst), and the log is how you see the cluster. LLM Engineering Topic 6.10 covers running many calls at once under a limit.
▸
Treat a model or chunker change as a project: estimate the cost before starting (for example 200,000 chunks of 300 tokens is 60 million tokens, which you multiply by your provider's price per million tokens), run it on a copy of the data, and evaluate retrieval on a golden set (Topic 9.2) before switching.

Remember this

1
Why this exists. The embedding step calls a model, usually over the network, and that call has a fixed overhead and limits on how much text one request may carry. Sending one chunk per request multiplies the overhead: 10,000 chunks in batches of 100 are 100 calls, not 10,000. And networks fail: a batch will sometimes time out or be refused for sending too many requests too fast. A loop that assumes every call succeeds works in a demo and breaks on the first large corpus.
2
The stages. load (Topic 4.1), clean (4.3), chunk and add context (4.4, 4.5), attach metadata (4.6), embed, upsert (4.7). Each stage takes the previous stage's output and produces the next. For most corpora a simple loop through the stages is enough. For large ones, stages become separate workers with a queue between them, so a slow embedding step does not hold up loading, and work can be spread over machines.
3
Batching. Group chunks into batches of a fixed size and send each batch as one request. The right size is set by the provider's limits and your memory, not by a magic number: check your provider's documentation for the maximum input per request, and keep a margin. The example uses batches of 4 for 10 chunks, so 3 batches: 4, 4 and 2.
4
Retries with waits. If a batch fails, wait and try again, up to a limit of tries, doubling the wait each time (exponential backoff: 100 ms, then 200 ms). LLM Engineering Topic 6.8 builds the full version with jitter and Retry-After; this example keeps the idea. The clock in the example is simulated (a counter), so it prints instantly: batch 2 fails once and succeeds on attempt 2 after a 100 ms wait.
5
Partial failure. Batch 3 in the example fails every time. After 3 attempts the pipeline gives up on that batch only, records its chunks as failed and continues. The summary line says stored 8 of 10, to retry next run: c8, c9 after 400 ms of simulated waiting. Because the sync from Topic 4.7 is idempotent, the next run notices that c8 and c9 are not stored and tries only those. A pipeline that aborts everything on one error would have thrown away eight good embeddings, and one that ignores the error would have silently lost two chunks.
6
Backfill when the recipe changes. Chunks are made by a recipe: a chunker version, a context format, an embedding model. Change any of them and the stored chunks are out of date. Stamp every record with the versions that made it (the example uses chunker-v1), then find the stale ones with one comparison: 8 of 8 here. Rebuild them in the background into a new index while the old one keeps serving, then switch over. Never mix vectors from two different embedding models in one index: distances between them mean nothing.

Explain it without notes

Why batch embedding requests, and what limits the batch size?

Explain exponential backoff and what the retry loop in the example does.

What should a pipeline do when one batch fails permanently?

How do you handle a change of embedding model for a corpus that is already indexed?

Practice

Write batches(items, size) returning a list of lists. Print the batches for [1, 2, 3, 4, 5, 6, 7] with size 3, one per line, joined by commas.

Write backoff_total(failures, base_ms) returning the total wait for a call that fails failures times before succeeding, with waits base_ms, 2 * base_ms, and so on. Print it for 3 failures with a base of 100.

Trade-offs

↔
More tries per batch: fewer failed chunks, but longer runs and more load on a service that may already be struggling.
↔
Queues between stages: smoother throughput and scaling, more moving parts to operate.
↔
Eager backfill after each change: always consistent, expensive. Lazy backfill by priority: cheaper, but the index is mixed for a while.

Done when you can

Done when you can draw the ingestion stages and say what each takes and returns.
Done when you can batch, retry with backoff and give up on one batch without stopping the run.
Done when you can produce a run report with stored count and failed ids.
Done when you can plan a backfill after a model or chunker change.

Command Palette

Words you'll meet

Step by step

01Step 1: draw the stages

02Step 2: send chunks in batches

03Step 3: retry a failed batch

04Step 4: record failures and carry on

05Step 5: stamp every record with the recipe

06Step 6: backfill without downtime

Try it yourself

Change the batch size

Give up sooner or later

Make the service recover

Code & diagrams

Break it on purpose

Retry after a success

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can