PHASE 4Intermediate ~22 min· topic 6 of 10

Topic 4.6

Metadata That Pays Off

In one line

Metadata is the labelled information stored next to every chunk: source, section, date, version, language and who may read it. Each field exists because a later step needs it, and a schema that is checked at ingestion keeps bad records out.

Think of it like this

Moving boxes. The contents are what you care about, but the labels on the outside (Kitchen, Fragile, Open first, Owner: Sam) decide where each box goes, who may open it and what to throw away. Metadata is the label on each chunk. Without labels a box of chunks is a heap.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Metadata: Information about a chunk (source, date, language, access) stored next to it but not part of its text.
Schema: The list of fields every chunk must have, with their types and allowed values.
Validation: Checking a record against the schema and rejecting it with a clear message if it breaks the rules.
Filter: A condition on metadata (lang = en) that removes chunks before ranking.
Freshness: Preferring the newest version of a source.
Access label: A field saying who may read a chunk, such as public or staff.
Citation: A reference to the source, section and date of the text an answer used.

Step by step

01Step 1: write the schema down

List each field, its type and what will read it. A field with no reader is a cost with no benefit.

schema.pywhole filepython

SCHEMA = {
    "source":  "str, file or url, used for citations and deletion",
    "section": "str, heading, used for citations",
    "updated": "str, YYYY-MM-DD, used for freshness and date filters",
    "version": "int >= 1, used to keep the newest",
    "lang":    "str, used by language filters",
    "access":  "public | staff, used to filter before retrieval",
}

02Step 2: validate every chunk as it is made

Return a list of problems instead of raising on the first one, so a whole batch can be reported at once. Reject or quarantine records with problems.

03Step 3: filter by access and language first

A function that keeps only chunks the reader may see in the requested language. It runs before any ranking.

Step 3: filter by access and language firstdiagram

Rendering diagram…

04Step 4: keep only the newest version

Group by source, keep the highest version. In the example that removes c1old from the public reader's results and leaves c1.

05Step 5: build a citation from metadata

refunds.md > Booking (updated 2026-03-01, version 2). Put the date in: readers trust a dated source more, and can spot an old one.

06Step 6: delete by source

When a document goes, remove all chunks whose source matches. The example ends with three chunks left after refunds.md is deleted.

Try it yourself

1
Add a rule
In the first example require lang to be en or fr and print a message otherwise. Add a chunk with lang set to de and check that it is reported.
2
Be the staff reader
In the second example call allowed for the staff reader and French (fr). What do you get? Why is newest_only still needed?
3
Date filter
Write a filter that keeps chunks updated on or after 2026-02-01. Because the dates are ISO strings, a plain >= works. Which chunks survive?

Code & diagrams

A metadata schema that is checked New tab

In Java the metadata is a Map<String, Object>, so the type checks use instanceof. A real project would use a typed class.

Expected output

chunk 1: ok
chunk 2: missing source; access must be public or staff, got internal
chunk 3: version must be a whole number from 1; updated must look like 2026-03-01, got 1 March 2026

Filters, freshness, citations and deletion New tab

Dates are ISO strings so they compare correctly as text. c4 is French and c2 is staff-only; c1old is an outdated version of c1.

Expected output

all:                    c1 c1old c2 c3 c4
public reader, English: c1 c1old c3
  newest version only:  c1 c3
staff reader, English:  c1 c2 c3
citation: refunds.md > Booking (updated 2026-03-01, version 2)
after deleting refunds.md: c2 c3 c4

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Let everyone read staff chunks

In the Python tab of the second example, change reader == "staff" to reader != "". Run.

terminal

$ Run

── what you'll see ──

all: c1 c1old c2 c3 c4

public reader, English: c1 c1old c2 c3

newest version only: c1 c2 c3

staff reader, English: c1 c2 c3

citation: refunds.md > Booking (updated 2026-03-01, version 2)

after deleting refunds.md: c2 c3 c4

Myth vs fact

Myth

Metadata is optional decoration.

Fact

Filters, citations, freshness, deletion and access control all depend on it. Adding it after the index is built means re-ingesting everything.

Myth

The model can sort out old and new versions by itself.

Fact

It sees two conflicting passages and may quote either. Remove the old version before the prompt is built.

Myth

Telling the model not to reveal staff-only text is an access control.

Fact

If the text is in the prompt the model has read it. Access control means the text is never retrieved for that user.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Treat the metadata schema as an API: version it, validate on write, and keep a migration path. A field renamed halfway through an ingestion leaves you with half the chunks unfilterable.
▸
Store dates as ISO 8601 strings (2026-03-01) or as numbers. Free-text dates (1 March 2026) cannot be compared or filtered, as the third test chunk shows.
▸
Store group ids, not user lists, in the access field, and resolve who belongs to a group at query time. If you copy a list of people into every chunk, a permission change means re-writing millions of chunks, and until it finishes a person who lost access can still read them. For several customers sharing one index, make a tenant id a required field and a mandatory filter (Topic 10.4).
▸
Keep metadata small and free of secrets. It travels with every search result and often with logs and traces. Reference sensitive details by id instead of copying them in.

Remember this

1
Why this exists. Search finds text that looks like the question. But some questions need more than that: only chunks the current user may see, only the newest version of a policy, only English, and a citation that says exactly where the answer came from. Those are facts about the chunk, not about its words, so they live in separate fields that filters, citations and deletion can use without reading any text.
2
A starter schema for AskDocs. source (file or URL, for citations and deletion), section (heading, for citations), updated (date as YYYY-MM-DD, which also sorts correctly as text), version (a whole number from 1, for keeping the newest), lang (en), access (public or staff). Later phases add the chunker version and the embedding model name (Topic 4.8). Every field has a reason; if you cannot say what will use a field, do not add it.
3
Validate at ingestion. The first example checks each chunk's metadata before it is stored: required fields present, version a positive whole number, access one of the known labels, updated in the exact date format. The three test chunks produce: ok; missing source; access must be public or staff, got internal; and version must be a whole number from 1; updated must look like 2026-03-01, got 1 March 2026. A bad record is rejected with a message you can act on, instead of stored and failing weeks later in a filter.
4
What each field is used for later. Filters (Phase 5): lang, access, date ranges. Citations (Phase 8): source and section, as in refunds.md > Booking (updated 2026-03-01, version 2). Freshness: keep only the newest version of each source, so an old rule and a new rule never arrive in the same prompt (the example drops c1old, which says 24 hours, when c1 says 2). Deletion: when a document is removed, delete every chunk with that source (Topic 4.7). Access control: show a reader only chunks their role allows.
5
Security: filter before retrieval, not after. The example removes staff chunks for a public reader before anything is ranked or shown. If you retrieve first and filter afterwards, restricted text has already been loaded next to the model, and a bug in the filter leaks it. Filtering inside the store's query (Topic 5.5) also avoids a ranking problem: filtering after the top results can leave fewer than k results (Topic 3.7). Labels are a mechanism, not a policy; the real rule comes from your access system (Phase 10).
6
Metadata is not embedded. Unlike the heading path (Topic 4.5), fields such as updated and access are stored next to the vector but are not part of the embedded text. That is why changing them does not need a new embedding, and why they are cheap to update (Topic 4.7).

Explain it without notes

Name four metadata fields and what each is used for later.

Why validate metadata when chunks are created and not when they are searched?

Why filter by access before ranking rather than after?

Why is metadata usually not part of the embedded text, and when is an exception sensible?

Practice

Write newest(chunks) for a list of (source, version) pairs that keeps only the pair with the highest version for each source, preserving order. Print the result for [("a", 1), ("b", 1), ("a", 2)].

Write is_iso_date(text) that returns True when the text looks like YYYY-MM-DD (digits only, with dashes in the right places). Print it for "2026-03-01" and "1 March 2026".

Trade-offs

↔
More metadata fields: more filters and better citations, more to validate and keep correct.
↔
Strict validation: clean data, but a bad file can block a run. Quarantine bad records and carry on.
↔
Keeping old versions: useful for history and rollback, harmful if they reach the prompt. Keep them in the store with a flag, and filter them out of search.

Done when you can

Done when you can write a metadata schema and say what reads each field.
Done when you can validate records and report all problems at once.
Done when you can filter by access, language and newest version before ranking.
Done when you can build a citation and delete all chunks of a source.

Command Palette

Words you'll meet

Step by step

01Step 1: write the schema down

02Step 2: validate every chunk as it is made

03Step 3: filter by access and language first

04Step 4: keep only the newest version

05Step 5: build a citation from metadata

06Step 6: delete by source

Try it yourself

Add a rule

Be the staff reader

Date filter

Code & diagrams

Break it on purpose

Let everyone read staff chunks

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can