Topic 11.2
Freshness
In one line
An index is a photograph of the documents at some moment. Freshness is how old that photograph may be when a question arrives. You decide the allowance, measure the lag between an edit and its appearance in answers, and show the reader the "as of" date.
Think of it like this
A bus timetable on a station wall is printed on Monday. On Wednesday the 8:15 bus is cancelled, but the wall still shows it. Nobody lied; the paper is simply older than the world. A RAG index is that paper timetable. The question is how long it may stay on the wall before someone is standing at the stop for a bus that is not coming.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Freshness
- How old the information behind an answer is compared with the source.
- Index lag
- The time between a document changing and the index containing the change.
- Change feed
- A stream of events ("document D2 changed") the ingestion path listens to.
- Schedule
- A job that runs at fixed times, such as nightly, and handles everything changed since the last run.
- As of
- The date or time the shown information was last known to be current.
- Tick
- A counter step used in the examples to stand for time. Real systems use real timestamps.
- Staleness allowance
- The longest lag you accept for a kind of content.
Step by step
01Step 1: write the allowance down
For each content type, one number. Example: hours and prices 60 minutes, policies 24 hours, archives 7 days. Without a number, "fresh enough" is a feeling, and feelings cannot trigger an alert.
02Step 2: record two timestamps per document
The source's updated_at is when the world changed. Your indexed_at is when the index caught up. Carry both on every chunk. Their difference is the lag for that document.
lag = chunk["indexed_at"] - chunk["updated_at"]03Step 3: try the schedule
A job at ticks 24, 48, 72 indexes everything changed before it. An edit at tick 5 waits 19 ticks, one at tick 50 waits 22. The worst lag is almost a whole period, and it is the one a user meets if they ask just after an edit.
04Step 4: try the change feed
Each edit becomes an event, a worker handles it 2 ticks later. Lag is the queue wait plus the work, so it stays small until the queue backs up (Topic 11.7). The cost is a queue, a worker and the code to handle duplicates and out-of-order events.
05Step 5: keep a slow schedule as a safety net
Events get lost: a webhook failed, a worker crashed. Run a slow full re-check (weekly, say) that compares source versions with indexed versions and repairs the differences. Events give speed, the schedule gives certainty.
06Step 6: show it to the reader
Put "as of" next to each source in the answer. In the example, at tick 40 a question about D2 (edited at tick 30) is answered from tick 0 under the schedule and from tick 30 under the feed. Showing the date turns a silent error into a visible one.
Try it yourself
- 1
Change the period
Change the schedule from 24 to 12 ticks (change the
23to11and the24to12innightly: the formula is(tick + period - 1) // period * period, so the next run after tick 5 is tick 12). Predict the worst lag, then run. What does a faster schedule cost in a real system? - 2
Slow the feed
Make the feed worker take 30 ticks (a backlog). Which policy is better now? Notice that the feed's advantage depends on the queue staying short.
- 3
Add an edit
Add
("D3", 70)toEDITS. At tick 74 what does each policy say about it? Add a third line asking about D3.
Code & diagrams
Time is a counter, never a real clock, so the three languages agree. The allowance of 20 ticks is an example value.
Expected output
nightly lag: D8=19, D2=18, D6=17, D1=22 | worst 22 | within allowance: no
feed lag: D8=2, D2=2, D6=2, D1=2 | worst 2 | within allowance: yes
tick 40 nightly: D2 answer is as of tick 0, D1 answer is as of tick 0
tick 40 feed: D2 answer is as of tick 30, D1 answer is as of tick 0
tick 74 nightly: D2 answer is as of tick 30, D1 answer is as of tick 50
tick 74 feed: D2 answer is as of tick 30, D1 answer is as of tick 50Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Round the schedule the wrong way
In the Python tab, change return (tick + 23) // 24 * 24 to return tick // 24 * 24 and run.
Myth vs fact
Myth
Re-indexing everything every night keeps answers fresh.
Fact
It keeps them at most a day old, and a day is long for a price or an opening time. Nightly is a safety net, not a freshness plan.
Myth
A change feed is always better than a schedule.
Fact
It is faster, but it needs a queue, retries, deduplication and a repair job for lost events. For documents that change monthly, a nightly job is the right amount of machinery.
Myth
If the answer cites a document, it is up to date.
Fact
A citation proves where the text came from, not when. Show the "as of" date to let the reader judge.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Measure lag at the percentile that matters: p50 tells you the pipeline works, the maximum tells you a document is stuck. A dashboard that shows only the average hides the stuck document.
- ▸
Handle out-of-order events: two edits of the same document can arrive reversed. Compare the event's source version with the version already indexed and ignore older ones, or you will index the older text over the newer.
- ▸
Content with a hard date (an offer that ends Friday) is better handled by metadata filters (Topic 4.6 covers the metadata fields) than by freshness alone: the chunk carries
valid_untiland retrieval skips it after that moment. - ▸
Cached answers add to the lag: an answer cached for an hour is up to an hour older than the index. Topic 11.3 ties cache invalidation to document changes so the two do not add up.
Remember this
- 1
Why this matters. A question-answering system sounds equally sure whether its source is a minute old or a month old. A refund policy changed on Friday, the index still has Thursday's text, and the system confidently quotes the old rule. Users trust the answer because it cites a document. Staleness is therefore a correctness bug, and it is invisible unless you measure it.
- 2
Decide the allowance first. Different content has different needs. Opening hours on a public holiday: minutes. A price list: an hour. An annual report: a day or a week. Write the allowance down per content type as a number of minutes (the example uses "ticks", a counter standing in for time). Then the rest of the topic is engineering to meet it.
- 3
Measure index lag. For each edit, index lag is the time from "the document changed" to "a question can see the change". Record both timestamps: the source system's
updated_atand the time your ingestion path wrote the new index version. The gap is the lag. Watch its worst case, not its average: one stuck document hurts one user a lot. - 4
Two ways to trigger ingestion. A schedule (a nightly job re-reads everything or everything changed) is simple and cheap, but its lag is up to the whole period. A change feed (webhooks, a database change stream, a modified-since poll every minute) puts each change on a queue and a worker handles it a few moments later. The example compares them with a clock that is only a counter: the schedule's worst lag is 22 ticks, the feed's is 2.
- 5
Show "as of". Because freshness is never perfect, say so in the answer: "Source: Refund policy, updated 3 Oct, indexed 3 Oct 14:02". It lets a user judge the answer and lets support explain a wrong one. It costs one metadata field per passage (the
updated_atandindexed_atcarried through ingestion, Phase 4) and one line in the prompt or the UI. - 6
Deletes and permissions are freshness too. A withdrawn policy that stays in the index, or a document whose access was removed and still gets quoted, is a stale answer with worse consequences. Process them on the same fast path as edits, never on the nightly one. Topic 11.9 turns a stale-answers report into a drill.
Explain it without notes
What is index lag, and how would you measure it?
Compare a nightly schedule with a change feed for keeping an index fresh.
Why show an "as of" date in an answer?
Why must deletes and permission changes take the fast path, not the nightly one?
Practice
Write pending(edits, indexed_at, now) returning the sorted list of documents that were edited at or before now but whose edit is not yet indexed. Try it with the schedule from the example at tick 40.
Write as_of_label(as_of_tick, now) returning text such as "as of tick 30 (10 ticks ago)", and "as of the original (never edited)" when as_of_tick is 0.
Trade-offs
- ↔
Frequent ingestion: fresher answers, more embedding cost and more load on the index (Topic 11.5).
- ↔
A change feed: low lag, but a queue and a repair job to run.
- ↔
A strict allowance: fewer stale answers, more alerts and more engineering effort for content that rarely matters.
Done when you can
Done when you can state a freshness allowance for a type of content as a number.
Done when you can compute index lag from two timestamps and explain why the worst case matters.
Done when you can compare a schedule and a change feed, including what each costs to run.
Done when you can explain why an "as of" date helps and where the data for it comes from.
Done when you can say why deletes and permissions need the fast path.