Command Palette

Search for a command to run...

Hectal
Back to the lesson: Topic 9.8 — Evaluating in Production
RAG and Vector Search · Example 1 of 1

From live sessions to rates and golden-set candidates

The ten sessions are made-up data. The sample uses the course's generator (state = state * 48271 % 2147483647, seed 42) and takes state % pool size, so all three languages pick the same two questions.

A golden set is made of yesterday's questions. Real users bring new ones, so the product also needs signals from live use: thumbs with reasons, how often it refuses, which sources get opened, and a sample of real traffic that feeds the golden set.

Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.

Ready for more? This phase has 5 hands-on exercises with automatic checks.

Practice questions

Write the code in the editor, run it, then open the model answer to compare.

01

Write rate(part, whole) returning the fraction as a 3-decimal string and raising a clear message when whole is 0. Print rate(4, 7).

02

Write top_reasons(reasons, n) that returns the n most common reasons, ties broken by first appearance. Print the top 2 for a list of five reasons.

Explain it without notes

01

Name three online signals and what each can and cannot tell you.

02

Why is the thumbs-down share computed over votes, and what bias remains?

03

How does production traffic improve the golden set?

04

Why can a very low no-answer rate be a warning sign?

From live sessions to rates and golden-set candidates
Sign in to run this example in your browser.

Expected output

sessions: 10
no-answer rate:        0.200
vote rate:             0.700
thumbs-down share:     0.571 of votes
opened a source:       0.625 of answered
down reasons: wrong source 2, no answer 1, out of date 1
add to the golden set: I forgot my login, how do I get back in? | Do you rent scooters? | Is there a student discount? | I scratched the bike, will I be charged? | Someone took my bike, what do I owe?
sampled for review:    How far can I go on the electric one? | Can I get my money back if I cancel?