Find images by caption and by the text inside them
Word overlap stands in for a real search, and the OCR text is typed in by hand. The point is what gets indexed.
Images can be searched by embedding them next to text, or, as a plain baseline, by indexing their captions and the text found inside them. The baseline is cheap, explainable and often enough.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write index_text(caption, ocr) that returns the caption and the text found in the image joined by one space, lower-cased, with no trailing space when there is no text. Print the result inside square brackets for an image with text and an image without.
Write best_image(question, images) where images is a list of (id, text) pairs. Return the id with the most shared words, or None if the best score is 0. Print the result for two questions.
Explain it without notes
What are two cheap ways to make images searchable in a text pipeline?
What does a shared embedding space allow?
What changes in ingestion when documents contain images?
Why can OCR text be risky in an answer?
Expected output
Q: How much is the monthly pass?
captions only: IMG3 (score 1)
captions + text: IMG2 (score 2)
Q: Where is the Hilltop shop?
captions only: IMG3 (score 2)
captions + text: IMG3 (score 3)