Topic 12.6
Images and Other Media
In one line
Images can be searched by embedding them next to text, or, as a plain baseline, by indexing their captions and the text found inside them. The baseline is cheap, explainable and often enough.
Think of it like this
A photo album with notes on the back. You can find the beach picture by reading the note, "Harbour, summer", without looking at any photo. A smarter album could also find it by what the picture shows. Both approaches are search; they differ in what they index.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Caption
- A short text describing an image.
- OCR
- Optical character recognition: turning the writing in an image into text.
- Multimodal
- Involving more than one kind of data, such as text and images.
- Shared embedding space
- One vector space in which both texts and images have a position, so they can be compared.
- CLIP
- A model (Radford et al., 2021) trained on 400 million image and text pairs to put them in one shared space.
- Alt text
- A text description attached to an image for people who cannot see it.
Step by step
01Step 1: list what the handbook pictures contain
A locked bike on a rack, a price board, a map of the shops. Their useful content is partly in what they show and partly in the text printed on them.
02Step 2: index the captions
Each image gets one line describing it. Search treats the line as any other passage.
03Step 3: add the text found in the image
The price board's OCR text is added to its entry, so that its words are searchable. The map's text lists the shop names.
04Step 4: compare the two modes
Run the question about the monthly pass with captions only, then with captions plus text. Only the second finds the price board.
05Step 5: return no match when there is none
If the best score is 0 the example says no match. Images are easy to retrieve wrongly, so the refusal path matters as much as it does for text.
06Step 6: decide whether to embed the images themselves
Try the baseline first and measure on questions about your pictures. Add an image embedding model only if the baseline fails on a question type that matters, such as pictures with no text and no useful caption.
Try it yourself
- 1
Ask your own
Add the question
"Is a helmet free?"to the list. Which image does each mode pick? Why does the answer differ? - 2
Remove the text
Set IMG2's OCR text to an empty string. What happens to the monthly pass question? What does it say about relying on captions only?
- 3
Add a diagram
Add an image of a diagram of the lock chain with the caption
"Diagram of the built-in lock chain". Write a question that should find it and test it.
Code & diagrams
Word overlap stands in for a real search, and the OCR text is typed in by hand. The point is what gets indexed.
Expected output
Q: How much is the monthly pass?
captions only: IMG3 (score 1)
captions + text: IMG2 (score 2)
Q: Where is the Hilltop shop?
captions only: IMG3 (score 2)
captions + text: IMG3 (score 3)Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Index only the text inside the image
In the Python tab, change text = caption + " " + ocr if use_ocr else caption to text = ocr if use_ocr else caption. Run.
Myth vs fact
Myth
To search images you must embed the images.
Fact
Captions and the text found in the image are a strong baseline and are fully explainable.
Myth
OCR text is as reliable as typed text.
Fact
OCR misreads digits and unusual fonts. Show the image when a number matters.
Myth
One shared embedding space is free to add.
Fact
It needs an image model, more storage and new evaluation questions. Add it when the baseline fails on a measured question type.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep a pointer from every image entry to its source page and position. Citations and deletions (Topic 10.6) depend on it.
- ▸
Test with questions whose answer is only in a picture. These rarely appear in a golden set unless you add them on purpose (Topic 9.2).
- ▸
Treat text found in images as untrusted like any document text (Topic 10.2): a picture can contain instructions as well as a price.
Remember this
- 1
Why this exists. Handbooks, manuals and slide decks contain pictures: price boards, maps, diagrams, screenshots. A text-only pipeline ignores them. Part of what a learner needs may only be there.
- 2
Baseline 1: captions. Write a short text for each image (a person, or a model that describes images) and index that text like any passage. It is cheap, works with all the tools you already have, and you can read exactly why an image matched.
- 3
Baseline 2: text inside the image. OCR (optical character recognition) reads the characters out of the picture. A price board becomes "Monthly pass 25 dollars". The example shows the difference: for "How much is the monthly pass?" captions alone pick the map, because "the" is the only shared word, and captions plus text found in the image pick the price board.
- 4
Embedding images and text together. Some models are trained to place images and text in one shared vector space, so a text question can be compared directly with an image's vector. CLIP (Radford et al., OpenAI, 2021) is the well-known example: it was trained on 400 million image and text pairs from the internet. A hosted API or an open model with the same idea lets you search pictures with no captions. Check a model's documentation for what it supports, the sizes it accepts and its licence before relying on it.
- 5
What changes in ingestion. Extract the images (and their position in the document), run captioning and OCR or an image embedder, store the result with a pointer to the source page and image, and keep the image's alt text if there is one. Tables inside images are hard: treat them as in Topic 4.2. Store a type field (text, image, table) so the retriever and the prompt know what each result is.
- 6
What changes at answer time. A text-only model can only read the caption and the OCR text, not the picture. A model that accepts images can be given the picture itself. Either way, cite the image and its page, and say when the answer relies on text read out of a picture, because OCR can misread numbers. For anything that matters, show the learner the image.
Explain it without notes
What are two cheap ways to make images searchable in a text pipeline?
What does a shared embedding space allow?
What changes in ingestion when documents contain images?
Why can OCR text be risky in an answer?
Practice
Write index_text(caption, ocr) that returns the caption and the text found in the image joined by one space, lower-cased, with no trailing space when there is no text. Print the result inside square brackets for an image with text and an image without.
Write best_image(question, images) where images is a list of (id, text) pairs. Return the id with the most shared words, or None if the best score is 0. Print the result for two questions.
Trade-offs
- ↔
Captions and OCR: cheap and explainable, limited to what the text says.
- ↔
Image embeddings: find what text misses, with more storage, cost and evaluation.
- ↔
Sending the image to the model: the richest answer and the highest token cost.
Done when you can
Done when you can index an image by caption and OCR text.
Done when you can explain what a shared embedding space is.
Done when you can list the ingestion changes for images.
Done when you can say why a number read from an image needs the image shown.