Word counts versus dense vectors
The dense table is a toy written by hand so you can read every slot. A real model learns hundreds of unnamed slots from data. The lab below uses a real one.
A vector is just a list of numbers. The simplest vector for a text counts its words, one slot per word in the language, and it cannot tell that two different words mean the same thing. A dense embedding fixes that by giving every text a short list of numbers where nearby numbers mean nearby meaning.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write bag(text, vocab) that returns the word counts for a given vocabulary, and print the bag-of-words vectors for 'cancel a ride' and 'ride cancelled' over the vocabulary ["a", "cancel", "cancelled", "ride"]. How many words do they share, and what does that say about word counting?
Using the dense table {"cancel": [1, 0], "cancelled": [1, 0], "ride": [0, 1]}, write embed(text) that sums word vectors (unknown words add nothing) and scales to length 1, and print the embeddings of 'cancel a ride' and 'ride cancelled' to three decimals.
Explain it without notes
What is a bag-of-words vector, and why does it fail on 'forgot my login' versus 'reset your password'?
What is the difference between a sparse and a dense vector, and which one is an embedding?
Why do two texts with no shared words get a high similarity in an embedding space?
Can you recover the original text from its embedding? Why does it matter?
Expected output
vocabulary: forgot, login, my, password, reset, your
question: 1 1 1 0 0 0
passage: 0 0 0 1 1 1
shared words: 0
dimensions: account, money, time
question vector: [0.959, 0.069, 0.274]
similarity to 'reset your password': 0.996
similarity to 'get a refund': 0.098