(Lesson 05.01 · 16 minutes · No sensor · For everyone)

Words become numbers.

A neural network only knows how to multiply numbers. Here you'll see how a sentence becomes pieces, how the pieces become numbers, and why in the end "cat" lands close to "kitty" and far from "bolt".

What you learn

What tokens are and why words get split
What an embedding is and why similar words end up close together
How to measure how close two sentences are

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The problem
A model only knows numbers, not words.

A neural network^01.03 does multiplications. A photo is already made of numbers, the pixels, and so is a sound. A sentence isn't. First of all, the text has to be cut into pieces, and each piece has to be turned into numbers the model can use. How you do that decides how much the model will be able to understand.

In practice. When a text model makes strange mistakes, first check how it cut up the sentence. That's often where the problem starts.
02 The tokens
Text gets cut into pieces called tokens.

The most common way of cutting doesn't follow whole words, but the pieces of words that show up often. Common words stay whole, and rare ones get split. "House" is a single piece, while "housewarming" might become two. Each piece is called a token, and a model has a vocabulary of a few tens of thousands of tokens, sometimes more.

In practice. Vocabularies were built mostly on English. The same sentence in Italian usually takes more tokens, so it costs more memory and more time.
03 The index
First, each token gets a slot number.

In the vocabulary, each token has a number, its position in the list. "Cat" might be number 18,402, and "kitty" number 51,337. But these numbers say nothing about meaning. Number 18,402 is no closer to 18,403 than to any other number. It's just the number of a drawer.

In practice. Never give a model the vocabulary numbers as if they were measurements. To the model, 18,402 and 18,403 would look almost the same, and they aren't.
04 The embedding
Inside each drawer there's a list of numbers.

In each token's drawer there's a list of a few hundred numbers, called the embedding. As the model learns, these numbers move. Words that get used in the same sentences end up close together. "Cat" and "kitty" show up in similar sentences, so their embeddings look alike. "Bolt" shows up in other sentences, and ends up far away.

In practice. Being close doesn't mean having the same meaning. "Hot" and "cold" show up in the same sentences and often end up close, even though they're opposites.
05 The map
Meaning becomes a distance.

Think of embeddings as points on a map with hundreds of directions. Animals sit in one area, tools in another, and food in yet another. How close two points are is measured by the angle between their two lists of numbers. It's called cosine similarity, and it goes from minus one to one.

In practice. The two-dimensional maps you see in articles are squashed. They help you understand, but two words that look close in the drawing can be far apart in the hundreds of real directions.
06 Sentences
A whole sentence becomes a list too.

A model can sum up a whole sentence in a single list of numbers. "Turn on the light" and "brighten up the kitchen" use different words but have close embeddings, because they ask for the same thing. That way you can search by meaning instead of by matching words. The photos app does something similar when you search for beach and it shows you the sea.

In practice. To search your own texts you don't need a model that writes. A model that makes sentence embeddings is enough, and it's much smaller, plus a comparison between lists of numbers.
07 In the phone
Words in your pocket cost very little.

A model that makes sentence embeddings in a single language can fit in a few dozen megabytes, and it answers in a few milliseconds. The phone's keyboard uses even smaller models to suggest your next word. The text stays on the phone. And that matters, because the messages you write are some of your most personal data.

In practice. Try the keyboard suggestions in airplane mode. If they keep showing up, the model that picks them lives inside the phone.

01 / 07

(Go deeper)

A few more numbers.

(01)

How a token vocabulary is built

The most common method is called byte pair encoding. You start from single letters and look, in a huge collection of texts, for the pair of pieces that most often shows up side by side. That pair becomes a new piece. You repeat this until the vocabulary reaches the size you want.

GPT-2, in 2019, had 50,257 tokens. Llama 3 has 128,256, and Gemma 3 about 262,000, to cover many languages better. A bigger vocabulary splits words less, but it makes the model heavier, because each token has its own embedding.

50,257tokens in GPT-2
128,256tokens in Llama 3
262,000tokens in Gemma 3, roughly

(02)

King minus man plus woman

In 2013 Tomas Mikolov and colleagues at Google published word2vec, a fast way to learn word embeddings. People found that the directions had a meaning. The embedding of king, minus the one for man, plus the one for woman, landed close to queen.

In 2016 another group showed the other side. With the same math, programmer minus man plus woman landed close to homemaker. Embeddings also learn from texts the biases of the people who wrote them. A model that uses them carries those biases along, unless someone looks for them and fixes them.

(03)

Cosine similarity, in code

For two lists of numbers of the same length, you multiply the elements in pairs and add them up. Then you divide by the lengths of the two lists. The result is the cosine of the angle between them. One means same direction, zero means no link, and minus one means opposite directions.

With sentence embeddings, real values almost always fall between 0.2 and 0.9. That's why you don't pick the threshold between similar and different in advance. You test it on a few pairs of sentences where you already know the answer.

function cosine(a, b) {
  let ab = 0, aa = 0, bb = 0;
  for (let i = 0; i < a.length; i++) { ab += a[i] * b[i]; aa += a[i] * a[i]; bb += b[i] * b[i]; }
  return ab / Math.sqrt(aa * bb);
}

(Try it on your phone)

Text your phone understands on its own

Put your phone in airplane mode and try these features. The ones that still respond use text models that live inside the phone.

(01) Type the start of a sentence and look at the words the keyboard suggests
(02) Type a word wrong on purpose and see if it gets corrected
(03) Search your photos for a word like beach, dog or cake
(04) Select the text inside a photo and copy it
(05) Translate a sentence with the translation app

Photo search is the most interesting case. It doesn't look for the word in the file name. It compares the meaning of the word with the meaning of the photos, like the embeddings in this lesson.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

Why does text have to be cut into tokens before it reaches the model?
(02)

"Cat" is token 18,402 and "cats" is 18,403. Does that mean they're close in meaning?
(03)

Why do "cat" and "kitty" have close embeddings?
(04)

You search your photos for beach and the sea shows up. What happened?

New words, and where they are explained

Tokenchapter 2
Embeddingchapter 4
Cosine similaritychapter 5

Sources and reading

To close the lesson, answer the quiz

← Predicting a number Back to the map

(Next lesson · 05.02) Classifying and searching text on the phone