(Lesson 03.04 · 16 minutes · Microphone · For everyone)

From voice to text, and back.

Dictating a message and having a page read to you are two models that travel the same road in opposite directions. Here you'll follow Whisper from the spectrogram to the words, learn to count its mistakes, and try what runs on your phone without a network.

What you learn

How a model turns voice into text, one word at a time
How you measure how often a transcription is wrong
How the road back works, from text to voice
What runs on the phone and what stays in the cloud

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 Transcription
From voice to text, word by word.

You dictate a message and the phone writes it down. This task is called transcription, or speech recognition. It's much harder than keyword spotting^03.03. There are tens of thousands of possible words and endless sentences, and everyone speaks with their own voice and their own accent.

In practice. An app with a few fixed commands doesn't need full transcription. A model that recognizes just those few commands is smaller and faster, and it makes fewer mistakes.
02 Whisper
A model that has listened to 680,000 hours.

Whisper, released by OpenAI in 2022, learned from 680,000 hours of audio collected on the web, in dozens of languages. It comes in several sizes. The smallest, tiny, has 39 million weights and runs on a phone too. The biggest has one and a half billion and needs a powerful computer.

In practice. Whisper tiny makes many more mistakes than the big one, especially in Italian. Try it on your own sentences before you choose the size, and count how many words it gets wrong.
03 Listening and writing
One half listens, the other writes one piece at a time.

Inside Whisper there are two halves. The first reads the mel spectrogram^03.01 of 30 seconds of sound and sums it up in a long list of numbers. The second writes the text one piece at a time. At each step it looks at the sound and at what it has already written, and picks the most likely next piece.

In practice. That's why in apps the transcription arrives in pieces and sometimes corrects itself. The model revises the start of the sentence once it has heard how it ends.
04 The mistakes
You measure it by counting the wrong words.

To grade a transcription, you compare the model's text with the correct one. You count the words that were changed, the ones that are missing and the extra ones, and divide by the number of words in the correct text. That's the word error rate, or WER for short. A WER of 10 percent means one wrong word in every ten.

In practice. Measure the WER with the sentences you'll really use, spoken by the people who will really say them. A model that does well with the news can make lots of mistakes in a repair shop.
05 Live
Captions while people talk, even without a network.

Many phones write down what you're hearing even without a network. Since 2019, Android has put captions on any video with Live Caption. The iPhone has something similar in the settings for people who are hard of hearing. They use models that listen in pieces a few tenths of a second long, not in 30-second blocks like Whisper.

In practice. Try dictation in airplane mode. If it works, the model is on the phone. If it stops, it was using the cloud^01.01.
06 Synthesis
The way back, from text to voice.

The road in the other direction is called speech synthesis. The model receives a text and turns it into the sounds of the language. From there it produces a spectrogram, and finally a wave to play. Kokoro, an open model from 2025 with 82 million weights, speaks in a natural way. The English voice you're hearing in these lessons is made the same way.

In practice. A synthetic voice mostly gets acronyms and foreign names wrong. You fix it by writing down how they should be pronounced, the way you would for an actor reading a script.
07 Where it runs
What stays on the phone and what doesn't.

Transcribing and speaking are heavier jobs than recognizing a sound. Whisper tiny on a recent phone transcribes faster than people speak, but with more mistakes. The big models often stay in the cloud. Many apps split the work. The keyword and the simple commands stay in your pocket, and the rest goes online when there's a connection.

In practice. If the app has to work in the countryside or in a warehouse, test transcription in airplane mode from day one. Finding out at the end costs a lot.

01 / 07

(Go deeper)

A few more numbers.

(01)

The sizes of Whisper

Whisper comes in five sizes. They all use the same mel spectrogram and the same way of writing, and only the number of layers and weights changes. The small size makes more mistakes but is much faster, and the big one does the opposite.

For the phone, the first two sizes are the ones that count, plus the third if you have a bit of patience. Quantized versions, with the weights written in fewer bits, take up less space and run faster, at the cost of a few more mistakes.

Size	Weights	On the phone
tiny	39 million	Yes, even in real time
base	74 million	Yes
small	244 million	With effort, on recent phones
medium	769 million	No, you need a computer
large	1.55 billion	No, you need a powerful computer

(02)

How to count the WER

The correct text is “turn on the kitchen light”, five words. The model writes “turn on kitchen lights”. The word “the” is missing, and “light” has turned into “lights”. That's two mistakes out of five words, so the WER is 40 percent.

The count is done with an automatic alignment that looks for the way to get from one text to the other with the fewest changes. Watch out for formatting details. If the model writes 100 and the correct text says one hundred, the WER counts it as a mistake even though the transcription is right. Before counting, you bring the two texts into the same form.

WER = (changed words + missing words + extra words) / words in the correct text

(03)

The voice of this course

You can listen to the lessons in The Basics in English, with a voice made by Kokoro on the computer of the person who writes the course. For acronyms there's a small lexicon, which says for example to read SoC letter by letter.

Then Whisper listens to each lesson again, and the text it hears is compared with the script, word by word. Across the five lessons, less than one percent of the words are different, and almost all of them are numbers written as digits instead of words. Only one was really wrong, epoch, which sounded like epic, and it was fixed in the lexicon.

(Try it on your phone)

Voice in airplane mode

Put the phone in airplane mode and try these features. The ones that still respond transcribe or speak inside the phone, and the others need the network.

(01) Dictate a message with the keyboard microphone
(02) Turn on automatic captions and play a video saved on the phone
(03) Record a voice memo and ask for the transcript
(04) Have the phone read a page aloud with its reading feature
(05) Ask the voice assistant what time it is

Some offline features reach other languages later than English. If one doesn't work, look in the keyboard or accessibility settings for a language to download.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

Why is transcribing much harder than recognizing a keyword?
(02)

The correct text is “turn on the kitchen light”. The model writes “turn on kitchen lights”. What's the WER?
(03)

Dictation works even in airplane mode. What does that mean?
(04)

What does a synthetic voice get wrong most often?

New words, and where they are explained

Transcriptionchapter 1
WERchapter 4
Speech synthesischapter 6

Sources and reading

To close the lesson, answer the quiz

← The word that wakes the phone The course review Back to the map

(Next lesson · 04.01) The phone senses motion