(Lesson 03.03 · 14 minutes · No sensor · For everyone)

The word that wakes the phone.

Your phone hears you even with the screen off, and yet the battery lasts all day. Here you'll find out how it does it, from the chip that listens on its own to the two models in a row, and why the right word is long and rare.

What you learn

How a phone is always listening and still uses very little power
Why you use two models in a row instead of one
How the phone recognizes your voice and not other people's
What makes a good keyword

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 Always listening
The phone is always listening, but only for one word.

When you say Hey Siri or OK Google, the phone answers even with the screen off. So the microphone is always on. But the phone doesn't understand everything you say. A tiny model looks for a single word, and until it hears it, it does nothing else. This task is called keyword spotting.

In practice. The keyword model is different from the one that understands your questions. The first always runs on the phone, the second only starts afterward, and sometimes it lives in the cloud.
02 The listening chip
A separate processor that uses very little power.

Keeping the main processor on just to listen would drain the battery. So the listening is done by a separate processor, small and slow, that uses a tiny fraction of the power. On the 2015 iPhone 6s it was the motion coprocessor, the same one that counts your steps. The rest of the chip^01.02 sleeps until the word arrives.

In practice. If you design something that's always listening, think first about how much power it uses. A model that runs every fraction of a second has to cost almost nothing, even if it makes a few more mistakes.
03 Two stages
A fast first model, and a more careful second one.

The tiny model often gets it wrong, and that's fine. When it thinks it heard the word, it wakes up a second model, bigger and more accurate, that listens to the same piece of sound again. Only if the second one agrees does the phone wake up. The first acts as a filter, and the second decides.

In practice. The same two-stage scheme works for lots of problems. A light check that's always on, and a heavy one only when it's needed.
04 Your voice
The phone learns your voice from a few phrases.

When you turn on the assistant, the phone asks you to repeat the word a few times. From those recordings it builds a voiceprint, a list of numbers that sums up your voice. Every time it's triggered, it compares the new voice with the voiceprint. If they're too different, the phone doesn't answer whoever spoke.

In practice. The voiceprint stays on the phone. It's a good example of data that doesn't need to travel, and of a model that adapts to a single person.
05 The mistakes
Two opposite mistakes, and you choose which one to put up with.

Keyword spotting goes wrong in two ways. Sometimes the phone wakes up on its own, maybe because of a line on TV that sounds similar. Sometimes it doesn't answer even though you said the word. The first mistake is measured in false alarms^03.02 per hour, the second in missed words out of a hundred. Move the threshold and one goes down while the other goes up.

In practice. A false alarm now and then is annoying but harmless. But if the phone then records and sends what it hears, it becomes a privacy problem.
06 The right word
A good keyword is long and rare.

Hey Siri, OK Google and Alexa weren't chosen at random. They have three or four syllables, with sounds that differ from each other, and people rarely say them in everyday life. A short, common word like hi would wake the phone all the time. The longer and more unusual the word, the easier it is to tell apart.

In practice. If you pick a keyword for a project of your own, try saying it inside normal sentences. If it sounds like words you use often, change it.
07 Beyond the voice
The same trick works for the other sensors.

A tiny model that's always on isn't only for the voice. The phone uses one to notice when you pick it up off the table, or to count your steps. Pixel phones hear the song playing on the radio and tell you its title. They look it up in a list saved on the phone, without sending anything over the network.

In practice. When you think about an app that listens, ask yourself what it has to hear all the time and what can wait. Whatever runs all the time has to fit in a few kilobytes.

01 / 07

(Go deeper)

A few more numbers.

(01)

Inside Hey Siri

In 2017 Apple explained how the iPhone keyword works. Every few milliseconds the sound goes through a small network on the always-on processor, with five layers of 32 units. When the score goes over a threshold, it wakes up the main processor, which runs a bigger network, five layers of 192 units, on the same piece of sound.

There are two thresholds, one for each stage. Apple tuned them by counting false alarms over many hours of normal speech where the word never comes up. A year later it added recognition of the voice of the phone's owner.

(02)

False alarms and missed words on the same curve

For each threshold you measure two numbers, the false alarms per hour over hours of ordinary audio, and the missed words out of a hundred real tries. If you put the pairs on a chart, you get a curve. The closer it sits to the bottom left, the better.

Whoever designs an assistant picks one point on the curve. A phone that wakes up on its own once a day is acceptable, one that does it every hour isn't. To keep false alarms that low, you accept that now and then you'll have to say the word again.

(03)

A keyword all your own

There are open tools for training a keyword you choose yourself. openWakeWord learns it from synthetic recordings, made with an artificial voice in thousands of variations, so you don't need to record hundreds of people. microWakeWord makes models small enough to run on an ESP32 board that costs a few euros.

The method is the one from this course. Lots of examples of the word, a huge number of examples of everything else, a mel spectrogram as input and a small network. The longest part is still measuring false alarms over hours of normal audio.

(Try it on your phone)

The keyword test

Turn on your phone's voice assistant, if you don't use it already. Then try these situations one at a time and mark what happens. You're measuring the two mistakes from this lesson on your own.

(01) You say the keyword from a meter away, with the screen off
(02) You say it from the other side of the room
(03) You say it with the TV or music on
(04) You have another person say it
(05) You say a phrase that sounds similar, for example hey Sara or OK Gigi
(06) You say it with the phone in airplane mode

If the phone woke up for the similar phrase, you saw a false alarm. If it stayed off when you were far away, that's a missed word. With another person's voice it usually stays off, because of your voiceprint.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

Why doesn't the phone use the main processor to wait for the keyword?
(02)

The first model thinks it heard the word. What happens next?
(03)

Which keyword works best?
(04)

The TV says a line and the phone wakes up. What kind of mistake is that?

New words, and where they are explained

Keyword spottingchapter 1

Sources and reading

To close the lesson, answer the quiz

← Recognizing a sound Back to the map

(Next lesson · 03.04) From voice to text, and back