(Lesson 03.02 · 22 minutes · Microphone · First model)

Recognizing a sound.

The doorbell, the whistling kettle and silence. Here you'll teach the phone to recognize three sounds from home, and find out why a model that's always listening is measured in false alarms per hour.

What you learn

How a model listens to sound as it flows by, one second at a time
Why a small network trained from scratch is enough
How background noise helps you train a better model
What false alarms are and how you choose the threshold

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The question
Three sounds, and a model that tells them apart.

Pick three sounds from home, for example the doorbell, the whistling kettle and silence. The phone listens, and every half second it says which one it hears. It's a classification^01.04 like the one for photos, but the input is the spectrogram^03.01 of one second. Silence is a real class, because it's what the microphone hears almost all the time.

In practice. Choose sounds that repeat the same way every time, like a doorbell or an alarm. A voice that's talking changes all the time and needs many more examples.
02 The windows
You listen one second at a time, moving on by half a second.

The microphone never stops, so the model looks at a piece of sound one second long, called a window. Every half second the window moves forward and the model answers again. That way, a sound that falls across two windows still gets heard whole by one of the two.

In practice. If the sound lasts less than half a second, like a click, shorten the step. The model works harder, but it doesn't miss it.
03 The network
A small network, trained from scratch.

The mel spectrogram of one second is small, 40 by 98 numbers. Four or five layers of filters are enough to read it. Each filter slides over the spectrogram looking for a pattern, like the straight line of a whistle. The whole network has a few tens of thousands of weights^01.03, and the phone trains it from scratch in a minute.

In practice. Record each sound in several rooms and at different distances. A doorbell heard from the hallway has a different spectrogram from one heard right next to the door.
04 Noise
The world is never silent.

In real life, the doorbell rings while the washing machine is running and someone is talking. A model that has only heard clean recordings gets confused. So during training, background noise is mixed into the examples at different volumes. It's a way of making up new examples, and the model learns the doorbell even over the washing machine.

In practice. Record ten seconds of the typical noises of the place where you'll use the model. The lab mixes them into the examples on its own during training.
05 False alarms
An occasional mistake becomes seven per hour.

A model that answers every half second gives 7,200 answers per hour. Even if it's wrong only once in a thousand, it sets off seven false alarms every hour. That's why, with sound, the share of right answers isn't enough. You count how many times in an hour the model goes off for nothing, and how many times it misses the real sound.

In practice. Ask for two matching answers in a row before raising the alarm. You lose half a second and get rid of almost all the isolated false alarms.
06 The threshold
Raising the threshold moves the mistakes around without removing them.

Every answer comes with a percentage. If you only raise the alarm above 90 percent, the false alarms drop, but a few real doorbells stay below it and get missed. If you go down to 60, you hear them all, with a few alarms for nothing. There's no threshold that's right for everyone. It depends on which mistake costs you more.

In practice. For a fire alarm it's better to miss nothing and accept a few false alarms. For a notification that bothers people, it's the other way around.
07 On the phone
Always listening costs little, if the model is small.

One second of sound becomes a spectrogram in a few milliseconds, and the small network answers in just as few. The phone works for a few milliseconds every half second and rests the rest of the time. The microphone stays on, but the sound never leaves the phone. Only a class comes out, every half second.

In practice. Always show when the microphone is listening. Even if the sound isn't saved, whoever is in the room has the right to know.

01 / 07

(Go deeper)

A few more numbers.

(01)

Speech Commands, the dataset for first steps

In 2018 Pete Warden, from Google, published Speech Commands, a dataset made for small models that are always listening. The second version has more than 105,000 one-second recordings of 35 short words like yes, no, up and down, spoken by more than 2,600 people.

All the clips last one second and use 16,000 samples per second, like the ones in this lesson. There's also a folder of background noises, from the sink to the bicycle, to mix into the examples. It's still the starting point for almost every comparison between keyword spotting models.

105,829one-second clips
35different words
2,618people who said them

(02)

How small a listening network can be

The example model that TensorFlow uses for microcontrollers recognizes two words, yes and no, and weighs about 18 KB. It runs on an Arduino board with 256 KB of memory, with no phone and no operating system. It's TinyML^01.01 in the strictest sense.

On the phone there's plenty of space, so you can use a slightly bigger and more accurate network. But the principle stays the same. The mel spectrogram takes the heaviest work off the model, and all the network has left to do is recognize a pattern in a small image.

(03)

A curve for choosing the threshold

Each threshold gives you a pair of numbers, the false alarms per hour and the real sounds missed out of a hundred. If you try every threshold and put the pairs on a chart, you get a curve. The good models sit at the bottom left, because they make few mistakes of both kinds.

The curve lets you compare two models without choosing the threshold first. If one model's curve sits entirely below the other's, the first one is better whatever threshold you use. Then you pick the point on the curve that matches the mistake you can live with.

(Try it on your phone)

Three sounds from home

In the lab you teach the phone to recognize three sounds you choose. You need ten minutes and three sounds that are easy to repeat, for example the doorbell, the kettle and silence.

Open the lab with the button below. It already starts with the Sound input.
Create three classes, two sounds and silence.
Record 20 one-second examples per class, in different rooms and at different distances.
Record 10 seconds of background noise, the kind you usually hear at home.
Press Train. Then in Use press Try it live, let the phone listen for five minutes and count the false alarms.

Open the lab →

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

Why does the one-second window move every half second and not every second?
(02)

A model answers twice per second and is wrong once in a thousand. How many false alarms does it make in an hour?
(03)

Why do you mix background noise into the examples during training?
(04)

You raise the threshold from 60 to 90 percent. What happens?

New words, and where they are explained

False alarmschapter 5

Sources and reading

To close the lesson, answer the quiz

← Sound becomes an image Back to the map

(Next lesson · 03.03) The word that wakes the phone