(Lesson 03.01 · 15 minutes · Microphone · For everyone)

Sound becomes an image.

To a model, sound is a row of numbers, and then an image made of frequencies. Here you'll follow a hand clap from the microphone to the spectrogram, and watch it scroll by live while you talk.

What you learn

What samples are, and why 16,000 per second are enough for the voice
How you get from the waveform to the frequencies
What a spectrogram is, and why it uses the mel scale

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 Vibrating air
A sound is air pushing and pulling.

When you clap your hands, the air squeezes together and spreads out, hundreds of times per second. The phone's microphone has a thin membrane that follows those movements. Each push becomes an electrical voltage, higher or lower. For the model, sound starts here, with a membrane that moves.

In practice. The phone has two or three microphones, one at the bottom and one near the camera. When you record examples, always hold the phone the same way.
02 The samples
One second of voice becomes 16,000 numbers.

The phone measures the voltage at regular intervals and writes each measurement down as a number. Each measurement is called a sample. The microphone records 48,000 per second, but for the voice 16,000 are enough. Almost all speech models use this rate. One second of voice becomes a row of 16,000 numbers.

In practice. Bring all your recordings to the same sample rate before training. With different rates, the model hears the same sound in two different ways.
03 The waveform
The row of numbers draws a wave.

Put the samples in a row on a chart and you get the waveform. You can see when the sound is loud and when there's silence. What you can't see is what's making the sound. A whistle and a word at the same volume have waveforms that look alike to the eye. To tell them apart, you need the frequencies.

In practice. The waveform is useful for trimming recordings. Cut the silence at the start and at the end, so every example really contains the sound.
04 Frequencies
Every sound is made of simpler vibrations.

Any sound can be broken down into many simple vibrations, slow and fast, each with its own strength. The fast ones are the high sounds, the slow ones the low sounds. How many times they vibrate in a second is measured in hertz. The calculation that does this breakdown is called the Fourier transform, and the phone does it in a flash.

In practice. The human voice sits almost entirely between 100 and 4,000 hertz. A voice model can ignore the rest and do less work.
05 The spectrogram
Sound becomes an image.

Now the sound is cut into pieces of 25 milliseconds, one every 10. For each piece, the frequencies are calculated and stacked in a column, the low ones at the bottom and the high ones at the top. The columns side by side make an image, the spectrogram. Time runs from left to right, and the color tells you how strong each frequency is.

In practice. A voice shows up as parallel horizontal stripes, a whistle as a single line, and a noise as a scattered blotch. Watch it live in the exercise below.
06 The mel scale
More rows where the ear hears better.

The ear easily tells 100 from 200 hertz, but it can barely hear the difference between 8,000 and 8,100. The mel scale imitates this sensitivity. It keeps lots of rows for the low frequencies and only a few for the high ones. A one-second mel spectrogram often has 40 rows and 98 columns, a small image that a model reads quickly.

In practice. You usually also take the logarithm of the values, the way the ear does with volume. A sound that's ten times stronger doesn't seem ten times louder to us.
07 The networks
A network made for images listens to the spectrogram.

From here on, sound is an image. Small filters slide over the spectrogram and light up where they find their pattern, for example a rising line. A network made of these filters is called convolutional. It's the same family of models that recognizes photos, and it works just as well on sound.

In practice. One second of mel spectrogram is 40 by 98 numbers, fewer than a photo of 64 by 64 pixels. That's why a sound model can be very small.

01 / 07

(Go deeper)

A few more numbers.

(01)

Why 16,000 samples per second

To record a vibration you need at least two samples for each of its swings, one up and one down. That's the sampling theorem. So with 16,000 samples per second you capture frequencies up to 8,000 hertz, which is enough to understand the voice.

The old home phone used 8,000 samples, and in fact you could hear the voice but not the difference between an f and an s. CDs use 44,100 because the ear reaches about 20,000 hertz. For a model, more samples mean more numbers to read, so you pick the smallest rate that's enough.

8,000samples per second, the landline phone
16,000samples per second, voice for models
44,100samples per second, CDs

(02)

The mel scale formula

The mel scale was born in the 1930s from listening experiments. People were asked to say when a sound seemed twice as high as another one. Today we use a formula that does almost the same thing. Up to a few hundred hertz, mels grow like hertz, and after that more and more slowly, like a logarithm.

Below, f is the frequency in hertz. A thousand hertz is about a thousand mels, and eight thousand hertz is a little over 2,800.

mel = 2595 × log10(1 + f / 700)

(03)

The spectrogram in the browser

In the browser you get the frequencies from the Web Audio API, which the phone already has. An AnalyserNode takes the latest samples from the microphone, does the Fourier transform and gives back the strength of each frequency. The exercise in this lesson draws a new column for every frame on the screen.

With fftSize set to 1024 you get 512 frequencies, from the lowest up to half the sample rate. To switch to the mel scale, you group them into bands that get wider and wider as you move up toward the high sounds.

const audio = new AudioContext();
const mic = await navigator.mediaDevices.getUserMedia({ audio: true });
const an = audio.createAnalyser(); an.fftSize = 1024;
audio.createMediaStreamSource(mic).connect(an);
const strength = new Uint8Array(an.frequencyBinCount);   // 512 frequencies
an.getByteFrequencyData(strength);                       // repeat on every frame

(Try it on your phone)

Live spectrogram

Press the button, then talk, whistle and clap your hands. At the top you see the waveform, and below it the spectrogram scrolling from right to left. With the switch you go from the linear scale to the mel scale and see how the rows change.

The tool needs JavaScript turned on.

The sound isn't recorded or sent anywhere. The page looks at it and forgets it. Try whistling a rising scale, then say the same word in a low voice and in a high voice.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

A voice model uses 16,000 samples per second. How many numbers does one second of voice become?
(02)

What does the spectrogram show that the waveform doesn't?
(03)

Why does the mel scale keep more rows for the low frequencies?
(04)

Why can a sound model be smaller than a photo model?

New words, and where they are explained

Samplingchapter 2
Frequencychapter 4
Spectrogramchapter 5
Mel scalechapter 6

Sources and reading

To close the lesson, answer the quiz

← Hands, body, face Back to the map

(Next lesson · 03.02) Recognizing a sound