(Lesson 01.03 · 18 minutes · No sensor · For everyone)

How a model learns.

Inside a model there are no rules written by hand. There are millions of numbers that start out random, and training adjusts them a little at a time. Here you'll follow a photo into the network and watch the numbers learn.

What you learn

How a photo becomes a table of numbers
What a neuron, a layer and the weights do
Why learning means bringing a loss down
Why training costs a lot and using the model costs little

You need to know

Model01.01 · chapter 1

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The numbers
To a model, a photo is a table of numbers.

The model doesn't see colors, it reads numbers. A photo of 224 by 224 pixels becomes a table with 224 rows and 224 columns. Each cell holds three numbers from 0 to 255, one for red, one for green and one for blue. A table like this is called a tensor.

In practice. Sound and motion become tables of numbers too. That's why the same models work with such different sensors.
02 The neuron
A neuron does a weighted sum.

The smallest piece of a network is the neuron. It receives a few numbers and multiplies each one by its own weight. Then it adds everything up and decides how much to pass on. The weights are the numbers the model adjusts as it learns. They're also called parameters.

In practice. MobileNet has about 3.5 million parameters. A model that writes text has a billion or more. That number tells you how much memory you'll need.
03 The layers
Layer after layer, from simple to complex.

Many neurons side by side make a layer, and many layers one after another make a neural network. With images, the first layers light up for edges and colors. The middle ones for shapes and textures. The last ones for whole things, like a leaf or a spot.

In practice. More layers means more calculations and more memory. Networks for phones have a few dozen, built to do few calculations in each one.
04 The answer
At the end comes a probability for each answer.

The last layer has one neuron for each possible answer, and each answer is called a class. The numbers that come out become probabilities that add up to one hundred. The model might say 92 percent healthy, 6 percent sick and 2 percent other.

In practice. A high probability means the model is confident, not that it's right. Sometimes it's very sure, and wrong.
05 The loss
Learning means being a little less wrong.

At first the weights are random, and the model is just guessing. You show it a photo together with the right answer, and measure with a number how wrong it was. That number is called the loss. All of training is about bringing it down.

In practice. The loss is computed on a small batch of photos at a time, usually 16 to 64. With a single photo, each correction would swing too far one way or the other.
06 The descent
You walk down the valley one step at a time.

Picture the loss as a valley in the fog. You can't see the bottom, but you can feel which way the ground slopes under your feet. For every weight, the model works out which way the loss goes down, and takes a small step in that direction. This way of learning is called gradient descent.

In practice. One full pass over all the photos is called an epoch. A small model does anywhere from a few dozen to a few hundred, and then you stop.
07 Using the model
Training costs a lot, using costs little.

Training takes lots of photos, lots of steps and lots of math, sometimes days on powerful computers. Using a model that's already trained is a different story. The photo goes in, the numbers pass through the layers just once, and the answer comes out. This is called inference, and it's the job your phone does.

In practice. You can train on the phone too, if you only teach the last part of the model. That's what you'll do in the lab, with a few dozen photos.

01 / 07

(Go deeper)

A few more numbers.

(01)

How many calculations a network does

ResNet-50, a famous network from 2015, has about 25 million parameters, and for each photo it does about 4 billion multiplications, each followed by an addition. MobileNet v2, built for phones, has 3.5 million parameters and does about 300 million. That's more than ten times fewer, with only slightly lower accuracy.

With the weights stored as 32-bit numbers with a decimal point, 4 bytes each, MobileNet v2 takes up about 14 MB. With 8-bit quantization it drops below 4 MB.

3.5 Mparameters in MobileNet v2
300 Mmultiplications for each photo
14 MBits weights at 32 bits

(02)

How long is a step

The length of the step in the descent is called the learning rate. If it's too long, the model jumps from one side of the valley to the other and never settles at the bottom. If it's too short, it goes down so slowly that it never gets there. A common starting value is 0.001.

Each step moves every weight with the same rule. The new weight is the old weight minus the learning rate times the slope of the loss for that weight.

weight = weight - learning_rate * slope

(03)

Where the slope comes from

To know how to move millions of weights, the model computes the loss at the end and then carries it back layer by layer, all the way to the start. Each layer gets its share of the blame and splits it among its weights. This backward pass is called backpropagation.

Nobody writes it by hand. Libraries like PyTorch and TensorFlow work out all the slopes on their own. In the browser the job is done by TensorFlow.js, the library the lab uses.

(Try it on your phone)

Teach a small network

Tap the sheet to place yellow dots and black dots, then press Train. Tap a dot you already placed to remove it. A tiny network tries to separate them, and the background color shows what it has learned. Below, you see the loss going down.

The tool needs JavaScript turned on.

Try removing the hidden neurons. Without a hidden layer the network can only draw a straight line, and it can't separate dots in a circle. Then put 8 neurons back and try a step that's too long. Watch the loss jump.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

What are the weights of a neural network?
(02)

The model says 97 percent sick. What can you conclude?
(03)

What does the phone do when it recognizes a photo with a model that's already trained?
(04)

During training the loss jumps up and down instead of going down. What do you try first?

New words, and where they are explained

Tensorchapter 1
Parameterchapter 2
Neural networkchapter 3
Classchapter 4
Losschapter 5
Gradient descentchapter 6
Epochchapter 6
Learning rateGo deeper
Inferencechapter 7

Sources and reading

To close the lesson, answer the quiz

← What's inside your phone Back to the map

(Next lesson · 01.04) The map of models