(Lesson 02.01 · 25 minutes · Camera · First model)

Classifying a photo.

A mug, a bunch of keys and the empty table. Follow a photo into MobileNet, from the square of pixels to the answer, and find out why twenty photos per class are all you need.

What you learn

How a network that works on images goes from pixels to shapes
What the body and the head of a model are, and which one you train
What the percentage next to the answer means, and what it doesn't mean

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The question
One photo, and one answer out of a few.

Put a mug and a bunch of keys on the table. You want the phone to look at them and say which of the two it sees. That's a classification^01.04. The model knows a few possible answers, called classes^01.03, and for each photo it picks one. Here there are three classes, mug, keys and nothing.

In practice. Before you collect the photos, write the classes down on a sheet of paper. If you mix up two classes yourself when you look at them, the model will mix them up even more.
02 The input
First of all, the photo gets small.

The camera takes photos with millions of pixels. The model, though, always wants the same size, 224 by 224. So the photo is cropped in the center and shrunk. What's left is 150,528 numbers, three for each pixel, one per color. That's the tensor^01.03 that goes into the network, and it's enough to recognize a mug.

In practice. The center crop cuts off the edges of the photo. When you take the picture, keep the object in the middle, or it might not end up in the square.
03 The filters
Small filters slide across the photo.

The first layer doesn't look at the whole photo at once. It uses filters of 3 by 3 pixels that slide over the entire image, like a magnifying glass. Each filter lights up where it finds its pattern, for example a vertical edge. This operation is called convolution. The later layers put the edges together and find curves, textures and shapes.

In practice. A network made of these filters is called convolutional, or a CNN for short. Almost all the image models in this course are.
04 The body
The body sums up the photo in 1,280 numbers.

After dozens of layers of filters, the photo has become a list of 1,280 numbers. They're not pixels anymore. Each number says how much there is of a certain thing, a handle, a shiny surface, a metal ring. This part of the network is called the body. Here it's MobileNet, already trained on more than a million photos.

In practice. You don't train the body again. It has already learned to see on a thousand kinds of objects, and you reuse that work without paying for it.
05 The head
The head chooses between your classes.

On top of the body you put a new head, a small one, with one output for each of your classes. It reads the 1,280 numbers and gives a score to mug, keys and nothing. Only the head gets trained, and it has 3,843 weights^01.03 instead of millions. Reusing a body that's already built is called transfer learning^01.05, and it's the reason a few photos are enough.

In practice. With 20 photos per class the phone trains the head in a few seconds. If you're not happy with the result, add photos before you change anything else.
06 Confidence
The model always answers, even when it doesn't know.

The last step turns the scores into percentages that add up to one hundred. Mug 91, keys 6, nothing 3. This percentage is called confidence. It's not the probability of being right. Point the camera at a banana and the model will still say mug or keys, maybe at 70 percent, because those are the only things it knows.

In practice. Pick a threshold. Below 60 percent, the app can say it isn't sure instead of guessing. Another option is a class made just for when there's nothing there.
07 Speed
On the phone the answer comes in a flash.

MobileNet does about 300 million multiplications for each photo. That sounds like a lot, but the network was designed for phones in the first place. In 2018 it needed 75 milliseconds on a single core of a 2016 Pixel. With the GPU of a phone today it fits inside a single video frame, so it recognizes the mug live.

In practice. In the lab, look at the milliseconds next to the answer. That's the latency^01.01 of your own phone, really measured and not read off a spec sheet.

01 / 07

(Go deeper)

A few more numbers.

(01)

MobileNet, a network made for the phone

The networks that won the image contests in 2015 had tens of millions of weights. MobileNet, presented by Google in 2017, splits each convolution into two lighter steps. First a separate filter for each channel, then a filter just one pixel wide that mixes them. The result is almost the same and takes eight to nine times fewer calculations.

Version 2, from 2018, has 3.4 million weights and does 300 million multiplications per photo. On ImageNet, which has a thousand classes, it gets the answer right on the first try 72 percent of the time. ResNet-50, seven times heavier, reaches 76.

3.4 Mweights in MobileNet v2
300 Mmultiplications for each photo
72%right answers on the first try on ImageNet

(02)

From scores to percentages

The head gives each class a score, which can be negative or as large as it likes. To turn the scores into percentages you use a function called softmax. First it makes all the scores positive with the exponential, then it divides each one by the sum of all of them. That way the percentages always add up to one hundred.

Written for class i, with z as the scores, the formula is the one below. Its flaw is the same one you saw in the chapter on confidence. The percentages are shared among the classes the model knows, even when the photo contains none of them.

p(i) = exp(z_i) / ( exp(z_1) + exp(z_2) + … + exp(z_n) )

(03)

The code, if you program

In the browser you can write all this in a few lines with TensorFlow.js, the library the lab uses too. The body is a MobileNet that's already trained and gives back the 1,280 numbers. The head is a single dense layer with three outputs and the softmax.

You compute the 1,280 numbers of each photo once, then you train only the head. That's why training on the phone takes seconds and not hours.

const features = body.predict(photo);           // 1280 numbers per photo
const head = tf.sequential({ layers: [
  tf.layers.dense({ inputShape: [1280], units: 3, activation: "softmax" }),
] });
head.compile({ optimizer: "adam", loss: "categoricalCrossentropy" });
await head.fit(allFeatures, labels, { epochs: 20 });

(Try it on your phone)

Your first model

In the lab you train the model from this lesson with your own photos. You need ten minutes, a mug, a bunch of keys and a table.

Open the lab with the button below. It already starts with the Photo input.
Create three classes and name them mug, keys and nothing.
Hold to take photos, 20 per class, while you move around the object. For nothing, point the camera at the empty table.
Press Train and watch the two curves go down.
In Use, try it in different light and on another table, and note when it gets it wrong.

Open the lab →

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

Why are about twenty photos per class enough, and not thousands?
(02)

A model only knows mug and keys. You point the camera at a banana. What happens?
(03)

What does a filter in the first layer do?
(04)

Why should you keep the object in the center when you take the photo?

New words, and where they are explained

Convolutionchapter 3
Confidencechapter 6

Sources and reading

To close the lesson, answer the quiz

← The method in seven steps Back to the map

(Next lesson · 02.02) A good image dataset