(Lesson 02.02 · 18 minutes · Camera · First model)

A good image dataset.

A model is only as good as the photos it learns from. Here you'll see the mistakes people make when they take them and how to avoid them, from the snow behind the wolves to the test photos that must never be shown.

What you learn

Why a model also learns things you didn't mean to teach it
How to choose the variety and the number of photos for each class
Why you need a class that says there's nothing there
How to split the photos without inflating the result

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 Shortcuts
The model learns everything you show it.

In 2016, three researchers deliberately trained a model to tell wolves from huskies, with the wolves always on snow. The model seemed good. In reality it was looking at the snow, not the animal. Learning a detail that comes along with the answer, instead of the answer itself, is called a shortcut. If all your mugs are in the kitchen, the model might learn the kitchen.

In practice. Look at all the photos of a class together. If they share something that isn't the object, like the background or the light, the model will use it.
02 Variety
Change the light, the background, the distance and the angle.

A good class looks like all the ways the object will really show up. Photograph the mug by day and in the evening, on the table and on the couch, up close and from far away, upright and upside down. If you have more than one mug, use them all. Twenty different photos are worth more than a hundred that are almost the same.

In practice. Take each group of photos in a different place. Five photos in four places beat twenty photos always taken in the same corner.
03 Balance
Every class with about the same number of photos.

Imagine 180 photos of mugs and 20 of keys. A model that always answers mug is right nine times out of ten. Its accuracy, meaning the share of right answers, is 90 percent. It looks great, and the model is useless. When one class has many more photos than the others, the classes are imbalanced, and accuracy on its own is misleading.

In practice. Don't let any class grow past twice the size of the others. If one has to stay small, look at the mistakes class by class as well as the total.
04 The nothing class
A class just for saying there's nothing there.

A model always chooses between its classes, even when the photo contains none of them. That's what the nothing class is for. It collects what the camera sees when the object isn't there, the empty table, the floor, a hand passing by. That way those moments have a right answer too, instead of a wrong answer given with high confidence^02.01.

In practice. The nothing class should be the most varied of all, because the rest of the world is big. Put in objects that look like yours too, like a glass next to the mugs.
05 Cleaning up
Get rid of duplicates, and watch out for blurry photos.

When you hold down the button, the phone takes ten photos per second, almost identical. For the model they count as one, but they weigh as much as ten. Move the phone between one shot and the next. Blurry photos, on the other hand, should only be removed if they won't happen in real use. If the phone will be in the hand of someone walking, keep a few.

In practice. Scroll through the photos of each class before training. In a minute you'll find the ones that ended up in the wrong place, like a mug among the keys.
06 The split
The test photos are never shown.

Before training, you set some photos aside, for example one in five. They're used for validation^01.05, and the model never sees them while it learns. But watch out for twin photos. If two shots from the same session end up one on each side, the model recognizes the twin and the score gets inflated. This mistake is called data leakage.

In practice. Split by session, not by photo. Shots taken on the same day in the same place all go together, on one side or the other.
07 Made-up data
One photo can become ten.

During training, each photo can be changed a little at random. It gets rotated a few degrees, cropped, brightened, flipped. To the model it's a new photo, and it learns that a mug is still a mug even when it's tilted or in the dark. This trick is called data augmentation.

In practice. Only use changes that also happen in real life. Rotating half a turn is fine for a leaf, but not for a 6 that turns into a 9.

01 / 07

(Go deeper)

A few more numbers.

(01)

How many photos you really need

There's no right number for every problem. With a body that's already trained, you start from 20 or 30 photos per class and look at where the model gets it wrong. Classes that look alike, like two olive varieties, need many more photos than two different objects like a mug and keys.

The honest way to find out is to measure. Train with 10, 20, 40 and 80 photos per class, and each time write down the accuracy on the test photos. If the curve is still going up, you need more photos. If it has flattened out, the problem is somewhere else, usually in the variety or in the labels.

(02)

A mistake that made it into journals

In 2021 a team at the University of Cambridge reviewed 62 studies that used artificial intelligence to spot COVID in chest X-rays and CT scans. None of them was ready for a hospital. One of the most common reasons was the data itself.

In some studies the images of healthy people came from an archive of children, so the model learned to recognize children instead of the disease. In others the same images showed up both among the ones for learning and among the test ones. Those are the shortcut and the data leakage from this lesson, made by expert researchers.

62studies reviewed
0ready for clinical use

(03)

Labels make mistakes too

In 2021 a team at MIT looked for label errors in ten famous datasets. Among the 50,000 test photos of ImageNet they found almost 3,000 with the wrong answer, about 6 percent. A dog marked as another kind of dog, an object with the name of the one next to it.

In your dataset the mistakes are easier to find, because you took the photos yourself. But they do happen, especially when you collect in a hurry. A photo in the wrong class teaches the model the very mistake you want to avoid.

(Try it on your phone)

The dataset check

Open the photos you took in the lab, or a gallery folder with photos for a problem of your own, and check these points one by one.

(01) Every class has at least 20 photos
(02) The biggest class has no more than twice as many photos as the smallest one
(03) Every class was photographed in at least four different places
(04) There's a nothing class, with varied backgrounds and objects that look like yours
(05) You removed the duplicates and the photos that ended up in the wrong class
(06) The test photos come from a separate session

Fix the missing points before you train again. It almost always improves the model more than any setting does.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

All the mugs are photographed in the kitchen, all the keys on the couch. What might the model learn?
(02)

You have 180 photos of mugs and 20 of keys, and the model has 90 percent accuracy. What do you check?
(03)

Why should the test photos come from a separate session?
(04)

Which change is a bad idea for augmenting the data of a model that reads digits?

New words, and where they are explained

Shortcutchapter 1
Imbalanced classeschapter 3
Nothing classchapter 4
Data leakagechapter 6
Data augmentationchapter 7

Sources and reading

To close the lesson, answer the quiz

← Classifying a photo Back to the map

(Next lesson · 02.03) Finding objects