(Lesson 02.03 · 20 minutes · Camera · First model)

Finding objects.

Counting the apples on the table means finding them one by one. Here you'll follow a detector from the YOLO family, from the thousands of boxes it proposes to the few that are left, and try the one that runs on your phone.

What you learn

How a model describes an object with a box and four numbers
What IoU measures and why it helps remove duplicates
What resolution costs and what you lose when you lower it

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The question
What's there, and where it is too.

On the table there are three apples and a mug. Classifying the photo isn't enough anymore, because there are many answers and each one has its own place. Detection^01.04 finds every object, gives it a class and puts it inside a box. Counting the apples gets easy, you just count the boxes.

In practice. If you only need to know whether something is there, classifying is simpler and faster. Detection is for when it matters where things are or how many there are.
02 The box
A box is made of four numbers.

To the model, a box is four numbers. Two say where the center is, and two say how wide and how tall it is. They're measured in proportion to the photo, from 0 to 1, so they work at any resolution. Each box carries a class and a confidence^02.01, which is how sure the model is.

In practice. When you draw boxes for training, keep them tight around the object. A loose box and a tight one on the same apple teach two different things.
03 The grid
The photo is split into cells, and each cell answers.

YOLO, one of the most widely used detectors, splits the photo into a grid of cells. Each cell takes care of the objects whose center falls inside it, and for each one it proposes a box, a class and a confidence. It all happens in a single pass through the network. That's where the name comes from, You Only Look Once.

In practice. Two small objects with their centers in the same cell are the hardest case. If that happens, move closer or use a bigger input image.
04 The candidates
Thousands of boxes, and almost all of them get thrown away.

A 640-pixel photo becomes three grids, of 80, 40 and 20 cells per side. That's 8,400 proposed boxes for each photo. Almost all of them have a very low confidence, because there's nothing in that cell. The first filter is a threshold. Below 25 percent, the box is thrown away.

In practice. You choose the threshold. Set it high and you lose the hard objects, set it low and boxes show up around nothing. Start at 25 percent and adjust it by looking at the mistakes.
05 The overlap
How much two boxes overlap, from 0 to 1.

After the threshold there are often three or four boxes left on the same apple, shifted a little. To tell whether they're about the same object, you measure how much they overlap. You divide the shared area by the total area the two of them cover. The number goes from 0, far apart, to 1, identical, and it's called IoU, intersection over union.

In practice. An IoU of 0.5 sounds low, but it's already a good overlap. Draw two equal squares shifted by a third of their side and do the math, it comes out to exactly 0.5.
06 Duplicates
Keep the best one and delete the others.

Now you remove the duplicates. You take the box with the highest confidence. You delete the ones that overlap it too much, usually with an IoU above 0.5 or 0.7. Then you move on to the best of the ones left, and so on. It's called non-maximum suppression, or NMS, and at the end there's one box per object.

In practice. If two apples close together become a single box, the IoU threshold is too low. If the same apple has two boxes, it's too high.
07 The phone
The smaller the photo, the faster it runs and the less accurate it gets.

YOLOv8 nano has 3.2 million weights, a little less than MobileNet. At 640 pixels it does almost 9 billion operations per photo. You can go down to 320 pixels, and the calculations drop to a quarter. The price is small, faraway objects, which at that resolution are only a few pixels and disappear.

In practice. Choose the resolution by looking at the smallest object you need. If you can still recognize it yourself in the shrunken photo, the model will recognize it too.

01 / 07

(Go deeper)

A few more numbers.

(01)

The YOLO family

Joseph Redmon presented YOLO in 2016. The detectors of the time looked at the photo thousands of times, one piece at a time, and took seconds. YOLO looked at it only once and reached 45 frames per second on a computer GPU.

Since then many versions have come out, written by different teams. Ultralytics distributes YOLOv8 and the versions after it in five sizes, from nano to extra large. Check the license before you use them in a product, because it's the AGPL and it asks you to publish your own code too. The detector you can try below is EfficientDet-Lite0 from Google, with an Apache license.

3.2 Mweights in YOLOv8 nano
8.7 Goperations per photo at 640 pixels
37.3mAP on COCO, the score for detectors

(02)

How you score a detector

A box counts as right if it has the right class and an IoU of at least 0.5 with the one drawn by hand. Then for each class you look at two things. How many of the model's answers are right, and how many of the real objects the model found. When you raise the confidence threshold, the first goes up and the second goes down.

You try every threshold and take the average, first for each class and then across the classes. The result is called mAP, mean average precision. The COCO dataset makes the score stricter and also averages over ten IoU thresholds, from 0.5 to 0.95.

(03)

COCO, the 80 starting classes

Almost all ready-made detectors are trained on COCO, a dataset published by Microsoft in 2014. It has 80 classes of common objects, like person, bicycle, cup and fork, and more than 200,000 photos with boxes drawn by hand.

If your objects are among those 80 classes, you can use a ready-made model. If not, you train it again on your own photos, starting from the weights it already learned on COCO. Here, though, taking photos isn't enough. On every photo you have to draw a box around each object, and that work is the longest part.

(Try it on your phone)

Live detector

Point the camera at the table or the room. A detector already trained on the 80 COCO classes draws the boxes live, with the class and the confidence. With the slider you change the threshold and see which boxes are left.

The tool needs JavaScript turned on.

The model downloads only once, then it works in airplane mode too. The camera images stay on your phone. Try counting small objects from far away and then up close.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

What does detection add compared to classification?
(02)

Two boxes match perfectly. What is their IoU?
(03)

Three boxes are left on the same apple. What reduces them to one?
(04)

You go from 640 to 320 pixels. What happens?

New words, and where they are explained

IoUchapter 5
NMSchapter 6

Sources and reading

To close the lesson, answer the quiz

← A good image dataset Back to the map

(Next lesson · 02.04) Cutting out pixel by pixel