(Lesson 01.04 · 14 minutes · No sensor · For everyone)

The map of models.

Recognizing a leaf, counting the people in a line, understanding a sentence. They look like different problems, but every model is described by two things, the data that goes in and the answer that comes out. Here you'll draw the map that holds the whole course together.

What you learn

The two questions that tell you which model you need
The six tasks, from classifying to generating
When you don't need a neural network

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The map
Two questions for every model.

To figure out which model you need, two questions are enough. What data goes in, and what answer has to come out. The data can be a photo, a sound, a movement, some text or a series of numbers. The answer can be a word from a list, a box, an outline, a number or something new.

In practice. Write down both answers before anything else. They're almost always enough to know which model to look for, and how hard it will be.
02 Classify
To classify is to choose from a list.

The most common task is classification. The model looks at the whole input and picks one answer from a fixed list of classes, for example healthy, sick or other. Photos, sounds, gestures and sentences all get classified, and the pattern is always the same.

In practice. Always add a class for things that don't belong, like nothing or other. Without it, the model is forced to choose, and it's confidently wrong.
03 Find
To count, you need to know where.

Sometimes knowing what's there isn't enough, you need to know where. Object detection finds every object in the image, draws a box around it and says which class it is. That way, objects can also be counted, like the apples on a table or the people in a line.

In practice. One variant finds points instead of boxes. A hand has 21 keypoints, one for each joint, and with those you can recognize gestures.
04 Segment
To segment is to cut along the edge.

Segmentation goes deeper. It decides which class every single pixel belongs to, as if it were cutting the object out with scissors along its edge. It's what your phone does when it blurs the background behind you on a video call.

In practice. Segmenting costs more than finding a box, both in computation and in labels. You pick it when the outline really matters, for example to measure a spot.
05 Estimate
Sometimes the answer is a number.

Not every answer is a word. How many steps you took, how many degrees the phone is tilted, how full the glass is. When the answer is a number, the task is called regression. Instead of counting right answers, you measure how far off the model is on average.

In practice. If you only need the number in a few bands, say empty, half and full, turn them into classes. The model often learns faster and makes fewer mistakes.
06 Anomalies
Noticing what doesn't fit.

Some problems have very few examples of failure. An engine breaks down once a year, and nobody has a thousand photos of faulty parts. So the model learns only what normal looks like, and flags anything that strays too far from it, without ever having seen a failure.

In practice. The hard part is the threshold. Too low and the alarm always goes off, too high and it never does. You set it by looking at the data, not by eye.
07 Generate
The newest arrivals write the answer.

The newest models on phones don't pick an answer, they write it. A generative model produces text, voice or images, one piece at a time. They're the heaviest, from a few hundred megabytes to a few gigabytes, but since 2023 they run on phones too.

In practice. A generative model can classify too, but it uses much more memory and battery. If the answers are few and always the same, a small classifier is enough.

01 / 07

(Go deeper)

A few more numbers.

(01)

The full map

Each row is a type of data, each column a task. The cells with an example are models that really run on a phone today, and each one has a lesson in the course. The dots are combinations that aren't used much.

Data	Classify	Find	Segment	Estimate a number	Anomalies	Generate
Images	Healthy or sick leaf	Boxes around people	Background cut away from the subject	How full the glass is	Faulty part on the conveyor belt	A description of the photo
Sound	Siren, dog, car horn	The moment of the keyword	Who speaks and when	Beats per minute of a song	The engine makes a strange noise	Text read aloud by the phone
Motion	Walking, running or standing still	The moment of a fall	The phases of each step	Steps, tilt	Unusual vibration	·
Text	Command, question or spam	Names and dates in an email	·	How positive a review is	A message that doesn't belong	Summaries and replies
Body points	Hand gesture	21 points of the hand	·	Elbow angle	Bad posture	·

(02)

They aren't all neural networks

Artificial intelligence is the broad name for programs that do things that usually take a person. Today almost all of it comes from machine learning, meaning programs that find the rules on their own by looking at lots of examples with the right answer. Neural networks are the most common kind of machine learning, but not the only one.

When the data is just a few numbers, like the readings of a sensor, a decision tree is often enough. It's a series of yes or no questions, like "is the vibration above this threshold?" It takes up a few kilobytes, answers in an instant, and you can read it to understand why it decided what it did.

Another simple model is nearest neighbors, or k-NN. It doesn't learn weights. It keeps the examples in memory, and for a new input it repeats the answer of the most similar ones. With few examples it works surprisingly well, and in the lab you can pick it instead of the network.

(03)

What labels cost

The more a task says where, the more a label costs. Picking the class of a photo takes a second or two. Drawing a precise box takes several seconds, up to half a minute with traditional methods. Tracing the outline of an object costs even more. For the COCO dataset it took 79 seconds per object on average.

That's why you start from the simplest task that answers the question. If you only need to know whether there's a spot in the photo, you classify. You move to boxes only if you need to count the spots, and to outlines only if you need to measure them.

1-2 sone class per photo
7-35 sone box per object
79 sone outline per object, COCO

(Try it on your phone)

Which box is it in?

Here are seven real problems. For each one, pick the data that goes in and the task the model has to do. After each answer I'll tell you whether it's the right box, and why.

The tool needs JavaScript turned on.

Some problems sit across two boxes, and that's fine. The question to ask is always the same. What's the simplest task that answers what I need?

(Quiz)

3 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

An app has to count the cars passing in front of a camera. Which task do you need?
(02)

Why is it worth adding a class like other or nothing?
(03)

In a factory, one part in ten thousand is faulty. What's the best move?

New words, and where they are explained

Artificial intelligenceGo deeper
Machine learningGo deeper
Classificationchapter 2
Object detectionchapter 3
Keypointschapter 3
Segmentationchapter 4
Regressionchapter 5
Anomalychapter 6
Generative modelchapter 7
Decision treeGo deeper
Nearest neighborsGo deeper

Sources and reading

To close the lesson, answer the quiz

← How a model learns Back to the map

(Next lesson · 01.05) The method in seven steps