(Lesson 04.04 · 16 minutes · Camera or sensors · First model)

Predicting a number.

How fast the rhythm is, how many steps, how much battery will be left tonight. Here a model answers with a number instead of a class, and you learn to measure how far off it is and where it shouldn't be trusted.

What you learn

What changes in a model when the answer is a number
How to measure the error of an estimate
Why a model gets it wrong outside the values it has seen
What a forecast over time is, and how to give the margin of an estimate

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 A number
When the answer is a number.

How many steps you've taken, how many degrees the phone is tilted, how fast the rhythm is as you shake it. These are questions with a number as the answer. The model doesn't pick from a list, it estimates a value that could be anything. That's regression^01.04. What changes is the last layer of the network, which gives a single number instead of a percentage for each class.

In practice. If the answer has only a few possible values, like empty, half and full, a classification is easier to train. Regression is for when the values in between really matter.
02 The labels
Each example has its own number, really measured.

To train, you need the right number for each example, which is its label^01.05. Here the hard part moves to the measuring. If you shake the phone along with a metronome at 90 beats per minute, the label is 90, and it's precise. A label guessed by eye that's off by ten teaches the model to be off by ten.

In practice. Pick a problem where the right number is easy to measure. A metronome, a scale or a ruler is worth more than any model.
03 The average error
You measure how far off you are, on average.

With numbers, counting the right answers makes no sense. Saying 91 when the value is 90 isn't the same kind of mistake as saying 150. You take the difference between the estimate and the true value, ignoring the sign, and average it over all the examples. That's the mean absolute error, or MAE, and you read it in the same unit as the answer.

In practice. Always compare your MAE with that of a fake model that always answers with the average. If yours doesn't beat it by a lot, it hasn't learned anything useful.
04 The square
Big errors weigh more.

During training, the model usually corrects its weights by looking at the square of the error. Being off by 10 counts as a hundred, being off by 2 counts as four. That way the big errors weigh much more. It's the most common loss^01.03 for regression, but a single example with a wrong label can pull the whole thing toward itself.

In practice. If your data has wrong measurements you can't remove, train with the error without the square. It gets dragged around much less.
05 Out of range
The model can't estimate what it hasn't seen.

You recorded shakes between 60 and 150 beats per minute. At 100 the model estimates well, because it's right among examples it has seen. At 200, though, it gets it wrong, and often answers somewhere around 150, the highest value it knows. Estimating inside the range you've seen is easy, outside it isn't. The examples have to cover the whole range you'll need.

In practice. Write down next to the model the range it was trained on. If something outside that range comes in, the app should say so instead of trusting the estimate.
06 The forecast
Predicting the future is an estimate over time.

One variation is estimating the next value in a series. How many steps you'll take tonight, how much battery will be left at midnight. The model looks at the last few hours and estimates the ones to come. It's still regression, but the input is the past of the same quantity. It's called time series forecasting.

In practice. Before training, try a simple rule like tomorrow equals today, at the same hour. Beating it is harder than it looks.
07 The margin
A good estimate comes with its margin.

An estimate without a margin is misleading. The model says 110 beats, but how far off could it be? The simplest way to find out is to look at the errors it made on the test examples. If nine times out of ten they're within 8 beats, every new estimate can be given as 110 plus or minus 8.

In practice. Always show the estimate together with its margin. Someone who reads 110 plus or minus 8 makes better decisions than someone who just reads 110.

01 / 07

(Go deeper)

A few more numbers.

(01)

MAE and RMSE

Next to the MAE you'll often see the RMSE, the square root of the average of the squared errors. You read it in the same unit, but it gives more weight to big errors. With four errors of 1, 1, 1 and 9, the MAE is 3 and the RMSE is about 4.6.

When the two numbers are close, the errors are all alike. When the RMSE is much bigger, somewhere there are a few huge errors worth tracking down. Looking at both costs nothing and tells you a lot.

MAE  = mean( |estimate − true| )
RMSE = sqrt( mean( (estimate − true)² ) )

(02)

The fake model to beat

The dumbest model for a regression always answers with the average of the labels. It doesn't look at the input, and it still has its own MAE. Every real model has to beat it, and the gap between the two tells you how much it learned.

There's a number that makes this comparison on its own, called R squared. It's 1 if the model is never wrong, 0 if it does as well as the fake model, and below zero if it does worse. In the lab, next to the MAE you'll also see the fake model's, so you can compare them right away.

(03)

Forecasts your phone already makes

Your phone has been making forecasts over time for years. On iPhone, Optimized Battery Charging learns what time you usually unplug the phone and holds the battery at 80 percent until shortly before. Apple says the learning happens on the device.

These forecasts rely on rhythms that repeat, like the day and the week. That's why the most useful features are the hour of the day and the day of the week, even before past values. A small model with those two pieces of information often beats a big one that doesn't have them.

(Try it on your phone)

The metronome's rhythm

In the lab you train a model that estimates how fast the rhythm is as you shake the phone. It takes ten minutes and a metronome, and a free app will do.

Open the lab, choose the Motion input and the Number task.
For each rhythm, 60, 90, 120 and 150 beats per minute, first type the beats in the value field. Then press Continuous recording and shake the phone in time for 10 seconds. Do it twice for each rhythm.
Press Train, then look at the average error in Evaluate.
In Use, press Try it live and shake at 100 beats, a rhythm the model has never seen. Watch the estimate.
Try 200 beats, out of range, and compare the error.

Open the lab →

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

What changes in the network when the answer is a number instead of a class?
(02)

The estimates are 92, 85 and 100, and the true values are 90, 90 and 90. What's the MAE?
(03)

You trained the model between 60 and 150 beats. What happens at 200?
(04)

Nine times out of ten, the error on the test examples is within 8 beats. How do you give an estimate of 110?

New words, and where they are explained

MAEchapter 3
Forecastingchapter 6

Sources and reading

To close the lesson, answer the quiz

← Noticing what's unusual The course review Back to the map

(Next lesson · 05.01) Words become numbers