(Lesson 06.01 · 18 minutes · No sensor · For everyone)

Measuring for real.

A score of 95 percent can hide a useless model. Here you'll learn to read the mistakes class by class, to choose the threshold, and to measure milliseconds, memory and battery properly, on the phone of the person who'll use the app.

What you learn

Why accuracy alone is misleading, and what to look at instead
What precision and recall are, and how the threshold moves them
How to measure latency, memory and battery the right way

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The misleading score
A single percentage hides almost everything.

The leaf model gets 95 percent of its answers right. That sounds great. But if 95 of the 100 test photos show healthy leaves, you can get there just by always answering healthy. This percentage is called accuracy. On its own, it doesn't tell you which mistakes the model makes, or what they cost.

In practice. Before you look at any score, count how many test examples there are for each class. If the numbers are very different, accuracy alone isn't enough.
02 The matrix
A table crosses the right answers with the ones the model gave.

To see the mistakes, you build a table. The rows hold the true class, and the columns hold the class the model said. The hits sit on the diagonal, and the mix-ups sit everywhere else. If 12 sick leaves end up in the healthy column, you see it right away. It's called a confusion matrix, and it's the first thing to look at.

In practice. Look first at the row of the class that costs you the most when it's wrong. For the leaf, that's the row of the sick leaves, not the healthy ones.
03 Precision and recall
Two different questions about the same mistakes.

For the sick class, you ask two questions. When the model says sick, how often is it right? That's precision. Of the leaves that really are sick, how many does it find? That's recall. When you move the threshold, one goes up and the other goes down. You choose by looking at which mistake costs more.

In practice. For a disease that spreads you want high recall, even with a few false alarms. For an alert that bothers people, high precision is the better choice.
04 Latency
Milliseconds are measured on the phone, after a warm-up.

A model's latency^01.01 is measured on a real phone, not on the computer. The first answers are slower, because the model is loading and the chip is getting ready. You do about ten dry runs, and then you measure a hundred. You look at the median, and at the slowest runs too, because those are the ones people notice.

In practice. Report the median and the time that 95 percent of the runs stay under. An average on its own hides the stutters.
05 Memory
How big the file is, and how much memory it uses while it works.

A model has two sizes. The size of the file, which matters for storage and for the download. And the size in memory while it's working, which is bigger, because it needs room for the results in between. On a phone with 4 gigabytes of RAM^01.02, an app that asks for too much gets closed by the system without any warning.

In practice. Test the model on the least powerful phone you want to support, with other apps open. That's where you find out whether there's enough memory.
06 Battery
You measure how much an hour of real use drains.

A fast model can still use a lot of power if it runs all the time. The honest way to measure it is simple. You charge the phone, use the app for an hour the way a real person would, and see how far the battery has dropped. Then you do the same with the app without the model. The difference is what the model costs.

In practice. Keep an eye on the temperature too. A hot phone slows the chip down, and a model that answered in 20 milliseconds in the first minute gets much slower after half an hour.
07 The average phone
You measure on the phone of a typical user.

Developers often have a top-of-the-line phone, and the people who'll use the app often don't. A model that answers in 15 milliseconds on a thousand-euro phone can take 80 on a two-hundred-euro one. That's why you pick a reference phone, mid-range or budget, and you measure every number on it.

In practice. Write the numbers in a table with the name of the phone, the date and the version of the model. A month from now you'll want to compare them, and you won't remember how you got them.

01 / 07

(Go deeper)

A few more numbers.

(01)

The math of precision and recall

Among the test photos there are 100 sick leaves and 900 healthy ones. The model says sick for 110 leaves, and 90 of them really are sick. Precision is 90 out of 110, about 82 percent. Recall is 90 out of 100, which is 90 percent.

When you need a single number, people often use F1, an average of the two that punishes a model that neglects either one. Here it's about 0.86. It's handy for comparing two models, but you should choose the threshold by looking at the two numbers separately.

precision = true positives / (true positives + false positives)
recall    = true positives / (true positives + false negatives)
F1        = 2 × precision × recall / (precision + recall)

(02)

Measuring latency the right way

LiteRT has a tool made for this, benchmark_model. It reports the loading time, the first answer, the warm-up runs and the average of the runs that follow, each one separately. On iPhone, Xcode does the same for Core ML models. In the browser, you wrap performance.now around the call to the model.

Whatever tool you use, keep the conditions the same. Screen on, phone not charging, no heavy apps open, the same starting temperature. And repeat the measurement on different days, because system updates change the numbers.

for (let i = 0; i < 10; i++) model(example);            // warm-up
const t = [];
for (let i = 0; i < 100; i++) { const t0 = performance.now(); model(example); t.push(performance.now() - t0); }
t.sort((a, b) => a - b);
console.log("median", t[50], "95 percent", t[95]);

(03)

MLPerf Mobile, comparing phones

MLCommons, a group of companies and universities, publishes MLPerf Mobile, a set of standard tests for models on the phone. Image classification, detection, segmentation and questions about a text, always with the same models and the same rules.

The results are public, and there's an app to measure your own phone. They're for comparing chips, not your model. Your model has to be measured on your reference phone, with your data, as in this lesson.

(Try it on your phone)

The threshold and the two errors

Here are a model's answers on 500 test leaves, 100 sick and 400 healthy. Each leaf has its own percentage for sick. Move the threshold and watch how the confusion matrix, precision and recall change.

The tool needs JavaScript turned on.

The percentages are made up, but they behave like real ones. Find the highest threshold that misses no more than five sick leaves, and see how many false alarms it costs.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

95 percent of the test leaves are healthy. The model has 95 percent accuracy. What can you conclude?
(02)

The model says sick for 50 leaves, and 40 of them really are. There are 80 sick leaves in all. What are precision and recall?
(03)

How do you measure a model's latency properly?
(04)

Which phone should you measure on?

New words, and where they are explained

Accuracychapter 1
Confusion matrixchapter 2
Precision and recallchapter 3

Sources and reading

To close the lesson, answer the quiz

← Models that see and speak Back to the map

(Next lesson · 06.02) Quantizing