(Lesson 04.02 · 22 minutes · Motion sensors · First model)

Recognizing a gesture or an activity.

Shaking the phone, drawing a circle in the air, walking. Here you'll record the examples, train the model on the phone, and learn to check that it also works with people it has never seen.

What you learn

The difference between a gesture and an activity, for a model
Why the class for everything else matters most
How to test a model on people it has never seen
How to avoid one-off answers and gestures counted twice

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 Gestures and activities
A gesture lasts a moment, an activity keeps going.

Shaking the phone, drawing a circle in the air, tapping twice on the back. These are gestures, they last a second and then they're over. Walking, running and sitting are activities, they go on for minutes. For the model, the difference is in the window^04.01. For a gesture the window has to hold all of it, for an activity a piece that repeats is enough.

In practice. If you're not sure whether your problem is a gesture or an activity, ask yourself if it has a beginning and an end. If it does, it's a gesture.
02 Everything else
The rest of the time matters more than the gestures.

For almost the whole day, the phone isn't making any gesture at all. It's in a pocket, on the table, in your hand while you read. A model that only knows shake and circle will always pick one of the two. You need a class^01.03 for everything else, with plenty of varied examples. It's often the class with the most examples of all.

In practice. Record the class for everything else while doing ordinary things, like sitting down or picking the phone up from the table. The movements that look like your gestures are exactly what keeps false alarms away.
03 Recording
Twenty examples per gesture, done in different ways.

In the lab, you hold down the button, do the gesture, and let go. Each recording becomes an example. Twenty per gesture are enough to start, as long as they're different from each other. Faster and slower, bigger and smaller, with the right hand and with the left. If you can, have another person do it too.

In practice. Ask someone to do your gesture without explaining it too much. You'll see right away how many ways there are to draw a circle.
04 The model
A small network on top of the features.

Each example becomes about thirty features^04.01, like the average and the swing of each axis. A network with two small layers learns to separate the classes from these numbers. It has a few thousand weights, and the phone trains it in a few seconds. Meanwhile you watch two curves, the loss on the examples and the loss on the examples kept aside.

In practice. If the first curve goes down while the second one climbs back up, the model is learning by heart. That's overfitting^01.05. Stop training earlier, or add more examples.
05 People
The model has to work with people it doesn't know.

A model trained only on your gestures recognizes you very well. With another person it often gets it wrong, because they move in a different way. To find out how it will really do, you keep aside all the examples from one person, and test the model on those alone. If it does well there too, it will work with new people.

In practice. With three people you run three tests, leaving out one person at a time. The average of the three tells you how good the model is with someone it has never seen.
06 The mix-ups
A table tells you which classes get confused.

The total number of right answers hides where the problem is. A table that lines up the right class against the one the model chose shows you. The hits sit on the diagonal, and the mix-ups sit off it. If the circle gets taken for everything else ten times, you know which gesture to record better.

In practice. Look at the mix-ups before adding examples at random. Usually two classes get confused with each other, and the rest are already fine.
07 Live
Three matching answers in a row, then the action.

Live, the model answers every half second, even while the gesture is only halfway done. A single answer on its own can be wrong. That's why you wait until two or three windows in a row give the same class with a high percentage. Then you pause for one second, so the same gesture doesn't get counted twice.

In practice. Choose gestures that are different from each other and from the rest of the day. A hard shake works better than a small turn of the wrist, which happens all the time.

01 / 07

(Go deeper)

A few more numbers.

(01)

What UCI HAR teaches you

In the UCI HAR dataset, thirty people walked, went up and down stairs, and sat, stood and lay down with a phone on their belt. The first models built on the 561 features made very few mistakes on the activities with movement. Almost all the errors were between sitting and standing.

That's the lesson of the mix-up table. One precise weak spot explains almost all the error. To tell sitting from standing with a phone on the belt, you don't need a bigger model. You need one more piece of information, for example the angle of the thigh, which you can't measure from the belt.

(02)

Networks that read the traces

Instead of features worked out by hand, you can give the network the whole window, 100 rows by 6 columns. Small filters slide along time, like with images but in one dimension only, and find the useful patterns on their own. This is called one-dimensional convolution.

It does better when there are many gestures or when they look alike, but it needs more examples and more computing. With three or four very different gestures and twenty examples each, features and a small network are still the best choice, and you can also see why the model gets things wrong.

(03)

Leaving one person out

If you mix everyone's examples and set aside one in five at random, the tests end up with gestures from the same people the model has already seen. The score goes up, but it tells you little about a new person.

The honest method is called leave one subject out. With N people you train N models, each one without one person, and you test each one on the person left out. The average of the scores is what you'll see with the people who use the app. It's usually lower, and it's good to know that beforehand.

(Try it on your phone)

Three gestures with the phone in your hand

In the lab you teach the phone three gestures you pick yourself. It takes ten minutes and, if you can, a second person for the last test.

Open the lab and choose the Motion input.
Create three classes, for example shake, circle and other.
Record 20 examples per gesture, changing speed and hand. For the other class, make two continuous recordings of 15 seconds. Press Continuous recording, move around normally, and press it again to stop.
Press Train and watch the two curves. In Evaluate you'll find the mix-up table.
In Use, press Try it live. Then have another person try the model and count how many times it gets it wrong.

Open the lab →

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

What's the difference between a gesture and an activity, for the model?
(02)

The model only knows shake and circle. What happens when you put the phone down on the table?
(03)

How do you find out if the model works with new people?
(04)

Why do you wait for two or three matching answers in a row before acting?

New words, and where they are explained

Gesture and activitychapter 1

Sources and reading

To close the lesson, answer the quiz

← The phone senses motion Back to the map

(Next lesson · 04.03) Noticing what's unusual