(Lesson 02.05 · 22 minutes · Camera · First model)

Hands, body, face.

A ready-made model turns your hand into 21 points. From there you teach the phone the gestures you want with a few examples, and with a bit of geometry you measure angles without training anything.

What you learn

How a model finds the points of the hands, the body and the face
Why a model that learns from points is small and fast
What it means to normalize the points before you use them
When a rule on the points is enough instead of a model

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The hand
A hand becomes 21 points.

Open your hand in front of the camera. A ready-made model from Google, MediaPipe, finds 21 key points^01.04. The wrist, then four points for each finger, from the base to the tip. For each point it gives three numbers, the position in the photo and an estimate of how close it is. That's 63 numbers in all, instead of millions of pixels.

In practice. The 21 points always come in the same order. Number 4 is the tip of the thumb, number 8 the tip of the index finger. With just these two you can already spot the pinch gesture.
02 Two models
First you find the hand, then its points.

The work is split between two models. The first looks for the palm in the whole photo, like a detector^01.04. The second looks only at the crop around the hand and finds the points. In the next frame, the hand is almost in the same place. The crop gets reused, and the first model only starts again when the hand disappears.

In practice. A hand that moves too fast leaves the crop, and the first model has to start over. When you record examples, move calmly.
03 Body and face
The body has 33 points, the face 478.

The same scheme works for the rest of the body. The pose model finds 33 points, from the shoulders to the ankles, all the way to the tips of the feet. The face model finds 478, enough to follow the lips and the eyebrows. Models like these are behind photo filters and the workout apps that count your reps.

In practice. For the whole body the camera has to see all of you. Set the phone down two or three meters away, more or less at waist height.
04 The gesture
Your model learns from the points, not from the pixels.

Now you can teach your own gestures, for example thumbs up, fist and open hand. The ready-made model turns each frame into 63 numbers, and your model learns only from those. It's small, it trains in a moment and a few examples are enough. The background and the light are gone, they stayed behind in the pixels.

In practice. Record each gesture with both hands and at different distances. The model only recognizes the positions you've shown it.
05 Normalization
You take away the position and the size of the hand.

The same fist can be in the top left or the bottom right, close up or far away. To the model those would be completely different numbers. So first everything is shifted so the wrist lands on zero, then it's divided by the size of the hand. Only the shapes are left. This step is called normalization.

In practice. If your model recognizes a gesture only in one corner of the screen, normalization is almost always what's missing.
06 Angles
With three points you can measure an angle.

Not everything needs training. Hip, knee and ankle are three points, and between them there's an angle you can work out with geometry. When the knee angle drops below 90 degrees and then comes back up, you've done a squat. You're estimating a number, like in regression^01.04, but without training anything.

In practice. Before training a model, ask yourself whether a rule on the points is enough. You can understand a rule and fix it in a minute, and it doesn't make random mistakes.
07 The limits
Where it goes wrong, against the light and with hidden fingers.

The points arrive even when the model can't see well. A hand half hidden behind a mug still gets 21 points, some of them guessed. Two hands that cross can swap their points. Against the light, the hand becomes a dark silhouette and the points shake. That's why next to the points there's a confidence^02.01 to keep an eye on.

In practice. Throw away frames with low confidence instead of passing them to the gesture model. Better a gesture recognized a moment later than a wrong gesture.

01 / 07

(Go deeper)

A few more numbers.

(01)

MediaPipe, Google's ready-made models

MediaPipe collects models already trained for hands, body, face, objects and more, ready for Android, iOS and the browser. The hand model is a single file of about 8 MB that holds two models, the one for the palm and the one for the points. It runs in real time even on mid-range phones.

To train it, Google marked the 21 points by hand on about 30,000 real photos. Then it added hands drawn by computer in many poses and kinds of light, because there the exact position of each point is already known and nobody has to mark it.

21points on the hand
33points on the body
478points on the face

(02)

Why 63 numbers beat 150,000 pixels

A model that classifies gestures from pixels has to learn from scratch what a hand is, and for that it needs thousands of photos. One that starts from the points gets that work already done. Two different gestures are far apart in the 63 numbers even when the photos look alike, and 20 or 30 examples per gesture are enough.

There's a privacy advantage too. In the gesture dataset you can save only the points, without faces, rooms or people in the background. For the body, and above all for the face, the points are still personal data, though, and you have to treat them that way.

(03)

Normalization in code

If you program, it takes a few lines. You subtract the wrist from every point, find the point farthest from the wrist and divide everything by that distance. The result is 63 numbers between minus one and one, ready for the gesture model.

You have to save the same numbers in the dataset and use them live. If you normalize the examples one way and the new frames another way, the model sees two different worlds and makes mistakes for no clear reason.

const wrist = points[0];
const shifted = points.map(p => [p.x - wrist.x, p.y - wrist.y, p.z - wrist.z]);
const scale = Math.max(...shifted.map(([x, y]) => Math.hypot(x, y)));
const numbers = shifted.flat().map(v => v / scale);   // 63 numbers

(Try it on your phone)

Three gestures of your own

In the lab you teach the phone three gestures you choose, starting from the 21 points of the hand. You need ten minutes and a room with normal light.

Open the lab with the button below. It already starts with the Hand input.
Create three classes, for example thumbs up, fist and open hand, plus a nothing class for when your hand isn't making any of the three gestures.
For each gesture record 20 examples, with both hands and at different distances.
Press Train. With points it only takes a few seconds.
In Use, try the gestures against the light and with one hand half hidden, and watch the confidence.

Open the lab →

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

How many numbers reach your gesture model for each frame?
(02)

Your model recognizes the fist only when the hand is in the top left. What's missing?
(03)

You want to count squats. What's the simplest way?
(04)

Half of the hand is hidden behind a mug. What does the points model do?

New words, and where they are explained

Normalizationchapter 5

Sources and reading

To close the lesson, answer the quiz

← Cutting out pixel by pixel The course review Back to the map

(Next lesson · 03.01) Sound becomes an image