One lab for all the sensors.
Here the phone collects the examples, trains the model, measures it, shrinks it and uses it offline. The path is always the same, only the sensor the data comes from changes. The lab opens with the intermediate courses, when you have finished the basics.
- ChooseThe sensor and the task. Classify, estimate a number or spot anomalies.
- CollectHold to record the examples, or import them from your gallery.
- TrainThe phone trains a small network and draws the two curves while it learns.
- EvaluateConfusion matrix, the examples it got wrong and milliseconds measured on your phone.
- ShrinkYou quantize the model to 8 bits and compare size and accuracy.
- UseYou try it live, even in airplane mode, and export it if you want to take it somewhere else.
Five challenges, one for each input.
Each challenge sets up the project with its classes and has a goal you can measure. After training, the Evaluate step tells you whether you reached it, and the challenge stays marked here. The challenges you pass also go on your certificate.
- Your three mugs Get the phone to recognize three mugs from your home, even when they look alike, plus a class for when there's no mug at all. Take the challenge →
- Your doorbell Teach the phone to recognize the doorbell and tell it apart from voices and silence. Take the challenge →
- Three gestures with the phone Shake it, draw a circle, do something else. The phone has to tell which gesture you're making, and fast. Take the challenge →
- Thumbs up, fist, open hand Get the phone to recognize three hand poses in front of the camera, plus a class for every other pose. Take the challenge →
- Your home assistant Get the phone to understand what you're asking about, the light, the weather or the music, even with sentences it has never seen. Take the challenge →
Five inputs.
| Input | Sensor | What it becomes | Starting model | Download | |
|---|---|---|---|---|---|
| Photo | Camera | A 256 × 256 pixel square | MobileNet v3 small, already trained, 1,024 numbers | about 16 MB | Open |
| Sound | Microphone | A one-second mel spectrogram, 40 × 98 | Small convolutional network trained from scratch | nothing to download | Open |
| Motion | Accelerometer and gyroscope | A two-second window on six axes | 32 features and a small network | nothing to download | Open |
| Hand | Camera | 21 points with three coordinates | MediaPipe Hand Landmarker | about 20 MB | Open |
| Text | Keyboard | A whole sentence, 384 numbers | Multilingual MiniLM | about 135 MB, from Hugging Face | Open |
Everything stays on your phone. The examples and the model are saved in the browser's memory and you can delete them whenever you want. The camera, the microphone and the motion sensors only work on a secure page, so the address has to start with https.