(Lesson 06.03 · 16 minutes · No sensor · For everyone)

Pruning, distilling, choosing the architecture.

Besides using fewer bits, you can make a model lighter by removing the parts that do little, by having a smaller model imitate it, or by starting from a network built for the phone. Here you'll see the three routes, and the order to use them in.

What you learn

What pruning is, and when it really makes the model faster
How a small model learns from a big one
Why it pays to start from an architecture designed for the phone

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 Three routes
There are three ways to get a smaller model.

A model that's too heavy for the phone can be made lighter in other ways besides quantization^01.05. You remove the parts that do little. You teach a small model to imitate a big one. Or you start from a network designed for the phone. The routes are often used together.

In practice. Before you slim anything down, write down how big the model can be and how fast it has to be. Without a target, you keep trimming without knowing when to stop.
02 Pruning
Many weights are almost zero, and you can remove them.

After training, many weights are very close to zero and matter very little. Pruning sets them to exactly zero. You prune a little, retrain to recover, and do it again. In many networks you can remove half the weights and lose almost nothing in accuracy.

In practice. Setting weights to zero isn't enough to make the model faster. The phone multiplies the zeros too, unless they're grouped in blocks or you remove whole channels.
03 Whole channels
Removing whole channels really does speed things up.

If you remove whole channels instead of scattered weights, meaning whole filters in a layer, the network really does get narrower. Each layer has less work to do, and the phone feels it right away. You pick the channels that contribute the least. You lose a bit more than with scattered pruning, but the gain shows up in the milliseconds.

In practice. Measure the latency^01.01 after every round of pruning. A model that's smaller on paper is sometimes no faster on the phone.
04 The teacher
A big model teaches a small one.

A big model, the teacher, answers each example with its own percentages. Mug 70, glass 25, keys 5. A small model, the student, learns to give the same percentages, as well as the right answer. Those shades hold what the teacher has understood, for example that a mug and a glass look alike. It's called distillation.

In practice. Distilling doesn't need any new labels. The teacher can label thousands of examples that have no answer, and the student learns from all of them.
05 Text too
Models that write can be distilled too.

Many small language models are made this way. The small sizes of Gemma 3 were trained partly by imitating a bigger model. In 2019, DistilBERT kept 97 percent of BERT's abilities with 40 percent fewer weights. For the phone, it's the most common way to get a model that's small but smart.

In practice. If you use a big model from an online service to create examples for a small one, read the terms of use. Some services don't allow their answers to be used to train other models.
06 The architecture
Some networks are born for the phone.

Instead of shrinking a big network, you can start from one made for the phone. MobileNet splits each filter into two light steps. EfficientNet grows in depth, width and resolution all together, in proportion. MobileNetV4, from 2024, was designed by measuring milliseconds on real phones and chips.

In practice. Before you slim down a big model, look for a network designed for the phone that fits your task. It often does better, and with less effort.
07 Together
The techniques add up, in the right order.

A typical path starts from an architecture made for the phone. You distill from a big model to win back accuracy. You prune the channels that matter least. Last of all, you quantize to 8 bits. Each step takes off a little size and a little accuracy, and after each one you measure again, on the phone.

In practice. Keep a sheet with the size, the milliseconds and the accuracy after every step. If a step saves little size and costs a lot of accuracy, go back.

01 / 07

(Go deeper)

A few more numbers.

(01)

The winning lottery ticket

In 2019 Jonathan Frankle and Michael Carbin at MIT ran a curious experiment. They pruned a trained network, then took the small network that was left, reset it to its starting weights and trained it on its own. With 10 or 20 percent of the weights, it reached the same accuracy as the full network.

They called it the lottery ticket hypothesis. A big network contains many small networks, and training finds the lucky one. It doesn't tell you how to find it before training, but it explains why pruning works so well.

(02)

Distillation with temperature

Geoffrey Hinton and his colleagues described distillation in 2015. The trick is the temperature. Before the softmax, the teacher's scores are divided by a number greater than one, and the percentages get softer. Classes that are wrong but plausible get more weight, and the student sees the similarities better.

The student learns with two goals at once. It imitates the teacher's soft percentages, and it gives the right answer on the true labels, when there are any.

p_soft(i) = exp(z_i / T) / sum of exp(z_j / T)        with T between 2 and 10

(03)

How you design a network for the phone

The first networks for the phone were designed by hand. Since 2019, with MobileNetV3, Google has used automatic architecture search. A program tries thousands of variants and measures them on a real phone, keeping the ones with the best balance between accuracy and milliseconds.

MobileNetV4 took the idea further. The search looked for a network that works well on CPUs, GPUs, NPUs and other accelerators all at once, instead of on a single phone. The result is simple blocks that almost any chip runs well, which is why people use them outside Google too.

(Try it on your phone)

A plan to slim it down

Think of a model you want to bring to the phone, maybe the one from your own project. Write down the goals and the first step you'd try. The card stays saved in your browser.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

You set half the weights to zero, but the model isn't any faster. Why?
(02)

In distillation, what does the student learn besides the right answer?
(03)

Where should you start for a model you want to bring to the phone?
(04)

In what order are the techniques usually used?

New words, and where they are explained

Pruningchapter 2
Distillationchapter 4

Sources and reading

To close the lesson, answer the quiz

← Quantizing Back to the map

(Next lesson · 06.04) Formats and engines