(Lesson 06.02 · 16 minutes · No sensor · For everyone)

Quantizing.

Writing the weights with fewer bits makes a model four times smaller, and often faster. Here you'll see how it's done, how much you really lose and what to do when you lose too much, all the way down to the 4 bits of the models that write.

What you learn

How you go from 32 to 8 bits, and why the error on each weight is tiny
How much space and speed you gain
What calibration and training on the steps are
Why language models go down to 4 bits

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The weights
Every weight is a decimal number.

After training, a model is a long list of weights^01.03, numbers like 0.0173 or minus 0.84. Usually each one is stored in 32 bits, which is 4 bytes, and has about seven significant digits. MobileNet v2 has 3.4 million weights, so it takes up almost 14 megabytes. Do you really need all that precision?

In practice. Here's how you estimate the size of a model. The number of weights times the bytes in each weight. With 32 bits that's 4 bytes, and with 8 bits it's just one.
02 Fewer digits
With 256 possible values, you lose very little.

Quantization^01.05 rewrites every weight with fewer bits. With 8 bits, a number can only take 256 different values. You take the range of the weights in a layer, say from minus 1 to 1, and divide it into 256 steps. Each weight moves to the nearest step. The error on each one is half a step at most.

In practice. You quantize layer by layer, and sometimes channel by channel. A layer with a few very large weights stretches the range and wastes steps, and that's where you lose the most.
03 A quarter
A quarter of the space, and faster math.

Going from 32 to 8 bits makes the model four times smaller. MobileNet v2 goes from 14 megabytes to less than 4. The calculations change too. Multiplying 8-bit integers takes much less energy than multiplying decimal numbers. And NPUs are built for exactly that. Often the model gets two or three times faster.

In practice. To use the phone's NPU^01.02, the model almost always has to be 8-bit. Many NPUs can't work with decimal numbers at all.
04 How much you lose
Usually less than a point, sometimes a lot more.

On many image models, 8-bit quantization costs less than one point of accuracy. That's not a rule. Small models that are already squeezed tight, like some versions of MobileNet, can lose quite a lot. The only way to know is to measure, on the same test photos, before and after.

In practice. Check the confusion matrix of the quantized model, not just the total. Sometimes the model loses almost everything on a single class.
05 Calibration
About a hundred examples help you choose the steps.

You already know the weights, but the values passing from one layer to the next have to be written in 8 bits too. To find out what range they fall in, you run about a hundred real examples through the model and see where they end up. This step is called calibration. With the wrong examples, the steps end up in the wrong place.

In practice. For calibration use examples taken from real use, with the same lighting and the same backgrounds. Not the cleanest photos you have.
06 Training on the steps
The model can learn to live with the steps.

If you lose too much after quantization, there's another way. While the model is learning, you simulate the steps, and the weights get rounded the way the phone will round them. That way the model learns to work well with those numbers too. It's called quantization aware training, and it usually wins back almost everything.

In practice. Try simple quantization after training first. Move to training with the steps only if you lose too much, because it means training all over again.
07 Four bits
For models that write, you go down to four.

For language models memory is everything, so you go down to 4 bits, which is 16 steps per weight. That takes a few extra tricks. The weights are quantized in small groups, each with its own steps, and some parts stay more precise. That's how a model with 3 billion weights goes from 6 gigabytes to less than 2.

In practice. Below 4 bits the quality drops fast. Before going any lower, try a smaller model at 4 bits and compare.

01 / 07

(Go deeper)

A few more numbers.

(01)

The formula for the steps

For each layer you choose two numbers. The scale, meaning how much one step is worth, and the zero point, meaning which step matches the real zero. With weights from minus 1 to 1 and 256 steps, the scale is 2 divided by 255, about 0.0078.

To write a weight, you divide it by the scale, round it and add the zero point. To read it back, you do the opposite. The weight you read back isn't identical to the original, but the difference is never more than half a step, which is about 0.004 in this example.

q = round(x / scale) + zero        (an integer from 0 to 255)
x ≈ scale × (q − zero)

(02)

Layer by layer or channel by channel

In 2018 Raghuraman Krishnamoorthi at Google compared the ways of quantizing networks for the phone. MobileNet v2, quantized after training with a single scale per layer, fell apart. With one scale for each channel it stayed close to the original, and with training on the steps it won back almost everything.

The reason lies in the MobileNet filters that work on one channel at a time. Some channels have tiny weights and others have large ones. With one scale for all of them, the small ones all land on the same step and get lost. Today almost every tool uses one scale per channel.

(03)

The tools

With LiteRT, you ask the converter for quantization after training, along with a function that supplies the examples for calibration. Core ML has coremltools, PyTorch has its own quantization tools, and ExecuTorch uses them to prepare the model.

Language models use formats made for the purpose. GGUF from llama.cpp offers many 4-bit variants, such as Q4_K_M, and phone apps usually start from those.

conv = tf.lite.TFLiteConverter.from_saved_model("model")
conv.optimizations = [tf.lite.Optimize.DEFAULT]
conv.representative_dataset = a_hundred_real_examples   # calibration
open("model_int8.tflite", "wb").write(conv.convert())

(Try it on your phone)

How many bits do you need?

Choose how many bits to use and watch two things. A photo written with fewer bits per color, and a row of weights sitting on the steps. Below, you see how big MobileNet v2 would be and how far off each weight is on average.

The tool needs JavaScript turned on.

The photo gives you the idea, but it exaggerates. A model copes with the steps much better than your eye does, because the small errors of many weights partly cancel each other out.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

A model has 3.4 million weights at 32 bits. How big is it at 8 bits?
(02)

With 8 bits, what's the largest error on each weight?
(03)

What is calibration for?
(04)

The quantized model loses too much. What's the most sensible next step?

New words, and where they are explained

Calibrationchapter 5
Training on the stepschapter 6

Sources and reading

To close the lesson, answer the quiz

← Measuring for real Back to the map

(Next lesson · 06.03) Pruning, distilling, choosing the architecture