(Lesson 01.02 · 15 minutes · No sensor · For everyone)

What's inside your phone.

A model runs inside a chip the size of a fingernail, with a battery that has to last all day. Here you'll open the phone layer by layer, and see what each part does when a neural network is at work.

What you learn

What the CPU, GPU and NPU do, and why networks prefer the last two
Why memory slows a model down more than the math does
Which sensors a model can use on your phone
What it costs in battery and heat to run a model

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The chip
A whole computer inside one chip.

If you open a phone, under the screen you'll find the battery and a board full of components. In the middle there's a chip smaller than a fingernail. It's called an SoC, a system on a chip, because a whole computer fits inside it, with all its processors.

In practice. In a phone's spec sheet, the name of the chip matters more than the brand. Two phones with the same chip run a model at roughly the same speed.
02 The CPU
The CPU does a few things at a time, very fast.

The CPU is the processor that runs the system and the apps. It has six to eight cores, some powerful and others that save battery. Each core runs one instruction after another. It can do anything, but a neural network asks for millions of identical multiplications, and one at a time, that's a lot.

In practice. A small model runs fine on the CPU too. In the browser, it's the fallback when the others aren't available.
03 The GPU
The GPU does many things at once.

The GPU was born to draw the screen, millions of pixels sixty times a second. To do that, it has hundreds of small calculators all working together. The multiplications in a neural network are a lot like pixels, many and all the same. That's why the GPU does them much faster than the CPU.

In practice. The course lab uses your phone's GPU through the browser. You don't need to install anything, the page takes care of it.
04 The NPU
The NPU is built only for neural networks.

The NPU is a part of the chip designed for one thing only, the multiplications of neural networks. It often works with small, 8-bit numbers, and for the same calculation it uses much less power than the GPU. Apple calls it the Neural Engine, Qualcomm calls it Hexagon.

In practice. An NPU's power is measured in TOPS, trillions of operations per second. Apple's A17 Pro chip claims 35. That's a ceiling, and real apps use only part of it.
05 Memory
The chip often waits for memory.

The model's numbers live in RAM, the phone's working memory. Before each calculation, they have to be brought into the chip. A phone moves between 50 and 80 gigabytes a second, a data center card more than 3,000. So the chip spends a lot of time waiting for data.

In practice. A 2-gigabyte model that writes text reads all of its numbers for every piece of a word. At 60 gigabytes a second, it can't go past 30 pieces a second, whatever the NPU.
06 The sensors
Eyes, ears and a sense of balance.

A model can start from anything the phone measures. The cameras give images, two or three microphones give sound, the accelerometer and the gyroscope give motion. Then there's the compass, the barometer, GPS and the light sensor. Each sensor is a possible input, and the courses on the path start from here.

In practice. The microphone records 48,000 numbers a second, the accelerometer a few hundred. The more numbers come in, the more work the model has to do to keep up.
07 The battery
The battery is the bill to pay.

A 4,500 milliamp-hour battery holds about 17 watt-hours. An app that keeps the camera and the NPU running can draw two or three watts, so the phone runs flat in a few hours. And it heats up. When it gets too hot, the system slows the chip down to protect it, and the model runs slower.

In practice. Measure a model's speed after five minutes of continuous use, not on the first try. A cold phone is always faster than it will be in the field.

01 / 07

(Go deeper)

A few more numbers.

(01)

Why memory costs more than math

In 2014 Mark Horowitz, at Stanford, measured how much energy the operations inside a chip cost. Multiplying two 32-bit numbers with a decimal point costs about 3.7 picojoules. Reading those same 32 bits from external memory costs about 640, almost two hundred times as much.

That's why models for phones try to move as little data as they can. They use smaller numbers, 8 bits instead of 32, and networks that reuse the same numbers many times once they reach the chip. With 8-bit numbers the multiplication costs less too, about 0.2 picojoules.

640 pJreading 32 bits from memory
3.7 pJmultiplying two 32-bit numbers
0.2 pJmultiplying two 8-bit numbers

(02)

Who can use the NPU, and when

Not every app can reach the NPU. On the iPhone you get there with Core ML, Apple's engine, which decides on its own whether to use the Neural Engine, the GPU or the CPU. On Android each maker has its own tools, and Google offers LiteRT, which tries to use the right accelerator for each phone.

For now the browser can't reach the NPU. There's a proposal to make that possible, called WebNN, but it isn't available on every phone yet. That's why the course lab uses the GPU, which the browser reaches through WebGL and WebGPU. It's a bit less efficient than the NPU, but it works everywhere.

(03)

Cold and hot

A phone has no fans. The chip's heat goes into the case and from there into the air, and if the chip makes more heat than can get out, the system lowers the speed of the processors. It happens after a few minutes of heavy work, sooner in summer or with the phone in a case.

For a model, this means the same photo can take 20 milliseconds right after you turn the phone on, and much longer after ten minutes. When you test your model in the lab, let it run for a while before you look at the numbers.

(Try it on your phone)

Measure your phone

Press the button. The page looks at what's inside your phone and measures how many calculations it does in one second, first with the CPU and then with the GPU. It doesn't send anything to anyone, the numbers stay here. Keep the page open until it's done.

The tool needs JavaScript turned on.

These are rough estimates, and they change from one run to the next, especially if the phone is warm or in battery saver mode. Try twice, once with the phone cold and once after a few minutes of continuous use. On the second run the page also reminds you of the numbers from the first.

(Quiz)

3 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

Why does a neural network run better on the GPU than on the CPU?
(02)

A model that writes text runs slowly even on a phone with a powerful NPU. What's the most likely cause?
(03)

When should you measure a model's speed on the phone?

New words, and where they are explained

SoCchapter 1
CPUchapter 2
GPUchapter 3
NPUchapter 4
TOPSchapter 4
RAMchapter 5
Sensorchapter 6

Sources and reading

To close the lesson, answer the quiz

← Where AI does its thinking Back to the map

(Next lesson · 01.03) How a model learns