(Lesson 05.03 · 20 minutes · No sensor · For everyone)

A language model in your pocket.

A model that writes can fit in a phone, if it's small enough. Here you'll work out memory and speed with real numbers, and find out what these models do well and where they make things up.

What you learn

How a language model writes, one token at a time
How to work out how much memory you need and how many tokens per second to expect
What the context is and why a phone can hold only a little of it
What a small model is good for, and what it isn't

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The game
A language model guesses the next piece.

A language model does just one thing. It reads a text and estimates which token^05.01 comes next. It adds that token, rereads everything, and estimates the next one, one piece at a time. Answers, summaries and translations all come out of this repeated game. A model of this kind that's small enough to fit in a phone is called an SLM.

In practice. When a model gets a fact wrong with total confidence, remember the game. It isn't looking for the truth, it picks the piece that sounds most likely.
02 The parameters
Billions of weights, all kept in memory.

A model's size is given in parameters^01.03. Google's Gemma 3 1B has one billion, and Meta's Llama 3.2 3B has three billion. The big models in data centers have hundreds of billions. To answer, all the parameters have to fit in RAM^01.02, the phone's working memory.

In practice. Check how much RAM your phone has in the settings. The system and the apps already use a good part of it, so about half is left for the model.
03 The bits
Each weight takes two bytes, or half a byte.

Stored the normal way, each weight takes 2 bytes. So a model with one billion weights weighs 2 gigabytes. With 4-bit quantization^01.05 each weight takes half a byte, and the same model drops to around 600 megabytes. It loses a little quality, but it fits in almost any recent phone.

In practice. Here's the quick math. Billions of parameters times the bits, divided by eight, gives you gigabytes. Three billion at 4 bits makes one and a half gigabytes, plus some room to spare.
04 Speed
Memory decides how many tokens per second.

For every token it writes, the model reads all of its weights from memory again. If the model weighs 1.5 gigabytes and the phone moves 60 gigabytes per second, you can't go past 40 tokens per second. Not even with the most powerful NPU^01.02. Real phones reach only part of that. To read comfortably, about ten are enough.

In practice. A smaller model is also faster at writing. On the same phone, half the weights means almost twice the tokens per second.
05 The context
The model only remembers a certain number of tokens.

Besides the weights, the model keeps the conversation so far in memory, and that's called the context. The longer it gets, the more memory it takes, and the longer you wait for the first word. On a phone, a few thousand tokens is fine, which means a few pages. Pasting a whole book into a small model doesn't work.

In practice. To work on long documents, don't paste them in whole. First find the right pieces with search by meaning^05.02, and give the model only those.
06 What it can do
Good at rewriting, weak on facts.

A model with one to three billion parameters can summarize an email and correct a text. It can also rewrite a sentence in a different tone, and pick out dates and names. On facts, though, it's often wrong, because there's little room in its weights to remember the world. It confidently makes up dates, quotes and names. People say it hallucinates.

In practice. Use small models on texts you give them, not on general knowledge questions. Summarizing what's in front of it is much safer than remembering.
07 In phones
Many phones already have one inside.

Since the end of 2023, some phones have had a language model built into the system. Google put Gemini Nano in its Pixel phones, and Apple put a model of about 3 billion parameters in iPhones with Apple Intelligence. They summarize notifications and correct text without sending anything over the network. For harder requests, they hand off to a bigger model in the cloud.

In practice. Try the notification summaries or the text correction in airplane mode. If it works, the model inside the phone is doing the job.

01 / 07

(Go deeper)

A few more numbers.

(01)

The memory math

The table shows how much some open models weigh. The 16-bit column is the full size, and the 4-bit column is the quantized version used on phones. On top of the weights you need memory for the context, a few hundred MB for a few thousand tokens.

On a phone with 8 GB of RAM, models up to 3 or 4 billion parameters at 4 bits fit comfortably. The 8 billion ones need a top-end phone with 12 or 16 GB.

Model	Parameters	At 16 bits	At 4 bits
Gemma 3 1B	1 billion	2 GB	about 0.6 GB
Llama 3.2 3B	3.2 billion	6.4 GB	about 1.8 GB
Phi-3.5 mini	3.8 billion	7.6 GB	about 2.2 GB
Gemma 3 4B	4 billion	8 GB	about 2.5 GB
Llama 3.1 8B	8 billion	16 GB	about 4.7 GB

(02)

Reading is fast, writing is slow

A model works in two phases. First it reads the whole question, and it does that in parallel, many tokens at once. Here computing power matters, and the NPU helps a lot. Then it writes, one token at a time, and each token means rereading all the weights. Here memory speed matters.

That's why on phones the answer starts after a moment and then flows at a steady speed. A long question makes the first wait longer, and a long answer makes the rest longer. The memory math from the chapter on speed applies to the second part.

(03)

Trying them on your phone

There are free apps that download an open model and run it on the phone. Google AI Edge Gallery, on Android, tries out Gemma models and measures tokens per second. PocketPal and MLC Chat also work on iPhone. In the browser, with WebGPU, there's WebLLM.

Each model downloads once, from a few hundred MB to a few GB, and then works in airplane mode. It's the most direct way to see the math of this lesson with your own eyes, including the phone that heats up and slows down after a few minutes.

(Try it on your phone)

Does it fit in my phone?

Pick a model, the bits of the weights, the RAM and the kind of phone. The tool does the math of this lesson. It tells you how much the model weighs, whether it fits in memory and how many tokens per second it can write at most.

The tool needs JavaScript turned on.

These are rough numbers, worked out with the rules of the lesson. The models you download, like the ones in the table in the first Go deeper section, often weigh a bit more. For your own phone, try one of the apps in the third Go deeper section and compare.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

A model has 3 billion parameters quantized to 4 bits. Roughly how much does it weigh?
(02)

Why does a smaller model also write faster?
(03)

What is a small language model good for?
(04)

You want it to use a 300-page manual. What do you do?

New words, and where they are explained

Small language modelchapter 1
Contextchapter 5
Hallucinationchapter 6

Sources and reading

To close the lesson, answer the quiz

← Classifying and searching text on the phone Back to the map

(Next lesson · 05.04) Models that see and speak