(Lesson 05.04 · 14 minutes · No sensor · For everyone)

Models that see and speak.

You take a photo of the fridge and ask what you can cook. Here you'll see how a small model brings a photo into a conversation, where it works well and where it goes wrong, and when a simpler model is the better choice.

What you learn

How a photo becomes something a language model can read
What CLIP does and why it also helps you search your photos
The typical mistakes of small models that look at images

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The question
A photo and a question, and a written answer.

You take a photo of the fridge and ask what you can cook. The model looks at the photo, reads the question, and answers with a sentence. A model that works with several kinds of data at once, here images and text, is called multimodal. For a few years now there have been small ones that run on a phone too.

In practice. Before you use one, ask yourself whether you really need a sentence. To know whether there's a cup, a model that finds objects is enough, and it's smaller and more reliable.
02 Eyes and words
The photo becomes a row of tokens.

The trick is to turn the photo into something the language model already knows how to read. A small image model cuts the photo into little squares and turns each one into an embedding^05.01. Those numbers go into the language model as if they were text tokens^05.01. To the model, the photo becomes the start of the conversation.

In practice. A photo can be worth hundreds of tokens. It fills the context^05.03 like a few paragraphs of text, and it makes you wait longer for the first word.
03 The bridge
Images and sentences in the same space.

To connect photos and words, you often start from a model trained on millions of pairs of photos and captions. It learns to place the photo of a dog close to the sentence "a dog on the grass". The best known one is called CLIP, released by OpenAI in 2021. The same space is also used to search your photos by typing a word.

In practice. If all you need is to search your photos with words, a model like CLIP is enough. It's much lighter than a model that writes answers.
04 The small ones
A few hundred million weights are enough to describe a photo.

SmolVLM, released by Hugging Face in 2025, also comes in a version with 256 million parameters. It describes a photo, reads a sign, and answers simple questions. Google's Gemma 3n, designed for phones, understands images, sounds and text. They do well with clear scenes and direct questions.

In practice. Ask precise questions. "What color is the car" works better than "describe everything you see".
05 Where they go wrong
They count badly, struggle with small print, and make things up.

Small models go wrong in predictable ways. They miscount objects once there are more than three or four. They struggle to read small text, because the photo gets shrunk before it goes in. And sometimes they describe things that aren't there, like a person in an empty room. It's the same hallucination^05.03 as in the models that write, applied to what they see.

In practice. If you need an exact number, use a model that finds objects one by one. The model that describes is fine for the general idea, not for counting.
06 Privacy
The photo that never leaves the phone.

Photos hold faces, documents, the inside of your home. Sending them to a cloud service means trusting whoever receives them. A small multimodal model looks at them inside the phone. It's slower and less capable than a big one, but for reading a label or describing a receipt it's often enough.

In practice. For someone with low vision, having their surroundings described with no network is very useful, even in places the connection doesn't reach. It's one of the cases where a small model in your pocket is worth more than a big one far away.
07 Choosing
The smallest model that answers your question.

For every problem with images there's a ladder. A class, a box, a mask, and at the top a written sentence. As you climb, the model gets bigger, slower and harder to keep in check. The multimodal model is the last rung. You use it when the answer really is a sentence, and not a name or a number.

In practice. Write down the answer you expect for ten different photos. If they're all words from a list, you need a classifier, not a model that writes.

01 / 07

(Go deeper)

A few more numbers.

(01)

How a photo gets into a language model

The image model most used in these systems is a vision transformer. It cuts the photo into little squares 14 or 16 pixels wide. A photo of 224 by 224 pixels with 16-pixel squares gives 196 squares, so 196 embeddings.

A small layer acts as a bridge and brings them to the size of the language model's embeddings. Many small models then merge nearby squares into groups, to spend fewer tokens on each photo. SmolVLM, for example, uses few tokens for each piece of the image, and that's one reason it fits in so few parameters.

196squares in a 224-pixel photo
256 Mparameters in the smallest version of SmolVLM

(02)

CLIP and photo search

CLIP was trained on 400 million pairs of photos and captions found on the web. It has two halves, one for photos and one for text. It learns to make the embeddings of each photo and its caption similar, and those of mismatched pairs different.

The result is a way to classify without training anything. You write "a photo of a cat" and "a photo of a dog", compute the embeddings of the two sentences, and see which one the new photo looks more like. It works with any class you can put into words, even if it isn't the most accurate method.

(03)

Trying them on your phone

Google AI Edge Gallery, on Android, runs Gemma 3n on the phone and accepts photos as well as text. In the browser, with WebGPU, there are SmolVLM demos that describe what the camera sees without sending anything over the network.

Try them with the same five photos and the same questions, and write down the wrong answers. You'll see the mistakes from this lesson, especially with counting and small print, and you'll get a sense of how much to trust them.

(Try it on your phone)

Photos your phone understands on its own

Put your phone in airplane mode and try these features. The ones that still respond use image and text models that live inside the phone.

(01) Search your photos for a word like sunset, dog or cake
(02) Select and copy the text inside a photo of a document
(03) Press and hold on a person or object in a photo to lift it off the background
(04) Point the camera at a sign in another language and translate it
(05) Ask your phone to describe a photo, if it has that feature

Searching your photos by word uses a shared space for photos and sentences, like CLIP. Lifting the subject off the background is segmentation. Describing a photo is the job of a multimodal model.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

How does a language model look at a photo?
(02)

You ask a small model how many people are in a group photo. What should you expect?
(03)

You just want to search your photos by typing a word. What do you need?
(04)

When is a multimodal model a better choice than a classifier?

New words, and where they are explained

Multimodalchapter 1

Sources and reading

To close the lesson, answer the quiz

← A language model in your pocket The course review Back to the map

(Next lesson · 06.01) Measuring for real