-
01 Every pixel
Every pixel gets its own class.
A spot on a leaf is not a box. To measure it, you need to know which pixels are spot and which aren't. That's exactly what segmentation01.04 does. It gives a class to every pixel of the photo, for example leaf, spot or background. The result is a second image, the same size as the first, made of colored areas.
In practice. Segmentation is for when the shape or the area matters. If you only need to know where an object is, a box costs much less.
-
02 The mask
The result is a mask, pixel by pixel.
For each pixel, the model gives a confidence02.01 to every class. Keeping the highest one gives you a mask, like a sheet of paper cut out with scissors. The model often works small, for example at 256 by 256, and the mask is then scaled up to the size of the photo. That's why the edges sometimes look soft.
In practice. With the mask the math gets simple. The spot pixels divided by the leaf pixels give you the sick part as a percentage, without a ruler.
-
03 The U shape
First it narrows down to understand, then it widens out to draw.
The network has two halves. The first, the encoder, shrinks the photo layer after layer and works out what's there. The second, the decoder, enlarges the result back to the starting size and says where things are. Bridges between the two halves bring back the fine details lost along the way. When you draw it, the network has the shape of a U.
In practice. If the mask finds the object but gets the edges wrong, the problem is the resolution. If it gets the object itself wrong, you need better photos.
-
04 Instances
Three apples, or apple one, apple two and apple three.
Semantic segmentation colors all the apples the same way, because they belong to the same class. It can't tell where one ends and the next begins. Instance segmentation, on the other hand, separates each object, with its own mask and its own number. It usually starts from a detector and draws the mask inside each box.
In practice. To count objects that touch each other, like coffee beans or cells, you need instance segmentation. Semantic segmentation merges them into a single blob.
-
05 Video calls
The blurred background is a model that's always on.
When the phone blurs the background of a video call, it's segmenting every frame into two classes, person and background. It has to do it 30 times per second, while it's also compressing the video and sending it. The MediaPipe model that does this job weighs about 250 kilobytes and looks at images of 256 by 256 pixels.
In practice. Hair and fingers are the hardest parts, because they're thin compared to the model's pixels. A light that makes the person stand out from the background helps more than a big model.
-
06 Depth
How far away each pixel is, from a single photo.
Another kind of model gives each pixel a number instead of a class, the distance. It's called depth estimation. From a single photo it doesn't know the real meters, but it can tell what's in front and what's behind, from shadows, sizes and lines. Many phones use it to blur the background in portraits.
In practice. iPhone Pro models since 2020 also have a LiDAR sensor, which really measures distance with light. Where there is one, the model fills in the details between one point and the next.
-
07 The choice
Class, box or mask, the question decides.
The more precise the answer, the more it costs, both in calculations and in labels. A class takes one tap to assign, a box a few seconds. In the COCO dataset, outlining a single object took more than 70 seconds. Before you choose, ask yourself what you need to know. Whether the leaf is sick, how many spots it has, or how big they are.
In practice. When in doubt, start with classification. Move to the box or the mask only when the question really calls for it.