(Lesson 06.04 · 18 minutes · No sensor · For everyone)

Formats and engines.

LiteRT, Core ML, ONNX Runtime, ExecuTorch, MediaPipe, WebGPU. Here you'll sort out the names, understand what's inside a model file, and see which engines can reach the phone's NPU.

What you learn

What a model file contains and what an inference engine does
Which engines are used on Android, on iPhone and in the browser
How to choose the right engine for a project

You need to know

If you don't remember one of these, the link takes you to the chapter that explains it.

How do you want to follow it

Preparing the scene…

01 The file and the engine
A model is a file, and an engine runs it.

After training, the model is saved to a file. Inside are the weights and the recipe, meaning the order of the operations. To use it on the phone you need a program that reads the file and does the math, and that's the inference^01.03 engine. Each engine reads its own formats and decides which part of the chip does the work.

In practice. Choose the engine before you train. Some operations exist in one format and not in another, and finding that out at the end costs you dearly.
02 LiteRT
Google's engine, born for Android.

LiteRT, which was called TensorFlow Lite until 2024, reads tflite files. It runs on Android, iPhone, computers and microcontrollers. To use the GPU and the NPU it hands the work to delegates, pieces of code that talk to the right accelerator. It's the engine behind MediaPipe and behind many of Google's models.

In practice. If a delegate can't do an operation, that operation goes back to the CPU. Too many trips between the NPU and the CPU can make the model slower than before.
03 Core ML
On iPhone there's Core ML, and it chooses on its own.

On iPhone, Apple's engine is called Core ML, and it reads mlpackage files. It decides by itself whether to use the Neural Engine, the GPU or the CPU, layer by layer. To bring a model over from PyTorch, you use coremltools. During the conversion you can also compress the model to 8 or 4 bits.

In practice. Core ML doesn't always tell you where the model is running. With the Instruments tool in Xcode, you can see whether it's really using the Neural Engine.
04 ONNX
ONNX, the format that travels from one world to another.

ONNX is an open format, created to move models between different libraries. You export to it from PyTorch and run it with ONNX Runtime, on Android, iPhone, computers and even in the browser. To reach the accelerators it uses execution providers, which work much like the delegates in LiteRT. For example Core ML on iPhone, or QNN on Qualcomm chips.

In practice. When you export to ONNX, test the file right away with a few examples and compare its answers with the original model's. You want to find the differences now, not later.
05 ExecuTorch
PyTorch's engine for the phone.

ExecuTorch is Meta's engine for bringing PyTorch models to phones and small devices without going through other formats. The model is prepared on the computer as a pte file, already split into the parts that will run on the CPU, the GPU or the NPU. Meta uses it in its own apps and for Llama models on the phone.

In practice. If your model starts out in PyTorch, ExecuTorch saves you a conversion. Fewer conversions mean fewer surprises.
06 In the browser
In the browser, you reach the GPU with WebGPU.

For now, a web page can't call the NPU. It uses the CPU through WebAssembly, and the GPU through WebGL or WebGPU. That's what TensorFlow.js, ONNX Runtime Web, MediaPipe and transformers.js are built on. It's the route this course's lab takes. There's nothing to install, and in exchange you give up a bit of efficiency.

In practice. WebNN is the proposal for reaching the NPU from the browser. Once it's available everywhere, the same pages will run faster without changing the model.
07 Choosing
You choose by looking at the phone and the model.

For an Android-only app, LiteRT is the simplest choice. For iPhone only, Core ML. For both with a single model, ONNX Runtime or LiteRT. If the model starts out in PyTorch and has to run on the NPU, ExecuTorch. If you don't want to install anything, the browser. MediaPipe covers the ready-made cases, like hands, faces and objects.

In practice. Whichever engine you choose, measure on the reference phone. The same model can run at very different speeds from one engine to another.

01 / 07

(Go deeper)

A few more numbers.

(01)

What's inside a tflite file

A tflite file is a single block of bytes, laid out so it can be read without being copied, in a format called FlatBuffers. It holds the list of operations, like convolutions and additions, the list of tensors with their shapes, the weights, and for quantized models the scale and zero point of each tensor.

To look inside, there's Netron, a free viewer that also works in the browser. It shows the network as a block diagram, with the shapes of the inputs and outputs. It also reads ONNX and Core ML files, and many other formats.

(02)

Which engine reaches what, on the phone and beyond

The table sums up how each engine reaches the three parts of the chip. On the CPU almost all of them use XNNPACK, a highly optimized library from Google. For the NPU you need pieces written by the companies that make the chip, like Qualcomm, MediaTek or Apple. That's why an NPU can work with one engine and not with another, or only with 8-bit models.

The same models also run outside the phone. On a microcontroller that costs a few euros, with LiteRT for Microcontrollers, a model of a few tens of KB recognizes a gesture or a sound on a small battery. On a Raspberry Pi, a model like the lab's runs with a camera and no screen. On boards with a GPU, like NVIDIA's Jetson, models follow a video in real time.

Engine	File	CPU	GPU	NPU
LiteRT	tflite	XNNPACK	GPU delegate	Chip makers' delegates
Core ML	mlpackage	Yes	Yes	Neural Engine
ONNX Runtime	onnx	Yes	Yes	Providers like QNN and Core ML
ExecuTorch	pte	XNNPACK	Vulkan and Metal	Chip makers' backends
Browser	various	WebAssembly	WebGL and WebGPU	Not yet, WebNN is on the way

The tutorials, one for each course →

(03)

Converting without surprises

Every conversion can change the answers a little or a lot. An operation translated in a different way, a color order flipped, a normalization step forgotten. The check is simple. You run the same hundred examples through the original model and the converted one, and compare the outputs.

For models with decimal numbers the differences should be tiny, in the third or fourth decimal place. For quantized ones you check that the chosen class is the same almost every time. If it doesn't match, the problem is almost always in how the input is prepared, not in the engine.

(Try it on your phone)

Inside a real model

Download the model that blurs the background in the segmentation lesson. It's 250 KB. Then open it in Netron, from your phone or your computer, and look for these things in the block diagram.

Download selfie_segmenter.tflite Open Netron ↗

(01) The input and its shape, an image of 256 by 256 pixels with 3 colors
(02) The output, the mask with a confidence for each pixel
(03) A convolution block, with the size of its filters
(04) The weights of a filter, when you tap the block
(05) The point where the network narrows and then widens again

You've seen the recipe the first chapter talks about. Each block is an operation that the engine has to know how to run on the CPU, the GPU or the NPU.

(Quiz)

4 questions to close the lesson.

When you have answered them all, the lesson counts as done and its stone on the map turns yellow. Answering is what counts, not the score.

(01)

What does a model file contain?
(02)

The NPU delegate doesn't support one of the operations. What happens?
(03)

You're making an app for iPhone only. Which engine is the most natural choice?
(04)

Why does the course lab use the GPU and not the NPU?

New words, and where they are explained

Inference enginechapter 1
LiteRTchapter 2
WebGPUchapter 6

Sources and reading

To close the lesson, answer the quiz

← Pruning, distilling, choosing the architecture Back to the map

(Next lesson · 06.05) An app that works without a network