From the lab to your own page.

1. (01)

   Export the model from the lab

   Open your project in the lab and go to the Use step. At the bottom, under the model card, tap Export the model and the card. The phone downloads a .json file with the project's name.

   Inside are the classes, the shape of the network, its weights as lists of numbers, the measurements and the card. It's everything the page needs, and for a model with three classes it weighs about 80 KB.

2. (02)

   Set up the folder

   Make a folder with two files, index.html and app.js. You can download them from this page and change them later. The page has a field to choose the exported file, a button for the camera, the video and a place for the answer.

   <main>
     <h1 data-en="My model, in my own page">Il mio modello, in una pagina mia</h1>
     <p class="soft" data-en="Everything runs in your browser. The photos never leave the phone.">Tutto gira nel tuo browser. Le foto non escono dal telefono.</p>
     <label class="file">
       <span data-en="The file exported from the lab (.json)">Il file esportato dal laboratorio (.json)</span>
       <input type="file" id="file" accept=".json,application/json">
     </label>
     <button type="button" id="start" disabled></button>
     <div class="view"><video id="video" muted playsinline></video></div>
     <p id="answer" aria-live="polite"></p>
     <ul id="bars"></ul>
     <p class="soft" id="ms"></p>
     <p class="soft" id="status" role="status"></p>
   </main>
   <script type="module" src="app.js"></script>

3. (03)

   The two libraries

   The lab's model has two parts. The body is MobileNet v3 small, a network Google already trained that turns a photo into 1,024 numbers. The head is the small network you trained, and it turns those numbers into the answer.

   MediaPipe runs the body and TensorFlow.js runs the head. The libraries come from jsDelivr with the version written in the address, so they don't change on their own. The body has to be the same file as in the lab, and Google's file is.

   import * as tf from "https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@4.22.0/dist/tf.fesm.min.js";
   import { FilesetResolver, ImageEmbedder } from "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@1.0.1/vision_bundle.mjs";
   
   const VISION_WASM = "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@1.0.1/wasm";
   // the same body as the lab. With a different one the head would get numbers it has never seen
   const BODY_MODEL = "https://storage.googleapis.com/mediapipe-models/image_embedder/mobilenet_v3_small/float32/latest/mobilenet_v3_small.tflite";

4. (04)

   Read the exported file

   The file has Italian or English keys, depending on the lab's language. The function takes either one and checks that the model comes from the Photo input.

   // The exported file has Italian or English keys, depending on the lab's language
   function readExport(json) {
     const get = (o, it, en) => (o && (o[it] ?? o[en]));
     if (!/^edge-in-tasca-(laboratorio|lab)\/1$/.test(get(json, "formato", "format") || "")) throw new Error(T.badFile);
     if (!/^(foto|photo)$/.test(get(json, "ingresso", "input"))) throw new Error(T.notPhoto);
     const m = get(json, "modello", "model");
     return {
       name: get(json, "nome", "name"),
       classes: get(json, "classi", "classes") || [],
       unit: get(json, "unita", "unit") || "",
       spec: get(m, "architettura", "architecture"),   // the kind of head and its sizes
       norm: get(m, "normalizzazione", "normalization"), // mean and spread of each number, or null
       weights: get(m, "pesi", "weights"),             // [{ shape, data }], one per weight tensor
     };
   }

5. (05)

   Rebuild the head

   You rebuild the head with the same layers as in the lab, in the same order. For a Photo model with classes it's a single layer, with 1,024 inputs and one output per class. Then setWeights puts the weights from the file back in their places, one after the other.

   // The same layers as the lab's head, in the same order. Dropout has no weights,
   // so the weights in the file go back into their places one after the other
   function buildHead(spec) {
     const m = tf.sequential(), L = tf.layers;
     const dense = (units, activation, first) => L.dense({ units, activation, ...(first ? { inputShape: [spec.dim] } : {}) });
     if (spec.tipo === "lineare") m.add(dense(spec.uscite, "softmax", true));
     else if (spec.tipo === "densa") { m.add(dense(32, "relu", true)); m.add(L.dropout({ rate: 0.2 })); m.add(dense(spec.uscite, "softmax")); }
     else if (spec.tipo === "densa2") { m.add(dense(48, "relu", true)); m.add(L.dropout({ rate: 0.2 })); m.add(dense(24, "relu")); m.add(dense(spec.uscite, "softmax")); }
     else if (spec.tipo === "numero-lineare") { m.add(dense(16, "relu", true)); m.add(dense(1)); }
     else if (spec.tipo === "numero") { m.add(dense(16, "relu", true)); m.add(L.dropout({ rate: 0.1 })); m.add(dense(8, "relu")); m.add(dense(1)); }
     else throw new Error(`Head "${spec.tipo}" isn't supported here`);
     return m;
   }
   function loadHead(model) {
     const head = buildHead(model.spec);
     head.setWeights(model.weights.map(w => tf.tensor(w.data, w.shape)));
     return head;
   }

6. (06)

   Load the body

   MediaPipe downloads the body, about 4 MB, and gets it ready. It tries the GPU first and uses the CPU if there isn't one. The options are the same as in the lab, with the 1,024 numbers normalized, otherwise the head gets numbers that differ from the ones it learned with.

   async function loadBody() {
     const files = await FilesetResolver.forVisionTasks(VISION_WASM);
     const options = { baseOptions: { modelAssetPath: BODY_MODEL }, runningMode: "IMAGE", l2Normalize: true, quantize: false };
     // the GPU if there is one, otherwise the CPU
     try { return await ImageEmbedder.createFromOptions(files, { ...options, baseOptions: { ...options.baseOptions, delegate: "GPU" } }); }
     catch (e) { return ImageEmbedder.createFromOptions(files, { ...options, baseOptions: { ...options.baseOptions, delegate: "CPU" } }); }
   }

7. (07)

   From the camera to the square

   The lab only looks at the square in the middle of the frame, shrunk to 256 by 256 pixels. The page does the same, so the model sees the photos the way it saw them while learning.

   const video = $("#video");
   const square = document.createElement("canvas");
   square.width = square.height = 256;
   async function startCamera() {
     const stream = await navigator.mediaDevices.getUserMedia({ audio: false, video: { facingMode: "environment", width: { ideal: 640 }, height: { ideal: 480 } } });
     video.srcObject = stream;
     await video.play();
   }
   // the central square of the frame, 256 by 256 pixels, as in the lab
   function grabSquare() {
     const W = video.videoWidth, H = video.videoHeight, s = Math.min(W, H);
     square.getContext("2d").drawImage(video, (W - s) / 2, (H - s) / 2, s, s, 0, 0, 256, 256);
     return square;
   }

8. (08)

   The answer

   Four times a second the page takes the square and passes it to the body, then to the head. Out comes a probability for each class. The page shows the highest one, the bars for all of them and how many milliseconds it took.

   function answer(body, head, model) {
     const t0 = performance.now();
     const feat = body.embed(grabSquare()).embeddings[0].floatEmbedding;    // 1,024 numbers
     const out = tf.tidy(() => {
       const x = model.norm ? Array.from(feat, (v, i) => (v - model.norm.mu[i]) / model.norm.sd[i]) : Array.from(feat);
       return Array.from(head.predict(tf.tensor2d([x])).dataSync());
     });
     return { out, ms: performance.now() - t0 };
   }
   function show({ out, ms }, model) {
     if (out.length === 1) {
       $("#answer").textContent = `${out[0].toLocaleString(EN ? "en" : "it", { maximumFractionDigits: 1 })} ${model.unit}`.trim();
     } else {
       const best = out.indexOf(Math.max(...out));
       $("#answer").textContent = `${model.classes[best]} · ${Math.round(out[best] * 100)}%`;
       $("#bars").innerHTML = model.classes.map((c, i) => `<li><span>${c}</span><i style="width:${Math.round(out[i] * 100)}%"></i><b>${Math.round(out[i] * 100)}%</b></li>`).join("");
     }
     $("#ms").textContent = `${ms.toLocaleString(EN ? "en" : "it", { maximumFractionDigits: 1 })} ${T.ms}`;
   }

9. (09)

   Put it together and try it

   The last piece connects the file field and the button. Then open a terminal in the folder and type npx serve, or python -m http.server if you have Python. Open the address that shows up, choose the exported file and turn on the camera.

   On the phone you need https. The simplest way is to publish the folder, for example on GitHub Pages or Netlify, and open the address on your phone.

   let model = null, head = null, body = null;
   $("#file").addEventListener("change", async e => {
     const file = e.target.files[0];
     if (!file) return;
     try {
       say(T.reading);
       model = readExport(JSON.parse(await file.text()));
       if (head) head.dispose();
       head = loadHead(model);
       if (!body) { say(T.body); body = await loadBody(); }
       say(T.ready(model.name));
       $("#start").disabled = false;
     } catch (err) { say(err.message); }
   });
   $("#start").addEventListener("click", async () => {
     try { await startCamera(); } catch (err) { say(T.denied); return; }
     $("#start").hidden = true;
     say(T.on);
     // four answers a second are enough, and they leave the phone time to breathe
     setInterval(() => { if (video.readyState >= 2) show(answer(body, head, model), model); }, 250);
   });

Now the model is yours. You can change the page, make the phone play a sound when it sees a class, or add a service worker^06.05 so it works offline too. The other tutorials →