7 Do’s and Don’ts of Using ML on the Web with MediaPipe

developers-%5BGD-BLOG%5D-7-dos-and-don%27ts-of-using-ML-on-the-web-with-MediaPipe.png” name=”twitter:image”/>
developers-%5BGD-BLOG%5D-7-dos-and-don%27ts-of-using-ML-on-the-web-with-MediaPipe.png” style=”display: none;”/>

Posted by: Jen person, Developer Relations Engineer

developers-%5BGD-BLOG%5D-7-dos-and-don%27ts-of-using-ML-on-the-web-with-MediaPipe.png”/>

Web developers looking to bring the power of machine learning (ML) to their web apps should check out MediaPipe solutions. MediaPipe solutions enable you to deploy custom tasks to solve common ML problems with just a few lines of code. Check out our guide in the documentation and try her web demo on Codepen to see how easy it is to get started. MediaPipe Solutions handles much of the complexity of ML on the web, but there are a few things to keep in mind that go beyond the usual JavaScript best practices. We’ve compiled these into a list of seven do’s and don’ts. Read on to get some good tips!

❌ Don’t bundle models with your app

Web developers are accustomed to making their apps as lightweight as possible to ensure the best user experience. You already know that when you have large items to load, you need to download them in a thoughtful way so that users can quickly interact with the content instead of waiting for long downloads. Strategies like quantization have made ML models smaller and accessible from edge devices, but they’re still large enough that you might not want to bundle them into a web app. Save your model to your chosen cloud storage solution. Then, when you initialize the task, the model and WebAssembly binaries are downloaded and initialized. After the first page load, use local storage or Indexed DB Cache models and binaries to make future page loads even faster. An example of this can be seen in the Touchless ATM sample app on GitHub.

✅ Please initialize the task early

Depending on the size of your model, connection speed, and device type, the task may take some time to initialize. Therefore, it is recommended that you initialize your solution before user interaction. Most of Codepen’s code samples perform initialization on page load. Please note that these samples are intended to be as simple as possible so that you can understand the code and apply it to your own use case. Initializing the model on page load may not make sense to you. Focus on finding the right place to spin up tasks so that the processing is hidden from the user.

After initialization, we need to pass a placeholder image to the model to warm up the task. This example shows a function that runs a 1×1 pixel canvas through a Pose Landmark task.

function dummyDetection(poseLandmarker: PoseLandmarker) const width = 1; const height = 1; const canvas = document.createElement('canvas'); canvas.width = width; canvas.height = height; const ctx = canvas.getContext('2d'); ctx.fillStyle = 'rgba(0, 0, 0, 1)'; ctx.fillRect(0, 0, width, height); poseLandmarker.detect(canvas);

✅ Please clean up your resources

One of my favorite parts of JavaScript is automatic garbage collection. In fact, I can’t remember the last time memory management crossed my mind. MediaPipe tasks require a little bit of information to get the most out of it, so it’s nice to have a little bit of memory cached in your memory. MediaPipe Solutions for Web uses WebAssembly (WASM) to run her C++ code inside the browser. You don’t need to know C++, but it’s helpful to know that C++ lets you pick up the garbage yourself. If you don’t free unused memory, your web pages will use more and more memory over time. You may experience performance issues or crashes.

Once the solution is complete, . close() Method.

For example, you can create a gesture recognizer using the following code:

const createGestureRecognizer = async () => const vision = await FilesetResolver.forVisionTasks( " ); gestureRecognizer = await GestureRecognizer.createFromOptions(vision, baseOptions: modelAssetPath: "https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/1/gesture_recognizer.task", delegate: "GPU" , ); ; createGestureRecognizer();

Once the gesture recognition is complete, destroy the gesture recognizer using the following command: close() Method:

gestureRecognizer.close();

Each task includes close Since it is a method, be sure to use it when necessary. Some tasks include close() There are methods for returned results, so please refer to the API documentation for more information.

✅ Try out tasks in MediaPipe Studio

When deciding on or customizing a solution, we recommend trying it out in MediaPipe Studio before writing your own code. MediaPipe Studio is a web-based application for evaluating and customizing on-device ML models and pipelines for your applications. This app allows you to quickly test MediaPipe solutions in your browser using your own data and your own customized ML models. Each solution demo also lets you experiment with model settings such as total number of results and minimum confidence threshold for reporting results. This is especially useful when customizing your solution and allows you to see how your model performs without creating test web pages.

✅ Test on different devices

While it’s always important to test your web app on a variety of devices and browsers to ensure it works as expected, it’s worth adding a reminder here to test early and often on a variety of platforms. I think there is. You can also test devices using MediaPipe Studio, so you can quickly see if your solution works on your users’ devices.

❌ Don’t default to the largest model

Each task lists one or more recommended models. For example, three different models are listed for the object detection task, each with advantages and disadvantages based on speed, size, and accuracy. We tend to think that the most important thing is to choose the model with the highest accuracy, but doing so comes at the cost of speed and increases the size of the model. Depending on your use case, your users may benefit from faster results more than more accurate results. The best way to compare model options is to use MediaPipe Studio. This may sound like an advertisement for MediaPipe Studio, but it’s really useful here.

✅Please contact us!

Are there any do’s or don’ts about ML on the web that you think I’ve missed? Have questions about how to get started? Or have a great project you’d like to share? Contact me on LinkedIn and let me know Please tell me everything about it.