MediaPipe Unity Plugin

This is a Unity (>= 2022.3) Native Plugin to use MediaPipe (0.10.20).

The goal of this project is to port the MediaPipe API (C++) one by one to C# so that it can be called from Unity.
This approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.

With this plugin, you can

• Write MediaPipe code in C#.
• Run MediaPipe's official solution on Unity.
• Run your custom Calculator and CalculatorGraph on Unity.
  • ⚠️ Depending on the type of input/output, you may need to write C++ code.

😸 Hello World!

Here is a Hello World! example.
Compare it with the official code!

using Mediapipe;
using UnityEngine;

public sealed class HelloWorld : MonoBehaviour
{
    private const string _ConfigText = @"
input_stream: ""in""
output_stream: ""out""
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""in""
  output_stream: ""out1""
}
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""out1""
  output_stream: ""out""
}
";

    private void Start()
    {
        using var graph = new CalculatorGraph(_ConfigText);
        using var poller = graph.AddOutputStreamPoller<string>("out");
        graph.StartRun();

        for (var i = 0; i < 10; i++)
        {
            graph.AddPacketToInputStream("in", Packet.CreateStringAt("Hello World!", i));
        }

        graph.CloseInputStream("in");
        var packet = new Packet<string>();

        while (poller.Next(packet))
        {
            Debug.Log(packet.Get());
        }
        graph.WaitUntilDone();
    }
}

For more detailed usage, see the API Overview page or the tutorials.

🛠️ Installation

Please first download the pre-built package from the releases page.

file	contents
MediaPipeUnityPlugin-all.zip	All the source code with required libraries.
MediaPipeUnityPlugin-all-stripped.zip	Same as MediaPipeUnityPlugin-all.zip but the symbols are stripped.
com.github.homuler.mediapipe-*.tgz	A tarball package
MediaPipeUnityPlugin.*.unitypackage	A .unitypackage file

If you need to run sample scenes on your mobile devices, prefer MediaPipeUnityPlugin-all.zip or MediaPipeUnityPlugin-all-stripped.zip.
To run sample scenes on your mobile devices, you need to place required models properly, but most required setup is already done in MediaPipeUnityPlugin-all.zip.

Build the plugin by yourself

⚠️ In most cases, you don't need to build the plugin by yourself. Only if the pre-built package doesn't work for you, please build the plugin by yourself.

This repository doesn't include required libraries or models, so if you clone this repository, you need to build the plugin by yourself.
See the build guide for more details.

Build a package by yourself

If you want, you can also build the plugin by yourself using MediaPipeUnityPlugin-all(-stripped).zip.

Build a unity package

1. Open this project
2. Click Tools > Export Unitypackage

• MediaPipeUnity.[version].unitypackage file will be created at the project root.

Build a local tarball file

1. Install npm command
2. Build a tarball file

cd Packages/com.github.homuler.mediapipe
npm pack
# com.github.homuler.mediapipe-[version].tgz will be created

mv com.github.homuler.mediapipe-[version].tgz your/favorite/path

Supported Platforms

⚠️ GPU mode is not supported on macOS and Windows.

	Editor	Linux (x86_64)	macOS (x86_64)	macOS (ARM64)	Windows (x86_64)	Android	iOS	WebGL
Linux (AMD64) 1	✔️	✔️				✔️
Intel Mac	✔️		✔️			✔️	✔️
M1 Mac	✔️			✔️		✔️	✔️
Windows 10/11 (AMD64) 2	✔️				✔️	✔️

Supported Solutions

This plugin implements the following MediaPipe Tasks C# APIs.

cf. The official available solutions

Solution	Android	iOS	Linux	macOS	Windows
LLM Inference API
Object detection	✔️	✔️	✔️	✔️	✔️
Image classification
Image segmentation	✔️	✔️	✔️	✔️	✔️
Interactive segmentation
Hand landmark detection	✔️	✔️	✔️	✔️	✔️
Gesture recognition
Image embedding
Face detection	✔️	✔️	✔️	✔️	✔️
Face landmark detection	✔️	✔️	✔️	✔️	✔️
Face stylization
Pose landmark detection	✔️	✔️	✔️	✔️	✔️
Image generation
Text classification
Text embedding
Language detector
Audio classification	✔️	✔️	✔️	✔️	✔️

Legacy Solutions

You can also use MediaPipe Framework, which allows you to run Legacy Solutions. However, please note that support for these solutions has ended.

📖 Usage

Once you've downloaded the pre-built package, please import the plugin into your project.

• .unitypackage
• .tar.gz

For Android

☠️ If you need to build your app for Android, please ensure you include libstdc++_shared.so in your APK³, otherwise DllNotFoundException will be thrown at runtime.

The easiest way to include libstdc++_shared.so in your APK is to place it in the Assets/Plugins/Android directory of your project.

You can also include libstdc++_shared.so at build time by adding the following code to your mainTemplate.gradle file, and the sample project is using this method.

// Include libc++_shared.so
task copyLibcppShared(type: Copy) {
    def ndkDir = android.ndkDirectory
    from("$ndkDir/sources/cxx-stl/llvm-libc++/libs") { include '**/libc++_shared.so' }
    into("$projectDir/src/main/jniLibs")
}
clean.dependsOn 'cleanCopyLibcppShared'

tasks.whenTaskAdded { task ->
    if (task.name == "mergeDebugJniLibFolders" || task.name == "mergeReleaseJniLibFolders") {
        task.dependsOn("copyLibcppShared")
    }
}

🍽️ Try the sample app

Before using the plugin in your project, it's strongly recommended that you check if sample scenes work.

Example Solutions

Some solutions (including Legacy solutions) can be tested using the sample app. Please check Assets/MediaPipeUnity/Samples/Scenes to see which solutions have samples.

UnityEditor

Select any scenes under Assets/MediaPipeUnity/Samples/Scenes and play.

Desktop, Android, iOS

Select proper Inference Mode and Asset Loader Type from the Inspector Window.

Preferable Inference Mode

If the native libraries are built for CPU (i.e. --desktop cpu), select CPU for inference mode.
For the libraries distributed on the release page, only the CPU is available for use on Windows and macOS.

Asset Loader Type

The default Asset Loader Type is set to Local, which only works on UnityEditor.
To run it on your devices, switch it to StreamingAssets and copy the required resources under StreamingAssets (if you're using MediaPipeUnityPlugin-all.zip, the StreamingAssets directory already contains them).

⚠️ Technical Limitations

UnityEditor / Your application may crash!

Since this plugin uses native libraries under the hood, if there is a bug in those libraries, the UnityEditor or the application may crash at runtime.

Additionally, in some cases, MediaPipe may crash the entire program by sending a SIGABRT signal instead of throwing an exception.

This may not be a problem in production since it usually happens when there's a fatal bug in the application code, and such bugs are probably fixed before release.
However, in a development environment, it is very annoying since the UnityEditor crashes.

On Linux and macOS, this plugin avoids UnityEditor crashing by handling SIGABRT, so if UnityEditor crashes, please let us know!
On Windows, there seem to be no ways to handle SIGABRT properly, so if you cannot tolerate this, use a different OS.

Graphics API

If you want to run inference using a GPU, you cannot use OpenGL Core API. Otherwise, you will encounter an error like the following:

InternalException: INTERNAL: ; eglMakeCurrent() returned error 0x3000_mediapipe/mediapipe/gpu/gl_context_egl.cc:261)

In practice, this error only occurs on PC standalone builds, and in such cases, please switch the Graphics API to Vulkan.

📜 LICENSE

MIT

Note that some files are distributed under other licenses.

• MediaPipe (Apache Licence 2.0)
• emscripten (MIT)
  • third_party/mediapipe_emscripten_patch.diff contains code copied from emscripten
• FontAwesome (LICENSE)
  • Sample scenes use Font Awesome fonts

See also Third Party Notices.md.

Footnotes

1. Tested on Arch Linux. ↩

2. Running MediaPipe on Windows is experimental. ↩

3. mediapipe_android.aar contains libopencv_java4.so and it depends on libstdc++_shared.so. However, some project or plugins may already include libstdc++_shared.so, so we don't include libstdc++_shared.so in mediapipe_android.aar. ↩