Face, Hand, Pose Tracking & More with MediaPipe GPU Plugin for TouchDesigner
Added 2023-09-09 00:18:36 +0000 UTCHey! In this tutorial, we'll go over the new free open-sourced MediaPipe plugin for TouchDesigner that's GPU accelerated and works on Mac and PC with no instillation. The plugin supports face, hand, pose and object tracking with multi-person face detection, object detection, and image segmentation / background removal.
Link to the GitHub to download the Plugin: https://github.com/torinmb/mediapipe-touchdesigner
Dom Scott's website: https://art.domscott.ca/home
Link to Torin's Patreon: https://patreon.com/tblankensmith
Link to the body tracking alternative: https://www.youtube.com/watch?v=83StND-y4fY
Link to OBS for using video with MediaPipe on Mac: https://obsproject.com/
Link to SpoutCam for using video on PC: https://github.com/leadedge/SpoutCam/releases
Comments
Is there a way to track the entire body? I need to track a body across a stage and if the person turns, then MediaPipe loses the point on the body I was tracking (left shoulder for example.) I need to track the body as an entire object.
David Henderson
2025-07-01 18:21:16 +0000 UTCOr to link to a 3d model e.g. fbx
HeavenArmy
2025-01-08 15:47:06 +0000 UTCHow can I create/rig a 2D character and have mapped to Mediapipe to create a avatar (like the mouse renderer example on Mediapipe's demo)
HeavenArmy
2025-01-08 07:29:04 +0000 UTCThis is great! But I'm still new to TD and can't figure out how I connect this to Particle GPU. I want to control particles by using your plugin, is there anyway you can make a tutorial about it in the future?
Akane Hiraoka
2024-12-11 08:09:03 +0000 UTCHi,Torin. I used the hand tracking part of the plugin and noticed that TD has some trouble with distinguishing the left and right hand(I mean, I attached different sounds for left and right gestures but while tracking it more likely triggers the sound for the left hand than for the right even if I show the gesture for the right side). What is the way to make side differentiation more accurate?
Runa Renard
2024-09-18 22:44:54 +0000 UTCIs there a Discord for troubleshooting or is here the best place? I'm having trouble matching the cameras in both MediaPipe and Video Device In. Is that what needs to happen? When I have MP set to Spoutcam and VDI set to my actual webcam, I'm able to play with Image Segmentation. However, when I set them both to Spoutcam, it goes black. I noticed this when trying to refine Image Segmentation as instructed in this video at 23:18. I also get errors that say "webcam failed to start. is it being used somewhere else?". I'm on PC, TD 2023.11880.
Desert Fox
2024-08-27 22:29:16 +0000 UTCMy software version is this(2022.32660) and I need to use this MediaPipe but i cant , please guide me, because I don't want to update it.
Behshad Jalalian
2024-05-04 22:29:26 +0000 UTCThanks Tori! This has been a lifesaver for me. I bought the Orbbec Gemini 2 camera thinking I could do skeleton tracking in TD on Mac M1, and when that did not work, thought I'd bought the wrong depth camera. I've got everything up and running now thanks to your Mediapipe .toe.
Alec Johnson
2024-03-16 14:03:33 +0000 UTCThanks! I'm a little stuck on the tutorial because I can't find the way to add the box with the "hand_trancking" (2:53 of the video) and continue with the visual possibilities. Sorry for my amateur question. If you have the way to share me, it would be great! thanks a lot!
Mecha MIO
2024-02-10 22:06:07 +0000 UTCHey! This is a quick example showing how to work with the hand tracking data: https://www.youtube.com/watch?v=NwqLnNyfOb4
Torin Blankensmith
2024-02-10 20:56:01 +0000 UTCHi Torin. Im new here! I from Argentina, buenos aires. i neet some help to this tutorial. i want to know if you can explain us how take the data of mediapipe to create some visuals with our hands or face, around Interactive Segmentation and Image Embedding. do you have that parte of the video to practice?
Mercedes Invernizzi Oviedo
2024-02-10 20:29:22 +0000 UTCYeah totally check out the helper component that we use to put a ring above the person's face
Torin Blankensmith
2023-10-25 22:02:57 +0000 UTCFor the face tracking, can I use a different 3D model, is it possible to use an obj file I made in Blender?
Ben Alderman
2023-10-25 22:01:51 +0000 UTCWe're about to release an update that adds a webcam flip 😄
Torin Blankensmith
2023-10-25 14:09:23 +0000 UTCHey! I haven't tested it out personally, but in theory it should still work with a black and white texture that you get from an infrared camera
Torin Blankensmith
2023-10-25 14:08:35 +0000 UTCThis might be a silly question, but I am wondering if this works with depth cameras such as Kinect V2. The reason I am asking is that the installation for trying to build is in the dark. If I use a regular webcam, the visual information won't register.
Ben Alderman
2023-10-25 12:25:09 +0000 UTCglad you got it working :)
Torin Blankensmith
2023-10-04 22:55:22 +0000 UTCGreat! But I had the first issue!! https://github.com/torinmb/mediapipe-touchdesigner/issues/35#issue-1906853027
Andrea Bucci
2023-09-21 16:29:35 +0000 UTCAmazing! Thank you so much!
Mini UV
2023-09-09 18:21:45 +0000 UTCthis looks amazing ! one question, is there an easy way to implement image flip? my webcam is flipped right now usually i just apply a transform but in this case it doesn't seem like you use a video in TOP. either that, or perhaps just make it so there is a TOP in and we can ever have the option to track people from a media file?
Adrien Oneiga
2023-09-09 02:01:56 +0000 UTC
