Automatic Face Tracking (new tool release!)
Added 2022-06-23 08:31:02 +0000 UTCHAHA! Very excited about this!! Nathan just put together a python script to get google's mediapipe face-tracking data into blender!
[quick edit- to help with any troubleshooting, if you're posting about having a problem, including your OS, whether you had problems installing python/the libraries, and a screenshot of the command terminal (both what you put in, and what it said), very much increases the odds we can help you :) ]
It's not useful for everything, but for some things, it's an absolute life saver.
Get the Face Tracker Python Script here at Eat The Future.
Here's a text version of what I cover in the video (along with some easy-to-copy-and-paste text):
FOR INSTALLATION:
Make sure Python is installed for your system
(you might have to uninstall any out-of-date versions)
Open a command prompt and install the libraries, one at a time, using:
pip install numpy
pip install opencv-python
pip install mediapipe
Create an environment variable:
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION
with the variable itself being
python
And finally, download the script linked above (and as seen in the video) from the Eat The Future github page.
TO USE THE TOOL (probably best explained in the video, honestly)
For Orthographic Mode:
Open a command prompt, and drag in the python script
Make sure there's a space.
Drag in the video to be tracked.
Another Space.
Set the Destination for the mdd file (I usually drag in the video again, and change the filetype to ".mdd")
Mine looks like this:
Hit Enter, and off it goes!
For Perspective Mode:
The same thing, but just one more bit of information.
Open a command prompt, and drag in the python script
Make sure there's a space.
Decide whether you want to use Field of View (FOV) or the camera settings (Sensor Size and Focal Length)
For FOV, put
--fov ##
(replace "##" with the estimated field of view)
For Focal Length, put
--focal_len ##/##
(replace "##" with the sensor size and focal length, respectively)
Add a space.
Drag in the video to be tracked.
Another Space.
Set the Destination for the mdd file (I usually drag in the video again, and change the filetype to ".mdd")
Mine looks like this
Hit Enter, and off it goes!
Anyways!! We hope this is useful to someone out there! At some point we'll be releasing it to a wider audience, but in the meantime just wanna make sure it works for all you here!
And unfortunately, as I say in the video, while we super want this to work on your machine, there's a lot of variables to account for, so no promises. That said- I've been able to successfully install it on every windows machine I've tried, so I'm optimistic!
Comments
Very late to this thread but wanted to give this a try on my Windows 10 but the scrip isn't working at all. After successfully downloading everything as instructed, when I drag the face_track.py and footage files into the CMD line, I get a syntax error for an unexpected character. Screenshot here: https://drive.google.com/drive/folders/1l4PStABkET0HacvUjwHSA8Nezp43F4FU?usp=drive_link Anyone else find a workaround?
2023-07-19 18:07:58 +0000 UTCNATHAN! why are you so frellin awesome?
Jack_Wolfe
2022-09-30 14:35:43 +0000 UTCIf i make a fight scene with photoscan characters, and use this techinique to put real footage faces on some scenes, it might work? Blend the photoscan character with the face footage
2022-09-26 23:16:56 +0000 UTCFor reasons I don't understand, the MDD file created only starts tracking on frame 73 of a 103 frame shot.
2022-09-06 01:12:01 +0000 UTCThis is so great Ian. One question though - do you have any idea how to get around the problem of "wrong vertex count" when trying to model the base face mesh ? and do you guys have any idea how to apply the track to another face - mesh? Thank you so much for your work!!
2022-09-03 18:49:40 +0000 UTCThis is amazing! Thank you so much. Is there a way how to project the video texture on the facemesh and zero out the head movement and rotation? I would like to track one face on another - deepfake style... Thank you for any tips in advance.
2022-08-27 17:08:38 +0000 UTCActually, it isn't receiving lights in ortho either which is interesting! Hmm. So if I have a spot on there or a hologram in front of the tracked face mesh, the face mesh is not affected by either. I'm wondering if it's a limitation of the mesh cache modifier?
2022-08-25 13:17:06 +0000 UTCOnly in perspective mode?? But it IS affected by the light when orthogonal? Cause that's bizarre! I can't think of anything that would make that happen!
Ian Hubert
2022-08-25 08:45:53 +0000 UTCAnybody else having issues with their face mesh NOT receiving lights when in perspective mode? I'm trying to have an HDRI/light/hologram shader light up the face mesh somewhat accurately but it never seems to be affected by it.
2022-08-24 20:43:58 +0000 UTCWhen this gets fully polished please post the final results! Fingers crossed for Mac owners
2022-08-02 08:41:27 +0000 UTCOoh BDG!
Boris
2022-07-31 14:52:12 +0000 UTCthis is such a cool concept - thx!
Frank Frohnhoefer
2022-07-30 18:50:54 +0000 UTCMy footage is always upsidedown when imported :/ anyone know how to fix this?
Curd Williams-Hertz
2022-07-24 19:28:41 +0000 UTCDid you ever get it goin?
Noneya D Biznazz
2022-07-17 05:08:21 +0000 UTCit's happening to me as well
Maddsketch
2022-07-13 01:29:06 +0000 UTCAm I the only getting parse error on input when trying the script? :( "face_track.py" "virgieinput.mov" "virgie.mdd" from the current folder gives me parsing error on video input and output (also tried absolute paths but no success)
2022-07-12 16:37:36 +0000 UTCput "python " in front of the script
Noneya D Biznazz
2022-07-12 16:37:21 +0000 UTCHi, Ian. When I ran cmd and put everything you showed. Cmd just opened the code, nothing happened.
2022-07-12 13:08:11 +0000 UTCHas anyone set up a place to pool info on this other than this thread?
Noneya D Biznazz
2022-07-10 06:18:56 +0000 UTC
