Gav Rymill

Telesnaps #37 & #38

Added 2024-03-31 15:41:18 +0000 UTC

Recently, one of the Machine Learning systems (which everyone is calling AI) called Runway released a little update with a new lip-sync tool. Previously I had experimented with the MyHeritage tool which was fine but it lost a great deal of the the likeness from the original photo.

The Runway tool is pretty cool and will generate lip movement either from a wav file (where it attempts to interpret the words) or a text input (which gives you more accurate mouth shapes). In this clip I've used a mixture of the two. The problem with the text input process is that the output doesn't match the timing of the original audio, so there's a lot of post-processing required to line it back up. The head movement isn't great, but overall I think there's potential here for certain shots when it works out nicely.

I think it's superior in appearance to Wav2Lip, but the drawback is that this only deals with a static image, whereas Wav2Lip will add lip movement to video. So what I need to do with this create some movement on the areas of the body which Runway is not affecting, and isolate the background so that the soldier behind him has some action. I think the extra movement throughout the whole image will make the ML/AI animated face less distracting.