Object Paint-Outs with a Moving Camera in Blender

I'd love a whole series on this sort of stuff in blender , "every day" vfx stuff like logo removal, object replacement, clean plating , etc etc

Hey Ian, I have a question. If i have to camera track a person's bust (waist up tight shot as opposed to full head to toe frame we usually see in tracked shots) in front of a wall, short of having available markers on the depth axis (lacking view of the floor), can i rely on marker's shapes to solve perspective and camera movement? Or is it not even possible to rely solely on the marker's shapes on a flat surface? Maybe I still need to add two or three master stands with markers in the foreground to supply enough tracking data in space? Could you direct me in the general direction? I am asking a tracking specialist on youtube called Matthew Merkovich. Just digging around as this is not standard conditiions for tracking. Cheers!

Raf Stahelin

WOW this has been so incredibly helpful, THANK YOU!

I also like the longer videos! I find that the longer ones have more information that (even if you find some things you talk about erroneous at the time) eliminates many questions people may have.

CmanXP

Just finished. You did a terrific job this time, I loved every part of that video. Very fluent with no boring parts. You deserve more Ian. Greetings from Italy.

Davide Gualmini

I think the longer the videos the better!

Lukas Sprehn

I might have agreed about dropping the tracking part right up until the moment where something you said made me think that I probably don't fully understand how solving and tracking really work. I find this is common with learning Blender. I watch many videos about the same task, but with different situations, and suddenly something specific just opens up a whole understanding about the software. That's just to say that repetition is not without value, in many ways. Plus, you might get more pity donuts!

Matt Kregor

Yeah I thought it would be as simple as setting those markers weight to 0, but once I start doing that with more than a couple of markers the solve errors shoots up like crazy.

OH! That's a super good question! Not off the top of my head, but I'll look into it, because that'd be really useful, yeah. Perhaps by setting it up as an object track maayyyybeeee- I'm not sure, but there's probably a way!

Ian Hubert

Absolutely worth maybe referring to another video instead of forever explaining what you're doing. Great content as always

This is incredibly smart.

Hogan

Yeah, I'm not against stuff being talked about that has already been talked about, even if it makes the vid longer, just because it is more fluid all together. Thanks Ian.

Re: recreating the mesh from tracking points by creating individual faces - I'd be very interested to see an automatic surface reconstruction. Meshlab has a bunch of options for this and I'd be super surprised if there wasn't a blender add on or five that went from point cloud to mesh so you could do it right in blender. Something to consider Thanks for the vid, always learn something new!

Max Cahill

This was such a great useful one. Keep up the good work. I get really excited whenever you post!

Noah Simpson

I'm all for long videos, the longer the better, I don't want quick snippets haha.

Hey Ian, I loved the idea of recreating an object in the scene by using tracking points and converting them to a mesh. It's brilliant. But is there a way that you know of to create extra tracking points that don't affect the end track but can still be used as mesh construction points?

Speaking as someone totally new to this process, I think since you have covered the process in several videos you can skip it going forward so long as the other videos are accessable, which I have no reason to doubt that they will be. However, I find it very helpful to see the problems that can pop up, as did in this video. I would personally appreciate it if you could mention these odd issues when they happen because it would help with troubleshooting later on if we find ourselves in a similar situation. I think just a quick tip in the intro of the video would be more than sufficient. Like a minute or two, or something in that area.

haha i was thinking the same thing

Safi Ullah

this left me with AWW

Safi Ullah

THanks man! I love knowing how to achieve something multiple ways. I can see this coming in handy for some of my paint outs vs mocha. I've been doing shots replacing an actor with a Ninja Turtle(rotomation) and I've been using Mocha in separate passes to reconstruct the background. ALSO man content aware fill in after effects has blown my mind on a few shots.

Tyler Adams

content aware fill is waiting for you

Sainath Reddy Seelam

Honestly I love seeing you do the tracking in each video, few people have already mentioned it as well, but you run into issues that I've had some version of before... and instead of just restarting or brute forcing it (like I do) you eloquently find the solution, or introduce a whole new panel I didn't know existed :D also holy cannoly that paint out is incredible

Gehrig Campbell-Dempsey

im a new sub, ive watched several tracking vids here ..howeverrr i enjoy seeing them it helps to see the problems, like the one Ian ran into with setting the lens he was using and how it gave a better solve. Being new to blender it would have probably took me a while to figure that out. But i get it .. from my perspective I quite enjoy the long videos...

I Like it Hardcore Tv

So happy to be able to finally get this patreon. Im going from The most old to the most recent and maaaaaaaaaaan its amazing! I saw this one just because i was really amazed on the other ones! keep it up man!

Absolutely! That moment reminded me of my wife bringing in comestibles when I'm deep in project stuff and how lucky I am.

cdilla

I think if you point people to your countless other tacking videos, you can skip it and spend more time with the new stuff!

Honestly, the more tracking tutorials I watch, the better, because every one of them I see introduces a different error, and I get to watch someone troubleshoot another problem for me that would have left me scratching my head. So many knobs and dials in there, and it's always nice to be able to reach for the right one instead of facerolling the keyboard and hoping for the best! Also: my jaw hit the floor on that cube holdout trick. Sure, tracking a matte might have been a little nicer, but you're right that seeing it in the viewport is really powerful. Thanks for the video!

Lonnon Foster

Part of what has always made vfx work seem really intimidating to me is this impression i got that you need a half dozen different programs, so seeing you do everything in blender (even if it wouldn't be practical in production) is really cool!

skes

Fantastic demo! I’ve been looking for a way to reach this technique without opening up Nuke, which is so darn expensive. So many are probably looking to become generalists, and sticking with blender for most things, and hopping over to Houdini for others is so perfect, especially for my workflow

I can't WAIT to mess around with that technology more. If we can do environment replacements without having to roto/key, that's going to be SO huge :D I'll check it out, thanks! :D

Ian Hubert

(setting the correct sensor size and focal length is of course essential for a good solve, what I was referring to was the optimize options that seems to be loose cannons)

Daniel defdac Larsson

Hi Ian, runwayml.com has super quick video rotoscoping/masking tools and stuff. Maybe that could come in handy when you gotta do stuff like painting that shoe back or creating greenscreen footage without actually having a greenscreen. You should check it out. Its free for SD exports i think.

Usually (with my limited experience with tracking) the reason that you get those mega big errors is because the K-twins is optimized out of control. So instead of setting the sensor size you could have reset the central point and the K-twins to 0 and hit solve again. Nowadays I rather add a humungus amount of points that I clean instead of optimizing center point/sensor size/k-twins. This strategy works in part because the new tracker is so much faster.

Daniel defdac Larsson

And also I want to mention how lucky you are that Kaitlin attacks you with donuts and kisses ;D I totally loved that moment of the video, it's cool you didn't edit it out.

Mornarion

I have always thought masking out an object in post would automatically mean rotoscoping your brain out, this is so cool. I'm not much into VFX (so far), but it's nice to know if I ever want to give it a try, it won't be an excruciating pain.

Mornarion

Ian, your explanation and teaching skills are awesome! But your ability to turn on keystrokes capture to see what you are clicking is... miserable. 🤪

Ruddy marvellous... I think this is my default comment these days for anything posted in your Patreon.... I mean you could post a dancing pizza with pink eyebrows and only one muscular arm (holding a donut... obviously)... and I'd be happy :)

Doug Brown

ZEKE! This is GREAT info, thank you! Someone told me to "check all the boxes why not" ages ago and I only just realized I should double check that, hahaha- thanks for that info! I think Sebastian Koenig told me to check "optical center" because most lenses (as you say) aren't ACTUALLY centered, but he's not happy with a track unless there's a zero after the period, hahaha. OKAY SO- at the 2019 Blender Conference I was talking with Sergey (the guy who basically MADE the blender tracker) about spiffifying Blender for VFX, and I bounced the exact idea of "auto-generating a mesh from the point clouds" and he said, and I quote, "yes, that would be easy", which I would think sounds overconfident (how do you know which vertices are supposed to go to which??) except that he's *ludicrously* smart. (Basically lots of people were doing their tracking in AE and copying the motion into blender, because AE has the idiot-proof, "TRACK CAMERA" button. And Sergey was like, "PSH yeah we could do that"- so. Who knows... maybe that's in the future! That said, I think there are a lot more things that Sergey could easily do than he can easily do at any given moment.)

Ian Hubert

You can always look up the specs of your phone's camera. Best to get as many clean points as possible and let the solver calculate these for you rather than trying to guess yourself. Biggest issue you're going to face is potential stabilization the phone is doing, and rolling shutter distortion.

Zeke Faust

8:50: There are two boxes you don't have to check! Optical center is only useful if you've cropped your footage in an irregular way that offsets the center (or if you're using an extremely cheap camera/lens combo). Distortion, as you pointed out, is only useful if you're planning on a undistort/distort workflow. Checking this box and not matching distortion on your render can actually cause sliding. Interestingly, this means that you can have a really low error and still have a bad track. So if you're not undistorting it's best to just leave it off. This caused me a ton of grief last year on the sci fi film i suped, they shot with anamorphic lenses and the distortion was crazy. PS You and kaitlin and so perfect BTW!!! PPS I wonder if you could use a shrinkwrap modifier to project a plane unto the point cloud mesh and match it exactly. Or otherwise do some kind of remesh on it that connects the nearest verts...

Zeke Faust

SAME! I feel like I'm a lot more likely to keep a support stand in a shot, or put something in the scene to stand in for a CG object, since it doesn't seem SO tricky to get rid of.

Ian Hubert

Seeing that Ian fixed the high solve error by inputting his camera's focal length. What if i used a mobile phone to record the video and I have no idea about it's focal length?

No, filter tracks looks at the 2D motion of the points and selects points with large spikes in the keyframe data. Clean tracks compares the screen position of the 3D point with the screen position of the 2D track and compares how different they are, in pixels.

Zeke Faust

Love this! I've always been scared of the concept of paint outs in VFX, so I've gone to great lengths to avoid them, but I feel a lot more confident after watching this! Might try some more adventurous shots in future short films now!

Kai Christensen

Also- & sorry if this is a silly question- can filter tracks/clean tracks be used interchangeably?

Chris Alldridge

Loved that little casablanca-esque interlude at 5:25.

Chris Alldridge

Ian is like a 5 star a la carte buffet at a fancy hotel.