Turning People into Digital Characters with Photogrammetry

NiCE

Wasn't expecting the dance lesson, I always get more than what I pay for

John Zwick

it went better than expected! my face is alittle messed up, but other than that it looks alright. definitely a background character for now, but with 32 pics, taken on my wifes new iphone. im not complaining!

Ah! Good luck!!! Let me know how it goes! :D

Ian Hubert

trying this out for the first time right now. fingers crossed

Thanks. I used the display.land app on my iphone for the theater and Quixel's Megascans for the foliage and random stuff. I also used your moths tutorial for the bugs.

Oh dang that looks good. So lush!

Ian Hubert

https://www.youtube.com/watch?v=7unlda6hUiY

Man this is just really awesome. Thanks Ian!!!

Phillip Park

Well I want to fix that for the project I want to do so the reality capture software looks more promising for that. But using it for these scenes is fun. Thanks for the tutorials.

Oh dang that's a cool style! A whole film like that would be rad! Yeah haha the missing bits of the mesh work super good for post apocalyptic stuff.

Ian Hubert

https://www.youtube.com/watch?v=kacymZN_EKk

https://www.youtube.com/watch?v=i-Xtep8MMpY

This is exactly what I needed. I've been testing Meshroom and I have been having trouble with people. It seems to really like the ground and walls best but not people. I am just learning Blender and I am fascinated by capturing folks. I've been making post-apocalyptic stuff because the meshes come out so crazy. I'm hoping to make a short film with dimensional stills using all this stuff. Thanks!

You could theoretically set it up in direct sunlight and get a huge overhead softbox...

Regarding your indoor vs outdoor, diffuse vs direct lighting situation: one idea for lighting would be to create a tent or canopy out of shower curtains or something similar that you could set up outside to get rid of any hotspots and get nice even lighting. If the subject stands on white for bounce on the lower half of the body, even better.

It's so handy! I also love that you can do it right in the timeline without having to open the graph editor ("Shift+F6" isn't exactly a hot key that rolls off the fingers).

Ian Hubert

I have to thank you for your Shift + E command! That's so helpful and thank you so much for that tip!

Colin Stein

I love this. The barnacles are taunting me.

Yeah! And if it's ever too expensive, you can try batch-reducing the resolution of the images and it'll knock the price WAY down (it doesnt' charge you till you decide you want to export the model, so experimenting doesn't cost anything). And oh! I think the big one is probably Node Wrangler- it's crazy handy for working with the the shader editor. Yeah Node Wrangler, Bool Tool, Copy Attributes Menu, F2 (great for making faces), and probably some that i'm forgetting, but those are the big ones, I think!

Ian Hubert

Dude yeah! I actually like the animation I came up with for the breakdown here more than the animation I used in the actual shot, haha. Just that little spine bounce works so much better than all those random keyframes I added.

Ian Hubert

It’s absolutely nuts how much life comes across from that little swing of the lower leg back and fourth. I don’t even have a good use for rigged characters yet but I think i gotta take a swing at it now anyway

Cavan Infante

This is awesome. Excited to mess around with this.

This is great stuff. I had no idea RealityCapture was so cheap. And screencast keys is very helpful, so I'm glad you've been turning it on. Do you have like a list of addons you use? There's been a few times I've been trying to recreate something, don't see the same options you used, and found them lurking in the addons lol

Omg this is so cool Ian ! thank you !

Thanks Ian! Very informative!

Erich Stenzel

Good idea but film loses resolution. Better idea is just hold down the camera button and walk around the model with the fast shutter.

Film at high FPS with high shutter speed instead of photographing? Then .mov to bunch of .png and pick each N image (Python script, for example)? That would speed up the process, so the model won't move.

Oh whoa thanks Peter! Very glad to have you here! And ahhh yeah, that's a good question! There's kind of two answers. There's "how do you model something that doesn't exist in real life" and "how do you imagine something that doesn't exist in real life", because a lot of times you have to be able to imagine the thing in order to model it. This tutorial covers a lot of how I go about it: https://youtu.be/t_c58ryJ-Sw The hardest part, I think, about making a whole street or city scene is the fact that it feels really overwhelming when you first start making the scene. When you're modeling a little detail, then zoom out and see all the rest of the empty street it's like- you have to add those details EVERYWHERE? And you can get overwhelmed. So, I like to block stuff in with gray cubes to get an idea of the overall shot, and then I usually throw textures on the cubes as quickly as possible- even if they're not very good (seeing a bunch of flat gray cubes is kind of intimidating, whereas I like "tweaking" stuff, so if I throw bad textures on it from the start it's easier for me to look at and think, "ah I just have to fix that at some point".) My goal is usually to get to a point where I'm excited about how the image is turning out. Once I get the image far enough, my desire to just "keep tweaking this one last thing" can keep me going for hours, so I just try to get to that point as soon as possible. But until I'm excited about it, if I stop working, a lot of times it's really hard for me to come back to it, haha Also! I have to force myself not to call things done too soon. A lot of times I just want to say it's "good enough", but if I sleep on it and look at it the next day, I'll usually add something that ends up being my favorite part of the image. Also, once you have a city street, you also have an entire city street full of assets that you can use the next time you try to make a street, so eventually you have a big library of random stuff :) Man, I wish I knew more about baking! I usually model/texture everything with as little geometry as I can, and with fairly small textures, so they render fairly quick, usually a couple minutes per frame (also, outdoor environments also render a ton faster than indoor ones, usually). What type of scenes are you trying to make?

Ian Hubert

I hope it's useful!! :D Also ooo- I've heard of it, but haven't used it (maybe I used it's predecessor? I feel like I used some Agisoft photogrammetry software a while back). Have you used it??

Ian Hubert

Oh man yeah I could NEVER get the hands before I did the C-stands thing. And yeaaahhhhhh- it seems to work? I feel like it didn't used to, and maybe it's a lot cleaner if you parent it to the specific bones? But I haven't noticed any problems (mostly I can just remember which layer the right bones are on, haha)

Ian Hubert

Haha! Yeah it's true- and if I were JUST trying to make that shot (especially one with so little change in camera perspective), greenscreen would be the way to go. This is kind of an experiment (the real project was making the T-pose version of the model), because my end goal is hopefully to have a collection of a couple dozen extras I can easily slap mocap on to fill out the mids-to-backgrounds of scenes. That said, going full digital does have a few advantages (though again, maybe not in this instance)- I can make the camera move however I want, the lighting in the scene affects the character a lot easier (it's really hard to anticipate the lighting needs of the final composition when you're filming someone in a green void), I get that lighting interaction for free (not only bounce from the environment onto the character, but from the character onto the environment. Normally I'd have to set up a full digital dummy person in the scene anyways to generate really accurate shadows) and it's a lot easier to copy/paste/move around the scene. And once I DO have a mocap-ready person, I can just put different motion data on them and drop them in the scene instead of having to get an entire greenscreen shoot going (weirdly hard to do, especially if you need to get other folks over). But if I'm honest, a lot of it's just because it's fun to finally be able to do anything resembling halfway-decent digital people (I've been scared of them for so many years, haha- the uncanny valley is a harsh place to live).

Ian Hubert

AH! That's awesome to hear! Yeah it's bizarrely straightforward, but so crazy powerful.

Ian Hubert

Yeah! Mostly cycles still. Sometimes you have to crank the samples, yeah- although I think I usually average around 300? There's lots of random things that can cause fireflies and such, if you can post a still, maybe we can see if there's anything that could be tweaked to avoid major render times.

Ian Hubert

Ooo- so yeah, in the case of meshroom, I definitely decimate (or- I think I just "merged vertices by distance", which is probably a dumb way to do it, but it worked alright in the past?) With RealityCapture, I just do the process in-program (under Simplify), so I can reduce the geometry before I generate the texture. It keeps the UVs from being weird. You CAN reduce the geometry later in blender, and I think it's reeeelatively forgiving? But if you can do it before the texturing process I think it's going to work out better overall :) AND YES! I'm very glad you're here! :D

Ian Hubert

Hey Ian! i'm a big fan of your work! you truly inspiring me, i come from music videos and i'm experimenting with blender! saw all of your videos! i have 2 question: 1. How do you scale up your work, from create an old tv to creating an entire cyberpunk city? i don't know if its clear (sorry but i'm italian and my english is so so bad) i mean, ok, you create a tv from a real tv, but how do you create something that doesnt exist from the real world? whats the process? ... and 2. do you bake light and things before render or how do you render the complex scenario? THANK YOU a lot! i will send you more cash to support your projects!

Wow awesome.... you're kinda like a Blender Wizard! Do you ever decimate the model? After you import the scanned model, and if so, does it keep it's form? So glad I joined this Patreon thingummy.

Doug Brown

Wow, I didn't know you could just parent to the control shapes of Rigify, I've always been unhiding the hidden armature layer and parenting to that. This is much easier. Also, clever to use the C-stands for his hands!

Jan van den Hemel

Great stuff. Thanks for the hint on reality capture. I took the same set of photos I was getting garbage from meshroom and got great results from RC.

Mark Miller

Thanks a lot Ian i asked you for the photogrammetry tut and you did with a rigging tut also....this really made my day....Have you tried Agisoft Metashape Pro another alternative to RealityCapture...

shonal bose

Ian, this is super awesome. But genuinely curious, why would you not shoot something like this on a green screen? It seems like the hardest way possible to put someone in a shot.

sooo kewl. Opens a new door for me. thanx

JØRGEN WETAAS BULL

Oh wow, the highest I've ever rendered on cycles was 600, and that's for stills. How much does samples size increase rendering time?

YakovlevArt

Turn your samples up in render settings, for my renders I'm usually at 1024 minimum (unless I have all diffuse surfaces) up to 10k rarely

Do you primarily render with cycles? I've been doing more animations and not seems impossible to get a clean image, without noise, or the "softness" that comes from the denoiser node.

YakovlevArt

"relieved from your forever"- is that a way of saying you lost your job?? Because you sound very excited, but that sounds also kind of tricky?? But also!! I'm glad the timing is good! :D I've been getting crazy into experimenting with this stuff- hopefully going to have a lot to post pretty quick here.

Ian Hubert

Oh awesome!!! You gotta send me a link when it's done! :D You stay safe too!

Ian Hubert

Man, that shot is so beautiful!

it's that guy!

THANK YOU SO MUCH IAN !!!!!!!!!!!!!!! PERFECT TIMING AS I WAS RELIEVED FROM MY FOREVER YESTERDAY !!!

Dannysh Rosli

This could not have come at a better time! I started working on a new project recently and have been catching up with your posts here on Patreon and it seems like every time I reach a milestone or a roadblock, all I have to do is continue scrolling down for more nuggets of information and inspiration. Love the way this looks and works, thanks for the breakdown!

Arturo J. Real

Yeah!! I've used DisplayLand a lot! I love the idea and all the work they've put into making it an easy experience- although I've never really had much luck getting production-type assets (although I'm still working on it)- everything's just a bit too low-poly. That said, for environment reconstruction, it's a great idea for snagging a scan, just so you have a low-poly version of where everything's at. The biggest issue I've had with it is that it (like meshroom) doesn't let you decide the area you want to reconstruct. If you accidentally film it, it'll probably be in the final thing. So I've tried to scan a friend just to have a quarter mile in each direction get scanned along with him. I should try one more time scanning someone in a small room, just to see if I can limit the amount of extra geometry that's allocated to the environment (I'm running under the assumption that there's a geometry "cap", and that it's fairly evenly distributed... it seems to match my experiences a bit?)- but I still want to do some tests!

Ian Hubert

Yeah!!! Basically just drag and drop, then babysit the program a little! And ooo- maybe?? Can you clarify a bit what you mean by "using footage to rig an armature"? You mean matching up a robotic arm with footage?

Ian Hubert

This is so cool! So much easier than I thought it would be! Do you know anything about using footage to rig an armature, like to give someone a robot arm? Thanks man!

Benjamin Lacey

Best dancing animation ever! By the way, Ian have you heard of the DisplayLand app? It may not have the same quality as DSLR and RealityCapture, but may work better than Meshroom in some cases, or just a quicker workflow for simple things.

Im trying to make a horror short completely in blender, and I have to say I really appreciate your insights and advice! You have been ridiculously helpful, stay safe Ian!

Saw this on Instagram and it looked awesome! Thanks for making a post about this!

Hudson