LightBox Game Day Presentation (Continued Virtual Production Schemes)

Ian, have you checked out VirtuCamera App yet? It allow the blender camera to be controlled by a phone! I have a demo just using my iMac screen on my instagram /davewrath but I think using a projector will be my next iteration. Ghetto Mando!

Great lighting ideas here, lookup DMX controllers (can totally make an Arduino output DMX scenes https://hackaday.io/project/5342-arduino-dmx-512-tester-and-controller) and calibrate all your light 'producers' so red is red is red.

👍👍

Thank you Ian, hope they will upload the video in youtube 👍

Dang- yeah! I was trying to re-find the link yesterday, and couldn't... it looks like they haven't put it back up? I'll let you know if I can find it! Sorry about that!

Ian Hubert

Where i can see the presentation? The video is taken down :(

No worries, nothing interesting as yet but schedule is less busy in January so will be doing more experiments then. I really wish there were more tools in Blender to do this sort of thing.

OH! I'm sorry I never responded to you about this! I checked it out back then, but never made it back to this comment! That's SO cool!!! Thanks for sharing that! You experimented any more with that technology since??

Ian Hubert

Hey, so I got the Vive tracking to work in blender using your BROIL scene check it out here https://youtu.be/xx3Okr3XAgo

@ Evan Barron I'm hoping to make some tutorials about cam tracking + live compositing + projected background etc. It's actually a significant part of my thesis project (I study Integrated Digital Technologies at Ryerson University). Open Source Virtual Production Pipeline for Blender is the goal. Am looking for collaborators for apple development and python scripting for Blender - if anybody's interested please reach out on instagram! Also, keep up the great work Ian!

I'm sure its possible for Android phones that have AR capabilities. But I'm not as familiar with Android devices/development.

This reminds me a lot of the virtual sets they used for the Mandalorian. Sick to see a more accessible version of that!

John Zwick

WOW!! That is very cool!!

Tony Rizo

Oh hey!!! This is some super great info! Thanks so much for tracking it down :D. I'm going to try to get my hands on a vive tracker as soon as possible! Very cool to know you can use them without the headset.

Ian Hubert

Oh whoa! That's awesome! I'd love to hear more about that, yeah- I don't have an iphone, but I think my next phone's going to have to be one. Between this and the facial motion tracking and all that, they seem pretty darn powerful.

Ian Hubert

This sounds extremely promising! Any chance of something similar for android? How would I go about replicating something like this?

Evan Barron

Great stuff, there is a way to use just a single Vive tracker and a single base station to get camera location into blender, been up a few hours looking into it find the info here for the addon https://blenderartists.org/t/openvr-tracker-streaming/1236428 If you don't wan to buy a whole Vive set up you can actually use it without the HMD info here from the company who make the vive trackers... http://help.triadsemi.com/en/articles/836917-steamvr-tracking-without-an-hmd A second source https://vvvv.org/blog/using-htc-vive-trackers-without-headset

Could also invest in a mechanical/robotic camera arm that repeats the same motion. Programmable of course. Unless that's too expensive - haven't looked into i. Or one that can be hooked up to a software like Blender and be moved through it.

Lukas Sprehn

Hey Ian - i’m using an iPhone and a little program I made inside TouchDesigner to stream camera location data real-time into blender via OSC. It works extremely well. And a lot cheaper than a VIVE setup. Am also using a projector and eevee viewport. Very exciting stuff! Next step - strap the iPhone to a camera and start shooting! I’d love to chat about it - and can send you some of my programming and setup for iPhone tracking to get you started if interested! @tylerrdillman on Instagram

Hi Ian, I enjoyed your chat with the kitbash guys, and this so reminded me of the whole rear screen projection VFX that Hitchcock used in a lot of his movies. But you are using front screen, I deal with projectors in the classroom myself and I agree the laser projector is definitely worth the money as it is cheaper than the LED panels in Mandalorian. And yes you definitely need a better screen with glass beads that provides better contrast. I was going to order a screen from these guys before covid hit, but no there is no money for that in my department. Check them out, maybe it will spark an idea? Thanks for all of the great work! Keep it up! https://elitescreens.com/index.php/products/ambient-light-rejecting-screens

Tony Rizo

Great share - thx man!

Frank Frohnhoefer

Hey Ian, I though I would share some of my research about screens here, this might interest you. Nothing fancy, just a bunch of links to products that I haven't been able to test for myself yet, but it seems very promising. It is simply various black paint brands designed especially for video projections and some of the results seems to be very impressive. https://www.youtube.com/watch?v=8XXefZHZCcI https://www.youtube.com/watch?v=XOpDOKDmli0&t=43s https://www.youtube.com/watch?v=KvHCiRoLMgk https://www.noxscreen.com/ http://normandesign.se/index.php/2015-10-31-09-46-28 I'd be very curious to get your take on this :) Thanks for all the greatness you share :)

Raging Heroes

What projector are you using? I didn't see the model being mentioned in the video.

another note: reflective projection screens - you lose gain if the camera and projector aren't aligned, but you don't lose all the gain. so.... Scotchlite 7610 is, I think, the gold standard for front-projection work - 900x gain! (700x gain at 45 degrees) and $$$$, I'm sure. That gray-wolf screen has a gain of 1.8 - hahaha.

oops - typo above: "There is also, I believe *special paints for projection" --- side note - I was in a bar once and they had projected a football game on a blank wall, and it looked great. When the game was over, the lights came on and I was shocked to see it was a burgundy colored wall! My brain made the whites look completely normal!

... for the fill light - a cheap and easy solution might be using additional monitors or TVs (cheap at st. vinnie's) :-) Killer stuff - keep it up, man!

Hey Ian! Some thoughts: for your blacks - rather than a white screen, try gray and then over drive the projector a bit. There is also, I believe. Ideally you'd want to do a silver screen type of thing where you've got a gray screen with reflective beads to increase the gain of the projected image - I've got a gray wolf screen you can borrow if you'd like, I forget the gain measurement on it. I think you can get the paints, but I don' t have any details on what's good and what isn't.

Like, basically you would just animate a simple camera move in Blender, and then you would render that out and play that on the projector while you film Sean in front of it trying to match the animated motion! Seems like it would be a great way to squeeze out a little more parallax

Kai Christensen

ok, so based on the success you've had with reprojecting and then re-animating tracked shots, maybe you could try animating the camera in a simple move for the background, exporting that, and sticking it on the projector while you try your best to 90% match it with the real camera? If those re-animated shots are anything to go by, a simple dolly from side to side might totally work! Feels like it would be the next best thing to actually hooking up a Vive tracker or streaming camera data in live...

Kai Christensen

ian,can you make a tutorial for lighting room? especially like tutorial from video wall.. thank you so much

Ian did you just get one kitbash? I saw a sale a few weeks ago and just didn’t know if it would really help in my workflow for the price that they’re going for... would you recommend it?

Greg Beaudry

I'd love to try out using projectors to light people on a green screen, because this is a bit out of my reach without a laser projector or the space to rear project. It would especially help shots with changing light, like water reflections.

Harry Robins

Very nice. I almost understood most of this. I think I'm learning.

I didn't really get it the first time, but this setup looks really promising! BTW- I made a Hubert-inspired science video that takes place in a crappy-utopia colony, and I think you'll like it: https://youtu.be/kdPZOWnkYK0

Gabriel Meilikhov

AH! Yeah those are all great ideas! I want to give it a shot. Maybe tonight? If I could get an HDRI to provide both a background and ambient light, that would be awesome. It's going to be crazy hard to not get wild light spill, but maybe that's okay, if it's a bright enough environment.

Ian Hubert

I think they're uploading the individual talks to YouTube- I'll post a link once they do! :D

Ian Hubert

Thanks, Ben! I'll keep that in mind!

Ian Hubert

Ah, thanks a bunch Carlos! :D

Ian Hubert

Great session on LightboxExpo. I'm not a regular Blender user, so I appreciate the fbx downloads too.

I've heard of solutions! I gotta dig through my messages, but somebody was putting me in touch with someone who created an addon that could do it (patreon isn't the best for organizing messages, sometimes). Especially for windows, if you just did a "copy location" constraint, I feel like it'd be that easy for doing digital extensions! It's trickier if you want DIRECT extension (like, a real floor continues on into a projection). I want to see how your stuff turns out!

Ian Hubert

Oh! You mean the ones on the video wall? I'm making some videos about robots real quick here! :D

Ian Hubert

Yeah!! That's one of the first effects I ever did- projecting a neon sign into a dark background. It worked alright! Especially out of focus. It'd be cool to do a TON of signs, but in a similar setup like this, just projecting over the whole scene and dragging and dropping the neon wherever it made sense. I want to try that now!

Ian Hubert

YES! Matt's stuff is incredible!!! Been very much enjoying seeing his progress- I want to try to follow in his footsteps regarding the vive-controller-on-a-camera! He's got great videos on YouTube :D

Ian Hubert

Matt Workman from Cinematography Database on YT has been doing all this amazing stuff with 'Virtual Production' like the Mandalorian sets... hes got a FB group & cool content. He's working w/ Unreal engine & has a software available for ??? I think setting up a VirtualSet w/ green screen or LED screens or back projection.... I'm sure you are aware, but especially because of covid, plus just how cool it is & affordable its becoming... plus how photo real its all looking, VR Sets are the new thing & I'm sure they'll just be more & more common.

Michael Patrick Ford

Great demo today! Thanks for sharing!

Awesome, will there be a recording of this afterwards or will it disappear like other twitch stuff after a few days? There's definitely some little bits and bobs you did here that Id like to refer back to later.

Jason

That's so cool! You could make a bunch of experiments, like using another projector with a diffuser (as you said) for ambient lighting, or maybe set the laser projector as a retroprojector with a white sheet between the actor and the projector, so part of that light is transfered to the actor in front of it... or even using multiple projectors and a projection stitching software like Immersive Display PRO for wide shots, and Unreal Engine 4 for real-time VFX compositing a-la Mandalorian. Endless posibilities, man!

Great presentation Ian, Really enjoyed it!

Phillip Park

this reminds me of unreal engine/ light setup for the Mandalorian : https://youtu.be/tpUI8uOsKTM?t=184 I get you want to use an htc vive, and that may work (https://www.vive.com/us/accessory/vive-tracker/) , but they are currently out of stock / you gotta run it through steamvr / set it up just right with the the bases (https://www.vive.com/us/accessory/base-station/) - its a way of doing things. Another way of doing things is to get a cell phone and mount it onto your camera - then record the stream OR send it via wireless stream to blender . https://medium.com/@JohnCoatesDev/visualizing-an-ios-device-in-blender-through-quantum-entanglement-ba8b6f0b47a5 can you give a price range/ what to look for when getting a laser projector? is there an off brand you go with or do you stick with the familiar brands?

Yes! :D

I'd also love to see that workflow!!!

Awesome! I really need to sleep since it's a work day tomorrow but now I also want to fire up the projector, stay up all night and try to create cool stuff!! Ah, the dilemma! 😄

Mario Taylor

i love your dynamo dream robots so how do you model them and animate them

would love to see caustic effects projected on the wall, or even one of your fancy neon signs as a backdrop. Nice work!

Whoa! So there is a way to make a virtual cam setup with Blender using a vive tracker? I have done this at a small scale with UE4. Would love to see how it works with Blender.

Amazing! I’ve been looking to do some rear projectiony stuff for some sci-fi windows recently. Have you seen a solution for getting Vive tracker data into Blender?

James Ballard

NICE ONE!!!

Thanks! And yeah- it's coming together! Still kind of looks projectiony- but I think with a bit more work it'll feel more integrated. Having it brighter really does seem to help (if it were the view through a window? I think that'd work excellent).

Ian Hubert

Ahahaha, yessss- I think it's the textures that really make the impact, though- each model has a lot of big ol 4k PNGs. If I were to use them a lot, I'd probably shrink them down a bit. I had to just a few from a different file, since I could barely even open the original kitbash file. Which isn't a knock against the assets, but maybe the organization??

Ian Hubert

So glad the talk went well! And the rear projection stuff is looking super freakin sweet!

Blake Rizzo

Woo!

it's that guy!

Thanks Ian, loved watching! Bet those KitBash3D buildings have more polys than you’re used to lol

sofia