From Green Screen to In-Game Character Animation

As you all surely know, I've decided quite a while ago to ditch the idea of using low-poly 3D models for actors, and instead go with a more true to my vision, real actor billboards approach for Call of Saregnar. While I knew that using 3D models would have saved me tons of headaches and manual work, it just didn't fit with the overall style of the game. Again, the idea of going with a high- or semi-high poly count felt like an abomination against the low-poly world.

Another thing suggested to me at the time, was modeling the actors in a 3D package, render them into frames and put those in the game. While it would definitely work from a technical standpoint, the tradeoff between ease of production vs. the overall look was again something I wasn't prepared to go for. My gut feel told me that nothing could replace the charm a real actor would create, and I've since learned to trust my instincts.

Now, what is the process of turning "actors on green" into an animated character in-game?

Let's start with one thing I believe contributes the most to the end result.

The Lighting

The technique used to light up the actor here is known as Three Point Lighting, which – as the name suggests – consists of using three lights for the illumination of the subject. 

The first and most important light is the key light, which is the main and strongest light in the scene. This light is placed on one side in front of the subject, generally at a 45deg angle. The key light determines the overall lighting design of the shot and could as well be the only light in the scene.

The second light is called the fill light, which is used to fill the dark areas that the key light didn't illuminate. This light is placed opposite the key light and somewhat lower, usually at the subject's face-level. You can completely eliminate this light from the equation if you desire deep shadows to achieve certain moods. Sometimes a reflective surface can be used instead of a light to achieve a similar effect. Such a surface can be anything from a purposely-built reflector to a white wall for example. I am currently considering using a reflector and use the current fill light to aid the key light in certain situations.

The last light in our triad is called the hair- or rim light. The hair light is mainly used to separate the subject from the background, which is very important when shooting in front of a green screen, otherwise you can get a lot of the green reflections on the subject and it makes it hard to separate cleanly. If you look at the photos I have somewhat overbalanced the hair light in those shots. The main mistakes are the reflections on the helmet, which should not be seen, and the shadows projected to the front of the character, which make it difficult to separate the subject from the background. I will have to tone this light down and place it lower in the future. Lesson learned.

The key light is a Yongnuo YN-600II, while the fill and hair light are two Yongnuo YN-300III. All the lights are daylight balanced at 5500k.

Having the three main lights out of the way, let's discuss the lighting of the background, or green screen.

Just like the main subject, the green screen also needs to be lit properly and uniformly. In my case I am using two regular shop lights, each fitted with a pair of (kinda almost) daylight balanced fluorescent tubes. They do a good enough job of lighting the green screen background, but I need to find a way to illuminate the screen beneath the actor's feet a bit as well. As you can probably imagine all those shadows, wrinkles and difference in light do make a headache when separating the background from the subject. I am still unsure on how to solve the problem without adding additional lights pointing at the floor. My ideas always turn into costly solutions.

The Green Screen

The green screen I am using is a cotton fabric measuring 3m wide and 6m tall/long. It is suspended on a couple of stands with a crossbar linking them. The canvas has a loop sawn on the top through which the crossbar is inserted, so it can be lifted up. The problem with my current stand is that the thing is way too weak and it can't be raised height enough for my needs. 

You can see in the photo above how the middle is flexing and it could barely cover the actor's head (that's me on the left, 6ft tall). I since purchased (yay expenses!) a much stronger and taller stand, which will do away with these problems.

Another issue are the wrinkles. I was told to iron the canvas, but for this first test I tried to go without. Alas, it was good advice and the wrinkles turned out to be a real problem in post-production. I do have clamps to stretch the canvas, but I did not take them with me at the time. The clamps would grab the canvas and stretch it around the stands, so that may be a valid alternative to ironing it, however I could not stretch the part of the canvas that folded away from the stand. To solve that, I found that good old duct tape to the floor takes good care of the problem. As for the horizontal wrinkles, I think most will be solved by the added weight of the canvas itself when raised higher on the new stands.

The Photoshoot

Before I continue, I must say that I've been blessed to have encountered a very passionate group of people who do historical reenactment in their spare time and I am inviting you to pay a visit their home page if you have a moment. I owe them BIG TIME for all they are providing me with:

• A big (warehouse-type) place situated at a 15min drive from my home, where I have semi-permanently erected the photoshoot stage.
• Potentially 20 members, including women willing to participate in the shoot.
• A big variety of clothing, armour and weapons.
• And most of all, they are friendly, reasonable, passionate, and avid old-school gamers!

There must have been some sort of divine providence that brought them to me...

Ok, but enough of that. Let's cut to the subject at hand.

When shooting in front of a green screen, you obviously have to pay attention that the subject does not wear any green, for that would create havoc in any kind of automated removal of the green screen from the photo. I guess I could use a blue screen instead of the green one for such cases, but that is way beyond what I am willing to do.

Since I am using a fairly old camera (a Nikon D90), I cannot afford to shoot at very fast shutter speeds in the lighting conditions I have, so the actors need to stop between photo frames, so I can take a picture. If the actor would move while the camera shutter is open, the part that is moving (too fast) would become blurry, and we don't want that. The problem with this stop and move and stop approach is that the movements can appear unnatural unless great caution (and several retries) is taken. But then again, I'm not sure a newer DSLR could cope with that much higher shutter speeds anyway to make fluid movement possible. I'm certainly not buying one to test out my theory!

A thing I didn't do this time, was to place a marker at the feet of the actor, so she knows where to stand, and I don't have to refocus the camera each time the pose or actor changes. The marker also tells the actor where the center of the frame is, so he can stay within bounds. I'll be sure to place some tape at the spot next time I do a photoshoot.

Of course clear instructions to the actors on the base technicalities behind the process are more than useful. If the actor knows what the limitations are, then the process can proceed much more smoothly than it would otherwise.

For the first (test) shoot the instructions were basic, the plan was barely there, but I still attempted to come up with some useful shots to be used in-game (#screenshotsaturday anyone?). Since while I was busy setting up the lights and the other equipment, the guys got themselves busy wearing armour and weapons, and I thought we should do guard roles! I took some idle guard motions, some drinking motions and of course some drunk guard motions. In the end they turned out pretty good.

For the upcoming photoshoots I came up with a big spreadsheet detailing the roles to play for each character present in the story. While it is not yet complete, it contains the main player characters, NPCs and enemies you'll likely encounter in the game. 

Each character has a number of fields: ID, actor's name, costume details, character age and sex, character name, type, the role she plays, alignment, and then, a number of checkboxes with different "shot types" (ST) as I call them, which determine what kind of shots need to be taken for this particular character:

• UI Portrait Shots (Shots taken from the chest up, focusing on the head.)
• Conversation Shots (Shots taken from the waist up to be used in conversations.)
• World Wondering Shots (Shots of characters wondering around the 3D world.)
• Full-body, Look-busy Shots (Shots of the whole body in various poses.)
• Combat Shots (Shots to be used in combat. Warriors have a different set of shots from the spellcasters.)
• Cutscene Shots (Shots used in cutscenes between chapters.)

Of course not all characters have to have all of these STs taken, but only a subset. I will not go into detail of what each ST consists of, since I don't want to bore you to death. The only interesting part is that some of the STs are to be taken from eight different directions starting from facing the camera directly, and then repeated by every turn of the actor by 45 degrees. This will come in handy when the in-game NPC is able to turn around herself and/or you are able to walk around her in the 3D world.

If you want details on the hardware, this "camera setup" is comprised of my good old Nikon D90 DSLR with a 1.8G 50mm prime lens connected to my Macbook Pro via USB. The camera is mounted on a Velbon EX-640 tripod, and to keep the shakes at minimum, I also use a wired trigger on the camera.

As far as software goes, I run an app called Sofortbild on the mac, which takes the photos from the camera and uploads them on my computer directly, bypassing the camera's SD card. It also allows me to review the taken picture almost immediately after I release the shutter.

Putting the characters in motion

After this photoshoot I ended up with 468 individual photos, which needed to have the green removed (or keyed out). I didn't have any fancy software to do this automatically, so I played a bit with an open-source command-line utility (chromakey), but I couldn't manage to remove all the green with it. I am sure the problem was the aforementioned wrinkles and shadows in the background.

Anyway, for this occasion my goal was not to process all the photos, but simply to see what the end result would look like in Unity. For that I booted up Affinity Photo (a more than valid and much cheaper Photoshop alternative) and manually paint-selected the green colour out of a series of photos taken.

In the keyed-out photo (above right) you can see the problem with armour and other reflective surfaces; They reflect the green all too well. That had to be removed as well. So, back into Affinity Photo, apply some filters and most of the green is gone.

After that, I placed the transparent character images onto a strip (a spritesheet really) and imported the thing into Unity.

In Unity I use a Sprite Renderer component to display the character in the 3D world. To test the animation out, I wrote a simple animator component which takes an array of sprites, and allows you to specify the speed of the animation in FPS, plus an option to loop and mirror (bounce) the animation. The result is the video in the head of this article. 

It is still all a work in progress, especially the animator component, which needs to be quite a bit more complex to allow for less repetitive and more natural animations, but at least what I have now allows me to test the basic animations.

A BIG Thank You

Worth mentioning is that the majority of the equipment I used in the green screen shoot was purchased with funds from your support here on Patreon. Buying it all by myself would probably cause a serious dent in my wallet, so THANK YOU!

Conclusion

I hope you enjoyed my rather lengthy plunge into the rather lengthy process I used to create those animations. If you did, please let others know of this devlog. One way or another, all the money received goes 100% into the making of this rather lengthy game of mine.