Anatomy of a Video Game Feature

Hi everyone! This is a case-study about a feature I added to kemorig, and more generally a look at how implementing features in video games often goes.

We've all played games that frustrated us when simple things were missing or wrong. Have you ever cried out out, "Can't they just add this! How hard can it be?" Criticism is always valid, but it's good to understand why features are not always as simple as they appear.

So, let's take a look at how adding Auto Eye Movement to kemorig went!

A Dead-Simple Feature

Last month I added a simple but impactful feature to kemorig: Auto Eye Movement. Kemorig has webcam and ARKit tracking, but sometimes eyes are tricky to track well, and this feature can be enabled to makes Avatars' eyes move automatically.

Auto Eye Movement has 2 main options:

• Make eyes turn in the direction the head turns.

• Make eyes look at the camera.

Thinking this through off the top of our heads: this is a really easy feature! Eyes turning with the head is trivial (just copy the head rotation into the eyes). And rotating the eyes to look at the camera is a simple vector math problem.

"How hard could it be? We'll get this done in an afternoon!"

The Plan

First, some background.

Kemorig's code base has the concept of Tracking Devices, and it's designed to be easy to add new devices. Any Tracking Device can broadcast tracking updates, for which there are 3 types: Face, Eyes, and Body tracking.

These updates are processed by the Game Instance, which is like a central brain for much of the logic in kemorig. The Game Instance will receive tracking updates and, depending on the user's settings, perform modifications to it and send it across to the Avatar.

How does the Avatar receive the tracking info and move? In animation code! An Avatar has an Anim Blueprint asset, which is a child of kemorig's custom Anim Instance class. An Avatar's Anim Blueprint can contain arbitrary stuff users author, but there is one mandatory requirement, which is to include a Kemorig Tracking node. This is a custom anim node that computes the pose resulting from tracking.

Now that we know the lay of the land, we can see how to implement Auto Eye Movement. We'll make a new Tracking Device for Auto Eye Movement (technically a "virtual" device), and it will do some simple math and send Eye updates.

tl;dr: kemorig has the groundwork already laid to add Auto Eye Movement. Easy!

This is how I implemented it, and it worked! Well, it kind-of worked...

We Need Some Changes

The first part of Auto Eye Movement I implemented is making eyes turn with the head. But a little problem arose with how to get the rotation of the head.

There's some subtlety with the head rotation we want. We don't want the final pose of the model, we only want the head rotation from tracking. Avatars may be playing arbitrary animations in addition to motion from head tracking, and we don't want those animations to rotate the eyes; we only want tracked head rotation to do-so.

Fortunately, kemorig already has a hook for this. When the Game Instance sends out the final tracking update, it is an event we can listen to. Therefore, we need our Auto Eye Movement code to handle that event in order to get the head rotation.

But there's a problem: the Game Instance sends all tracking updates out, and the Anim Instance filters them and decides which ones to actually use on the Avatar. For example, the user might have Webcam and ARKit active, and we only actually apply 1 of them to the Avatar. This means when we handle the Game Instance's event, we don't know which updates we care about and which ones to ignore!

Thinking through this problem, I decided that I should change this logic, and have the Game Instance filter tracking updates before it sends its event out. So, I went ahead and removed that filtering logic from the Anim Instance, and moved it to the Game Instance's processing. This also required some re-organization of the Game Instance code (for the better).

Changing a Core System

You might be thinking: "okay, so what? Why even bother explaining this little road bump?"

Well, a few paragraphs up we thought this was a trivial feature, and suddenly it's challenged an assumption in the code base and requires a modification to a core system.

Changing a core system means:

• A previously tested and bug-free system is changing, opening the door to new bugs.

• It's a system at the center of the app, influencing lots of parts of it, so those new bugs might be subtle and hard to spot.

Now, I'm a solo developer and kemorig is in early access. I can approve my own feature and make the changes, and a new bug isn't the end of the world. But with most games that is not the case:

• In most dev teams you can't modify an important core system at the drop of a hat -- There will be meetings and code reviews, it takes time.

• If the game is already released, it's probably too risky to change a core system, and the new feature will probably be cut.

Our easy feature has turned into a liability.

Complexity Grows

Next, I moved-on to getting eyes looking at the camera. As stated above, this requires some simple vector math. We need the location of the eye, the location of the camera, and the forward direction of the head to calculate a rotation.

Unlike the head rotation thing, we do want the final pose of the Avatar for this. That is, we want the pose the Anim Instance produces after whatever animations on the Avatar are applied.

Fortunately, anywhere in code where we can access the Avatar, we can then get a reference to its skeleton and look up the transform of a given bone.

Unfortunately, kemorig is an app that uses user-created Avatars, and looking up information about the skeleton poses extra challenges:

• What are the head bones and eye bones?

  • Users specify these bone names in the Kemorig Tracking node (in the Avatar's Anim Blueprint). No other part of the code knows these bone names, because they never needed to.

• When observing a bone, what does "forward" mean?

  • Kemorig is designed to accommodate any model for Avatars, and there are no requirements for bone orientation. That is, on your Avatar model, the head bone's "forward" direction may be along its X, Y, or Z axis, or the inverse of one of those axes. The app calculates each bone's orientation automatically at runtime, so that tracking movements are applied correctly. This is also calculated in the Kemorig Tracking node.

So this crucial information only exists in the Kemorig Tracking node, and some of it is supplied by users when they author their Avatar. I do not want to change how users author Avatars for this one little feature. Our solution needs to work within the constraints of how the Kemorig Tracking node works.

I decided to add 2 new events that the Kemorig Tracking node broadcasts. The first contains the head's transform and orientation, and the second contains the 2 eyes' transforms and orientations. Our Auto Eye Movement code can handle these events and get the info it needs.

This was Hard

That last paragraph was the most difficult part of this entire thing to implement. Animation Node code is very poorly documented, and very few people talk about them online. Implementing these new events required sleuthing around in Unreal's engine code to find examples, and it took a few tries to get it right.

I was not planning on writing animation node code for this feature.

Our easy feature has grown considerably in scope. But at least we're almost done.

Just Kidding, There was More Complexity

I actually glossed over another detail: Avatars may or may not have eye bones. Even if an avatar does not have eye bones, our Auto Eye Movement code still needs to send eye rotation along,

So, our Auto Eye Movement code has to do a little more jockeying with the events it receives. We prefer to calculate rotation with eye bones, but if we only receive information about a head bone, we will calculate the eye rotation differently.

We also don't know how many milliseconds will pass between events the Anim Node sends. This means we have to implement a time-out where the code will wait X milliseconds without an eye update, after-which it toggles to the head-only logic.

It's important we support Avatars without eye bones. After all, what if an Avatar only uses morph targets to rotate their eyes?

Hang On, Morph Targets? Oh No...

We now arrive at the big thing I overlooked before implementing Auto Eye Movement. Because after I implemented everything above, I tested out the feature and was struck by something.

It didn't look right... Something was wrong with the eyes...

I forgot about the morph targets. I forgot that eye rotation not only involves rotating the eyeballs, but also eye lid morph targets for looking up, down, left, and right. (Plus, as alluded-to above, you could author an Avatar that does all eye motion via these morph targets.)

So Auto Eye Movement would tell the Avatar to look down, and without the corresponding eye lid morph target, they looked freaky.

This made me realize a big a problem with how kemorig's Face, Eye, and Body tracking updates were implemented: The Eye update only sent eye rotation, and the eye lid morph targets were part of the Face update. This oversight was not an issue before, because there wasn't a reason to split Face and Eye updates between different devices before.

More Core System Changes

So, onward we go.

The solution to this problem is to change Face and Eye updates, such that Eye updates contain eyelid morph target info, and Face updates do not.

This change required an even bigger change to core systems than we made before. And it required reorganizing quite a bit of code (even ARKit and Webcam code) that was built with the assumption that Eye updates did not contain blend shapes.

Our easy feature has grown into a refactor of kemorig's tracking code.

We Did It!

At this point, the journey of getting Auto Eye Motion reached its end. This feature looked simple, but:

• It challenged assumptions made when core systems were authored.

• It grew in scope as it was implemented.

• The first implementation didn't look good and more work had to be done.

Again, I'm a solo indie dev, and kemorig is in early access. Implementing this feature was a good thing! It meant I improved a lot of code, and made the app more robust. This is a story with a happy ending.

What Did We Learn Here Today

So why am I sharing this story?

I feel it's a useful case study that gives a taste of what gamedev is like. In my experience, this is a typical story of implementing a feature in a game -- it is like this most of the time.

Not to mention, depending on the circumstances this is a also typical story of how features get cut.

Video game engines and game code are imperfect creations, built within deadlines to satisfy specific requirements. When new requirements are invented later, that imperfect code gets pushed, and dominoes start to fall. This is also why it's notoriously hard to predict how long a feature will take to implement; even a "simple" feature.

The difference with kemorig versus most industry work is that I have the time to properly refactor core systems when this happens. More often than not, game studios do not have the time to that, so if the feature is necessary it will come with hacks and one-off solutions to get it out the door.

"Hacks? But won't that cause more problems later?"

Thanks for reading!