PROTOWLF

Billing Paused for 1 Month - Fires, Vacation

Hi everyone, I decided to pause Patreon billing for 1 month starting today. Two reasons for this:

• I had to evacuate my home for a couple days due to the ongoing fires in Los Angeles. Luckily our neighborhood is safe, but it's been a major disruption, and I've still got stuff packed and ready to go in case I need to evac again (the dangerous winds are expected to last through Tuesday).

• Later in January I'm leaving for vacation in Japan, where I'll be for roughly 2.5 weeks. I might do a couple impromptu streams out there, but I'm not going to be working.

I was originally thinking I might crunch before the vacation to keep the normal Patreon schedule, but the fire situation scrapped that plan. So, billing is paused for 1 month and will resume February 12th.

During this time, new members can still join if they wish (for example, if you want access to kemorig).

If you have questions about how paused billing works, this is Patreon's help page about it (Note that my page uses "subscription billing"): https://support.patreon.com/hc/en-us/articles/214013323-How-to-pause-billing-for-your-members

Thanks!

Anatomy of a Video Game Feature

Hi everyone! This is a case-study about a feature I added to kemorig, and more generally a look at how implementing features in video games often goes.

We've all played games that frustrated us when simple things were missing or wrong. Have you ever cried out out, "Can't they just add this! How hard can it be?" Criticism is always valid, but it's good to understand why features are not always as simple as they appear.

So, let's take a look at how adding Auto Eye Movement to kemorig went!

A Dead-Simple Feature

Last month I added a simple but impactful feature to kemorig: Auto Eye Movement. Kemorig has webcam and ARKit tracking, but sometimes eyes are tricky to track well, and this feature can be enabled to makes Avatars' eyes move automatically.

Auto Eye Movement has 2 main options:

• Make eyes turn in the direction the head turns.

• Make eyes look at the camera.

Thinking this through off the top of our heads: this is a really easy feature! Eyes turning with the head is trivial (just copy the head rotation into the eyes). And rotating the eyes to look at the camera is a simple vector math problem.

"How hard could it be? We'll get this done in an afternoon!"

The Plan

First, some background.

Kemorig's code base has the concept of Tracking Devices, and it's designed to be easy to add new devices. Any Tracking Device can broadcast tracking updates, for which there are 3 types: Face, Eyes, and Body tracking.

These updates are processed by the Game Instance, which is like a central brain for much of the logic in kemorig. The Game Instance will receive tracking updates and, depending on the user's settings, perform modifications to it and send it across to the Avatar.

How does the Avatar receive the tracking info and move? In animation code! An Avatar has an Anim Blueprint asset, which is a child of kemorig's custom Anim Instance class. An Avatar's Anim Blueprint can contain arbitrary stuff users author, but there is one mandatory requirement, which is to include a Kemorig Tracking node. This is a custom anim node that computes the pose resulting from tracking.

Now that we know the lay of the land, we can see how to implement Auto Eye Movement. We'll make a new Tracking Device for Auto Eye Movement (technically a "virtual" device), and it will do some simple math and send Eye updates.

tl;dr: kemorig has the groundwork already laid to add Auto Eye Movement. Easy!

This is how I implemented it, and it worked! Well, it kind-of worked...

We Need Some Changes

The first part of Auto Eye Movement I implemented is making eyes turn with the head. But a little problem arose with how to get the rotation of the head.

There's some subtlety with the head rotation we want. We don't want the final pose of the model, we only want the head rotation from tracking. Avatars may be playing arbitrary animations in addition to motion from head tracking, and we don't want those animations to rotate the eyes; we only want tracked head rotation to do-so.

Fortunately, kemorig already has a hook for this. When the Game Instance sends out the final tracking update, it is an event we can listen to. Therefore, we need our Auto Eye Movement code to handle that event in order to get the head rotation.

But there's a problem: the Game Instance sends all tracking updates out, and the Anim Instance filters them and decides which ones to actually use on the Avatar. For example, the user might have Webcam and ARKit active, and we only actually apply 1 of them to the Avatar. This means when we handle the Game Instance's event, we don't know which updates we care about and which ones to ignore!

Thinking through this problem, I decided that I should change this logic, and have the Game Instance filter tracking updates before it sends its event out. So, I went ahead and removed that filtering logic from the Anim Instance, and moved it to the Game Instance's processing. This also required some re-organization of the Game Instance code (for the better).

Changing a Core System

You might be thinking: "okay, so what? Why even bother explaining this little road bump?"

Well, a few paragraphs up we thought this was a trivial feature, and suddenly it's challenged an assumption in the code base and requires a modification to a core system.

Changing a core system means:

• A previously tested and bug-free system is changing, opening the door to new bugs.

• It's a system at the center of the app, influencing lots of parts of it, so those new bugs might be subtle and hard to spot.

Now, I'm a solo developer and kemorig is in early access. I can approve my own feature and make the changes, and a new bug isn't the end of the world. But with most games that is not the case:

• In most dev teams you can't modify an important core system at the drop of a hat -- There will be meetings and code reviews, it takes time.

• If the game is already released, it's probably too risky to change a core system, and the new feature will probably be cut.

Our easy feature has turned into a liability.

Complexity Grows

Next, I moved-on to getting eyes looking at the camera. As stated above, this requires some simple vector math. We need the location of the eye, the location of the camera, and the forward direction of the head to calculate a rotation.

Unlike the head rotation thing, we do want the final pose of the Avatar for this. That is, we want the pose the Anim Instance produces after whatever animations on the Avatar are applied.

Fortunately, anywhere in code where we can access the Avatar, we can then get a reference to its skeleton and look up the transform of a given bone.

Unfortunately, kemorig is an app that uses user-created Avatars, and looking up information about the skeleton poses extra challenges:

• What are the head bones and eye bones?

  • Users specify these bone names in the Kemorig Tracking node (in the Avatar's Anim Blueprint). No other part of the code knows these bone names, because they never needed to.

• When observing a bone, what does "forward" mean?

  • Kemorig is designed to accommodate any model for Avatars, and there are no requirements for bone orientation. That is, on your Avatar model, the head bone's "forward" direction may be along its X, Y, or Z axis, or the inverse of one of those axes. The app calculates each bone's orientation automatically at runtime, so that tracking movements are applied correctly. This is also calculated in the Kemorig Tracking node.

So this crucial information only exists in the Kemorig Tracking node, and some of it is supplied by users when they author their Avatar. I do not want to change how users author Avatars for this one little feature. Our solution needs to work within the constraints of how the Kemorig Tracking node works.

I decided to add 2 new events that the Kemorig Tracking node broadcasts. The first contains the head's transform and orientation, and the second contains the 2 eyes' transforms and orientations. Our Auto Eye Movement code can handle these events and get the info it needs.

This was Hard

That last paragraph was the most difficult part of this entire thing to implement. Animation Node code is very poorly documented, and very few people talk about them online. Implementing these new events required sleuthing around in Unreal's engine code to find examples, and it took a few tries to get it right.

I was not planning on writing animation node code for this feature.

Our easy feature has grown considerably in scope. But at least we're almost done.

Just Kidding, There was More Complexity

I actually glossed over another detail: Avatars may or may not have eye bones. Even if an avatar does not have eye bones, our Auto Eye Movement code still needs to send eye rotation along,

So, our Auto Eye Movement code has to do a little more jockeying with the events it receives. We prefer to calculate rotation with eye bones, but if we only receive information about a head bone, we will calculate the eye rotation differently.

We also don't know how many milliseconds will pass between events the Anim Node sends. This means we have to implement a time-out where the code will wait X milliseconds without an eye update, after-which it toggles to the head-only logic.

It's important we support Avatars without eye bones. After all, what if an Avatar only uses morph targets to rotate their eyes?

Hang On, Morph Targets? Oh No...

We now arrive at the big thing I overlooked before implementing Auto Eye Movement. Because after I implemented everything above, I tested out the feature and was struck by something.

It didn't look right... Something was wrong with the eyes...

I forgot about the morph targets. I forgot that eye rotation not only involves rotating the eyeballs, but also eye lid morph targets for looking up, down, left, and right. (Plus, as alluded-to above, you could author an Avatar that does all eye motion via these morph targets.)

So Auto Eye Movement would tell the Avatar to look down, and without the corresponding eye lid morph target, they looked freaky.

This made me realize a big a problem with how kemorig's Face, Eye, and Body tracking updates were implemented: The Eye update only sent eye rotation, and the eye lid morph targets were part of the Face update. This oversight was not an issue before, because there wasn't a reason to split Face and Eye updates between different devices before.

More Core System Changes

So, onward we go.

The solution to this problem is to change Face and Eye updates, such that Eye updates contain eyelid morph target info, and Face updates do not.

This change required an even bigger change to core systems than we made before. And it required reorganizing quite a bit of code (even ARKit and Webcam code) that was built with the assumption that Eye updates did not contain blend shapes.

Our easy feature has grown into a refactor of kemorig's tracking code.

We Did It!

At this point, the journey of getting Auto Eye Motion reached its end. This feature looked simple, but:

• It challenged assumptions made when core systems were authored.

• It grew in scope as it was implemented.

• The first implementation didn't look good and more work had to be done.

Again, I'm a solo indie dev, and kemorig is in early access. Implementing this feature was a good thing! It meant I improved a lot of code, and made the app more robust. This is a story with a happy ending.

What Did We Learn Here Today

So why am I sharing this story?

I feel it's a useful case study that gives a taste of what gamedev is like. In my experience, this is a typical story of implementing a feature in a game -- it is like this most of the time.

Not to mention, depending on the circumstances this is a also typical story of how features get cut.

Video game engines and game code are imperfect creations, built within deadlines to satisfy specific requirements. When new requirements are invented later, that imperfect code gets pushed, and dominoes start to fall. This is also why it's notoriously hard to predict how long a feature will take to implement; even a "simple" feature.

The difference with kemorig versus most industry work is that I have the time to properly refactor core systems when this happens. More often than not, game studios do not have the time to that, so if the feature is necessary it will come with hacks and one-off solutions to get it out the door.

"Hacks? But won't that cause more problems later?"

Thanks for reading!

kemorig v0.5 release

Happy new year! The plan was to get this build out before end of year, and if you're in in the US, I succeeded! 😅

New Features

• Auto Eye Movement

  • Optionally have eyes automatically move (instead of using tracking).

  • 3 modes: look at camera, rotate with head, rotate oposite of head.

  • Can also toggle on override in Avatar and Level blueprints, and look at custom targets.

• Tweakable eye rotation multipliers

  • In KemorigTracking node, you can now set a multiplier for how far eyes rotate up/down/in/out.

  • Useful for stylized models, where "realistic" eye rotation looks off.

Bug Fixes:

• Fixed bug where the ObjectLauncher blueprint would not compile correctly in-editor.

• Fixed various Level and Avatar internal functions not firing when testing in-editor.

• Fixed the "Shading Quality" graphics option accidentally setting shadow quality instead.

------------------------------------

DOWNLOAD LINK - password is "eyes" (no quotes)

CUSTOM CONTENT NEEDS TO BE RE-EXPORTED TO WORK IN v0.5

UPGRADING YOUR EDITOR PROJECT

DOCUMENTATION

DISCORD

Status Update - December 21

Happy holidays everyone! I hope you're having a safe comfortable end of the year.

First, a little life update, the last couple months have been very busy and hectic, but I'm doing okay and pushing through. In November, my partner and I had to say goodbye to our dog Cody, who passed away with a sudden illness. Later in November we also adopted a new dog, who is a ball of husky energy. We love him, but he does need lots of our time for training! I've also been chasing doctor appointments for various things, along with typical holiday stuff and family gatherings. It's a bitter-sweet December, kemorig progress has been slow, but I'm doing okay and continuing to get work done when I can.

Still, the current build of kemorig seems to be bug-free! And I hope the new Blender add-on is helpful for those of you getting your avatars into Unreal.

Here's what I'm working on now, and what you can expect soon:

• I'm shooting to get a new kemorig build out before the end of the year. This will be a smaller build with a couple features:

  • Auto Eye Movement. You will be able to make avatar eyes look at the camera, or look in the direction you turn your head. This is very useful for webcam tracking, which is not always the best at tracking eyes.

    • This has some complexity, because I want it to be toggle-able (in the app or blueprints) and key-bindable. You may only want this on sometimes, or in some stances.

    • I also want to let you set custom look-at targets in levels (this may or may not make it in the first version)

  • Better camera tools in blueprints. The current stuff is pretty basic, and I've run into limitations with my avatar. I want it to be easier to make great looking camera transitions between complicated stances.

• I'm working on a big collection of guides for making character models in Unreal.

  • See screenshot above.

  • Unreal's documentation is not good, and the educational resources on youtube are bad. It's hard to learn this stuff, so I'm making a one-stop-shop collection with all the info you need, whether you're a beginner, new to unreal, or an experienced dev that needs to look something up.

  • I realized when making kemorig docs, that there's a big overlap with general Unreal knowledge, and these new guides make sense on my personal site as general Unreal reference. This makes them useful to everyone, not just kemorig users.

  • As such, this collection has information about more than just making kemorig avatars -- it's more framed around a game character, which is more complicated. The kemorig docs will probably get some kemorig-tailored versions of these guides, and link out to my site when appropriate.

And that's what I'm working on.

By the way, you can always ask for help on our Discord server. I'm happy to help, and I really like seeing the progress on your avatars.

Happy holidays!
-PROTO

kemorig v0.4 release

kemorig v0.4

Introducing Base-Kun -- a comprehensive example avatar! Test out kemorig with him, and use him as a reference in-editor for setting up your own avatars.

There is also more editor content, and bug fixes.

New Features

• Base-Kun

  • A new example avatar with a cel shaded art style. Base-Kun is an example (in the app and the editor project) that features:

    •  Jiggle bones.

    • Multiple skeletal meshes animating together.

    • Idle and transition animations.

    • A ragdoll interaction.

    • Throwing props interactions, with physics impulses when hit.

• Kemorig Cel Shade

  • A suite of materials for cel shading.

  • Base-Kun uses it, and there are a few other examples showing what is possible.

• Object Launcher

  • A built-in actor you can place in levels and avatar blueprints for throwing props. Allows you to tweak throwing position and angle, and randomize thrown props from a list.

  • See how it's used in Base-Kun's blueprint!

Bug Fixes/Changes

• Fixed Level Blueprints not receiving kemorig events (this broke the level interactions in the example level).

• Fixed a bug where eye-wide weight sometimes did not calculated properly and got stuck at or close to 0.

• Fixed Eye Tracking not preferring ARKit when set to "Auto".

------------------------------------

DOWNLOAD LINK - password is "character" (no quotes)

CUSTOM CONTENT NEEDS TO BE RE-EXPORTED TO WORK IN v0.4

UPGRADING YOUR EDITOR PROJECT

DOCUMENTATION

DISCORD

Blender add-on -- docs, update, and new logo

Hey everyone, I've done some more work on the PROTO Game Asset Tools Blender add-on:

• Updated to V1.1.0. Improved vertex color tools.

• Documentation is now available on my site: https://protowlf.com/blendergameassettools/

• There's a logo now!

I've updated the download post, and you can download the add-on there: https://www.patreon.com/posts/blender-proto-115751827

Again, I'm making the add-on patron-exclusive for 1 month. It'll be available to everyone for free in mid-December. Thank you for your support!

Next up, I'm working on animations for Base-Kun in kemorig, and going back to working on the big Blender-For-Unreal guide.

[Blender] PROTO Game Asset Tools - Download

This is the download for PROTO Game Asset Tools - a Blender add-on.

This add-on makes game asset authoring in Blender easier, and fixes issues with using Blender with Unreal. Features include:

• FBX Exporter with unit-conversions -- fixes unit conversion issues with Unreal and other engines.

• Quick Export panel to easily export meshes and animations.

• Quick Action panel to easily manage animations in Blender.

• Vertex Color tools to pack information into vertex color channels for games.

Video

Watch a video overview here: https://www.youtube.com/watch?v=MaZWdJQCnLM

Compatible with

Blender 4.2 - https://www.blender.org/download/releases/4-2/

Documentation

https://protowlf.com/blendergameassettools/

Download

Status Update - November 10

Hi everyone!

This month the status update is visible for free and paid Patrons.

Blender Add-on

Today I'm releasing the v1 of my new Blender add-on: PROTO Game Asset Tools. It's exciting for a few reasons:

• It will make using Blender for kemorig much easier.

• It is (as far as I know!) the only free add-on for Blender that exports to Unreal without bugs.

• It streamlines the Blender workflow for animating for games.

Here's a video overview: https://youtu.be/MaZWdJQCnLM

It will be available for all paid Patrons exclusively for 1 month, before free release to everyone. Thank you for your support!

Huh? Why a Blender Add-on?

My current goal is to give you an comprehensive example avatar (source Blender files included) and write a full guide on how to work with Blender for Unreal. Base-Kun is that avatar -- pictured above!

Unfortunately, I discovered more issues with Unreal's importer, and concluded that the guide was impossible without purchasing paid Blender add-ons. That sucks! So I set out to make my own free add-on as part of the guide.

When is the next kemorig Update?

There will be an update this month (November), and it will come with Base-Kun, my cel-shader materials, bug fixes, and possible some minor features that sneak in.

Unfortunately, last month I kind of got blind-sided by the Unreal importer issues, and decided I needed more time to give you this update. I have still been working full-time on kemorig and its related stuff. I try really hard to give you something every month. I hope the status updates kept you in the loop and that you're all still on-board!

Your Support is Making This Possible

I want to end this status update with another enthusiastic thank you to everyone supporting me financially, either here on Patreon or on Twitch. It's kind of crazy that I could spend 2 weeks suddenly making a Blender add-on to fix Unreal's problems. I think it's going to help you guys a bunch, and I'm hopeful that Unreal game devs in-general will appreciate it too.

Blender Addon

As I've been working on an example avatar, I realized that I need to do additional work to make the Blender to Unreal process more accessible.

The Problem

I discovered additional bugs with Unreal's FBX importer, that are specifically related to unit-conversions. I thought that Unreal 5.4 (which kemorig uses) fixed these problems, but turns out it didn't fix everything. A model exported from Blender's default FBX exporter will mostly work, but there's some bad data in the model. And when you try to make a physics asset, Unreal's physics editor won't generate shapes correctly.

I don't have this problem with my avatars because I use the Blender addon "Auto Rig Pro", which has its own FBX exporter that handles the unit conversion during the export.

The Plan

I'm going to try making a simple free Blender addon that adds what you need to the FBX exporter. It'll be a tool you can use for making kemorig avatars, and also a tool for Unreal devs in general. I'm hoping it won't take that long to make, but I've never made a Blender addon before, so I have to wrap my head around it.

(If you're not aware, FBX bugs when using Blender for Unreal has been a problem forever. It's really frustrating, and I wish Epic would fix it already. But hopefully with this addon I can give everyone a simple free solution).

In addition, I have a couple other scripts I use when making characters in Blender. I'm going to bundle these in with it too.

This might delay the next kemorig update -- I'm still shooting for end of October but it might slip past. The whole point of the update is to give you Base-Kun, a crystal-clear example of how to make characters for Unreal. And it turns out I need to do this extra work to achieve that.

Status Update - October 21

Hi everyone! Here's what I've been working on and what to expect soon.

Base-Kun

I've spent the better part of the last 2 months working on Base-Kun. This new 3D character is for 2 things:

• He will be a new example avatar in kemorig, source Blender files included, to serve as a reference for setting up a complex cel shaded avatar. The goal is that you can see a clean straight-forward example for all steps in the avatar pipeline.

  • He's replacing Face-Chan. Sorry Face-Chan fans 😔

• He is a mesh to showcase my cel shaded materials in Blender and Unreal. I'm going to use him for other things, but I'm not 100% decided how yet. I might simply release him for free on his own.

"Why a human character?"

It's been a while since I made a human, and this was a very valuable exercise. I think it's harder to make a human look good than an anthro. Truth be told, it took a long time to write this status update because I wasn't happy with how he looked for a while. (I'm actually still not 100% satisfied!)

This kind of 3D that's trying to look 2D is challenging, because if anything is a bit off the whole character can look wrong and unappealing. I also wanted his design to be super simple (he's a base mesh!), and I think this made it even more unforgiving.

Also, having a human cel-shaded reference is more valuable to most people. A human better represents the typical anime art style than an anthro.

Next kemorig Update

The next update for kemorig will probably be content-only and add Base-Kun, as well as the cel shaded materials he uses. I'm shooting to get it out before the end of October.

The materials will be organized so that you can use for your own stuff!

Thanks everyone! I hope you can forgive me for putting a human on your timeline.

What kemorig Example Content Do You Want?

Hi everyone, quick question:

What example content do you want to see included with kemorig? Comment below!

By example content, I mean stuff in the editor project to help you make your avatars and levels.

I'm already working on the following:

• A feature-rich cel shading "toon" material (based on what I use with Proto)

• A full body version of Face-Chan, with multiple stances.

• A ragdoll interaction.

• A throw stuff interaction.

• A jiggle bones example.

(One thing to note: I unfortunately can't give you the editor version of the Manny mannequin avatar that comes with the app, because I don't have the rights to distribute Epic's source assets. This is part of why I want to make my own full body example with Face-Chan)

Status Update - September 16 (+ new kemorig docs)

Hello again, here's the status update for September.

New kemorig docs

I just updated the kemorig site with new documentation pages:

• Tracking Settings - A new page about the Tracking Settings feature (you may not have known this existed!)

• Updated Quick Start Guide - updated with info on setting up webcam, and new images reflecting the current state of the app.

• Webcam, ARKit, and Microphone - 3 new pages with info on setting up tracking in the app, and some tips on getting the best results.

Documentation is always a moving target, and I'm doing my best to keep it up to date and tell you everything you need to know. Better documentation = fewer support issues! As always, if there are things you're still confused about or need help with, speak up on the discord or leave some feedback here on Patreon.

Next kemorig Update

The last few months of kemorig saw 3 huge releases in a row: the initial early access launch, then the levels feature, then the webcam feature.

The next update needs to be a smaller one. I'm going to focus on bug fixes and giving some love to smaller features, like camera scripting and some annoyances I've run into with stances and interactions.

I may also get in some more example content to help with authoring avatars and levels.

The next big feature on the horizon is hand tracking. I'm not diving right into this one yet, and I want to take my time with this one. It's probably going to be a few months before hand tracking gets in.

Art

I've been all-in on kemorig lately and want to spend a bit more time in the upcoming month on 3D art. Somewhat ironically, part of that is I want to make more promotional material for kemorig now that the webcam update is out. But I also want to take some more time to keep my sculpting and general art skills dusted off.

Thanks everyone. I hope September is treating you well!

kemorig on my stream

Hi everyone!

This is a video with two new examples of how I'm using kemorig in my own stream. It's a glimpse at what I've been working on lately, and I hope it can also serve as inspiration for what you can do with it with your own VTuber!

Last Monday was a special stream on Twitch for me, so I prepared new features for my VTuber. The two big ones were these:

Fighting Game Mode

My cel shaded style is very much inspired by Guilty Gear, and I wanted to take a stab at my own GG-style animations for Proto. In addition, I thought it would be cool to let Twitch Chat control Proto, and give chatters something to interact with during down times.

How it's done:

• This feature is set up as a 2nd Stance Group (the 1st is the main group I use for standing, sitting, etc while streaming) with a Neutral, Crouch, WalkForward, and WalkBackward stance.

• I have an additional Toggle-Interaction to enable Fighting Mode. When Fighting Mode is not enabled, the avatar's Anim Graph toggles off the slot that Fighting Mode uses.

• The stances are bound to hotkeys, which streamer.bot (a program for interacting with Twitch) triggers in response to chat commands.

• There is a little bit of blueprint script in my avatar to move him, automatically switch back to Neutral stance, and stop him from moving off the side of the screen.

PSP Mode

I stream a lot of retro games, and wanted to stream off my PSP for the first time! It was an opportunity to do something special with Proto, and I had the idea to literally have him holding a PSP with the illusion that we're looking over his shoulder.

How it's done:

• The PSP sitting pose is an additional stance added to the avatar's main stance group. (not shown in the video: it also has transition anims authored to/from other stances).

• When entering or leaving this stance, the avatar's Blueprint toggles visibility of a skeletal mesh (the psp model), which is attached to a bone on the avatar.

• The zoomed-in PSP framing the video feed is a rendered image from Blender (it does not animate).

I'm really happy with how these features turned out. Plus, they help me improve kemorig, as I run into little annoyances here and there while implementing them.

Hope this was interesting, and can't wait to see more of YOUR creations with kemorig!

kemorig v0.3.1 HOTFIX

This is a hotfix release fixing the following issues:

• Fixed avatars exported with Blender's default FBX exporter having extreme movement with webcam.

  • The Kemorig Tracking node now auto-calculates a movement factor based on the mesh. You have the option to manually enter it as well.

• Fixed Avatar Tracking Settings not being applied before stance changes.

Thank you for your patience and swift bug reporting!

------------------------------------

DOWNLOAD LINK - password is "newcam" (no quotes)

CUSTOM CONTENT NEEDS TO BE RE-EXPORTED TO WORK IN v0.3.1

UPGRADING YOUR EDITOR PROJECT

DOCUMENTATION

DISCORD

kemorig v0.3 release - the webcam update

kemorig v0.3 is here!

Bringing webcam support, quality of life updates, and more!

Here's the changelog, download and more info at the bottom:

New Features

• Webcam Tracking

  • You can now use a webcam to track head, eye, and face motion. It also tracks your head position in addition to rotation.

  •  Webcam face tracking tracks a sub-set of facial morph targets. The same morph targets you set up for ARKit work for Webcam and vice-versa.

  • Tracking uses a version of Google MediaPipe integrated into the app.

• Tracking Selection

  • You can now combine tracking methods (currently ARKit and Webcam) and route different methods to body, eye, and face tracking.

  • For example, you can use ARKit for eye and face tracking, and Webcam for body tracking.

• "Kemorig Tracking" anim graph node

  • When authoring your avatar, the old "ARKit Tracking" anim graph node is gone and replaced with "Kemorig Tracking", handling all skeletal motion from any tracking method.

  • You no longer need to manually enter bone orientations! Just enter what bone corresponds with head, neck, spine, etc.

• Improved UI

  • The app's UI now features collapsible/expandable categories, making it easier to navigate big menus.

• Miscellaneous

  • Eyelid settings now has an interactive graph for dialing in your settings. You can also now share settings between both eyes.

  • New feature for pausing eye motion when eyelids are closed (threshold tweakable in menu)

  • New feature to help correct incorrect head roll when a tracking device is not centered (tweakable in menu).

  • kemorig_editor project now lets you recenter and mirror tracking when testing.

  • Help menu now has a link to documentation, and a Tips section.

  • Various menus options now have a "reset to defaults" button.

  • Tracking Settings reorganized with many new options (Tracking Settings the collection of settings you can apply to your avatar per-stance to tweak how tracking behaves).

Bug Fixes/Changes

• Fixed some bad math with rotations when re-centering tracking.

• Fixed bug where the ARKit status icon erroneously showed tracking was active.

• When tracking methods are toggled off, the app now re-centers the avatar.

------------------------------------

DOWNLOAD LINK - password is "newcam" (no quotes)

CUSTOM CONTENT NEEDS TO BE RE-EXPORTED TO WORK IN v0.3

UPGRADING YOUR EDITOR PROJECT

DOCUMENTATION

DISCORD

Status Update - August 24

Hi everyone!

This month's status update is all about the webcam update coming to kemorig! Check out the video above for a preview of what to expect.

There's a lot coming in the new update:

• webcam tracking - this is the big one! Webcam will track your head rotation and position, eyes, and a sub-set of face blendshapes.

• tracking selection - the app has been reorganized to easily let route different tracking methods to parts of the avatar (currently ARKit and webcam). You can independently select which method influence body, eyes, and face.

• better eyelid control - there's now an interactive graph to tweak eye blinking and wide-open blendshapes.

• better menus - menus now have expandable/collapsible sections to more easily navigate (especially important as the number of options to tweak tracking grows).

• easier avatar bone setup - you no longer have to manually enter bone orientations in the anim graph node to get your avatar tracking.

I'm still hoping to get this released before the end of August. The features themselves are done, but I'm running into a snag with packaging the editor version (some technical difficulties with Unreal and packaging plugins). Hopefully I'll get those resolved quickly, but if not I might have to reorganize my code again before I can release.

Thanks everyone again for your support! Hope to get this in your hands soon.

-PROTO

kemorig Webcam WIP

Hey everyone, webcam tracking is working! 🎉

This is a quick work-in-progress video showing the current state. It's using Google MediaPipe (as alluded-to in the last post), and runs inside the kemorig app.

Right now you're seeing it track head and shoulders. Finally, we can see avatars move around instead of rotating in-place! It will also be able to track eyes, and influence some (but not all) face blendshapes.

Now, don't get too excited by the words "shoulder tracking". Although MediaPipe can technically track the whole body, it's very rough, and for the most part the only stable data we get is for the face. On top of that, depth (i.e. detecting what body parts are closer to the camera) is extremely rough, so rough that it's virtually unusable data. So, think of this as head rotation and position, with a tiny bit of shoulder information sprinkled on-top.

Combining Webcam with iPhone ARKit

You will be able to use webcam tracking at the same time as existing ARKit tracking. This is very cool! I plan to use this in my own setup, to get great face tracking from ARKit and get more body motion from the webcam.

There's going to be a new menu that lets you select which tracking methods affect Body, Eye, and Face tracking independently.

Easier Avatar Setup

Did you find it annoying manually entering bone orientations to setup your avatar? Me too, it sucked! With the webcam update, that's gone and the animation code now auto-detects this.

Code Refactor

I've spent the last couple weeks doing a lot of work reorganizing kemorig's code to support multiple tracking methods. Turns out, many of my assumptions about how to structure the app were challenged with the introduction of webcam tracking.

This refactor is basically done now. This is behind-the-scenes stuff, but the way tracking data flows through the app is vastly improved.

Thanks everyone, can't wait to get the next build in your hands!

Status Update - July 21

Hello!

I hope the v0.2 kemorig update has been fun to play around with.

First I'm going to share some cool stuff being done with kemorig, then go over what I'm working on right now.

Cool kemorig Stuff!

• Here's some clips of other streamers using kemorig!

  • Djano has a super-detailed wolf using fur rendering: https://www.twitch.tv/djano__/clip/ArtsyPlacidButterflyWutFace-i1NaiAq9UZcuqQX3

  • Kofu has a stylized borzoi that they've done some really neat shader work with: https://www.twitch.tv/kofu/clip/ChillyImpossibleTermiteDxAbomb-ojV7AshFSaFnlwxC

• Have a clip of what you're doing in kemorig? Post it down below!

Personal Site

After the v0.2 kemorig release, I went on a 2-week side quest to fix up my personal website: https://protowlf.com 

This is where I occasionally post tutorials, and now I've updated it to host documentation for my Unreal plugins. It also functions as a home-base for me, and I might expand it in the future. (Speaking of, I really need to post more tutorials...).

kemorig Webcam Tracking

I started looking into solutions for webcam tracking in kemorig. I've learned some things about the current landscape of this:

• There appear to be 3 main technologies out there that could realistically be used:

  • OpenSeeFace (this is what VSeeFace uses) - a small machine learning project using a relatively small (hundreds of thousands) dataset of images. Good at tracking where your head is, can only track rudimentary expressions.

  • MediaPipe - Google's machine learning stuff. Pretty sophisticated face expression tracking (still not as good as a depth sensing camera, but not a surprise). This is what most other apps other than VSeeFace are using.

    • However, Google's API is indecipherable, their documentation sucks (and a lot of it has been deleted), and it's really hard to build this into a C++ application. Hard as in: I am a professional software engineer and I have no idea where to start.

    • There are some open source projects and that implemented an older iteration of MediaPipe face/hand tracking into C++ apps, and I suspect this is what the apps we're seeing are using.

  • NVIDIA - NVIDIA is inching into face tracking with their machine learning stuff. I don't know a lot about it, but it turns out iFacialMocap has created a desktop app that uses it. Of course, this requires an NVIDIA gpu to use.

I think going forward with kemorig, these are the 2 things that will probably happen:

• 1) Implement an open source MediaPipe solution into kemorig, and support it natively.

• 2) Add support for iFacialMocap's desktop app, which just so happens to support Unreal out of the box.

  • This isn't an idea solution, as it requires running an extra app alongside kemorig. But I want to support iFacialMocap's iPhone app anyway, and if implementing support for the desktop app is easy, that's a win-win.

WHEN will webcam tracking come to kemorig?

Short answer: I'm not sure. But probably in August.

If it starts really taking a long time, I might add iFacialMocap support first, because that's undoubtedly an easier task.

A Note on Head Position Tracking

One of kemorig's current problems it that the Live Link iPhone app does not track head position (only expressions and rotation).

iFacialMocap's iPhone app does track head position, making it a better solution that Epic's Live Link app (note that iFacialMocap is not free, but it's pretty cheap). I plan to add support for this app at some point.

I might also look into a webcam mode that only tracks your head/torso, and you still use an iPhone for face expression. By combining the two, that might produce the highest fidelity motion. I don't have concrete plans for this, but it's on my mind.

Alright, long post! Thanks for reading everybody, and I hope you have a good day.

-PROTO

Next Steps for kemorig

I hope you're enjoying the new kemorig build! I'd like to share what I'm planning to work on next, and also extend a question:

What are you looking for in the next builds?

If you have thoughts, please comment on this post! What missing features are most important for you? What are the problems you've run into so far? What do you like most or dislike most?

My Current Plan

Here what I'm planning on diving into next:

• Webcam Tracking - I think webcam tracking is a make-or-break feature for a lot of people. I want to get this in next.

• Porting Unity Models - For many people, "how do I get my VRChat/VRM avatar into kemorig" is their first question! The good news is I don't think it's that hard to do. But not everyone is a 3d artist, I don't have a step-by-step guide, and there are some stumbling points. Some things I want to offer help on are:

  • Shaders - You can set up PBR materials in Unreal easily, but what about popular cel shaders from Unity? I'd like to make guides on how to get started, and I might add an included cel shading material.

  • Do I look like I know what an FBX is? - If you aren't a 3D artist, it's probably not obvious what an FBX is, or how to get it, or what exactly you got when you bought an avatar in the past. I'd like to help educate people on the basics, so they can understand how to work with Unreal. You shouldn't need to be a 3d artist to port your model!

• Hand Tracking - Multiple streamers have told me that without hand tracking, they can't start using kemorig. I understand! I consider webcam tracking higher priority at the moment, but my feeling is that this is the next big-ticket item in line.
  

Let me know what you think! Do you agree with these priorities, am I missing something big? I'd love to hear from you down below.

As a reminder, you can also join the protowlf DISCORD

kemorig v0.2 release

kemorig 0.2 is here

This is a big update!

New Features

• Custom levels

  • Levels are streamed in and out while running the app. You can switch levels behind your avatar without stopping your broadcast.

  • Level transition effects are customizable and can be disabled if desired.

  • Levels can define interactions and run Blueprint script. They also support their own hotkeys and camera positions.

• Talk detection

  • Now your Blueprints get Start/Stop talking events, allowing you to create logic similar to PNGTubers. You can get creative and blend in different animations when speaking.

• New Microphone menu

  • Added a mic monitor, with adjustable noise gate. Also added an interactive widget to adjust talk detection.

• More Blueprint Support

  • Added convenient events to handle inside of Avatar Blueprints and Level Blueprints for Interaction Played, Stance Changed, Level Begin, Level Swap Started, Level End, Avatar Spawned, Avatar DeSpawned, Mic Start Talking, and Mic Stop Talking.

  • You can now switch avatars and levels in your Blueprint logic, by calling SelectAvatar or SelectLevel.

  • Added a Camera Position actor you can place in levels, and utility functions to blend the camera between them. See the example in the Kemorig Day level. (Note: the camera functions are a little limited and will be expanded in future updates).

• Content Error Validation

  • Custom Avatars/Levels will now be flagged with errors in the app UI if any are detected, to help narrow down issues.

  • Currently only detects one (if you forget to define a Name in your DataAsset), but this will be expanded over time.

Bug Fixes/Changes

• Fixed issues with previously saved hotkeys not appearing in the UI.

• Fixed hotkeys not accepting gamepad hotkeys, and displaying modifier keys in the key box.

DOWNLOAD LINK - password is "levels" (no quotes)

CUSTOM CONTENT NEEDS TO BE RE-EXPORTED TO WORK IN v0.2

UPGRADING YOUR EDITOR PROJECT: guide

DOCUMENTATION

DISCORD

Status Update - June 17

Hello!

Thank you for supporting my Patreon! The response to kemorig has been tremendous and I can't thank you enough.

Since there are many new faces, I'll use this status update to let you in on the various things I work on.

This is what I do these days:

• kemorig - This is my main project right now. The vast majority of my dev hours are devoted to kemorig.

• Streaming - I stream on Twitch every weekday! Sometimes I stream work, and sometimes I stream games. When I went indie I didn't initially have Twitch in-mind, but the way the cookie crumbled, turns out Twitch has been a very worthwhile and fulfilling part of my indie life. And I discovered VTubers are super cool!

• Unreal Plugins - I have released 2 code plugins on the Unreal Marketplace, and will probably release more over time. These are things I wrote for my own projects, and were worth polishing up for release. These take up a bit of my time, doing support for users, updating, and bug fixing.

• 3D Art/Animation - I do 3D art, especially cel shaded characters, in-part to support future indie games. With kemorig's release, art has taken a back-seat, but over time I expect to shift back into art more.

• Game Projects? - I have a game prototype I was previously working on, but once I decided to make kemorig, the game had to get put on pause. What I was working on was a short experimental narrative game, which was something of a test-bed for my 3D art style. This can is kicked down the road until kemorig no longer requires my full attention.

kemorig status 

The next kemorig update is on-track to release this month, and the big new feature is custom levels! And it's going to be very fancy: levels are streamed-in and loaded without interrupting your avatar! Check out the video above for a sneak peek (it uses a level from the Unreal Marketplace).

Even if you don't want an environment for your avatar, levels are still a useful way to make your own lighting setup, and your own scripted camera positions. For example, for my stream I will be using a custom level to place camera positions for my stances, and blend between them during transitions.

After levels are done, I will most likely be tackling webcam tracking.

Thanks for reading!

-PROTO

kemorig guide - Create Poses in Unreal

I just uploaded a new guide for kemorig: Creating Poses in Unreal

It's possible to make pose animations in Unreal, and this guide goes over how to do it. Very useful if you aren't an animator, and just want to get your avatar out of T-pose.

Future Guides

One thing I'm also working on is a few pages on using Blender for Unreal. Blender is what I use, and it can work very well! But for whatever reason quality information on this is hard to come by. Blender mostly works out of the box... but there are some gotchas.

(spoilers: I recommend purchasing Auto Rig Pro and using its FBX exporter if you use Blender for Unreal)

Discord

I have a discord! The protowlf discord has a kemorig section, and some of you have already hopped in, reporting issues and getting help.

Join here

kemorig HOTFIX v0.1.1

 This is a hotfix release fixing the following issues:

• Fixed the editor project being set to deferred shading mode (it should have been forward shading like the app).

• Fixed custom materials appearing black when loaded into the app.

• Fixed a crash that sporadically occurred during normal use of the app (garbage collection was invalidating an important property).

• Improved the lighting setup of the app's map, and the test map in the editor. (Later, you'll be able to adjust the lighting in the app, for now it is a static setup).

• Updated app to engine version 5.4.1 (turns out this didn't fix any issues, but it's updated anyway)

Thanks everyone for the support, and for quickly reporting bugs! Hopefully this hotfix goes smoother.

(Note: I didn't update the contributor credits in this hotfix, wanted to push it asap. The credits will get updated next release)

DOWNLOAD LINK - password is "hotfix" (no quotes)

UPGRADING YOUR EDITOR PROJECT: guide

DOCUMENTATION

DISCORD

What is Supported? - kemorig Early Access Status

Making this public post to highlight the current status of kemorig. It's not done yet! You probably want to know:

What tracking features are currently supported?

• ARKit Face Tracking - using an iPhone X or newer using Epic's Live Link App. This tracks face expression and head rotation (not translation).

• Lip Sync - using a microphone

What tracking features are NOT supported yet, but plan to be?

• Webcam Tracking - webcam tracking is a big feature for many, and it's not implemented yet!

• Leap Motion - for hand tracking.

• Other ARKit Apps - I'd like to support other ARKit apps (such as iFacialMocap). This would also allow for head translation tracking.

Other things I plan to add support for:

• Custom Levels - You'll notice the app has a tab for levels, and it's all disabled. This is an important feature (probably the next feature that will be added).

• Twitch Integration - Right now you can bind hotkeys to interactions, but eventually I want twitch integration directly in the app. Until then, you can use other apps (like streamer.bot or SAMMI) to interact with twitch and press hotkeys virtually. I use streamer.bot in my setup right now and plan to make a guide for that soon.

• More support for microphone-only avatars - It's possible to make avatars that simply respond to mic input and don't do face or body tracking. I actually think a lot of people will enjoy this! But I want to add some explicit support for this to make it easy to set up.

If you plan to try out kemorig in early access, read over this list!

Thanks again for all the kind words and support! I hope you'll be able to make awesome stuff with kemorig, and more features are on the way.

kemorig v0.1

The first early-access release of kemorig is here!

This is the very first build, I'm so excited to share it! I hope it's a smooth experience for everyone.

Notes on Early Access:

• Don't share the files publicly, this is exclusively available for patrons.

• Be wary that things might change in early access, and in future builds you may have to re-export avatars, or even re-do parts of them. I want to minimize this of course, but be advised!

• The tracking methods currently supported are limited! More methods will be implemented during early access.

Discord

kemorig doesn't have it's own discord, but you can join the PROTOWLF discord and find a channel in there for kemorig: here

Share cool stuff that you make!

If you need help, please refer to the documentation below first, and I can answer follow-up questions on the discord.

I'm very much looking for feedback! Let me know what you love and what you don't quite love.

DOWNLOAD LINK HERE - password is "firstbuild" (no quotes)

DOCUMENTATION - Go here to learn how to use it!

Status Update - May 13

Here's a little update on what I'm working on! Moving forward, I plan to give you an update like this once a month.

As you'd expect, I'm working hard on getting kemorig ready for its first release to patrons. It might be ready by end of May, but might come in June.

The big task that needs to be completed is documentation. This might seem strange, "just release it and finish the docs later!" But this is critically important before release, otherwise you'd get it, start trying to make an avatar, and immediately hit brick walls.

So far, I've got the guides for setting up the editor done. Next up is finishing the steps on how to set up all the features of avatars.

This will be on a kemorig website! The docs, as well as a nice landing page, will be up on a new website. The site will probably go online once the first build is ready for release.

As a side note, I'm pretty happy with the site layout I've put together. I'm not a web-dev by trade but I have scrapped together sites before. It's a static site using Jekyll and the popular theme Minimal Mistakes, and I'm customizing a ton of stuff on top of it to make a nice documentation interface. The cool thing is, it's a custom static website, not using a commercial site builder. So it's super simple and fast, and isn't loaded with bullshit and tracking.

I'm fixing kemorig bugs. I've been using kemorig in my own streams, and I'm catching lots of oversights and bugs. I'm resolving as many of them as I can before the first release.

Thanks everyone!

kemorig Early Access Preview

Hey there! This is a video going over what to expect from the first early access build of kemorig that will be released to patrons.

The video shows what the app itself will support. In addition, the first release will come with an editor project for authoring custom avatars, and documentation on how to do it.

I'm excited to get it into your hands! Thanks everyone.

-PROTO

Hello, Patreon!