SpeechRecognition and EvilCorpHQ demo scene

A few weeks back I promised to look into speech recognition, as it would be an obvious good addition for LogicBricks. This can recognize simple keyword lists as well as more complex grammar using Windows 10 and Unity's speech recognition features. When starting with this project the assumption was I can wrap the simple looking API in a plugin and build a simple demo scene in just a few hours. However, making a reasonable plugin UI, implementing error handling and necessary performance features as well as circumvent VaM quirks with file dependencies took a bit longer.

Features

• This is not "AI", it does only understand phrases you teach it to understand. It simply triggers whatever you hook up to it when a phrase is recognized.
• Keyword Phrase: Simple list of keyword variants. Works great for things like "Yes", "Yeah!", "Yep", ... or "No", "Nope", ...
• Grammar Phrase: Provide an XML file that defines a grammar structure, which helps system to make more sense of what you are saying, provide synonyms for individual words, etc. I recommend the Microsoft documentation here for more details what is possible.
• The plugin allows you to listen for multiple phrases simultaneously. Each have their own trigger actions. To provide context you can enable/disable phrases via triggers depending on what makes sense at the time in the scene.
• UI should be familiar if you used LogicBricks before.

Examples

Some sentences that should be recognized by the demo scene's grammar definitions:

• "launch the terminator robots"
• "deploy killer androids"
• "release the mutant penguins"
• "retrieve the sharks"
• "recall ninja sharks"

In this case I did stick to simple sentences structured as "action - fluff - object". Of course you could define variation not just in words but also in sentence structure.

Notes / FAQ

• This plugin is using your system default microphone, not your microphone setting in Oculus or SteamVR software! Make sure your mic is enabled and has a reasonable volume setting.
• WHY on earth does this use a ".json" file extension for ".xml" files??! Because....VaM. When creating a VAR package this allows dependencies to be handled automatically, making sure your scene does not break. As the list of extensions properly supported by VaM is fixed, ".json" is simply the least terrible choice.
• If you XML files refer to other XML files using the ruleref tag, you can/should add a dependency hint for each file needed, so VaM can find them. For each file add a comment line like this: <!-- "VaM Dependency Hint" : "common.json" -->
• Of course Windows 10 Speech Recognition does not know what a VAR package is and how to get files out of it. Therefore, when the plugin encounters XML files (".json") that are located inside a VAR package, all ".json" files in that package are extracted to "Custom\PluginData\SpeechRecognition" to make them accessible. Sadly that includes other files likes scenes that don't really need to be extracted. If you have lots of those, you might want to put your XML files into a separate VAR package to avoid extracting more than needed.
• The problem with speech recognition is that a creator will have a hard time to think of all the possible things a player could say and setup recognition and reactions for those. There is simply no way to implement everything. Usually you will have to give the user some kind of hint what types actions can be done.

Credits for EvilCorpHQ scene

• Speech synthesis for assistant voice by Google (via voicegenerator.io)
• LipSync by AdamAnt5
• Hair by NoStage3

Dependencies for EvilCorpHQ scene

• You obviously need LogicBricks.8 and SpeechRecognition.1 both attached below, but otherwise all the needed dependencies can be found for free on the Hub, just use VaM's handy "Scan Hub For Missing Packages" function.

Download

• This is an EarlyAccess release! Download will be available for free under CC BY-SA license after 2021-06-14. You are allowed to reference this package in your own VAR packages, even if they are paid or use a different license. Links to my Patreon are always appreciated.
• Unusually long 4-week EarlyAccess phase because I expect this to have a rather small target user group. If you like content like this, you should probably subscribe.
• Beware of some browsers mistakenly renaming .var files to .zip when downloading from Patreon. More details on how to install VaM content and missing dependencies here.