xvasynth

Interim batch (14 voices); Second community voice actor model; xVATrainer nearing completion; Hardware upgrade

A fair bit to update on! Let's get into it.

First of all, I've been going through more v1->v2 voice re-training. The 13 re-trained voices are (all Skyrim):

- Serana
- FemaleCondescending
- MaleBrute
- MaleYoungEager
- Frea
- MaleEvenTonedAccented
- MaleDrunk
- Galmar
- Esbern
- Tulius
- Ulfric
- Kodlak Whitemane
- Arngeir

I will use these to update the FuzRoBork integration plugin with pre-cache files for these voices, in the next few d...

Interim voices (5); First community voice actor model

So the strawpoll in the last post apparently doesn't work (fab), but I got another poll going on the discord server, and the vote was overwhelmingly (17 for, 6 against) in favour of doing more voice re-training v1->v2, rather than a new set of voices. The re-training is underway for a good number of voices, but a couple are done already that I can share already, and there were also some novel requests.

The voices included are:
- GTA Vice City: Steve
- GTA Vice City: Victor
-...

Fuz Ro Bork integration! ; Meta-poll #2

This one's been a long time coming, but it's finally here (and finished). 

This is an xVASynth plugin to integrate xVASynth voices into the Fuz Ro Bork mod which adds real-time TTS to the player in Skyrim. This has been a joint effort with the FuzRoBork author to get it to work. Check the video out to see it in action.

The non-existent latency was done through the use of pre-cached files for several v2 voices, which ship with this initial release, though more could come in th...

Batch #34 - 6 (5+1 bonus) new voices; xVASynth now public on Steam

The next voices are here!

Had a slight set-back. While integrating the training script into xVATrainer, I found some ways to optimize the training scripts, which also affects the quality (I think for the better - let me know!). This meant a few days of (necessary) re-calibration, but the scripts are now faster, which should hopefully mean future voices should come out a little bit more often.

Anyway, here's the voices, including a request:

- Cyberpunk: Evelyn
- Cyberpunk:...

Interim batch of 12 voices re-trained to v2

While the poll was going, I spent some gpu time re-training some of the other older v1 voices over to v2, in addition to the ones mentioned in the last post. With this update, we're now at about 20% of the voices having been re-trained to v2. The training scripts are getting faster and faster, with every optimization and tweak. Seems like the weeks spent on that are actually amounting to something!

Anyway, the full list of updated voices:

- Fallout 4: Nate
- Fallout 4: Nora
...

1 year anniversary of v1.0 | xVASynth on Steam!

Well, it's now been an entire year since I first posted the initial full release of the app, on 11th Jan  2021!

Things have come a long way for xVASynth, from its early days in 2018 (YouTube showcase of v0.1). Since this time last year, we've reached:

• Over 450 voices trained (not including re-trained models)
• Support for 39 games and game series
• Almost 2000 members on Di...
  2022-01-11 12:02:28 +0000 UTC View Post

Batch #33 - 6.5 new voices

Some new voices from the last poll are here, along with some further v1->v2 re-trained voices.

I actually included one more voice from the poll that didn't win, because I had already trained it before the poll ratings changed order, right at the end. The voices are as follows:

- Skyrim: Florentius
- Fallout New Vegas: FemaleAdult09
- Fallout New Vegas: FemaleAdult09 (Joana)
- Fallout New Vegas: FemaleAdult10
- Fallout New Vegas: MaleGhoul02
- Fallout 3: FemaleGen...

Interim 11 (4 new and 7 v2 re-trained) voices;

I hope everyone's having a great holiday.

I've been taking it a little easier on the voice training, and focused a bit more on re-training some main voices from v1 to v2. I've included them here, as well as some new voices.

The new voices:

- GTA Vice City: Colonel
- GTA Vice City: Diaz
- GTA Vice City: Lance
- Age of Empires: Narrator (IV)

Voices re-trained to v2:

- Witcher: Triss
- Skyrim: Isran
- Skyrim: FemaleOrc
- Skyrim: FemaleArgonian
...

Batch #32 - 7 new voices; +1 game

This batch is now ready! Aside from the private requests, the winning voices are:

- Cyberpunk: Rogue
- Fallout New Vegas: FemaleOld03
- Dragon Age: Flemeth
- Dragon Age: Greagoir
- Dragon Age: Russel Dorsey

From requests, we also have:

- Cyberpunk: River (aka Sobchak)
- GTA Vice City: Tommy

This introduces app support for GTA Vice City, so remember to also add the asset files (to the ./resources/app/assets folder) if installing this voice!

I also r...

Batch #31 - 5.5 new voices

The next lot of voices is here (with some v2 re-training of older voices on the way). The winning voices this time are:

- Fallout 4: Travis
- Fallout 3: Moira
- Fallout New Vegas: Ulysses
- Dragon Age: Ser Gilmore
- Fallout 76: Mort

Now, for the Travis voice, if you've played Fallout 4 you might know, there are two stages of the character development, one where he's really shy, and one where he's really confident. And this is reflected in the voice, quite distinctly. S...

Interim batch, 6 voices; +1 game; Hardware upgrade

I've been going through some requests (of which one was private), as well as some v1 re-training to v2 models. The voices:

- Fallout 76: GenericSettler03
- (v2 re-trained) Skyrim: FemaleCommander
- (v2 re-trained) Skyrim: MaleNord
- (v2 re-trained) Fallout 4: MrGutsy
- (v2 re-trained) Cyberpunk: Johnny
- Star Wars Jedi Fallen Order: Saw Gerrera

This brings support to a new game: Star Wars Jedi Fallen Order!
Install the new asset files (to the assets folder) if do...

Technical post #? - New AI dataset pre-processing tools

I spent some time recently, putting together and finalising a series of new tools for data preparation/pre-processing. These are tools that I've already finished integrating into xVATrainer (which is how I use them now!), so I thought I'd quickly cover them here.

Some of these heavily build on top of the previous technical post, so check that out if you haven't already.

Speaker clustering

As alluded to in that previous post, I've now finalized a speaker clustering script, ...

Batch #30 - 10 voices; Fallout 4 : Alternative Player Voices; +1 new game

The voices from the first v2 poll is here! This covers the top voted voices from Poll #30, one of the requested voices, and some re-trained v1 voices. The list:

- Skyrim: Brynjolf
- Skyrim: Hadvar
- Fallout 3: Harkness
- Baldur's Gate: Narrator (BG3)
- Mass Effect: Kelly
- (requested) Fallout 76: MaleGenericSettler02

Re-trained from v1 to v2:
- Fallout 4: Supermutant
- Fallout 4: MaleGhoul
- Fallout 4: FemaleGhoul
- Mass Effect: FemaleShepard

Thi...

xVASynth v2.0.0!

After several months of development, v2.0 of xVASynth is now finally live! The update is now live on the Nexus (Skyrim already, the rest are also coming today, I'm working my way through the slow uploads).

Also releasing today is the initial release of the xVADict community project, for the Elder Scrolls words. This is the ARPAbet dictionary, with Elder Scrolls words labelled with the correct ARPAbet pronunciation by myself, and others in the community (let me know if you'd like to take...

More v2.0 models; Public v2.0 release date; xVATrainer

The training scripts have been whirring away, and I've actually gotten them to be faster and more high quality still, with some further work. 

I've been working through some base voices to use as pre-training, in future voices, as well as some voices for use in the v2.0 showcase video (which is now finished). The v2.0 voices are as follows:

- Skyrim: FemaleEvenToned
- Skyrim: MaleEvenToned
- Oblivion: FemaleAltmerBosmerDunmer, and Morrowind: FemaleDunmer
- Fallout 3:...

Progress update; Another v2.0 voice

Just wanted to make a quick post to update people with what's going on at the moment.

I've been hard at work optimizing the training scripts, and I think I've now got it as good as it's going to get for now. It's definitely faster, now, which is great, but still slower than the v1.0 training script, simply due to the fact that the model is bigger (more components to train), and that there is an alphabet about twice as big to learn (normal + the ARPAbet symbols).

I've gone through ...

Batch #29 - 13 (5+8 bonus) new voices; Meta-poll #1

The next round of voices is here, with the winning 5 voices from the poll, as well as a bunch from requests. The poll voices are:

- Mass Effect: Jack
- Fallout New Vegas: FemaleOld01
- Fallout New Vegas: FemaleOld02
- Fallout New Vegas: MaleAdult09
- Fallout 3: ElderLyons

And the voices from requests are:

- Mass Effect: Morinth
- Mass Effect: Legion
- Mass Effect: The Illusive Man
- Mass Effect: Jacob
- Borderlands: Troy
- Portal: Wheatley
- Por...

v2.0 pre-release; Success with v2 FastPitch models

Here we go - v2.0 is here! And after a long time trying, so is the new training script for the v2.0 models, using the new and improved FastPitch architecture!

v2.0 is a huge update, with lots of things I've been wanting to add for quite some time, so I'm super excited to get this out.

I'll say a bit about the main features (and a new showcase video will be out when the app is out of the beta pre-release), but let's get the changelog out of the way:

Changelog:

2021-09-30 11:01:37 +0000 UTC View Post

Interim batch (6 voices); 2 new games; v2 release date; xVASynth + Skyrim Real Time Player TTS mod

Got a few things, for this post. Let's start with some new voices, to come from requests:

- Borderlands: Tyreen
- Mass Effect: Samara
- Persona: Ren
- Persona: Morgana
- Persona: Kasumi
- Portal: GLaDOS

These now introduce two new game series: Persona, and Portal! Please remember to install the asset files for these, if you're going to install the voices! 

-- 

I'll post the next poll this week, and go back to the normal cycle (and there are a c...

Batch #28 - 15 (6+9 bonus) new voices; +1 game; 400+ voices! ; short break to prepare release

The next batch is here! A fairly big one, as there was a spike in requests (one of which private). The voices from the poll are:

- Fallout 3: MaleAdult06
- Fallout 3: Butch
- Cyberpunk: Rhino
- Cyberpunk: Kerry
- Borderlands: Claptrap
- Borderlands: Jack

The voices from the public requests are:

- Fallout3/4/New Vegas/76: Narrator (aka Newscaster for F4)
- Final Fantasy: Aerith
- Final Fantasy: Cloud
- Hyperdimension: Cave
- Hyperdimension: Compa 2021-09-10 16:19:49 +0000 UTC View Post

Batch #27 - 10 (7+3 bonus) new voices; more v2.0 progress

A few more voices are now finished, from requests and the previous poll:

- Skyrim: Adril
- Skyrim: Galmar
- Fallout New Vegas: FemaleAdult01Default
- Fallout New Vegas: MaleAdult08
- Fallout New Vegas: Veronica
- Fallout New Vegas: Mr. House
- Cyberpunk: Sebastian
- Dragon Age: Sten
- Dragon Age: Wynne

[EDIT] Also, I forgot: - Team Fortress 2: Sniper

Since the poll post, the update I was waiting for, from NVIDIA has finally come out. I've already adde...

Batch #26 - 7 (6+1 bonus) new voices; +1 game

The votes are in, and the voices are done! The winning poll choices are:

- Skyrim: Dexion
- Fallout New Vegas: Dean
- Fallout New Vegas: MaleOld01
- Fallout 3: FemaleAdult01Default
- Borderlands: Lilith
- Borderlands: Brick

I also did a voice I myself personally wanted, for Humankind, the new 4x game, just because I tried it out, and I really enjoyed it (I recommend it, if you like 4x games), and the game is rumoured to have upcoming good mod support. I got the Narr...

Poll #26 - Pick your top 3 choices

We're nearing the end of the Skyrim voice list, with most/all of the remaining ones being voices from previous polls which did not get picked. 

If you've missed it, I've also updated some of the existing Skyrim voices to have Tacotron2 pre-training and HiFi models:

- FemaleDarkElfCommoner
- MaleSoldier
- FemaleOldKindly
- FemaleOldGrumpy
- FemaleShrill

Batch #25 - 16 (6 + 10 bonus) voices ready; 3 new games

[EDIT 23/10/2021]: The team fortress 2 voices are really not that good at the moment, but I think I can greatly improve them with the v2.0 models. I will make another pass on them soon.

Another big round, having finished off more voices from a spike in requests, and the poll. This time covering an additional 3 newly supported games. The voices from the polls are:

- Skyrim: Storn
- Witcher: Lambert
- Cyberpunk: Elizabeth
- Fallout 3: Tenpenny
- Fallout New Vegas: Female...

Overwatch extravaganza! 30 new voices interim batch

Got a big batch of requested voices, covering all the voices from the Overwatch game (apart from bastion, which is just robot beeps). The Overwatch voices were for some reason really quick to train (maybe shorter lines?), so I got through them comparatively quite quickly.

So the list is:

- Ana
- Ashe
- Baptiste
- Brigitte
- D.va
- Doomfist
- Echo
- Genji
- Hanzo
- Junkrat
- Lucio
- McCree
- Mei (already in the previous post, but I included it he...

Batch #24 - 16 (7+9 bonus) voices ready

A big one this time. I had a spike in requests, so I've been quite busy! 

As I do more and more voices, I keep finding myself learning new things about the games I thought I knew well. I never knew that for GTA:SA, the main character (CJ) actually has a different voice (acted differently, same voice actor), for when you play him as overweight (having of course played CJ as maximum ripped all the time, ofc). So there's a bonus voice in there.

From the poll, the winning voices ...

Changes to the Patreon structure

Hi everyone, I just wanted to make a quick announcement post regarding some changes to the Patreon structure.

The next big update for the app will be a big one. In fact, I re-wrote the majority of the app, and there are enough big new features that I think it's time we move up to v2.0!

I didn't want to reveal anything too early, but one of the new things I wanted to add to the app is nexusmods.com API integration, to allow people to log in and view from within xVA...

Batch #23 - 9 (7+2 bonus) voices ready; Over 300 voices!

More voices dropping:

- Skyrim: Modyn
- Skyrim: Elenwen
- Skyrim: FemaleVampire
- Skyrim: Fura
- Fallout New Vegas: MaleAdult06
- Fallout New Vegas: MaleAdult07
- Fallout New Vegas: FemaleGenericGhoul

There are also two extra voices included, from requests:

- Skyrim: Astrid
- Skyrim: Valerica

These two voices replace the old Astrid+Valerica model from v1.0 days, where I trained the two voices together (same VA). 

Also, I forg...

Batch #22 - 7 new voices; Brief break, to focus on s2s

The latest poll is now ready. This includes the following voices:

- Skyrim: Delvin
- Skyrim: MaleVampire
- Skyrim: Nazir
- Skyrim: Odahviing
- Fallout 3: FemaleOld01
- Cyberpunk: Misty
- Cyberpunk: Placide

The next poll will follow very soon. I do however have some time off from work this coming week, so I will aim to spend it all doing research and development for the Speech-to-Speech model design. That means my GPU will be taken up by this, and I can't train vo...

(very) Technical post - Representation learning of speakers

Ok, this post will go a little deeper than the others. I've been working on something else, in the background, called representation learning. This actually very closely aligns with what I actually do day-to-day as part of my PhD, so it's been quite a bit of fun.

What representation learning is

Representation learning is a whole massive field of deep learning. It encompasses the work we do to make neural networks condense information about something in a very small, compressed s...