VNSound2.0: Spectrum setup guide

Spectrum is a new feature in VNSound v2.0. Basic demo is in the video, and this document will talk the detail of it.

中文版本说明：https://www.patreon.com/posts/vnsound2-0-pin-72146667

First in the setup, the sampling setting:

• Target audio:
  Here set how spectrum link to a sound.
  PlayingAudio - MMDD/VNSound only: default setting, it will search MMDD then VNSound, link to the first playing sound found.
  PlayingAudio - Any audio found: This option can found sound playing in studio. But can not set which one, it will link to the first one found. BGM and chara voice can be found by spectrum, but can not find mp4 video's sound.
  MMD Director: Set a index and spectrum will link to that sound. MMDD can load multi sound, the first one is 0 and 2nd one is 1 and so on.
  VNSound: Set a index or id and spectrum will link to that sound. VNSound set id for every sound, also you can set index 0 to link to the first one.
• Sample Count:
  To get spectrum data, a Fast Fourier Transformation (FFT) will be performed every frame. This option select how many sample in FFT's result. More samples get you better resolution in frequency, but slower. Note the sampling frequency is fixed by Unity.
• Sample channel:
  Which channel to sample. If you choose stereo, it will do FFT twice every frame. In most case Left only is good enough.
• Sample window:
  Windor function for FFT, it is a math problem. I just say Hanning or Blackman or BlackmanHarris looks good and familar. Rectangular and Triangle looks bad.
  

Then Spectrum band in setup:

Here you can put samples into band. For example we get 512 samples by default, that too many to process. We can put some of them together to get the level of a frequency range, called band.

Turn on "Show band frequency range" can help check the frequency of each band. The default setting is just a sample of dividing bands. You can delete or add band on your need. Frequency range of band can overlap, and no need to be continuous.

And here you can set "Action clip" link to each band. New button can help create clip from this UI, and use < > to choose an existed one.

Next Algorithm options of setup:

• Band statistics:
  Select how calculate samples in band.
  Sum: add all sample value up, default one because it is fast and looks good.
  Avg: get the mean value of samples
  RSS: Root Sum Square of all samples, someone says this is the right algorithm
  RMS: Root Mean Square of all samples.
• Audio profile:
  The output of band is a normalized value, between 0 to 1, so we need a maximum value to do the normalization. The maximum value is updated in run-time, but we need an initial max value, called audio profile. Or if we start from max value = 0, the output will jumping violently at the beginning of sound.
  The value of audio profile is depending on the sound and other setting. For example if choose Sum of samples, the output value will be bigger than Mean value, means you may need bigger profile value. You can refer to spectrum preview, it will tell you the max value collected in the sound. Then you can set the profile value to the half of actual max value. Smaller profile cause the output value moves bigger, but only in the beginning, before the collected max value reachs profile value.
• Audio normalize:
  Independent: default, every band has it's own max value to do normalize
  Unified: all bands share the same max value to do normalize
  Unified mode can show the real level of bands. But we are not doing science here, and in the independent mode bars jumping more happier.
• Audio drop down:
  By default this option is ON, and when current band value greater then output value, output value will raise immediately. And when current value lower then output value, output falling down on it's own speed.
  Drop speed is the initial speed, and Drop accelerate makes the speed increase over frames. These two parameter help you create a smooth animation. If the value drop down too slow, set a bigger value.
  And if turn off this option off, the output value is always the current value.
• Avg Amplitude:
  Enable this option you will get an extra bar for volume. That is the sum of all bands, response to the total volume of sound. You can assign a clip to volume just like other band.
• Import/Export:
  After you done the setting, you can export it to a json file. And import to another scene.

Then the spectrum preview in manager

In the vnsound manager, the spectrum setting looks like following screenshot

In first line show the sample count and band count, and which sound is linked to it. In second line are conctol buttons:

• Enabled: Enable or disable this spectrum
• Edit: go to the setup ui
• Reset: reset to initial state
• Preview: preview how the spectrum band works.
• X: delete setting

Click preview will expand the ui and show every bands.

Each band has a bar, shows the normalized output of it, and a max value collected until now. The max value can be a reference when you want to adjust audio profile.

Under the bands info, the drop speed and accelerate rate can be adjust on fly. They appear only when you enabled them in setup.

Check the More info option beside preview button can give you more detail about bands.

As you can see, frequency range and current value are added into band info. If you are using unified normalize mode, the total max value is also displayed.

Assign clip and tips:

Spectrum itself is meaningless. You need assign clip to band. 

Clip which link with band defines something changing according to sound. The length, fps, speed and loop can be ignored, because it is not actually played. Band will drive the clip with normalized output. just like you drag the navigator bar by mouse. For example, if you create a clip with 10 frame length, band will play frame #5 when output is 0.5, and #10 if output is 1. So if clip length is 100, band will play frame #50 when output 0.5 and #100 if output is 1.

Technically, everything works in clip now can work which sound. But since it will update every frame, create as few keyframes as you need!

Some reference about FFT and spectrum:

For more tech and math

https://www.nti-audio.com/en/support/know-how/fast-fourier-transform-fft

https://www.teachmeaudio.com/mixing/techniques/audio-spectrum