Patreon Peeks 49: A Pikmin Cave MIDI Visualizer

Hello everyone! Today I wanted to illustrate how I created a custom MIDI visualizer for the What I Love video about Pikmin 2's cave music. I could have used an existing MIDI visualizer to create something to look at while I explained how the randomly ordered gestures work, but I wanted to specifically illustrate a fantastic detail that my friend Waffles had painstakingly put into each MIDI recreation they had made of Pikmin 2 music:

This is a screenshot of one instrument in Waffles's "Soil 1" recreation. Notice the line of MIDI notes at the bottom? The colors represent different velocities, but these notes don't actually play; instead, Waffles used them as a visual guide for which gesture is currently playing! I thought that was really cool, and I knew I could convey that information while still cleaning up the visuals into something simpler and clearer than Logic Pro's UI. So, I exported each MIDI track that I needed, making sure any concurrent tracks were assigned to different MIDI channels, and I opened Blender.

There's a large third-party plug-in for Blender called Animation Nodes, which adds a library of programming nodes that can be plugged in to drive a Blender scene's animations. Blender's native Geometry Nodes have been coming close to offering all the features of Animation Nodes, but there's one key advantage still exclusive to Animation Nodes: taking MIDI information as an input.

I built a node system firstly to take in the MIDI from the cave music and populate it in a Blender scene as an object. In the following screenshot I attempt to comment the code:

It's complicated already and it's only half done! This at least got a MIDI track to appear in Blender, as infinitely thin line meshes (I'll get to that in a moment):

Next, I needed to get this system to recognize the special notes at the bottom that indicated gestures. For this, I needed a whole extra set of Animation Nodes that could define text to be displayed on a text label object. This is where the long brown connector in the above node image is headed:

Now I had a system entirely in Blender that can literally read Waffles's gesture indicators! The next step: make it pretty to look at...

You may have noticed an orphaned "Edge to Tube" node in the first node structure I posted. That was a remnant of figuring out how to make these notes look any more interesting than flat lines or rectangles. I really wanted them to have round caps, so they could somewhat resemble the bubbly UI for which Pikmin 2 is iconic. So on top of this Animation Nodes system, I ended up creating a Geometry Nodes system. It makes cylindrical tubes out of each line segment, and adds a slightly squished sphere on each end:

Trying to merge or weld these objects to get rid of their seams proved way too taxing on this already complex system, so instead, I had the Geometry Nodes also create a sphere, store its normal data, and delete the sphere. Then, the shader for these objects used that normal data to make each note look like a solid object, even though it wasn't! This glossy bubble shader also emits a global color from within, or a Z-location-based color if it happens to be where the special gesture indicators are on the X axis.

So, at last, I had a cool-looking custom MIDI player that could read which gesture was active. But only for one MIDI track... and duplicating all of this to read more than one MIDI track would definitely put a lot of pressure on Blender, which could cause crashes during rendering. So instead, I rendered recordings of each individual MIDI track I needed, to be assembled later.

Well, normally that would be the point where you put footage together in a compositing application, and I could probably have done that in DaVinci Resolve's Fusion system... but at this point I could already easily visualize the workflow in Blender, it's just that familiar to me. So, that whole scene was built in 3D in Blender as well!!

I imagine this is going to be a common sight for future What I Love videos now that I've been constructing the visuals in free software, and I'm absolutely excited for it! But I wanted to showcase—and also myself just step back and fully grasp—how complex the underlying systems can get. I'm happy to answer any questions about it in the comment section below, or in the patron Discord server!