StreamFX: The struggles with the Spout/Sink system

It's been a hot minute since the last major update to this, so I'll explain what's currently problematic. I've implemented and scrapped several iterations of the feature already, due to simply not working out as expected. And there's a few reasons for that:

1. libOBS works in creation order, not dependency order

Possibly the most difficult thing to solve right now is the rendering/update order of filters. libOBS updates things in creation order, which usually does not matter. But the new system allows you to inject a Spout output back into the Filter chain of any other source. We now have a problem, as when we cross-reference a scene, we'd ideally need the other scene to be rendered first.

This however simply can't be guaranteed without wasting lots of performance like Source Mirror does. Source Mirror forces the referenced Source to update, ignoring the order set by libOBS entirely. We can't do this in the middle of a filter chain, so right now my solution is to always force a "frame" of delay.

Edit: For Sources in the same scene, the order is the scene item order - usually.

2. Asynchronous and Synchronous Sources behave differently

While synchronous Sources tick and render at the configured frame rate, asynchronous Sources aren't restricted by that. As long as they eventually push data out to libOBS, libOBS will be happy to silently adjust the internal buffering delay up to a few seconds. If you've ever had a stream with delayed audio but perfect video that wouldn't go away until you restart OBS, this is why.

The problem with this is that our filter chains usually expect data to be consistently coming in. I've currently solved this by disallowing re-injecting asynchronous data back into the filter chain, as I have no actual solution for this problem at the current time. libOBS simply is not designed for the node-graph style of effects that StreamFX enables you to do. I'd have to reimplement half of what makes libOBS a Broadcasting software to get this feature to work in the intended way.

I might actually do this in the future, but right now this is one limitation that the new system has.

3. How can I find Sources/Filters/Transitions/... with Spouts?

What should've been the easiest, ended up being very difficult. I've planned the feature to allow you to get the state of any effect without requiring extra Spout filters to work. That means that if you add a Blur Filter to a Source, that Filter will automatically publish the input and output states to any Sink that is willing to listen. Since libOBS is not designed for this at all, I have to get a bit creative.

At the moment, I'm using a central singleton registry object that maps UUIDv7 strings back to their obs_source_t objects. This will obviously not work out for Sources or Filters with multiple outputs, so I'm still working on a different solution that supports that. It's likely that I won't be referencing the obs_source_t object at all, and instead opt for sub-objects that have common functionality for this type of work.

4. Optional: Optimizing StreamFX -> StreamFX filter chains

As an added bonus to the whole thing, with Spout/Sink I can finally optimized filter chains consisting of only StreamFX filters. At the moment I have to capture the "target" state in a render target, apply filters to it, then render that when the next filter calls video_render on the current filter. This wastes a ton of GPU time for absolutely no reason, and is one of the problems that I brought up 7 years ago to libOBS maintainers - it was shot down within a single hour twice.

I plan to detect when the previous filter supports the Spout/Sink system, and then automatically "listen" to the output of the Spout it provides. This will allow me to skip an entire rendering step, saving both VRAM and GPU time.

5. Optional: Public API for Spout/Sink

While StreamFX is currently paid, I plan to make a public facing API for this feature that any plugin can use to optimize further. Since it is obvious that libOBS maintainers aren't interested in this optimization, this is the best I can do. It's not the primary focus of the work, but I'll try to get it in anyway.

With all that said...

I hope you understand a little bit on how difficult this feature actually is to implement. I hadn't expected half the difficulties I ran into when I originally planned this to be part of v0.11, but now that I actually work on it, it feels like slamming my head face first into a meat grinder and turning it on. Working outside the box has its ups and downs, most of the downs come from having to come up with solutions to problems you wouldn't have without the box itself.

Anyway, I'll go back to programming now. Until next time.