Update 51: Upgrades

Hello everyone,

Going to be a talky, tech focused update as I talk about all the computer stuff I've been doing.

Not many pictures either due to some hardware issues.

You've been warned.

Along my journey of evolving as a 3d artist there is one aspect you must give careful consideration: The hardware you run it all on.

I've been fortunate, I've been successful, and the majority of the money I earn from this I tend to reinvest in hardware that will make my creations easier, and remove limitations from me. I want my imagination to be the limit, not my computers.

So enter some upgrade time, along with some fairly major system changes.

Now, this is one of those jobs I've been putting off for ages, not least because animation is such a time consuming process that takes up much of the time I spend 'at work'

But also because performing it usually means taking a lot of systems down for a while, it's not really practical to do that when you're mid way through a project, or rendering it.

That has been painfully demonstrated for me over the last month when I posed to myself 'Oh I'll just switch them all to Linux, it'll be fine!'

Knowing I had some time to dedicate to it that's what I did.

First, let me explain why all of this was even necessary:

When it comes to render time, I'm usually a bit stressed, I've spent longer on the project than I'd like to have (though that seems always to be the case!) and I just want it -done-

And render time is kind of a hard limit, you can't really make it faster than it is, but sometimes you can, you can experiment and try new methods, sample counts, a lot of stuff contributes to the per-frame render time.

And when you're rendering 19,000 of them, every second counts!

How I normally set up my machines for rendering is a very brute force approach, I'd literally remote desktop into each of them (running windows) and use a CLI rendering tool called B-renderon to queue up the files.

This would give me an output (after some fiddling around arranging windows just so) a bit like this:

Now as cool to watch as this is (and it is cool, in a nerdy way) It's not very efficient, there's a lot of overhead here, and there's also a few key details we can figure out, mainly that the majority of the machines are hauling ass, but one is taking almost 10 seconds longer per frame than the rest.

Now that might not sound like much, but, 19,000 frames remember.

Now of course this is 4 machines doing it so we have to assume that machine isn't responsible or have a hand in -all- of them

But we can say it probably did at least a quarter of them.

So we take that 10 seconds, multiply it by the 4500 ish frames it would have had a hand in, we get 45,000 seconds, 750 minutes, 12.5 hours!

That's half of an entire day lost to an old CPU.

That needed upgrading!

But this wasn't all, for some reason I decided it was a good time to learn linux, and install that on the machines as well, Figuring it would help speed things up.

And it did! A little, not -as much- as I'd hoped (and I found out if I had rendered in the latest version of blender at the time instead of the version I built the scene in I would have saved that 10 seconds on that but there we are)

But that lead me down another rabbit hole of considering running linux on my main machine, I did, briefly, but since that was a brand new one (with a brand spanking new 5090 ready to render my frames -even faster-) I ran into driver issues when it came to using houdini, so I'm back on windows.

Not wanting to stop my journey of efficiency there, I decided to see what I could do about handling these rendering jobs.

This is what it looks like now:

At a glance updates, detailed information available when required, full control over the nodes, there's a lot more functionality that I've added since (and continue to add) but the gist of it's functionality is there.

And this is how I queue up jobs:

It's actually turned into a pretty robust rendering management system which I think is better than a lot of the other options available, everything else I found just wouldn't meet my needs in some way.

But what is it I'm rendering there..... I hear nobody ask..... as I write this alone.

There's a couple foibles about completed frames that have been bugging me for a while.

Now as I write this my NAS is unfortunately.... kind of dead (more upgrades, long story, still in the middle of it, kind of stressed about it but it'll be fine.... probably) So I can't supply the screengrabs I took to illustrate the problem right now (I may update this post in the future to I include them: watch this space.

It's part of the reason this update is a little later than usual, I'd hoped to have things up and running by now to show you all, but this is why you don't do these kinds of things during project time!

To try and paint a word picture, I would end up with frames that were scuffed, and that I'd have to manually go through each frame, find the offending frame, and replace it with a slightly less optimal, but still better looking frame than I had before.

I'd traced it back to the motion blur I was adding in post spazzing out due to blender outputting the odd very bad frame of vector information.

Now my rendering process is, again, quite antiquated by this point, it's the 'it works' solution I found after messing around and trying a bunch of different solutions.

But this was not the first time I'd encountered this issue, and it's not the first time I've tried to fix it either.

I've experimented with motion blur added in post (doesn't respect sub frame times, causes massive artifacts on camera changes)

I've experimented with various implementations of them, but I'd never been able to solve it properly, whilst getting the effect that was, in my mind, not a compromise in some way.

Well this time, I believe I solved it for good! It took a bit of doing, but I figured it out, so that was another box checked off!

<Insert relevant image that I don't currently have access to here, but it's fine, the OS isn't working but all my data is still on my drives its fine, it'll be fine.>

The question I was trying to answer was: what denoising solution was the most temporally stable.

If you don't know what that means, Temporal is basically = over time.

So you might have seen this (perhaps by other artists, or maybe you weren't looking at the kinds of details we end up focusing on)

But the denoising sometimes creates artifacts that make the scene look a bit... wobbly... for lack of a better description, it looks pretty bad when it all gets put together, individual frames won't show an issue but you line them all up next to each other and things start looking just a bit crap.

Figuring this out is not easy, the simplest solution is to simply crank the samples and let the scene render at however long it takes to resolve the noise properly.

But... 19,000 frames remember, we don't have a month just for the render job alone.

So you use denoising, and there are about 11 different ways to accomplish this.

You can:

-Use an inbuilt denoiser (there are 2 of them)

-Use the inbuilt temporal denoiser function, 2 of them too

-Denoise the frames using a compositor node based setup at rendertime.

-Denoise the frames using a different compositor node based setup (I have 4 to test)

-Denoise the frames post render, using either a compositing program (nuke) or in a NLE (like resolve)

There's a lot more sub-categories within those and a lot of different ways to do it all, but my plan is to basically test all of them, there's about 11 different iterations of tests I've done.

<I would show you all these had I taken a screenshot of this and placed it somewhere other than my nas (it was supposed to be a quick upgrade!)>

So, from the animation I've re-rendered the section that was producing all the squiffy frames (which I believe I've solved now, think it was how I was exporting it from houdini)

And I intend to comb through each and every one of those to figure out what the best, most effective rendering solution is!

So... that's all..

It really doesn't feel like a months worth of work, and full disclosure, it wasn't, I took some time off to do some home improvement that again, in the washes of the crazy hours of animation I put in, get left to the side until I have the time again.

But coding and development and... well... anything worth doing takes it's time apparently.

On the outset of it, I've the rest of this week to do my analysis and figure out what denoising solution is going to provide me the best results with the least headaches, then I'm attending Annecy Animation festival to get the ol' creative juices flowing again, so that I might be best placed to work on the next scene.

And.... that's important too, because there's something else I realised after that last episode.

I could have done better.

And that's not to say I wasn't perfectly happy with the standard I delivered, and delighted with the results overall, but there was a couple sticking points that, had I spent a bit more time on, I know I could have improved.

I think it's maybe why I was a little sensitive after finishing it, though to be fair:

Giving a 10 bullet point list of things that you think could have been done differently, however, is definitely not the way to give 'feedback'.

And that's really what matters here, the focus this past month has been about getting all the things that cost me time and effort and removing them (having to comb through 18000 frames to find fucked ones and replace them takes some time, believe me)

And with a nicely optimised (and most importantly, cool looking) rendering setup it's even less for me to worry about and take my attention away from the things that matter.

And that is creating the best damn content I can, within reasonable limits of course :)

Massive thank you's to all my supporters, all of this wouldn't be possible without ya'll.

Caio for now!