Update 52: Problems

Hello Everyone! 

Another techy one for you all because... well that's what i've been doing this last month!

Last month I managed to resolve the majority of the last remaining software issues that were preventing me from being able to transfer fully to using Linux as my main operating system.

This took a LOT of trial and error, I’m very happy to say now though everything ‘works’ (refinement is possible, of course)  but it’s enough for me to get started again, I still have some issues to  iron out and some parts are still not working -perfectly- but those are things I’m going to continue to chip at when I get free periods between animating.

This meant I’ve turned my attention to addressing the last batch of issues, join me as I describe them, and how I plan to solve every one of them over the next 2 weeks:

Issue 1: The NAS

Those of you who read my last update will know, I had a critical hardware failure last month which resulted in loosing access to the data on my NAS.

For those not informed : A NAS is basically a hard drive connected to your local network

My projects are so large and complex and have so many dependencies now, it’s essential to have these files stored centrally and redundantly.

For an understanding of why this is important. This is how much disk space the last project took up:

Yes thats terrabytes, with a T

It’s a crucial part of my infrastructure, all of my machines rely on data from it, and do not function well independently of it. 

And then it crashed! when I was, of all things, trying to perform maintenance and software upgrades on it!

The good news was: it just needed a new motherboard, and I didn’t lose any data, the outage, however brief, opened my eyes to a stark reality: My entire business, ability to work and create was reliant on this machine staying up, had this crash happened mid production, it would have been disastrous. 

This is not a situation I like and It needed to change.

This meant a new chassis. Which meant a server grade chassis, complete with redundant power supplies, redundant boot disks, easy access to  hard drives for replacements as needed, I needed a grown up server grade chassis.

The trouble is with these types of chassis, they’re normally destined for data centres, where how loud they are is not a primary design consideration.

Small, fast, loud fans.

Now my server/render stack sits right next to where I work (like literally less than an arms length away), which brings me to problem number 2:

Issue 2: The Noise

I’ve been fortunate that I can afford the kind of infrastructure most 3D artists dream of, but putting it to proper use has been a problem for a while now.

Hitting render means 3-4 days of 4-5 machines running their fans on full tilt, in the space I have to exist, work and live in.

The server rack with all the machines that render my work, this picture is taken from my desk

I’ve built all these machines with a ‘noise first’ approach, so I’ve invested in the nicest, quietest fans and coolers I can find, to guarantee sound and effective long term cooling, without being too offensive to the ears.

A render node

But there are limits, and the amount of hard compute necessary to bring these animations to life nowadays is pretty huge, when I hit render, I -hear- it.

And… I don’t really want to, it’s not easy to feel creative in a space that’s constantly under that level of noise pollution, it’s not just annoying, it actually starts getting in the way of my work, I don’t want to have to worry about these things, I just want to be able to create.

Render time used to mean a time I could step away from the computer and spend some time sketching ideas, writing scripts, thinking about where to take the series in the future, but, well, you try focusing when you have 40-50db of noise pollution in your office space.

So my next upgrade was a fancy, acoustically treated server rack.

The possibilities this introduces are pretty incredible for me, as well as silencing all my rendering and networking hardware, It allows me to place my workstation right next to the NAS inside the cabinet itself, which enables me to add an extremely fast network link between the two machines, so when it comes to pulling the completed frames from the NAS, to scrub footage and perform post production compositing work, (which is some 800GB of information, perhaps more depending on planned upgrades)

I can now pull that information 25 times faster than I could before.

Whereas before I could expect the final composition and output of the file to take a day (allowing for iterations and 2-3 hours per try) Now, that process could be completed in as little as 10 minutes, just enough time to make a proper cup of tea.

Its an upgrade I’ve been drooling over for years. I have had to put it off because they're very expensive, and very large.

It does however, present another problem:

Problem 3: Power 

At this stage my render stack pulls about 2.5 KW of energy at full tilt, putting my workstation into that stack and upgrading the 4090’s to 5090’s (a progressive upgrade process I’m continuing to do) means this is going to go up, somewhere in the region of 3.5-4KW based on my rough calculations.

This was with 4 nodes running, I plan to add another 1, plus my workstation into the stack, ergo 3-3.5K watts, maybe more when the 5090's go in.

That’s a lot of juice, which means the single extension chord that powers the rack that I have to run from a socket in the next room (because I have an old house that was never built with the amount of sockets modern living demands) wasn’t going to cut it anymore.

So I’m having proper sockets installed in my office, right where they’re needed, fused and on their own circuit (that I can charge to the company, you try boiling a kettle for 4 days straight and see your power bill at the end)  

This also offers me an enormous peace of mind that I’m not going to have a fuse go, or a power chord overheat, or a short circuit blow up the kit (which has been close to happening before during rendertime)

I remember being woken one night during a render during the most recent production to my UPS complaining that the power draw was exceeding it’s limits, I had to quickly get out of bed at 4am and re-jig the power connections to bypass the UPS (I’ve bought a bigger capacity UPS since)

Then, the socket the AC was plugged into was overheating, due to (as I’ve found out since an electrician has been, an improperly wired socket) which caused the AC to cut out!

It’s again… another thing that makes render time particularly stressful, one that I soon won’t need to worry about anymore.

There is, however, a problem with running 4.5KW of hardware in a house in the UK:

Problem 4: The Heat

I don’t typically like working in what passes for summer here in the UK, for one. It is a nice time to take time off. Touch grass, get inspired, nurture and feed my creativity.

For the other, it’s so damn hot! In the UK our homes are typically built to retain heat, and very rarely is AC installed.

I have a portable unit to keep on top of the particularly hot times, but it’s very loud and distracting, and terribly inefficient, setting it up means running a long hose, obstructing my movement through the space, and hearing a constant, 60db ish compressor from the other side of the room, all day, all night.

In addition, it dumps a lot of heat (not all, i’m venting up the chimley mostly) into my bedroom.

Slipping into the mocap suit (which is basically a huge onesie with sensors. Batteries and hot things strapped to it) is not a desirable thought when things are this sticky.

Coupled with the fact that. I am owned by a cat (who loves to chase lights) meaning me prancing around in a onesie covered in shiny lights is kind of like painting ‘attack me’ on my back.

However, due to the way my rooms are setup. The only place I can restrict her to that has access to all her needs, is the place where I must vent the hot air from the AC 

So locking the cat out to mocap means no AC. 

Now this… I have managed before, but let’s just say I’m in a hurry to get it done. 

And, that’s not how I want to approach my art, I don’t want to rush through it because I’m overheating, the restriction should be my imagination, not the environment.

So I’m getting proper. Legit. Split air conditioning installed right next to the server rack, a significant investment for sure, but one that’s going to solve this problem for good.

So, what am I going to do with all this power? 

Mainly, this is all about optimising what I have, but it is presenting an interesting opportunity, with this much rendering horsepower, I could look to up the render resolution somewhat.

I’ve run some experiments, and I’d like some pixel peepers to examine some frames and for you all to tell me in the comments what you think.

What follows is the same frame rendered in a different resolution:

1080p

1440p

4K

Increasing the resolution significantly increases the time taken to render a frame, for reference, the time to render each frame in my case is:

1080 Frame: 1m09s

1440 Frame: 2m06s

4K Frame: 4m22s

So with some napkin math (these figures are not accurate based on current hardware, but for our purposes it’s close enough) total render times for a project the length of AimeeXJuno 

1080p: 3 Days

1440p: 5.5 Days

4K: 11.5 Days

This is assuming 5x 5090’s working continuously, currently I have 2x 5090’s and 4x 4090’s

So personally, I think 1440p might be the sweet spot here, over a week just for a render, whilst doable, seems like we’re hitting a point of diminishing returns.

I don’t know how many of my supporters put this up on a big screen or really get all that close to the action, I don’t know how much benefit that might have, these are stills, video is different of course.

This would effectively mean the new ‘higher’ resolution format would replace the file in the ‘master’ tier,  and the ‘base’ tier will see a version that’s been downscaled to 1080p.

Due to the way rendering works (because it’s definitely not worth rendering the whole thing twice just for the different resolution) it effectively means a bump in quality for everyone, but it does lend some quantifiable increase in quality to the higher tier, more so than providing the master files does, something I was never really keen on, it doesn’t feel like much of a value proposition. 

I plan to experiment with this a little when I get to rendering my next project, but in the mean time, please do let me know what you think in the comments, is this worth it? Is 1080p enough? Is that time better spent elsewhere?

I welcome any and all thoughts on it.

Amongst the upgrades

I’ve also bought a 3D printer. I’m planning to experiment with producing figurines, scenes and stills from my animations (original characters only for copyright reasons)

I plan to experiment with a few different scenes, see what comes together well and is practically printable, they might be cool to sell, or perhaps I’ll provide the models for printing to a certain tier.

Maybe in a couple months time I’ll have some fun things to show on that front, for now it’s still in the box waiting for me to put it together (building work needs finishing first)

Overall

I can’t really describe how much of a  game changer all these upgrades are for me. From where I was literally 2 months ago, the improvements in my workflow and the possibilities this opens up are so exciting!

Linux alone was a game changer, when I wrangled it to my will and figured it out, I've never felt so empowered and in control of my hardware.

I have managed to solve every single practical, software and mental limitation standing in the way of my productions, I expect my work to enjoy a significant uptick in quality as a result.

Not having to worry about rendering also means more chance for experiments, shorter segments, off cuff little situations that various characters find themselves in, all the little projects I’ve been wanting to set off but couldn’t due to all the problems I listed today.

I can now do so with no concerns about overheating or power shorts, I can record mocap whenever I feel inspired or need to.

Whilst all this is happening, it’s an opportunity to finish decorating my office, a project I’ve been working on since the beginning of the year, but kept being put off. 

Largely because, doing all of this work means I cannot work! It means tearing down all of my infrastructure, going without desktop computing (I still have an iPad and a phone. I’m not a savage) and in some cases, internet (shock I know) whilst I rewire, re-organise and resolve these last few problems. 

But, the reason these things have been such a problem is because I’ve always been looking for the most immediate solution, and not routing out the root cause. I was using a half measure, and I should have been using a full one.

Overall, I find myself stood on the edge of an extremely exciting precipice, It has been a LOT of work,  and not the kind of work where I can show you all pretty pictures and previews and such.

I am extremely excited to  work with the most advanced hardware I’ve had access to, that has never been as well optimised, in an environment that has never been more comfortable and inspiring.

My time in Annecy was also extremely inspiring, and has enabled me to solidify the plan for my entire Juno Arc, I’ve never felt as sure and confident about where the series is headed!

We’re going to mix in some new elements of the genre (chastity anyone?), there’s also plenty of new toys and environments in the works, I’ve scattered a few sketches I did when on holiday of some of the parts we’re going to be producing, work will continue on that as I cast myself briefly into the stone age. 

I write this as possibly the last interaction I will have with my PC for a while, as I prepare to clear the room ready for all the building works that will be happening over the coming weeks.

Next time I see you. I should have some results to actually show, I’ve not had a chance to review my denoising experiments yet, so I will have previews of all that put together by then, this is again, something I’d very much like the community’s opinion on.

Please do take the time to cast your vote for your opinion on the render resolution, I've also attached the files directly for download, as I don't know how embedding changes the resolution/quality.

Ciao for now!