Update 54: Catharsis

Hello everyone!

Let's get straight to business, I promised you all some server and computer porn, followed by actual previews, and I managed to get both done! Despite -literally so many things- going wrong.1

Let's get into it.

This is where I left off, A quick tidy up, and after a lot of grunting and struggling, I put the shell of the new rack together:

You VS the guy she tells you not to worry about

First, the fancy matching patch panel (that costs way too much for what it is) but hey, ubiquiti, the apple of the networking world:

Despite me mentioning it -at every opportunity- the rack manufacturers omitted to add brushing to the small square cutout, luckily, I had some brushing strips hanging around (that I mistakenly ordered too many of) that fit just fine with a bit of epoxy:

Next up, I ran all the cables, they had these toolless keystone things that matched the aesthetic, a bit fiddly, and I managed to mess one up (which led to a mini crisis later down the line) but a few hours later:

1-8 were for all the KVM's/Network interfaces in the rack (for remote access) 9-16 were for 10gb connections to each node + workstation (the NAS gets an aggregated link) and the final 8 go out of the rack to the AP's / Cameras / Other hard networked devices

Enter some seriously satisfying cable management:

Cable management that I definitely didn't get wrong at this stage, nope, everything here is just fine

Just as a reminder of where I came from:

I did give it a good try, honestly, I had these cable clips going down the sides, but, and this is the single biggest mistake I've ever made when it comes to making custom patch cables:

NEVER BUY THE CHEAP CABLE

I had this solid core cable, which was the only type I could find that came in lengths that were reasonably close to what I needed.

You can buy stranded (which is always the type to buy for this application) only in boxes of like, 300 metres, whilst not especially expensive per metre, the fact that you have to buy 300 metres of it in one go means its quite the investment.

Using solid core within an environment that required flexibility was a huge mistake, if you make too tight of a turn, it -will- break , and I'm pretty sure that was responsible for a good chunk of all the errors I've had to fight over the years, network failures and the like, every time I made a patch cable up I'd have to re-terminate at least one end at least once before I got a good test result.

It's fine if you're embedding it in a wall or something, but given I had to buy a 300m roll of the stranded stuff, it's what I used everywhere in this reno.

Moving on, I installed the main rack switch (a new one to replace the old one, which I definitely didn't buy because it lights up when it gets a link, no siree)

It's because it has POE delivery across the entire switch, which was going to solve some messy cable management I had previously in the old rack.

And my 'old' router goes in, which has been a proper trooper for me over the years.

All lovely and matching, please ignore the fact that I had to use rack studs for the top two units and screws for the switch, it came with rails, using these screws was the only compatible solution

Server rails are another one of those things that, as well as being seemingly unnecessarily expensive (they are usually at least £100 for a pair!), is also some of the most proprietary stuff you're likely to come across, as I'm about to find out.

I decided to test fit the new NAS chassis first, I'd measured this out a bunch of times when planning this build, it was about 10mm longer than my previous NAS chassis (which only just fit in my previous rack, and whereas this new rack was taller and wider, it was the same depth as the old one)

And this one came with rails (which it should for £1400!) nevertheless, a moment of panic when I fit it into the rack to find:

The rails are too long for the rack, I did consider cutting them down, but there were mechanics embedded in them, so that sadly was not an option.

At this point I reached out to both the retailer whom I purchased the chassis from, and the manufacturer of the chassis to ask if they had a shorter, compatible rail kit.

But, I thought, hey, I've got a tonne of spare rails from various server chassis lying around, the ones for the old NAS fitted OK, maybe I'll use them?

The server literally came off the rails as I pulled it, twisting and warping both the frame and the rails

OK maybe not those rails then... I also had another 3 sets of rails lying around, none of which fitted, so OK no worries, lets leave that job for now and return to terminating all those cable runs for the rack.

Whilst testing terminations my totally solid plan of : make each run slightly longer than the last (except pairs, each node gets 2 runs, one for it's KVM, one for it's actual network connection) and thread them through the cable management bar as I placed them, as it turns out, I got a lot of them very wrong.

As it happens the -first- run I tried failed validation, which lead to a mini-panic attack (luckily, thanks to my personal development, sans the extreme rough edges it had before) as I spiralled, thinking I'd somehow messed up every single one of them (I didn't, it just happened to be the first one I tried!)

So I was reminded to never do your cable management -before- you validate the runs:

Each of these jobs at this point is taking roughly a day to complete, it's the 'fun' stuff, but it is very time consuming, nevertheless by the end of the day I had all the runs inside the cabinet terminated and labelled:

White labels for data runs, yellow for KVM connections

Whilst the rack was empty, I started my plan for sound deadening, starting with this large acoustic panel sat right behind the rack, which I'd hoped would cut down on the bounces behind the cabinet:

This was all I could do in this area for now, I had to order some 'shelves' for the NAS at this point, since it had been a couple days since I'd contacted the chassis/supplier about the shorter rails, and I couldn't really keep waiting around for it anymore, so I placed that order.

In addition, since I believed I had learned my lesson from last time (IE don't order all the fun stuff in advance for it to sit in the hallway and take up space long before you can actually use it) I went to order he second UPS I would need for all the planned power delivery on the rack.

Only to find out, nobody had any in stock! I found one eventually, but then later got a call from them to say that, this was a mistake and it might be 2-3 weeks before they got one in. In the end I had to order an upgraded model which they did have in stock, features I don't want or need, but times a wastin', I needed to get back up and running.

The run on effect of this was I couldn't continue building machines and populating the rack, I wanted to populate from the bottom up, and since the UPS's were central to power delivery, it would limit by ability to test machines and solve problems before I slotted every machine into the rack

Buying a non matching model was obviously, not a viable solution.

At this point the room was still in a very much, not inhabitable by humans state:

Which was really starting to get to me at this point, I'm not going to lie, my bedroom was still holding the bulk of the large items that I took out to do this reno, and living in it was a depressing lifestyle, ok for a few weeks, but we're up to 5 at this point, which as it turns out is apparently my own personal limit of patience.

So moving on, I decided to get the switch that was going to split networking to my AV stack and move out the hubs and stuff (that were previously stacked in my old rack)

Messy, and it's going to get a whole lot worse, but I did what I could at this stage.

Then I put up the cat gym around the room for noodles acrobatics:

This started clearing up some space in my bedroom, and I was finally able to move some of the bulky stuff into the room to make it at least semi-inhabitable again:

At this point I just had to slot in the empty chassis into the rack, sans any actual computers in them, I'd ripped out the old LED strip from the old rack and put it in there, but it had gotten damaged during that process, so part of it had gone green:

This was obviously not acceptable, and functioning LED's were clearly, absolutely essential to the smooth running of this machine, so I had to order some new ones.

I managed to, around this time, get all the AV stuff in, and unpack all of my new speakers and such:

Which coming from the rats next of unlabelled speaker connectors was seriously cathartic

Moving on, I was delighted to find out the socket I'd positioned fit juuuust about right into the notch in the new side table I'd ordered (another bulky item that had been sat in the hallway for the past month)

It could have been a few mm higher, but I had to guess how high everything might be, so I'm quite pleased my guesswork got me at least kind of close.

By this point the UPS had arrived (finally) so I had to do one more bit of installation before proceeding, I'm mentioning prices here and there but the following section goes into just how insane pricing gets on this level of hardware

So these UPS's (that cost £1200 each!) don't actually come with network access, you have to buy a card separately, that costs another £110 on top.

The reasoning behind it is simple, power loss can cause failures, these aren't meant to keep the farm running during a render, just give the devices time to shut down, not loose data, and hopefully, protect from any power surges/ripples.

With that in place, I move to the NAS, that beautiful flawless NAS that I'd spent a few weeks repairing a month or so back.

Little did I know that was just the beginning of the issues it was going to throw at me.

The old chassis for the NAS, looks good right? so why did I have to spend £1400 on a new one?

A few reasons for it, mainly it was about accessibility and expandability, the idea was I should very rarely have to remove the NAS from the rack to work on it, and especially, replace failed drives, if you see the two silver cards in this chassis, that holds my NVME storage, which is what I use to store all the project files, rendered frames and so on.

As it stood, if one of those drives failed (which does happen, thankfully, I've not had one fail just yet) but if it does, it means the whole NAS has to be shut down, I need to find the correct card, find the drive, replace it, and put it all back together again, I needed a more viable long term solution.

The new chassis has 8 slots for NVME drives that use the U.2 connector, this is the connector used by the super high end enterprise disks.

I wasn't quite ready to upgrade to all of those, not least because buying the drives alone (8 of them) would have cost something in the region of £16,000

Since my application of drive space relies on read rather than consistent writes, these consumer M.2's are more than sufficient for the job, and I literally just bought 4 more of them the last time I repaired this thing, so I didn't want to throw away that investment just yet.

So I found these m.2 to U.2 carrier card things that enabled me to get that functionality, without having to sell my kidney for the fancy enterprise drives:

That doesn't mean, however, that my woes are over, in order to facilitate this, you need HBA's, to cut a long explanation short, it's another bunch of money you need to spend.

The HBA's cost around £400 each, and when I first put them all in, I discovered that, due to the delightful way that server standards are communicated (in the way that there appears to be an unpublished bible somewhere that, everyone is supposed to just.... know) I ended up with the wrong cables.

I was able to find some in a supplier here in the UK, but since I wasn't getting them from China as surplus (which is where I get a lot of this equipment from, since it's a LOT cheaper) I ended up paying a premium for them.

How much of a premium you may ask? Well it cost me £360 for 4 of them...

But I got it all put together, installed the new upgraded CPU (which is about to bite me in the ass in the biggest possible way) and got it all put together, anxiously eyeing the big chonky 80mm fans I mentioned last month that cool this thing.

These fancy motherboards have something called IPMI, which is basically BIOS access over the network, so I plugged it in and decided to use that method to upgrade the BIOS, since I knew it wouldn't support the new CPU.

During the -brief- time I heard the fans running and oh my god they were loud

Like, being on an aeroplane loud, it was totally unbearable. Not for a machine that would run 24/7

So I looked at the stock fans and wouldn't you know it they of course have a proprietary connector on them:

Unlike my previous case which would just accept any 3-4 pin standard fan header:

I considered for a time seeing if I could do a bit of custom case modding and simply use the fan bracket from the old chassis in the new one, but there were one too many fitment issues, and enough had gone wrong so far, I didn't want to push my luck.

So in the end, in the jankiest possible solution, I dropped 4x Noctua 80mm fans I had lying around into the holes:

This made the noise perfectly reasonable, but caused some further issues that are going to come up later down the line....

So, IPMI> BIOS update, no need to put old chip in right? No need to plug in a GPU and monitor and go through all that rigmarole right? these things are MADE to be updated over the network right?

Long story short, it went really wrong, managed to corrupt the BIOS completely, and it wouldn't boot or even POST.

I even attempted pulling the BIOS chip and flashing it using a chip reprogrammer (which, if what I've done before is outside of my comfort zone, this was outside of my comfort city)

You're supposed to orient the chip correctly in the programmer, but of course, how to do that exists in that unwritten manual everyone besides me has read, there's a marking on the chip itself, but not the thing it plugs into.

And of course if you get it wrong, you have a chance to fry the chip, so not anxiety inducing at all.

And the first time I put it in I definitely got it the wrong way around, which definitely didn't spike my anxiety at all.

But I eventually got a successful flash and read, so, enthused I installed it back into the motherboard aaaaand it didn't work.

Then I rolled back to a previous version, that didn't work either.

Tried updating the IPMI firmware, nogo.

Nothing worked, so I had to deal with another few days of it cluttering up the space my desk was supposed to go in, at which time I built the final node for the rack:

This is definitely the 'most noctua node' out of the lot, I'm pretty sensitive to noise and want this place to be peaceful whenever possible.

Back to the NAS I wrote to ASROCK to ask them for a BIOS file to see if maybe the image I'd gotten off their website wasn't appropriate for my use case (which I wasn't sure it was) I gave it a day for their response (and to try and relax a little, now I had, you know, a seat to sit on) but no dice, the NAS is central to the entire network, the other machines and as a subsequent result: rendering, did not function without it, it's hard not to feel a bit on edge when it's not working.

In the end, as a last ditch effort, I used the old motherboard I had previously replaced (a part I'd attributed the previous failure to) and surprise of surprise, it worked!

With that done, I moved to my workstation, which was pretty much a straightforward transplant into a new chassis, nothing crazy there, at this point I could put my desk back together and actually get back to desktop computing:

Fitting the batons to support the desk

I'd bought and planned to use some threaded inserts to mount the desk, previously it was just held in place with some wood screws, but my latest desk surface is a nice piece of solid walnut, and drilling into and removing screws from them just seemed like a bit heretical, Unfortunately however, when I lined it up and made the marks in the wood, I found it was too close to the edge of the wood to facilitate it, so it had to be wood screws in the end after all.

But this enabled me to get my desktop back up and running at least.

With sweet, sweet pointer precision returned to me, I was -technically- able to return to work at this point, but there was still a bunch of cleaning and tidying to do.

Plus, I've not had anything resembling actual rest in over a month at this point, so I decided to (and this was maybe... thursday?) leave it till the start of the following week.

More computer building was ahead of me, during all this time, I've been living basically out of my housemates room, And he'd been playing X-com 2 again on his switch, and the experience was terrible.

He's been really great, tolerating all my mess and clutter around the house and providing me with a chill place to rest in, so I wanted to do something nice for him, so I built him a steam machine:

He's still not actually used it, since he's finishing off his save, but I will drag him kicking and screaming into the world of PC based gaming.

So now power distribution was in the rack, it was ready to start plugging everything in, except there was another problem, the clearance wasn't enough to mount the strip flat and not have the cables crushed by the side panel:

I eventually found some brackets that enabled me to twist the PDU 90 degrees so they didn't interfere with it:

Then made some custom length C13-C14 cables to cut down on all the excess:

Spent some time coming up with a nice arrangement for the acoustic panels that I was going to mount on the side panel:

Got all the AV stuff back into the room, some cool ass lighting, and returned it to functionality:

Did my best to cable manage everything nicely behind it:

Got the place cleaned up and my paintings back up on the wall.

Assembled my new 3D printer, which enabled me to correct this travesty of a KVM mounting solution:

Surprisingly it only took a few prototypes to get something functional:

Which I then refined into something a bit smoother:

Sure beats the cable tie solution!

And everything was now tidy and ready for use!

I ran into -so many more- issues that I had to resolve, software wise, that took up the majority of the week I'd planned to return to animation (my fancy NIC's were way too hot for my desire for silence, so they've had to go sadly)

I also had to bring the workstation back to Windows, sadly I hadn't accounted for Rokoko Studio in my software requirements.

Sorting all this and accommodating for the necessary rest before returning to animation, it left me with only a couple days to work on my next piece before writing this update.

Nevertheless, I got straight into my next project, I had set this out before I went on hiatus, It was going to be a simple set of sleepy orgasms from Juno.

The full piece is going to come in somewhere around 10 minutes or so, but to facilitate the return of some rapid content, I'm going to split this up into the first segment, around 3 minutes or so, and re-release it in full once I've gone through the entire scene.

After all this time I had some anxiety about returning to animation, It's normal I think to install some self doubt after you've been away for so long... Luckily, I needn't have worried, I slipped right back into it like a pair of old comfortable shoes, I was surprised at the quality I was able to achieve even in such a short period:

The facial animation is just the raw mocap at the moment, but given i've been working with it for a while now, I'm definitely getting to grips with getting more and more usable animation in a first pass!

I also explored a different workflow, and did the entire performance capture in one shot, which was a surprisingly effective workflow, nice when I can put the AC on and spend the time getting the takes right!

Some more polish and details to add obviously but making rapid progress, and given that the scene is mature at this point, all the problems are solved already, that 3 minute segment should be done this month, I hope.

So, there we are, I am back and doing what I love once more, and am very excited to bring you all some new content!

Amongst this, I am going to explore some old files, and see if I can't do a little remaster/supercut of some highlights from previous animations, just a little package to thank you all for your patience during this time, I know the wait must have been frustrating, I will try my best to ensure it is worth it!

More news as it comes, ciao for niaou!