160TB Server with Linus Tech Tips! - Smarter Every Day 222

As others have said there is plenty of ways to do it. I believe it is important to have. I don't know all of your financial obligations, but since I believe it is so important, I am increasing my contribution. All I ask is to please consider it to some offsite storage. I am available if you need to talk through stuff (though admittedly there are more technical people here that understand the system better than me).

Andrew Krob

Destin, you'll probably get different opinions, but here are a couple of options for data at your scale: 1. Backup to M-Disc Blu-Ray or something similar. This would allow you to offload 25-50 GB per disc. M-Disc are rated for archival purposes with a shelf life if 50+ years. 2. Use tapes. LTO-7 & 8 give you multiple TB (depending on how compressible the data is) per tape. Offload as many tapes as you need for archival footage and just store those in a cool, dry place. (IMO, this is probably the best option from a cost and usability perspective) 3. Duplicate hardware offsite - probably not what you'd need but at a previous job we essentially did this. Grabbed a duplicate of our hardware, stuck it in a datacenter, and then just mirrored the data over to it. You'd need fast internet but it is effective and basically just requires that you scale both at the same time.

Yep. Very much so. Even in my job I have to be very mindful of security. It's really unfortunate that it has to be that way.

Andrew Krob

You kinda have to have the same basic approach as dealing with security. What is your threat model? What are you worried about? Are you worried about bit-rot? Fire? Theft? Are you worried about a hacker encrypting your data? Your threat model will have a lot to say about how you approach those things. I've been watching Linus for about 6 months and he's pragmatic and IMO he has a good trade-off between solutions that are good and also cost-effective. He did a video about the Jellyfish a couple of days ago that I think underlines what he does and why he does it. Something like Synology or even Drobo makes high-redundancy really accessible and the Stornado + the Enterprise drives are a great option. They will protected from what Destin's most likely issues are: bit-rot and misplaced drives.

Doh! Sorry about that. Here's a decent pop-press intro to bit rot: https://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/ The TL;DR: is that after roughly 10^14 bytes read can cause an uncorrectable read error because the underlying bits of a hard drive flip for one reason or another. That's only once every ~12TB. The actual name of the stat is called the "UBER", or Uncorrectable Bit Error Rate (a rather difficult term to search for in this day and age). The other interesting metric is the MTBF, or Mean Time Before Failure. But, this is a probability game and there needs to be some active correction to prevent entropy from causing data loss. I've managed petabytes of storage for $JOB and see this in data centers as well as in residential situations. Uncorrectable drive errors are a foe and constant threat. This whole area of computers is a descent into madness because at this scale of density, physics is not your friend unless you start to embrace statistics. With 160TB of data, these probable failure rates are something you will almost certainly have to contend with. https://wccftech.com/backblaze-releases-hard-drive-findings-for-q1-2019-hard-drive-failure-rates-for-all-major-suppliers/ https://www.computerworld.com/article/2846009/the-5-smart-stats-that-actually-predict-hard-drive-failure.html Which is all just a long way of saying, you probably want to check to make sure that your unraid install is using ZFS under the hood. Even 45 Drives recommends the use of ZFS or some way of detecting and remediating this all-but-guaranteed effect: https://45drives.blogspot.com/2019/04/hardware-raid-vs-software-raid.html

Sean Chittenden

You're talking over my head.

Smarter Every Day

You get me.

Smarter Every Day

Yes backblaze was something I investigated but it's cost prohibitive.... and I need to have something local anyways to backup. What's your backup solution for your 18TB?

Smarter Every Day

Exactly.

Smarter Every Day

You guys together in a video? This is madness!

Invisible Jazz

I love the channel, but this video left me concerned. As someone involved in storage, for this amount of capacity, three things come to mind: 1) Unraid uses ReiserFS. How does this system handle "bit-rot"? What interval is the data being scrubbed to ensure the integrity of the bits? 2) Is there something on top of ReiserFS doing erasure-coded storage? 3) Depending on the quality of your power, I'd plug that server into a small UPS to do power line-conditioning in order to prevent brownouts or short-strokes of the HDD drive-heads. I know when I use ZFS at home on my storage box, I was running into dozens of correctable storage errors per week until I put a UPS in and the number of reported errors from a zfs scrub went to zero. Regardless, for 160TB, I'd look into something like ZFS or something erasure-coded based like minio that can do scrubbing and self-healing of corrupted blocks. Good luck!!!

Sean Chittenden

I realized your phantom generated a ton of data, but I never really thought about all of the old footage until now. By the way, as someone who abuses Excel sheets for everything I approve of your catalogue system.

Dan & Jen Baizel

3-2-1 rule! At least 3 copies of your data, on at least 2 different types of media, and at least 1 copy in a different (offsite) location. In addition to drive failures impacting your RAID array you can experience other hardware failures that can impact the integrity and availability of your data, and having it directly accessible to your computers locally leaves your data vulnerable to ransomware.

Nick K

Agreed with Andrew! Make sure you've got off-site backup, as I've always been told... and learned the hard way... raid != Backups

Ben V

Yeah... The problem with your scale is that you can't expand that storage server any more with all of those drive bays full. If you want more storage there, you just need a bigger/another server.

Rich Jeanes

I can't wait to see the next one!!!! :)

I was wondering what happend to the Apollo collab between the two of you.

Maarten Daalder

oh man, the old system would give me so much anxiety. I hope you expand so you can add the old data onto the server like the original plan.

Adam D'Alessandro

I haven't watched the video yet. I wanted to reach out and offer my opinions or knowledge for data storage. I'm a Sr. Storage Engineer with a large health Care company. ***Update*** I just watched the video, and came away with a few opinions. First, let me say it's really cool he got you hooked up. He created a nice system for you. I don't want this to make it sound like I'm trashing his thinking at all, as that's not my desire. I work with a lot of smart techie type people. However, the problem with working with techie type people is you think everyone is techie. Therefore, when someone who doesn't understand RAID's or how to manage servers comes in (admittedly I don't know unRaid, I use enterprise class storage arrays) there is a level of disconnect. That's why I have recommended a Synology for another project I'm associated with. They get support and you don't have to try it on your own. Ok, so that's not necessary the biggest thing I see here. Expansion will be necessary at some point. Yes, you can migrate from one server to another, but that will take time. Just be aware of that. The biggest thing I see here is relying on Raid as a backup. Yes if you do a RAID 5 you can loose 1 drive, if you have a RAID 6 you could loose 2 drives. What happens when you loose 3 drives, or heaven forbid the motherboard or raid controller goes belly up? Theoretically you can get those back. Since I'm a Storage Engineer in my day job, but a photographer on the side, I've been struggling with this issue as well. I currently have a 18TB RAID in my computer for the same reason. However, I'm still researching to see if there is someway to do offsite storage for disaster recovery. I don't want a tornado, fire, etc. to come through and all my data is lost. I'm considering www.BlackBlaze.com for that solution, but have yet to try it. Please consider your data and what is replaceable vs. what is not. I hope you all the best.

Andrew Krob

I viewed the original tweet before watching the video. Not going to lie... The pictures made me gasp a bit. Glad you got your data organization handled.

Hey Destin: totally random question, but that poster on the wall behind you... is that one of those massive electromagnetic spectrum posters?

But seriously, this is nifty! Sounds like you got a great setup and it should make your life a LOT less stressful.

Dude, we have a couple of these at DCMP and they're amazing. It took a bit to tune the NICs in our editing machines to take full advantage of the10Gbps network, but the effort was really worth it.

I'm waiting for Linus to drop something...