Server troubles (and upcomming upgrade!)
Added 2021-03-23 18:26:33 +0000 UTCHey everyone,
We're getting to the point where our current server is hitting its limit.
It's over 10 year old hardware (TorrentFreak wrote an article about it previously, give it a read: TorrentFreak: Opentrackr Coordinates Millions of Torrent Downloads on an Old Dell Server) and we're severely CPU bottlenecked. This is due to many reasons but mostly the cpu pipeline failing to advance because of slow memory and context switches.
(For reference, we get an effective 0.31 instructions per clock cycle, this is not too good)
We've previously upgraded the CPUs, but there is not much of an upgrade path left that would not put us in drawing too much power. We could change the memory to DDR3-1333 ECC but that also won't give us a significant boost. (Timings are hardly better)
Our current specs are as follows:
- Case: Dell R410
- CPU: 2x Xeon L5640 (6c/12t)
- RAM: 4x DDR3-1066 ECC 16GB
- Network: 1Gbps Integrated (Broadcom NetXtreme II type)
This might have been really pretty back in the days, but it doesn't cut it any more.
I've been playing with the idea of buying a new server for a good year and I'm planning on going forward with it in the upcoming months. This server will make use of mostly consumer-grade hardware, with the benefit that it allows us to upgrade to much better hardware for the money.
I have not finalized the specs, but my current target looks like this:
Case: 2U Enclosure
CPU: AMD Ryzen 9 Pro 3900 (12c/24t, 65W TDP)
Motherboard: Asrock Rack X470D4U
Memory: 2x DDR4-1600 (CL14-14-14-30, 1.35V) non-ECC 8GB
Network: 10Gbps Fibre (Broadcom BCM57810S based)
Our datacenter is able to provide us 10Gbps network connection free of charge, given we use a fibre card. Having a 10Gbps network card gives us more possibilities for growth and I'm confident the faster and newer network asic will allow us to off-load a lot to the card, saving CPU time.
We will need to pay an additional €12,10/month for 2U, instead of 1U. But that is acceptable.
If anyone has any recommendations they'd wish to make, feel free to comment! Any questions behind my choices are fine too, I'd happily answer them.
When I have an update on the situation I will create a new post.
Comments
Thanks for your comment Matt! We're not memory-bound (capacity wise). At peak usage we're at about 9GB of ram used, if I currently check it's at 7.3GB. I'm considering moving up to 2x16GB, which would also be our maximum for this cpu/chipset if we don't want to give up memory latency. On the upside buying more straight away saves me a future datacenter trip, so it may not even be that bad. I've also considered adding more servers to scale up that way, but in the end it still comes down to buying new hardware (and a doubled colocation bill). This also ignores how we might distribute the incomming connections, adding a load balancer would become even more expensive and complex so the likely solution would just be DNS based round robin. (Which is fine, but likely results in a non-even distribution) If we combine the costs, additional complexity and overhead I'd rather keep one server for as long as its sufficient. Although if hardware was free, I would absolutely pick the more redundant option :)
Isa S
2021-03-25 15:07:44 +0000 UTCAm I reading right that you'd drop from 64GB on the current box to 16GB? Or are you just not memory-bound enough for it to matter? The torrent.eu.org folks are apparently using opentracker's clustering capability to scale horizontally across a few servers, though I suspect they're seeing a lot less traffic. You'd also obviously end up with double the colo bill to spread the load across two servers, which isn't a great option if you're still paying out of pocket. But thanks for the update! Sounds like you've got a pretty good plan.
Matt W.
2021-03-25 01:43:57 +0000 UTC
