Recent Hardware Upgrades

Hi Patrons!

We’ve been experiencing higher than normal downtime recently, and I wanted to connect with you all to discuss the cause of that, and also the corrective action that we took to prevent this issue going forward.

BF4DB is hosted on a physical server that we own and colocate in a local data center in the United States. We have owned this server for a couple of years now, since just after we launched in May of 2017. This was not a new machine, but most of the internal components were purchased new. The chassis and motherboard were the only original components. This is an enterprise-grade server from IBM so it was expected to serve us well into the future.

To save costs, we initially installed two SSD’s in RAID 1, and one large HDD as a backup and storage drive. These worked well with no issues. As we grew, our infrastructure plan expanded as well to cover additional VM’s and we eventually moved a few systems to the backup drive. Among those were the redis server, our backup DB, and a few other projects.

Over the last few months, we’ve had a number of issues with that backup HDD crashing and not coming back up without a full server reboot. As time went on, this failure occurred more and more often until it’s final failure a couple of weeks ago. At significant expense that we covered out of pocket, a technician had to go physically reseat the drive in an attempt to bring it back up. After a few hours, skulls was able to move what was needed off the dying drive to bring the website back up.

The decision was made to procure new drives to replace the failed one (which failed again afterwards, permanently), but this time we were going to get two drives to replace the one, and put them in RAID as well. Since we would be conducting maintenance anyways, we decided to connect our servers remote management port so the server itself could be managed remotely again. This is something we’d used in the past, but wasn’t reconfigured after our last move.

During our first scheduled outage, the management port was configured which required the server as a whole to come offline. The new drives were scheduled to be installed at that time as well, but had not yet arrived. Skulls performed some maintenance on the virtual machines before bringing everything back up. 

Last week, the new drives arrived and were installed by the data center technician at our direction. The installation was successful and reallocating servers to the other drives will commence over the next few weeks. 

At this time, the server is running smoothly with no known issues and is expected to continue for another few years before another hardware upgrade. The drives we installed were not cheap, and completely depleted our bank account, and then some that we covered additionally out of pocket. But the majority of it was covered, thanks to the generosity of our Patrons.

Without you, this wouldn’t have been possible. All of our work is done voluntarily and none of our staff are paid. So THANK YOU to those who continue to be patrons, and support our goals and mission as an anti-cheat service for Battlefield 4.

Starting this month, we will make monthly financial reports available via Patreon for transparency. The first report will be published 3/1/20. If you have any questions, please feel free to reach out. Patreon is about to become a little more interesting for you guys. More details to follow.

- Sovereign_Skies