Development Update June 2024

Hey everyone,

Firstly, welcome to all the new Patrons who joined over the last few weeks. And returning patrons, thanks for your continued support.  If you haven't already, connect your Patreon account to your site and Discord accounts to claim your benefits.

It's been a while since the last development update. Here's what's been keeping me busy these last few weeks.

Download integrity

If you've been around the sites long enough, you may know that it uses a process to synchronize files to servers in different continents. These servers help reduce latency for users around the world. For those familiar with tech, it's like a shitty version of a CDN.

Some of you may have noticed that sometimes, the downloads are broken on some servers, while downloading from one of the other servers works fine. This has to do with the way the synchronization works. At all times, there are task workers running to handle synchronization tasks. Sometimes, when a lot of tasks are running and the system is under a lot of pressure, a worker process may get interrupted. The sync tasks are unfortunately non-atomic, meaning that if they are interrupted, they may leave the system in a partial state. And that results in a task that appears to be completed in the database, but leaves a corrupted/partial file on the destination server. There's currently no system that detects broken files automatically. Only when someone reports a broken file to the mods, can it be manually fixed.

To mitigate this, I've been working on a system that verifies file integrity using checksums. Essentially, once a file has been uploaded by a user, we calculate a checksum using a cryptographic hash function like MD5. We will then sync the file to the other servers, and recalculate the MD5 checksum for each of those as well. If the checksums match the original file, we can assume that the transfer was successful. This should make the synchronization process more robust, and prevent file corruption and wasted downloads.

I'm currently in the process of hashing every file on all Open3DLab servers. Future files will be hashed automatically after uploading as well. Once hashing is completed, we'll automatically compare hashes to see if the transfer was successful, and retry a transfer if needed.

Once a file hash has been generated, I will start displaying the MD5 checksum on the download confirmation page. You can then independently verify if the file you downloaded is correct, by simply generating a checksum of the file you downloaded, and comparing it to the one displayed on the site.

Note that checksum generation will take up some resources on our task workers, and as such, some site actions may be delayed, including the synchronization of Patreon/Subscribestar benefits. However, this should be resolved within a few days.

Other improvements

Most of my work recently has been spent on site reliability. I hope to get back to making improvements to site usability in the next few weeks. I've also been fixing a few bugs that were introduced as a result of the UUID changes to projects.

Thank you for your continued support. If you have any questions or concerns, please do not hesitate to contact us at admin@sfmlab.com, or through the relevant channels on Discord.

Sincerely,

Salaryman