Development roundup: March 2021

Hi everyone,

First of all, welcome to all the new Patrons who joined over the last few weeks. And returning patrons, thanks for your continued support.

If you haven't already, connect your Patreon account to your site and Discord accounts to claim your benefits.

First update of the year

2021 feels like it's only just started, but March is already here. While my main job has demanded most of my attention for the past two months by far, I feel very much obliged to give you guys an update on what's happening with the site this year so far.

The focus for this year is primarily on improving search, and letting people find content. While I am currently experimenting with a tagging system behind the scenes, it is not yet ready for production. To further complicate things, due to external factors acting upon the sites, the most important changes in these early 2021 days are related to reliability and security.

Earlier this year, I noticed an increase in load on the account registration and activation system. The number of bot registrations soared explosively, causing increased load on the servers, sometimes to the point of causing outages. To further complicate things, this caused issues for the email infrastructure too. The volume of bot registration emails sent out worsened the email reputation of the site, meaning there's a higher likelihood of activation mails for legitimate signups getting sent to spam folders. And since the site uses a third party, transactional email service, this incurred a small fee for each email sent, accumulating in an additional $100 on top of the regular invoice.

As a solution to this, I implemented rate-limiting for the registration and activation forms. And to go one step further, I decided it would probably be smart to add tighter rate limiting for the login pages. In order to have insights into rate limiting going forward, I created dashboards for tracking this information.

In addition, a security feature that was already enabled for admins has now been enabled for other users as well. It is also mandatory for Verified uploaders and Patrons, since there are rewards and privileges associated with these accounts.

Two factor Authentication (2FA) is an additional login step that requires the use of a dedicated 2FA device. This feature is intended to secure your account in case your password is compromised. You setup a device that generates a code every time you want to log in. Without the device, it is not possible to log in.

While I understand this makes logging in a bit more complicated, I think it is very important for people to get used to. Passwords alone are not enough, and 2FA helps provide an additional layer of security that's much more difficult to forge for online criminals. The current implementation of 2FA on the site is functional, but lacks the finishing touches. Initial feedback has shown that people don't understand they'll need to backup their security tokens in the app and keep the app installed if they want to be able to log in. In addition, it is not very clear that people should save their backup tokens in case they lose the app. This will probably need further clarification in a future update. I'm also looking for other options for people to receive their backup tokens.

Next, we had issues with a lot of spam in the comments. As some of you might have seen, some folks took issue a with a specific model and decided to use a script to post a page full of racist remarks in the comment. While I've always been partial to a "hands-off" approach to the comments section, (I'm not particularly in favor of having comments at all), I've taken action. I've implemented rate-limiting for the comment forms, which should prevent most common scripts. In addition, I've put in place a word filter that flags your comment for moderation if your post contains specific bad words. Unfortunately this is not applied retro-actively, so old posts containing these words might still be found on the site. And no, I will not reveal the banned word list. You can probably guess what's on there. Just be nice to people.

In more uplifting news, and actually in line with making content easier to find, I added a "featured" uploads bar at the top of the index page. It was primarily a reaction to high quality models being pushed down the page by a barrage of samey sound packs. But I think it's a useful feature to highlight some content on the site. The content that gets featured reflects some of the best works on the site made by veterans, as well as promising uploads from new folks in the community.

Hope this motivates you all to put some more effort into your preview pictures too, because even if your model is great, your chances of getting featured drop significantly if your preview pictures are too simplistic or bad.

Unfortunately, it's not all roses. We ran into storage issues with the EU servers, which caused issues with uploading to the site, as well as syncing of downloads to other servers. I was able to fix this after a couple of days, but it required me to write a smarter cleanup script for old, deleted uploads. We've got some headroom left for the next few months, so we're in no immediate trouble. But if upload frequency remains as it is, we might fill up the server before the end of the year. Your support is invaluable to keeping the servers running. I'm still looking at further improvements, like compressing files that aren't already zipped, and perhaps even putting files in cold-storage (making them only available on the Patron server, freeing space on the public ones) if they haven't seen downloads in a long time.

Lastly, you may have had issues accessing the site over the past few days. I'm still not entirely sure of the cause. As far as I can tell there was a sudden uptick in traffic to the site. This caused a problematic increase memory usage, resulting in the site becoming unavailable until the server was manually restarted, or the server automatically manages to kill the processes, reclaim its memory and restart the site software. I've run into similar issues in the past, but this time, it was difficult to isolate the problem. To further complicate things, a busy work week made it difficult for me to act quickly when the site went down. And then spending all my time after work failing to come up with a permanent fix took its toll. I rebooted the servers knowing full well they would probably crash again in a couple of hours, and just went to bed to refresh. Considering we had a pretty good uptime record this year so far, I'm kind of frustrated that it got to this bad over the past few days. At this time, memory usage seems stable, and the site seems to be running normally. I'm still gathering statistics, but it seems like the worst is past.

In closing, thanks again for your support. I hope to provide you with an update on the tagging system very soon. Unless fate decides to throw me another curveball.