Development roundup: Summer 2020

TLDR: I discuss the site goings on since March, and sum up the numerous features and site improvements, like the Migration, the upload process, Verified uploaders, automated upload validation and other ongoing improvements. I also talk about some of the site performance issues of the past few months.

----

Hey everyone,

I'd like to provide you with a summary of my activities for the past few months, and where we're headed from here. While I'm not usually someone to seek prolonged exposure to sunlight, the global pandemic has made this a lot easier. And as a result a lot of my time has been spent towards the development of the site.

SFMLab Migration

At the start of the pandemic, I spent several weeks working on Migrating SFMLab towards the newer codebase that's used by Open3DLab and SmutBase. While this lead to some extended unavailability, I have to say that the transition went by with relatively smoothly once the site was running on the new codebase. It has helped tremendously with the maintainability of the sites as bugfixes are deployed on all sites at once.

Plus, it's easier than ever to add new features. As mentioned in an earlier post, some of those old SFMLab features didn't make it into the new codebase yet at the time of the migration, and especially the comments section was subject to some bugs. If you've kept up with recent developments on the site, you might have noticed that some of these features are now being rolled into all the sites at once.

Upload Process & Verified Uploaders

One of the most common complaints heard both before and after the migration, was the moderation process. This was exacerbated once SFMLab's Verified Uploader functionality fell away. The process was unclear and moderation/approval speed slowed to a "whenever the moderators have time" sort of thing. Who would have thought that "a human looks at your upload" system was as slow as the humans looking at your upload! In addition, when a moderator approved your upload, it would just immediatelly be visible to the public, ruining any possibility of controlling the "release" or "launch" of your project.

After a few months of biting through this, the upload and verification process has seen major improvements. Once an uploader has created their page, there's helpful steps that clarify how the process works, and what they need to do before they can submit their files. Once they've submitted all their files, they explicitly mark the file as being ready for moderation. If there are issues found with the upload, a short description of the issue with the upload will be provided, plus a few links for finding help. Once a file has been approved by a moderator, it does not instantly become public. Instead, the uploader can publish/launch their project at their own convenience.

This is where Verified Uploaders make their return. Verified uploaders get to skip the moderation step and get to immediately publish on their own accord without having to wait for approval. At this time, Verified uploader status is granted automatically based on the number of approved uploads you have on any of the sites. As of right now, that threshold is set at 6, but this may be adjusted up or down depending on other factors that can help prevent abuse or ease the requirements for reliable creators.

Lastly, I've put a lot of work into making the approval process as quick as possible for the moderators. The most important one is that we now do some automated analysis on uploaded files. For example, we automatically check the contents of zip files. This makes it easier to check if the uploader has included the necessary files and kept to a specific file structure. There are also automatic checks for Blender files. For example, there is a check that lists the external files and textures in a .blend file, and whether or not they are packed into the file itself. If an upload is missing textures, moderation is easy, and only takes a few seconds.

At this time, these kinds of automated checks are only visible for moderators, but in the future, more of these can be improved and automatically provide feedback for the uploader. That's not to say that the process will be made fully automatic, but it makes a lot of sense to remove barriers that make approval time consuming.

Ongoing issues

As mentioned in the previous post, work was done to sync Patreon perks to the sites. If you've not done so, make sure to connect your Patreon account to the site. Be sure to let me know if you run into issues, Patreon's API is pretty unreliable. The initial integration from back in July managed to break down within a month after it launched. Credit to the fine folks at Patreon! I've deployed a new and improved coupling that will hopefully ensure that we won't be dealing with these issues going forward. Again, if you run into issues, be sure to send a message.

Lastly, the comments section. At this time, development for the comments section is still ongoing. you may have noticed some changes happening already and I hope to be able to show you more before the end of the year. As a word of comfort for some of you; I've not forgotten about the "Latest Updates and Comments" section! Many have asked, and yes, this will make its re-appearance on the homepage eventually. However, this is a feature that leans heavily on the implementation of the comments section itself, and will thus have to wait a little bit longer.

The Ugly

If you're a regular user of the site, you can not have missed the severe degradation of the site speed happening over the last summer. While all this talk of "new features" might have lead you to believe that this is just due to my bad programming and introducing feature bloat, I can assure you this is not the case. The problems are primarily related to issues with our hosting provider in the EU. While we're very happy with their service quality overall, their bulk file storage platform has been suffering due to a switch to more advanced software and hardware.

Long story short: It was time to migrate all the files from the old system to a new, faster server. This took several hours, but once completed and switched over to the new systems, the site load was instantly back to acceptable levels. This is an expensive monthly investment, but necessary to keep the site performing well.

If you had trouble accessing the site, you have my most sincere apologies. I have worked hard to solve the performance issues as they appeared, and hope to provide you with a more reliable site going forward.

Google has found Adsense policy violations

To make things worse, last month, Google decided to kneecap ad revenue for the SFMLab. Google Adsense provided good clean ad revenue for a good price on the SFW pages on SFMLab. Their system runs automatic policy violation checks, and sometimes that means they will restrict ad display on some pages. However, in July the site reached a point where the number of "severe" violations in the eyes of our advertising overlords reached a level where it was time for them to restrict ad serving on the entire site, and essentially stop generating income. I took measures in the past to ensure Google ads don't get displayed on pages with SFW content, but apparently Google marks some pages as containing sexual or offensive content without a specific reason or element of the page that is considered offensive. If you're familiar with Youtube's content policies, this feels similarly shitty. It's completely arbitrary and super annoying to deal with.

As such, I've switched all advertising on the site over to JuicyAds, the adult ad provider that was already active on NSFW pages and Smutbase. Unfortunately, that means you might get "questionable" ads in your face now. But if you're a Patreon, you get to view the site without ads anyway. I hope those who do not support the Patreon will not take offense.

Closing thoughts

I hope you found this roundup interesting and hope you will make use of the site and its new and improved features. None of this would be possible without you, the Patreon supporters, as well as the people who regularly upload content to the site. As exhausting as maintaining it can be, I take pride in hosting this site in my spare time, and I'm thankful that you are willing support it in this incredible journey. It is an important learning tool for me, and helps me sharpen my skills as a developer and sysadmin. But most importantly, it is a vital resource to many people in the online 3D community, and I hope it will be there to support them for many years to come.

If you have a question or comment, leave it under this post. Alternatively, you can let me know on Discord or Twitter. And if you can; spread the word about the site. Ask your favorite model makers to share their files through the sites, so their models don't disappear when their MEGA page gets taken down.

Thanks for your support, and stay safe.

- Ganonmaster