SFMLab Migrations 2020

Hi everyone,

It's been a busy month, but we're almost out of the woods.  You may have noticed some major changes and severe service disruption  over the past few weeks. To those who asked, my answers have been terse  and vague. The banner on the homepage explained little, and naturally  that had people worried. As such, I owe everyone an explanation  regarding the state of SFMLab over the past month, and how we will  continue going forward.

The Secret History of SFMLab

But first, I'll provide some historical context. I started  developing SFMLab six years ago, in January of 2014. The idea was to  create a site where people could safely upload SFM models rather than  relying on sites like Mega, MediaFire and FileSmelt. Links to those  sites tended to expire after some time, which lead to models getting  lost after periods of inactivity. Building a site that mirrors the Steam  Workshop but allows for adult content as well? A no-brainer.

But the project also served another purpose. At the time, my  Python and devops skills were limited. I wanted to use the site as an  opportunity to improve my familiarity with Python. This is where the  issues start to come in. Building the first version of the site was easy  enough. But when you're learning about effectively using certain  frameworks by only reading the documentation and using your own  intuition, mistakes start to slip through. Especially early on in 2014, I  made a bunch of poor technical design decisions that came back to haunt  me every time I would like to make a change to the site. This made it  difficult to keep development of the site moving forward.

In 2016, my interests had shifted from SFM towards Blender,  and I noticed I wasn't the only one in the community frustrated by SFM's  limitations and Valve's apparent disinterest in maintaining project.  What kept people from switching to Blender was the hike in learning  curve (especially before version 2.8x) and a lack of models to start  animating with. This is where SmutBase comes into the picture. SmutBase  was intended to be a separate site where people could easily share NSFW  models for more professional animation tools.

I started considering the technical aspects of the site. 2  additional years of Python and server management experience under my  belt, having learned many lessons from SFMLab's development, I decided  to design SmutBase with a more flexible and future-proof backend design.  This paid off. SmutBa.se launched in 2016 and was much easier to  maintain than SFMLab.

Cut to Winter of 2019. SmutBase now has a sister site,  Open3DLab. It runs on the same backend as SmutBase and was built in a  single weekend, without causing additional strain for the system. There  are many features and improvements to be made to SmutBase still, but  because the sites are running the same code and share the same database,  any features added to one of them, will immediately be active for the  other as well.

SFMLab on the other hand was still a problem child. It's  still got a large active user base, new content gets uploaded very  frequently. Making changes requires a lot of effort on my part, and the  site still grows every day. And there are many features that I would  like SFMLab to have as well. SmutBase has this great "chunked" upload  form that allows us to display a progress bar for the upload.  Implementing that for SFMLab was impossible in its current state. Design  decisions I made 6 years ago were still biting me in the ass.

Shoveling myself out of the shit.

In order to be able to maintain SFMLab in the future, it  would need to be moved to the SmutBase codebase. I knew this from the  day SmutBase launched. But it would be a massive operation, involve  service disruption, headaches and frustrations. I would also need to  take time off from work in order to make it happen in a reasonable  timespan. Remember: maintaining these sites is a side-project made  possible by the limited ad revenue and the people on Patreon.  Moving to a new place, switching jobs and the subsequent burn-out were  contributing factors in the first two years after launching SmutBase.  After working through that, I still didn't know where to start. While I  did feel urgency, I kept delaying and putting it off.

2019 was a turning point for me. There are many reasons, some  best discussed elsewhere, but the main takeaway is that I felt a need  to move forward. The later half of 2019 was dedicated to slowly create  feature parity between SmutBase and SFMLab, and introduce some changes  to the SmutBase code to allow the same code to host multiple sites.  Open3DLab served as a trial run. Launch a new site on the same codebase  and look for potential issues. This plan went off almost without a  hitch. The next step would prove the most difficult. Moving SFMLab's  content to the new site.

Moving site content is never a 1 to 1 operation. Because the  backends have different data structures, there are difficulties. Each  entry needs to be processed individually, and transformed to the new  data model. In addition, any assets like downloads and preview pictures  need to be imported and moved as well. I set up a task queue to  facilitate this. Several worker processes would run through items,  images, uploads and comments in parallel. Refining this process would  take some time, and running the final import and sync also took longer  than expected. Also, once the import was started, no additional changes  to the database should be made, as new changes would not be synced over  and be lost. This is why the site was set to read-only mode.

I had taken a week off from work to build, deploy and  finalize this migration process. In hindsight, too ambitious. Once the  first import had started, edge cases manifested, most of them related to  matching fields from one database to the other, as well as the task  queue getting overloaded. This left the database in a sort of half  imported/half failed state, which I spent the tail end of the first week  cleaning up. As a result, the migration window had to be extended for  some time, since I had to get back to work on my day job the next week,  spending only the weekends on the migrations. That's why the read-only  mode on SFMLab.com persisted for so long.

Doppelgänger

You may have noticed that I've left out the migration user  accounts from the previous explanation. Because SFMLab and SmutBase were  launched as separate sites, they have their own user database.  Convenient at the time, but a hurdle to overcome at this point. One way  to import the data would be to just copy over all the usernames, email  addresses and passwords, right? Wrong! While the databases are separate  there's a lot of overlap in the users. Although there are many slight  differences. Some people have used different usernames, email addresses  and passwords for signing up. Some usernames have been taken by  completely different people. These types of collissions are difficult to  reconcile.

That's why I have opted for a different approach. All uploads  are assigned to a system user. People who have a SmutBase account can  log in and people who only had a SFMLab account can sign up for a new  "SmutBase" account. The next step is to enter the old SFMLab credentials  to connect their SFMLab account to their SmutBase account. All uploads  will then be re-assigned to their SmutBase account. A newsletter will be  sent out to all SFMLab users, notifying them of the change and provide  instructions for what to do next.

I had my doubts about the approach I described above. It is  disruptive and inconvenient for users. However, the main benefit would  be speed of the entire process, and it would allow for shutting off the  old site as soon as the migration is completed. Any resolution of  accounts, uploads and comments could happen in the months after, at the  convenience of the user. It's a trade-off that I still feel confident  about. My only worry is that people will not understand and blindly ask  questions on Discord without reading any of the messages I've sprinkled  all over the site.

Unfinished business

You may have noticed that some things did not get transfered over, like Patreon perks and Verified Uploader status.  With the new codebase, a new system becomes accessible to us. This  allows you to connect your account directly to the site. Patreon perks  will then be granted automatically, rather than manually. The Patreon  tiers will be revised, with new perks to be introduced soon. Please stay  tuned for the reveal of these perks, and send in your suggestions.

In addition, we will be making an official application form  for Verified Uploaders. The old system was loosely defined, poorly  documented and involved some effort to enable for each individual user.  With the new system, we'll have an checklist that must follow before  Verified status is granted automatically. The benefit of this is that it  will also finally introduce Verified users to SmutBase and Open3DLab at  the same time. Having the sites on the same codebase means that these  features can be rolled out for all sites at the same time, while only  requiring development time once. While this is being built, please take  some time to get used to the new site layout, upload forms and  moderation process. More information is coming soon.

Lastly, you might have noticed that the TOR/Onion links do  not work as expected. There's some adjustments to be made to the  configuration on the TOR endpoints, as well as the site configuration.  These changes are mainly related to the lack of HTTP to HTTPS redirects  when visiting the site on the clearnet. The current TOR setup relies on  direct access to the unencrypted HTTP version, since accessing via the  HTTPS version would result in a certificate mismatch. A solution for  this is in the works.

Moving forward

For the next three months, I'll be watching the amount of  users that have successfully connected their account, and monitoring  whether or not the re-assigning of projects goes off without a hitch. In  the mean time, there will be more technical improvements, as well as  new features introduced to the site in the coming months. With the sites  now fully integrated, maintenance tasks will be easier to perform, and  new updates can be added with less effort. To me, this is incredibly  exciting, and I hope you will stick with us as we move to the future.

I'll also be looking to hear from everyone if they find  missing content or data errors that might have been introduced as a  result of the migration. If you have additional questions, feel free to  ask in our Discord chat, or send inquiries directly to admin@sfmlab.com.

Kind regards,

Ganonmaster
SFMLab, Open3DLab & SmutBase Administrator