I am sorry. Search is hard.

Hey everyone, over the past few weeks, I've been working on the site search for all three websites. This has obviously caused some frustration for a lot of you, as the new search hasn't been as specific as it was before. This is because search is actually quite difficult, the larger the dataset becomes. Site search has been a major performance bottleneck over the past few years, and complex queries tended to result in higher database load and longer load times. Into this context, I started implementing changes to the search.

In practical terms, we've gone from a straight-up database search (where every search query by the user runs a search query on a specific database table) to a dedicated, document-based search backend. This type of search backend is faster and more scalable. For little over a month, I've been working to implement this search backend and get it deployed to the website. This is the new search that has been active on the site for a couple of weeks now. Unfortunately, it's not been without its drawbacks, and in the process of updating some search capabilities were disabled and rebuilt from scratch.

Today, however, I'm thrilled to report I updated the site search with the ability to select tags to narrow down your search query. This is similar to how it used to be on the old search backend. However, thanks to the new search backend, we get some cool benefits. Mainly in the department of speed and server load.

On the old system, search queries would take an average of 1.2 seconds to load. While that sounds like it's pretty quick, for a single JSON request it's actually too long by modern standards. And the more parameters you added to your search query, the more it would stress the database. The new search is much, much faster. The average search query is reduced from 1.2 seconds to 150 milliseconds.

Now, this was already the case when I deployed the new search backend a couple of weeks ago. Unfortunately, it also required overhauling the search UI to accommodate the new backend. I wasn't able to complete the full revamped UI due to personal circumstances, and that meant the search was deployed in a partial state where search queries were very inexact, and largely unhelpful in narrowing down specific results. As of today, I've added the tags back in, which should improve the specificity of your search queries. If you have feedback on its functionality or the user experience, please let me know, as I plan to make further adjustments towards the end of the year.

One drawback that this new system has is that it depends heavily on the quality of the content that is being indexed. Meaning, that the ability to find the content you're looking for depends on the quality of the upload title, description, and most importantly, the tags being applied to uploads. The search backend can only index text content. And if your upload doesn't have many unique words that people might search for, it will hurt visibility.

Currently, many uploads on the sites have been tagged extremely poorly. And the systems currently in place to shape and moderate tag usage are lacking. This is an area where more development will take place. In the next few months. I am aware that people have a lot of opinions about how tags should work, and how they should be applied to uploads. I also can't tag uploads by myself, so I am thinking of a mechanism to utilize the wisdom of crowds for this purpose. If you want to give your opinion on how to best achieve this, please leave it in the Discord. In the meantime, it is up to uploaders to properly follow the tagging guidelines. If you want your uploads to be found, you need to tag your stuff appropriately.

My sincere apologies for the limited search functionality over the last couple of weeks, it was arguably too rough of a product to launch at that early stage. Search is one of the more difficult problems in technology today. It requires a lot of tinkering, and I'm learning as I go. I hope you'll stick with me through the process. For now, thank you for using the site.