booru-on-rails, Philomena and the future of Derpibooru

Rails, though not without its warts, has been a fairly pleasant framework to use for Derpibooru for a long time. However, I currently believe that for a number of reasons, Derpibooru's future does not lie with Rails.

Normalization

A few weeks ago, I followed through with a routine normalization change for tags. The change, fundamentally, was very simple: turn an array column of tag IDs into a join table between images and tags.

As usual, I pulled out all the stops. There weren't many code changes to make, so I spent my time testing and retesting the migration to actually do the work for this change.

The migration worked flawlessly and executed completely within eighty seconds.

After bringing the site back online, and waiting for our application performance monitor to catch up to the new deployment, I discovered that most image index pages on the site had roughly doubled in response time. This baffled me; why would a change that added 1-2ms to database response times almost double the render time for the application?

Unexpectedly, this performance regression came from a feature in ActiveRecord actually intended to improve performance. Before the migration, ActiveRecord loaded tags for each image in an approximately N+1 fashion, because at the time, there was no alternative. You cannot preload an association that is contained in an array column. But after the migration, I had introduced an association preload for image tags to prevent making N+1 queries.

ActiveRecord is so painfully slow at preloading associations that it is often faster to make N+1 queries than to try to preload. This is a major red flag.

Integrity

Anyone who has followed this blog for a while will know that I am nothing short of obsessed with low-level enforcement of integrity. This is extremely difficult to realize with a Rails schema, because the schema dumper does not support many features that would be able to effectively perform this enforcement (for example, check constraints).

ActiveRecord is overzealous to an absurd degree about not enforcing any constraints at a low level, because it has subscribed to the false prophet of database independence. This paradigm does not make sense for a permanent application, which will probably change databases either once or never.

The holdover from when we used MongoDB, combined with the lack of strong integrity constraints in ActiveRecord, has left our database in a sorry state. I find new integrity violations almost on a daily basis.

Performance and budget constraints

Compared to similarly-sized fandom websites, Derpibooru runs on a shoestring budget, underspending nearly every single one of them on server and bandwidth costs. This is not necessarily a good sign, because it means that even though we may be running things more cost-effectively than other sites, we don't necessarily have the headroom to survive long-term traffic peaks—which are a serious possibility, given that our traffic is increasing between 15-20% every year.

In this respect, Rails is a serious bottleneck. It is by far the slowest component of our web stack, accounting for 70-95% of your page load times for each request. Rails proponents often consider the raw performance argument to be a moot point for a few reasons.

First, it is commonly argued Rails allows you to make fewer developers more productive, and additional servers to deal with are a lot less expensive than talent to write a more performant website. This doesn't apply here because other than donations (which are greatly appreciated), we don't make money from running the site. Other websites are generally tied directly into a usage-based pricing model, and we don't do that.

Another common argument is that if you manage to render your pages in under 100ms, it doesn't matter how much faster you get, because it's already faster than you can blink. Sure, it doesn't matter to your eyes, but it matters to my server. With Rails, each server instance is independent, and can only serve one client at at time. This means that every millisecond I spend processing your request is one I can't spend doing any other useful work for other clients.

Finally, I often hear it argued by members of a certain orange themed programming website that server side performance is not a problem if you use a single page application (SPA), which bypasses the server interactivity delay. Empirically, we know this isn't true; users can tell that your data isn't coming in any faster if you don't actually make your backend faster. But that wasn't even the point; it doesn't make it fast enough to justify the continued use of Rails, and you can certainly still create a SPA without Rails.

Moving on

After evaluating alternatives, I have made a decision on the successor to Rails for Derpibooru. I have selected the Phoenix framework, using the Elixir language, and running on the Erlang VM. I have worked with Elixir many times in the past, and it offers the performance of an ahead-of-time compiled language, with more syntactic expressiveness than even Ruby.

The name for this new project is Philomena, based on our canon phoenix in MLP:FiM. You can find the source code here. 

We'll be running Derpibooru and Philomena in parallel for a while to assert that Philomena works without major hiccups. The project will be more permanently hosted at philomena.derpibooru.org once it reaches any meaningful level of feature completeness. For now, it will not be online very often, but that will change in the future as we gain confidence in it.

I promised you a source release for booru-on-rails. I aim to finish the work for Philomena and get it deployed by the end of November. The definitive date for the booru-on-rails release will be December 25, 2019—don't miss it! Like Philomena, it will be published on GitHub, and licensed under the GNU Affero GPL v3.

Hang tight. It'll be here before you know it.