Status update (2020/09/22)

It's been a long time since I posted a status update. So long, in fact, that some of you might have thought I died or dropped off the web, and my monthly patronage has dropped a bit -- though this is probably more substantially the result of Patreon's recent foot-shooting exercise and various tax collection changes.

In the wake of the events of the past few months, which I'm sure I won't need to recap here, I voluntarily left the moderation team at Derpibooru, and now exclusively work on Philomena. But don't worry, I'm doing just fine! School has started again, and I'm making good grades. The free-time crunch has been real, but on the plus side, working on web app code is a great way to procrastinate between assignments.

And speaking of that, some of the most substantial changes to Philomena are on my schedule, and will hopefully come to fruition sometime within the next few months:

- Separate application and background workers
- Merge together excessive database calls during requests
- Use "serverless" functions for image processing to prevent resource bottlenecks
- Store image files in an object storage server (AWS S3 compatible)
- Write production Kubernetes configuration files

All of this is in preparation for the demands of Derpibooru and Furbooru to move to fully cloud-based architectures. (The two sites are separate properties, they are not merging, nor will they run on identical infrastructure.) For those of you reading and shaking your heads, thinking I've lost my mind going to a cloud architecture, fear not. You will still be able to run the application from a single server if you desire, and it will be easier than ever to do it.

In addition, cloud Philomena will get some additional standalone tooling to ensure durability at scale, generally using a separate object storage server:

- Continuous file upload backups
- Automatic daily database dumps performed with pg_dump, rotated out every two weeks
- Automatic backups performed with pg_basebackup and continuous WAL archiving, rotated out every week
- Automatic daily backup testing for PITR
- Automatic daily public dump creation (for analytical/research purposes)
- Scripts to automatically restore database as it was at an exact point in time

Boorus will have, in their Kubernetes cluster, an app and image serving container (assigned to small nodes without much storage needed), a database container (preferably assigned to a large node that uses fast local NVMe storage), and a "serverless" container for running FFmpeg and similar utilities. I plan to allow starting multiple application containers. Standalone backup services should be run on a different provider to protect against data loss in the event of service termination.

Some of you might be wondering what's happening with Protodash. I did publish a GitHub repository with some initial work, and planned a lot more, but the events of this summer made me lose steam. I have thought and learned more about how to properly architect the new search engine, and I will reattempt it later, but I have decided that the base should be written in C, instead of Rust, due to the need to essentially implement my own on-disk binary formats and use lots of low level operating system calls. While this is possible in Rust, it is annoyingly difficult and requires writing too much code to do it, so I will stick to C for the base, handling disk serialization, memory mapping, and cross-thread synchronization, and write the rest on top of that C base in Rust.

I think that's about all I have to say for now. Thank you for your continued support over the past few months (it's been rough, I know) and rest assured that things are going well on my end.