byteslice

Validating randomness

In my previous post, I demonstrated the viability of a format-preserving encryption construction used to generate a random permutation of a set, using FNV as the PRF in a four-round Feistel network. While it does generate permutations that look reasonable at first glance, looks can be deceiving, so I decided to run more tests on the quality of that randomness.

There are two desirable properties...

Random sorting using cryptography

It's been a while since I've provided any updates on Protodash. I admit that it's descended into the role of vaporware, mostly because implementing it in an efficient way has turned out to be extremely complicated. My original idea was to implement a new database backend with the bitmap tree as its native data structure, but I didn't have the energy to bring that to completion, and then school started again and I lost the ability to consistently focus on it.

But I've gotten at least som...

Status update (2020/09/22)

It's been a long time since I posted a status update. So long, in fact, that some of you might have thought I died or dropped off the web, and my monthly patronage has dropped a bit -- though this is probably more substantially the result of Patreon's recent foot-shooting exercise and various tax collection changes.

In the wake of the events of the past few months, which I'm sure I won't need to recap here, I voluntarily left the moderation team at Derpibooru, and now exclusively work o...

Status update (2020/03/11)

My university, following suit with a wide number of other universities, is now kicking us out of the dorms, so I will be moving home tomorrow. I expect to return to my normal level of activity soon.

Status update (2020/02/20)

Cool date, isn't it?

The reading and writing demands from university have been ratcheting up this semester to the point where I haven't really been able to spend as much time I would have liked on the site at all.

Some of you have may noticed that I am not currently in my position as admin on the site staff list. In order to help myself properly destress and focus on my work, I have cut myself off from administration and development for the time being. Don't worry, it's only tempo...

The covering bitmap tree

Because I was asked several times about how the implementation of this data structure is able to search and sort so fast, here is an overview of how it works. By publicizing the data structure and algorithm, the hope is that one day search libraries like Lucene will also adopt this particular method of sorting by natural fields of the data.

The above diagram is a simplified view of a special 2020-01-22 05:13:07 +0000 UTC View Post

Faster searching

Someone new to Derpibooru may find themselves somewhat impressed by the sheer amount of functionality enabled by our search system. (Also, we apologize if you were ever confused by the minutia of the syntax.) You can search any number of our 450000 tags in any combination and number you please. You can use logical operators like AND, NOT, and OR to filter set membership any way you desire. You can sort by almost any numeric field you want. At the time of writing, I'm not aware of any other we...

booru-on-rails

Here it is.

https://github.com/derpibooru/booru-on-rails

Thanks to all the contributors who made this release possible:

- Clover the Clever
- K_A
- Nebbie Zebbie
- JoeyH3
- deliciousblackink
- Meow
- Arcaire
- DoublePipe
- jspenguin
- Carcer
- Raptorshy
- DJDavid98
- WingbeatPony
- Marker
- DragonMaus
- AppleDash
- Xena
- Raiinbow Dash
- ...

It's (A)live!

Philomena is now running live as the main backend for Derpibooru, with booru-on-rails still running behind the scenes to power requests to the old APIs.

There is still a fair amount of work to be done, and things to implement that were missed in the beta thread, but as far as I can tell, it's here to stay and while I didn't implement absolutely everything, there's no need to roll back. Success!

I am now preparing the Rails source code for a Christmas release. The moment you've all...

Philomena open beta

Philomena is in open beta. Check it out here. Get the source code here. 

Thanks to all my patrons for their continued support while I hammered it out. I'd like to also give a special thanks to the patrons who worked with me to test the site while it was in "alpha," if you could call it ...

On my recent absence

If you were watching the Philomena repository, you may have noticed a startling lack of commits in the past few weeks. I apologize for this; I've been extremely sick, and am just now starting to recover.

Status update (2019-08-23)

This will be a short one.

I have moved back into my residence for school, and I start classes again on Monday.

My birthday is tomorrow, and I will be taking the day off to celebrate.

Thank you all for your continued support throughout the summer. Your generosity helped me recover from my mistakes and get my finances back on track for the year.

Until next time,
byte[]

booru-on-rails, Philomena and the future of Derpibooru

Rails, though not without its warts, has been a fairly pleasant framework to use for Derpibooru for a long time. However, I currently believe that for a number of reasons, Derpibooru's future does not lie with Rails.

Normalization

A few weeks ago, I followed through with a routine normalization change for tags. The change, fundamentally, was very simple: turn an array column of tag IDs into a join table between images and tags.

As usual, I pulled out all t...

Hardware failure

My 250GB Intel SSD 510 bit the dust today when I logged on to my desktop to check my email.

This shouldn't really be a surprise, since I (obliviously) exceeded the MTBF of the SSD by 2 years. Intel doesn't provide a TBW rating for this drive on their specification sheet for it. However, based on similar drives of the time, the rating would likely be around 30 TBW, and I definitely exceeded that over 9 years of use as well.

This throws a bit of a wrench into the works. I didn't los...

Rails

Ruby on Rails is a "batteries-included" framework. Out of the box in a 6.0.0.rc1 install, you get the following support libraries included directly with the `rails` dependency:

- Active Model, an attribute validation and callback provider
- Active Job, a queueing system for background processing
- Active Record, a unified ORM for Rails with pluggable database adapters
- Active Storage, a manager for uploading files to external hosting services such as B2 and S3
- Ac...

2019-06-24 21:34:52 +0000 UTC View Post

The MongoDB zombie

This is my first attempt at a longer dev blog for the site. Hopefully some will find it interesting.

---

For the past few days, I decided to put the video processing work I had planned on the side, and focused on Derpibooru's database for a bit.

Those involved in the history of the site may know that Derpibooru did not originally use PostgreSQL as its primary datastore. In the past, we used MongoDB[1]. During the time period when the site was in its infacy, MongoDB enjoyed c...

Status update (2019-04-13)

Hello all,

After Patreon finished manually reviewing my account for suspicious activity, I received a total payment of $263.29 this month on the ninth. I had initially intended to write this status update the day I received my payment, but due to a demanding schoolwork and midterm schedule this week, I did not feel able to write it at the time.

In addition, I have been informed by other site staff that I am significantly undervaluing my work for the site, and that my goal tier of ...