LegendKeeper "Hydra" Progress Update #3

Hey folks! This is a progress update for "Hydra" , an upcoming upgrade for LegendKeeper that adds offline support, and a number of bugfixes and UI updates.

I've solved the main backend issues that caused Hydra to be postponed from its initial launch. The new backend is an absolute monster and should be able to handle whatever we throw at it. 

The largest remaining problem is that large (500+ pages) projects take a long time to sync from zero. I.e., if you reset your local client and re-sync with the server, and your project is big, it can take an unacceptable amount of time. This is because the sync algorithm is made to carefully merge changes between the client and server. When you're syncing from zero however, there's nothing to merge, so I'm writing a separate sync algorithm for the first-time-sync. This new "first-time sync" algorithm is orders of magnitude faster and will solve this issue.

I appreciate everyone's kindness in regards to the initial postponement of the update! I was really stressing out, and having everyone be so kind and supportive really made me feel a lot better. So just wanted to say I appreciate you! I'll continue to post updates as we get closer to launch.

I'm getting my second COVID vaccination today. Knowing me, it'll probably make me sick so I might be low-contact for a day or two.

Braden

Tech Details4Nerdz:

The issue with the first Hydra backend was one of throughput. In a synchronization scheme, the client and server generally only sync differences since the last time they synced. Most of the time the client and server are connected, so they are already in sync, and a small part of the time, a few things have changed. But, there are occasions where a lot has changed. For example, if the client erases its local database, it needs to download everything from the server again. Or maybe the client has created hundreds of articles offline using duplication and templates, and when they come back offline, they blast the server with all their new data. LK's database worked great for those first two scenarios, but would buckle at the weight of multiple clients syncing tons of data all at once. I thought at our scale, the architecture could handle it, but in my pre-launch tests I realized that wasn't the case.

One solution is to just slow the whole sync process down, but the amount of slowdown required to ease IOPS pressure lead to an unacceptable amount of waiting. Instead, I re-thought my backend architecture and implemented something specifically designed for massive throughput. A tiny sync step that would take 5-10 seconds to complete under heavy server load, now takes 5 milliseconds, and that was with a puny development machine.  So a huge improvement and much more Hydra-worthy. ☺️