v3 Groups & Local Data Storage

Hello all,

Finished another significant chunk of the v3 overhaul and wanted to make a post about it. I've built the framework for better Pilot/NPC/etc group handling, which nicely dovetails into how v3 will handle local data storage, which might be interesting to read about.

Pilot Groups now hold information and image data (another long-asked-for feature), and can be exported and imported along with (or without) their pilots. The same will be true for NPCs and forthcoming Campaign features, though these will also be able to take user-defined sets of properties (tags, in any other app) that will allow for generated or composite groupings within and between groups. Confusing, but I think once I've got in online it'll make more sense. Essentially, your non-Pilot data can be set into one "real" group and then any number of dynamic groupings based on whatever (campaign, type, tier, location, faction, etc. etc. etc)

Also, the "mass delete" dialog has been replaced with an "organize" modal that also contains the sorting/reordering functionality. This is less optimal in terms of user-clicks, but I'm going to take the hit here to make it easier to maintain a clean mobile interface, and cleaner drag and sort code in general. Might be a little clunkier than current functionality on PC, but -- hopefully -- worth it in terms of robustness.

Anyway, that's it with the sort of user-facing stuff in this update. A little light, sorry, but if you're not interested in the data storage changes, which made up the bulk of my work since the last update, you can stop here.

Data Management in v3

One of the problems I've struggled with in v1/2 is C/C's reliance on localStorage: a backup for local OS storage from way back when C/C was an electron app, press ganged into its only storage scheme. It's slow, finnicky, and extremely limited: 5mb max, which has become especially onerous as the LCP library grows.

In my defense: I didn't realize how popular LANCER and C/C would become!

So, if you've worked with localStorage before, you'll recall that it's a simple key-value dictionary that lives in your browser that persists between sites. If you interrogate your localStorage right now you probably have a decent collection of stuff from different websites and apps. COMP/CON saves all its stuff in there, as a small handful of key-value pairs.

This isn't great for two main reasons:

1) 5mb isn't a lot

2) Writing lots of data often gets weird

What you want to do is write as little as possible as frequently as is necessary. Obvious, right? If something changes, record exactly that thing and move along. Except because we're working with a key-value pair setup, we have to record in however big a chunk we decide "value" is going to represent. C/C saves each different data type (pilot + mechs, npcs, encounters, missions, lcps) as a single key-value chunk, which it has to fully rewrite every time it saves. Extremely non-optimal.

I probably could have split everything into item type + ID pairs then grouped and ordered them on load, but polluting users' localStorage with dozens or hundreds of C/C related items seemed... rude. It'd be a real pain in the ass if you routinely touch localStorage for work or something on the same device you use C/C.

This needed to change for v3 and, thankfully, there is now widely-available browser tech that allows this to happen: modern browsers* can hook into a persistent storage database that's way more useful than localStorage. I can keep all of my C/C data sectioned off into it's own little bucket and instance indexed dictionaries of data in smaller value chunks than I could reasonably get with localStorage.

Now, when eg. pilot data changes, I can go in and write changes to that specific pilot, rather than all pilots in your roster. I can add or delete a single pilot's data, specifically, without having to touch anything else in storage. Way safer, cleaner, faster. Now that there's a lot more space available, I also have a lot more leeway in terms of saving image, LCP, and campaign data. Just a very solid and very necessary upgrade all around. 

If you're thinking that it should have been written this way from the start, you're absolutely correct. I've made myself a note to simply make the code good the first time.

Sorry if this is rambling, I've been working some really awful hours thanks to the day job and am fully out of it. Hopefully I've said something interesting, or at least parse-able.

Thank you, as always, for your continued generosity

Beef

*some browsers, by design or user preference, can lock this sort of usage down. C/C will try and prompt you to allow for these storage permissions and, at worst, fall back to localStorage. In testing, however, almost all browsers allowed me at least 100mb of space when after C/C was granted storage permissions.

**achievement submission guidelines are coming very soon, for Patreon supporters.