Chapter 7: a sort of post mortem in one part (or two?)

While I'm busy figuring out the direction for the next part of Light of my Life, blocking out chapter 8, and prototyping a convenient, unintrusive way to continue in a new, clean installation for the next chapter, I've also taken some time to do a bit of a post mortem. I'm not usually in the habit of doing so, but I felt it was in order in this case, as I felt this release was particularly bumpy and unpleasant.

And while I fully intended to also include a section here about the content of chapter 7, after writing it up, I felt there was a lot of navel staring, the author is rising from the dead more often than Dracula to berate the audience, and ultimately it's lacking in hard or useful conclusions. Also, it turned into another wall of text.
So I'll leave that up to you to decide if you still wanna have a look or not through the poll below. If there is enough of an interest in my musings on the subject, I'll put it up in a separate post. If not, it's been a helpful helpful experience to collect my thoughts on the subject, and that's enough.

So, we ended up with just a mostly technical write-up that you see below, which isn't the most compelling reading I must admit, but it's nevertheless good to share.

Alright, post mortem, technical issues, where to begin?

Plenty to pick from. Let's start with the number one offender, as usual:

Android.

That's should be no surprise, because it's been a pain during every release. But usually, problems get fixed in the end, and this time, they were ultimately not, at least not for everyone.

The problem?

Refusing to load up. Crashes during play-through.

Why?

The game's just become too big at this point.

Why did it become too big?

This is a bit of conjecture, but the load on memory doesn't come directly from the images, as they're loaded runtime, nor from the in memory image index ren'py keeps, because even at about 30k/40K-ish entries (it indexes all explicit declarations as well as all individual files), its memory footprint is still only a few hundreds of kilobytes from what I can tell.

So you know what it isn't the problem, but any clue as to what is?

Drawing a parallel to PC where it's easier to look at memory, it does seem the memory footprint was almost doubled there since the last chapter, and the number of images also doubled, so there's a correlation right there. My guess is (and it's just a guess), it's the image declarations and the transforms scripted around them that are taking up a lot of memory, either as opcode or more likely as instantiated objects (sans dynamically loaded image data). For what that info's worth.

Solved?

Partially. Teching up to ren'py 8.1.1 made it slightly better behaved on Android, but ultimately, devices with modestly spec'd out resources were left behind. Instructions on how to help make as much RAM available as possible is about all that can be done.

Could it have been prevented?

Maybe at the start of production of chapter 7, by splitting chapter 7 off to its own install. But at the end of production, and with one foot in release, there was no more time for adding several weeks of refactoring and testing.

And how about testing sooner? Could that have prevented it?

Possibly, if it set on very early on in development. But it likely would have only manifested at a late moment in development, and by that time it would have still be too late to change it all around without significant effort.

Lesson learned?

Don't make big games.

Chapter 8 will be a separate install for that reason (because the same issue is also rearing its head elsewhere). I did some research and prototyping to make jumping from one installation to the next as seamless as possible, and hopefully it will not cause any frustrations with players (although specifically for android users, some manual steps will have to be performed by users that desktop users will not have to perform, so there's already a bit of a headache brewing there, urgh).

Older models PC/Laptops, and 32 bits OS.

Lots of parallels with the Android problems, but some differences. And this issue was compounded by the next one down.

The problem?

Crashes during play.

Why?

The problem basically boils down to lack of memory in both cases (same or similar to the Android issues).

But why was it a problem here too?

32 Bits OS doesn't allow for more than 4GB of RAM, and the game wants a really large chunk of that at this point. Also, if an aging laptop only has a similar (or less) amount of RAM, the game has a hard time running in that as well.

Solved?

Not really. Teching up to ren'py 8.1.1 also meant losing 32 bit support completely, but since those were machines already experiencing memory issues, the extra damage caused was limited I think, and it might have solved issues for people running older 64 bit systems that were juuust falling short, so that evens out maybe?

Could it have been prevented?

Not at this stage of development. See Android.

Lesson?

Again, don't make big installs. Chapter 8 will avoid this.

Anti-virus warnings.

That one gave me a bit of a scare, and some unexpected repercussions.

Problem?

Particular virus scanning software (McAffee mostly) detected malicious software, showing warning to users, and, equally bad, quarantined executables.

Why?

Apparently, some parts of ren'py's code triggered an heuristic check, that is to say, a check that looks at application behavior, rather than looking at a particular signature for known malicious software and virusses. In this case, it didn't like ren'py/python writing in AppData it seems, and that's where it stores its saves (which, as we learned, are pickles and seem to count as unsafe code by some definition).

Why was that a problem?

Primary problem was that players were alarmed, and many probably deleted the game. But that was not the end of it.
The secondary problem was that it seemed to target the 64 bits executable for quarantining, so many players that decided the alert was a false positive, ended up playing the 32 bits version because the 64 bit executable wasn't there anymore.

And why was that a problem?

Because that gave them all the same memory problems mentioned in the first two items, even if they had shiny new, beefed up machines, and had no way of knowing their anti virus software was ultimately responsible for that.

Solved?

No, unless teching up to the latest ren'py version did, as I haven't gotten much reports lately. Users were informed, but really the issue was beyond my control.

Could it have been avoided?

Theoretically, by feeding all executables through virus total beforehand? Yes/maybe (depending on if there even is a fix). Practically, not likely.

Lessons learned?

Some virus scanners suck. But no lessons here.

8 GB Updates

Problem?

Whenever there's a fix during the early release period, that's another 8 GB download for the entire game

Why is that a problem?

Many updates in short succession is a pain for users where bandwidth is at a premium.

Solved?

Yes, after the fact. I've chopped the one big .rpa file holding all the scripts and resources up into a bunch of smaller ones, and the one holding just script (which is 90% of the fixes during a release) is about 64 MB. So that's a reasonable file to patch. So long as the images remain untouched, that's a viable solution. But chapter 7 had its final release, so it's a moot point for now.

Also, it can be solved by using the Itch app if you have the game on itch (it updates incremental).

And it's now trivially solved in chapter 8, as that's a fresh install and will be a lot smaller without all that baggage of the previous install.

Could it have been avoided?

Yeah, if I'd done that sooner (and tested it, which was the main reason not to change this during release).

Lessons learned?

Chop up archives into smaller bits right from the get-go, so updates can be small under the right conditions.

Sometimes old saves can't be loaded

Problem?

Loading up saves made in previous chapters would crash for some saves. It got worse with later releases of chapter 7.

Why did that happen?

I take care not to mess with the call stack or source file names with released code, as that is also known to give these type of issues, so in this case it was caused by upgrading the ren'py engine to a later version.

Why did it get worse?

The first versions of chapter 7 went from ren'py 7.3.5 to 7.6.1, but later ones went all the way up to 8.1.1 (and that's a big leap, as that also updated the python version). That seems to have caused a bigger number of incompatibilities the otherwise very robust load system couldn't account for.

Solved?

No. Saves that are made on choice menu's or near program labels are usually fine, but some saves are not, and there's nothing that can be done to salvage those.

Could it have been avoided?

Not upgrading wasn't an option as that solved some other critical issues. So, no.

-----------------

Right, that's it. Feel free to tack on things that you felt went rocky, and I might amend this post with an analysis of that. And if you're interested in my musings about the content of chapter 7, vote below (I won't be addressing that in this post).

Next week we should see a new progress infographic going up as I get ready to start producing content for the next chapter.