Hug of death

Last week a post went up on Gizmodo from Alex Cranz reviewing the ecobee. The post contained a link to beestat with some nice comments. Thanks, Alex, and welcome to all the new beestat users and Patrons!

First of all, it was exciting to have beestat shared this way. Typically I see referrals from smaller communities like /r/ecobee or various HVAC forums. This article has been the biggest to date and I got to experience a small "hug of death" firsthand.

The first indication of a problem was some strange errors on the application server. It was running out of memory and, interestingly, sometimes unable to fork a process used only for error reporting.

The first thing I did was take a look at user volume. The previous day peak was at about 150, and I was suddenly approaching 250. This is number of unique users per hour. Honestly, the application server should probably be able to handle at least 2-3 times my current volume before I need to look at any upgrades. The peak was telling, but the user count was not the problem. 

What really causes problems is the number of *new* users. When a new user signs up for beestat, the past year of data from their ecobee account is synced down and the GUI spins and checks for updates every few seconds. This created a massive spike (14x more per hour) in API calls to my app server which began to kill it.

You can see the problem if you look at the server load. On this server, anything above 1 is bad. I didn't respond as quickly as I would have liked, but I upgraded the application server to help it process requests faster which got things under control until the peak subsided.

The database server shows a similar spike in load, but it handled it like a champ. Instead of throwing errors it just slowed down due to the volume. The number on the left is query count. Instead of 20M queries that day, it did 45M queries.

The problem only lasted a few hours and mostly resulted in slowness for many, and errors for a handful of users. 

In other news, things have been slow on some of the new updates. School is getting ready to start for my kids, rec soccer has started (I coach and play), and I had a patio installed which took a lot of my free time. Looking forward to being able to work on beestat some more. I have a lot of small requests that I can't mess with until the metric update is released.