Changes to background syncing

Today's Infrastructure

When you open beestat, your thermostat, sensor, and historical data is downloaded from ecobee.

Every five minutes, beestat checks locally for any updated data and updates the dashboard. This does not reach out to ecobee.

In order to keep local data up to date, a background sync runs for all users that reaches out to ecobee and updates the beestat database. Users who are actively using the application are set to sync once every 5 minutes, and users who have been away for a while sync at a slower rate.

Problems

This worked great at first...but as the user count grew so did the length of time it took the background sync to run. Here's a log of thermostat syncs for one Patron. This is supposed to execute every five minutes. If this user had their dashboard open it would take over an hour to show any changes. Whoops.

The Fix

There are a couple ways to fix this:

• Sync less frequently
• Move important syncing to another location 
• Add more threads

I took the nuclear option and did all three. First, I drastically reduced the sync frequency. As an example, instead attempting to sync thermostats every 5 minutes, I reduced it to once per week. Runtime data syncs every hour for active users and Patrons. For non-Patrons runtime data drops off to once per week as you become less active. the net effect is that Patrons have speedier load times if they're away for a while.

Now that the background sync is no longer real-time, I moved the thermostat & sensor sync into the GUI. This allows the application itself to request updates from ecobee rather than waiting on a delayed sync. This makes the GUI more responsive to changes and actually allows you to use it for live data again.

Finally, to help future-proof things, I added a second background sync thread. Instead of having one script loop over 7,000+ users, I now have two scripts looping over half that amount. In this case they just pick odd/even users, but adding more threads is trivial.

This has actually been running for some time and has eliminated the issues people have had not getting live updates in beestat. Success!