Making sure beestat isn't broken (redux)

Original Post:  https://www.patreon.com/posts/23245016 

Back in December I talked about how I had built some graphs to display information from my logs that I could keep an eye on to make sure everything was working fine. Overall that process was working fine, but I wasn't quite happy with it for a number of reasons:

1. Logs were all stored on my application database server
2. Viewing the logs required code and a special permission for me on beestat.io
3. A nightly process was required to aggregate log data
4. No alerting

Let's discuss how I fixed all those things and had fun doing it.

Logs were all stored on my application database server

This was undoubtedly the biggest change. Logs were eating up disk space and writing to the log files was adding load to my database server. Reading from the logs required loading chunks of those tables into memory thus removing other things and decreasing performance. The first order of business was to get these moved.

One option would have been to just build my own external logging service and GUI, but I didn't really care to bother with that. There are many I could purchase...but I didn't really care to pay for that. Instead I explored some free alternatives and landed on InfluxDB. InfluxDB is a relatively new database engine designed for storing time-series data (like log files or even thermostat data). 

After you configure it, you just throw data at it and everything is handled automatically for you. It's missing some features I would like but it's overall very nice and easy to use. This database server sits on a completely separate host and I simply fork a background process to send logs to it. Beestat is no longer affected by logs.

Viewing the logs required code and a special permission for me on beestat.io 

There are two main problems with having my own code inside beestat to view logs:

1. I have to write the code which takes time
2. If something happens to beestat I have no easy access to the logs

There happens to be several options for viewing data from InfluxDB. A popular one is Grafana, which works well. I tried it but wasn't happy with the level of polish. Another is Chronograf, which is the official GUI for InfluxDB. This has a lot more polish. Both of these options make it trivial to query and display data. Now if I want to query some specific information from the logs I can very quickly hop in and generate a new graph.

For example, here's me quickly generating a graph for API errors in the past 7 days grouped by hour. I can save this to a dashboard for viewing any time if I like.

A nightly process was required to aggregate log data so they were quick to view 

InfluxDB has two nice features built in: retention policies and continuous queries. Retention policies allow you to define how long to store data. After the set limit, old stuff just gets deleted. Continuous queries do periodic aggregation by hour, day, week, etc. These two ideas can be combined so that you store detailed logs for a month, and then anything older than a month gets condensed down into fewer rows. These condensed views decrease the size of your logs and make it very fast to query data over long periods of time.

Here's a fun one: Unique users per day since I started beestat. The large spike on the right is the day of the Home Comparisons announcement where I hit nearly 800 unique users.

No alerting

Yes, I could have gone and built this myself, too. Or...I could just build off the work someone else has already done. Chronograf has a simple interface that allows me to trivially add alerts when things go bad. A few days ago ecobee had a brief outage. Here's a screenshot from my Slack notification channel about it.

And here's the official email about it:

Guess who knew ecobee was down 11 minutes before the official email? :)

I can now set up all kinds of alerts that watch over beestat for me and tell me when things go wrong. This is important for me to improve the stability of my platform and provide the best experience I possibly can. I was able to research and implement all of the above in less than two weeks worth of spare time. It was a nice break from staring at thermostat data.

Thanks for reading! More news coming soon to your email about what's coming next for beestat. Make sure you're subscribed to the mailing list!