Predictions - so many predictions

We recently made some changes to how our radiosonde predictions (for SondeHub not SondeHub Amateur) are delivered to clients.

Previously the web client would request predictions for every single radiosonde. On the backend predictions are run on a schedule, so from a server load perspective having the client request all the data didn't create much additional load. However as the number of active radiosondes has grown over time, this has resulted in a larger and larger payload over time. Sometimes reaching 1MB. This is requested from the client every minute. Obviously this quickly adds up in total bandwidth, not to mention this can be annoying for mobile users both from network and memory usage perspective. This polling method also meant that there could be a delay of up to 2 minutes before the latest prediction was shown.

After moving to a culling system for radiosondes we recognised that the next big change we could make to save on bandwidth and improve performance was to also cull predictions. However culling predictions using the API would decrease performance as we wouldn't be able to heavily cache predictions like we did before.

It made sense for us to start delivering predictions via our websockets delivery method. This has some bandwidth benefits which should help reduce our overall AWS spend, but also helps with the polling problem. As predictions are generated they are pushed out to websockets, allowing for clients to see the latest prediction as soon as it's available. We've already had some anecdotal feedback reporting the improvement - which is great news.

We can already see a massive improvement in the amount of data transferred to clients in our backend metrics, along with a reduction in total number of requests. Both of these have a financial impact. Hopefully the web/mobile client experience has also improved however as we don't any tracking systems on the site we don't have any qualitative measurement to confirm this.

That's all for today, happy sonde hunting.

~ Michaela