sondehub

Nearing v2 Release

Added 2021-04-20 11:22:33 +0000 UTC

Thanks for supporting SondeHub! As we reach closer to v2 release I wanted to share how some of our infrastructure is built.

Our infrastructure runs on Amazon Web Services for several reasons:
- It's what I'm most familiar with
- I'm able source discounts for AWS
- AWS provides high amounts of availability and scalability, letting us experiment quickly (though this is true for several other platforms as well, like Azure or GCP)

v2 infrastructure overview

SondeHub v1 grew organically in its design, however it did allow us to prove out some of our design decisions. One of the goals I had with the design is allow third parties to interface easily. The advantages SondeHub platform provides compared to other services that provide upper air data is that our data is provided live as data is collected.

To let third parties gain access to this live data we utilise AWS IoT MQTT over WebSockets. A third party can use our API to connect to the websocket stream and start receiving telemetry with less than 30 seconds latency.

ElasticSearch Kibana showing altitude over 100 radiosondes

We also wanted users to be able to query the data interactively. Which is why we've invested much time in getting the data into an ELK stack where it can be queried using the popular Kibana dashboards.

11,000 messages receiver per minute during peak load

In SondeHub v2 we collect 30x more data. Every single frame is uploaded. This jump up in data processing caused us to hit a bottleneck in AWS ElasticSearch.

You have exceeded the number of permissible concurrent requests with unique IAM Identities. Please retry.

Working out instance sizes and ingestion method has taken a significant amount of time.

This is caused from the number of messages being received being too many for a single node to handle in AWS ElasticSearch. This can be worked around by batching up the messages prior to being inserted into ElasticSearch. We've built out that pipeline and over the last couple of days testing it seems to be handling the work load a lot better.

The other way to access our data is for historic purposes. Our dataset is growing quickly and we've been able to partner with AWS to store all our telemetry in S3. These buckets don't cost SondeHub so we are encouraging everyone to utilise data from here where possible.

And finally we have the SondeHub UI. Historically the way the SondeHub UI worked was proxying requests to and from HabHub however we have severely overgrown utilizing their backend. Mark (VK5QI) has done a huge amount of work forking the HabHub UI and adjusting it for our needs. For now the backend has a bunch of endpoints that provide HabHub compatible data format to make getting the UI up and running as fast as possible.

Since we have more control over the data formats and the UI we can start introducing new features. For example we can now report the SNR from receivers, a much requested feature.