42Bytes

Migration to golang: Api v2, WebSockets, Discord

Added 2023-05-07 11:18:50 +0000 UTC

Hello fellow traders!

It has been a while since the last Patreon post, like a few years. 😨

You may already have figured out, I am not a fan of writing articles, yes, my bad.

Psst only between you and me, I’ll share the reason: I am not feeling confident enough in my language skills and articles are big and scary things, better to not mess them up right?

And you can't mess up if there's nothing to mess with, the perfect logic is right there.

Okay, but today is another case.
Today I'll tell you a story about golang. I like to tell ppl about golang.
And about our development plans and what we are doing.

Background

For people who are curious

Currently WFM consists of a few docker services launched across a bunch of servers:

Http \\ api backend, using inhouse modified Flask framework
WebSockets, on gunicorn, on gevent, based on half assed ws protocol implementation. Sockets are responsible for status management and real time events.
Celery, for some periodic background jobs, like calculator stats, process bans \\ unbans, orders clearing, verification, stuff like that.
Discord bot, for our beautiful moderators
Manager, CLI tool for manipulating data, updating items, triggering some jobs like synchronizing DE manifests and drop sources, etc.
Not talking about: db, amqp, cache, load balancer, they are out of context for today.

Http + WebSockets + Celery + Manager, all working on python3, all using Flask or its extensions to some extent

The rising problem

WebSockets

Occasionally, especially in recent days, you might have noticed that you could get stuck in a "Connecting..." state and not be able to update your status for 5-20 minutes, and the site would become unresponsive for a short time after that.

The reason behind all of this is that websockets suck and they can’t handle today's load.

After a certain load, they begin to leak memory really hard, sometimes they can derp and slowly exhaust their own ability to accept incoming connections.

It also doesn't help that they are based on a very old and outdated technology that is damn hard to debug and frankly, not even worth trying.

Conclusion: nothing could help them, nothing but a full rewrite on something more modern and more suitable for such a task.

Solution and why it’s golang

Okay, we need to rewrite our WebSockets, but what technology should we choose for such a task?

Initially, I tried to write an extensive comparison between several languages and why I picked golang, but it quickly became 5 pages worth of text and probably needed to write five more on top of that, so I decided to skip this.
But if you would like to see that, let me know.

Here is the short version, as a condensed summary of all my conclusions:

Python: very slow and clunky
Javascript \ Typescript: I have enough TS on frontend
Rust: very good, hype train of 2023, but it will take a lot of time to learn it and write something that works.
Golang: perfect

A bit about Golang:

Simple as a brick, if you have some programming background, you could pick up Golang in 1-3 weeks and be able to put out reasonable code.
Simplicity is comparable to Python.
Very performant, with naturally built-in concurrency in the form of “goroutines”.
Goroutines and their synchronization are very simple, in comprehension to other languages.
And in WFM everything is about concurrency and async operations.
Statically typed and compiled.
Very good development tools, profiling tools, IDEs integration.
Rich Stdlib, you can build your web app by using only Stdlib.
Less dependencies to manage, track, update.
Opinionated language, harder to write incomprehensible mess of a code.
Usually you have very few solutions to the problem.
It forces developers to follow good practices.
Some people might find it boring, but in my opinion it's very good. In the end, our goal is to create a reliable product, not to play with the language.
But, ofc, I guess a very talented developer will be able to find a way to screw everything up.
Final docker image is around 100mb

Why not go even further?

This question popped in my head after a decision of rewriting WS was made.

The thing is, we already have a discord bot working on golang, we will have WebSockets on golang…

Logical conclusion would be to rewrite the API in Golang, right?

And the answer is YES.

Usually, it would not be a good solution to rewrite a well established project entirely, as you are going to waste so much resources and time on this.

But i have some counterpoints:

It’s fun.
We will have a fresh new stack with a lot of new interesting possibilities (I will write about them below)
We will be able to reuse this stack in other projects.
Unified language allows me to create modular monorepo for all services and microservices.

Benefits

What we will get from migration to golang

Blazingly fast execution times

Golang is way faster than python.
We will get x2-x10 performance gain in comparison to the python, even for simple operations that are usually bounded by I\O.

More real time updates

With new websockets we could finally do more real time updates.
Like updating all orders in real time.
Setup trackers and send real time notifications
And everything like that.

Tighter inter-service integration

For example,
You want to get a notification if someone creates an order below a certain price.

Link your discord account to WFM
Some user creates order
we send this info to discord via amqp\\grpc
Discord sends you a PM

Another case is, discord will be able to track your in game status => better status management for some ppl.

Simplified schema on interconnections:

Faster and more reliable development

What is the most interesting, developing in golang gives faster writing time and better code quality thanks to codecoverage and good testing tools and a forced "TEST YOUR F** CODE" policy.
So we are raising TTM and quality.

Looser coupling between parts of the code

Working in a team becomes less problematic, since we are now working on smaller portions of code that is not tightly coupled together and could be developed in parallel and unit-tested independently (still have to do integration test at the end ofc)

You can read about it here, we don't use the same one, but it’s a very similar approach: Hexagonal Architecture

What about new functionality?

I would like to concentrate on the new API and new WS server for now and reimplement what we already have, but at the same time, working on a new backend and especially new api, allows me to adjust many things right away.

So we will have some, in the new API I would like to add:

Bulk trading for certain items
Extended statuses, add a bit of additional info to your status, like “on mission” \\ “in dojo”
Real time order updates to item pages.

API v2 and v1

Anticipating your question, APIv1 is now deprecated and will not be updated.

But it will still work, at the very end of v2 development we will notify everyone and there will be 60 days to migrate to the new API.

Websockets

With WS I can’t be so generous, I’ll try to maintain some level of compatibility, but no promises.

All because I have to replace it as soon as possible, due to recent issues.

OAuth2.0

Neat part for all 3rd party devs and users of these applications.
We will have an Oauth server.

What does that mean ?
You will have the ability to register your application within WFM, enabling users to log in without needing their email and password or manually copying a JWT token from cookies.

Each app could have different access rights.

You can read about OAuth here: OAuth2.0
Overall this will allow us to work in a more professional and reliable manner with 3rd party developers and provide more refined access to the api and introduce or alleviate some API limitations like access rights, RPS and everything like that.

More about that later.

Okay, but WHEN ?

Most intriguing question isn’t it?
I would like to say SOON™ …

But, I'll break this tradition today.

New websockets

All core parts are ready, though I will still need to add a few missing actions (such as chat and auctions)

It should take about 1-2 weeks.

I want to replace WS as soon as possible, but I need to be sure that everything is going to work.

New API

Right after websockets deployment.

But it won't be one huge update for the whole v2 API, instead we will deploy v2 and slowly migrate WFM to it, endpoint by endpoint, because development will take some time.