Back on Track with Small and unfinished progress

Well it has been now 3 weeks since the last post, which was intentionally made because of getting back into things, also a big part of the engine was rewritten again (and I bet it will happen a lot more this year).

Instead of starting at the end, the stuff should be explained forward.
So what happen after the last post?
Quite a lot actually.

After realizing from this picture where my Memory Issues came from. It was clearly that the Entity System, renderer and also the storage of Entities itself were the issue.

As you can see those 1.048.576 Static Entities on the map (a 1024x1024) were taking around 2-3GB of memory. Mainly for the reason that each entity (even a simple Tree) took at least 300 bytes.

And after a lot of talks with people and researching a lot of things, I decided to for now temporary delete the entire Entity System from the game.

Instead of that I implemented a 2D Tile System that uses a 24 bit ID System for the Tiles.
The picture seen here is already using that new system.

What is the up and downside to this new system?

- 1: Since its 24 bit, there are 8.4 Million Tile IDs provided. On top of that there is a dynamic Metadata System which can allocate metadata dynamically between 0-8.4 Million, that just uses up extra IDs instead of having a fixed amount of space.

- 2: This System uses on its own (without rendering) on a 4096x4096 Map only 160MB (which half of that is at least the terrain), and when completely filled it eats around 600-1GB when rendering is included.

- 3: The rendering is a bit more easy to implement now from the inside & outside with a request system that allows to define what type of rendering is used. (Dynamic or Instanced),
Which wasn't a possible before.

-4: Spawning Tiles compared the Entity System is creating now a lot less garbage which should improve overall performance but also total memory allocation.

-5: Right now there is a limit of how much you can do with the tiles but there is a plan to add data storage to that too so that complex logical Tiles become possible.

While that System is a lot more memory efficient it is right now completely broken in terms of rendering. It doesn't work properly at all right now. What you see there is my TestShader running the rendering, which isn't made for rendering "100k" Tiles at once. That one is designed to render simple stuff and barely any of it. (Dynamic shaders are slow on giant batches)

So that needs to be fixed, and I have to try and error what is actually preventing the TileShader from rendering the models at all.

At least some progress in this endless looking loop of designing & learning & rebuilding a mini engine.

On another note what happened next to the redesigning of the Object Systems, I found out that javas internal Memory management of Primitives (numbers and flags) in map/list formats is actually quite bad. As you saw in the "Memory Usage" picture, the most issue were that LinkedHashMapEntries (basically a sorted Map (Insertion order)) was using a lot of memory just to hold data. In fact it was the most usage that I had. Mainly holding just references together.

So I needed a better solution, and while doing research for the Entity Problem I found a library that would maybe take care of that.
For those who want to know, that was Trove.
So after doing research about that one I noticed that it had a lot of issues and was also not implementing a lot of java interfaces that I would wanted to keep. (Call it bias)
So yeah that wasn't sadly a solution, and while trove wasn't a good library, it  at least lead me to the better library:

FastUtil.

That library is implementing a lot of features of trove, and improving on those a lot.
It is also more memory efficient then Trove. I have only a couple issues with it. (Mainly the size of the library because its bigger then all my other libraries + code + assets right now)
But other then some tiny "Me" issues, the library is perfect for me.

And after a surprise night where I started at 8-10PM reading in that library to learn how it works and ending it at 6AM without noticing where the time went, I can be pretty sure that I am going to use that library for a lot of my projects.

So yeah that is the reason why it took 3 weeks instead of 2, I was rewriting a lot of Systems and learning a lot and after 2 weeks the tile system wasn't even implemented for the whole reason that my head needed to wrap its head around the way I wanted to implement it.     

That covers this post and the next one I am not sure when it comes out, Especially because of holidays that are coming up. So either in 2 or 4 weeks the next post will come. That's depending on how much progress will happen next week.

Thanks for reading. Speiger