Question about extreme data outputs

Started by
11 comments, last by frob 8 years, 1 month ago

So, someone was asking about the game Factorio and how it keeps tons of data and AI loaded in memory, seemingly without having enough space in memory to do so. So with a simple image example, I think I understand what they do but would like to hear some input (because I'm a newbie and could be talking nonsense):

So, instead of having:
1 address per tilesheet or all of them (cause you may or may not use only 1 per map type)
alt. x addresses for x individual tiles of that/those sheets ('cause you may not call/search the tilesheets every time, especially if we're talking about other datastructures that are bigger and/or more abstract/complex than images).

2 for double buffer (for anything that needs buffering)
1 for applied data (rendered image, etc)

You now have:
1 address per tilesheet

1 for ALL tiles
1 for preBuffer (that you put the tile address' data into before it changes to the new data)

2 for double buffer (the first of which receives the preBuffer data)

1 for rendered image

Now, you can probably understand where I'm going with this, when considering things that are more dynamic across the worldspace, and not just output onto the screen (things like logical changes, AI, and such).

My question is:
While this should (I assume) should be good for saving memory, I'm thinking that it'll probably decrease the processing budget a lot, since you're doing more stuff per second? But in general, having multiple scopes of memory management within one scope of statements, can be useful?

I mean, for Factorio it shouldn't be that bad, cause it's probably not the most demanding code anyways and you may still want to design LOD/mipmapping not only for rendering, but for AI and other things as well.

Just a newbie/sophomoric question. Thanks for any input.

Advertisement

For AI, it could be something like having an address declared to a function/object and then call that adress by some preBuffer, then declare another function/object, then call the new data by the preBuffer, and so on and then, finally, call the preBuffer set of functions/objects.

Again, I'm just asking generally here. I don't know too much about this.

Edit: Seems like what I'm talking about may be what the industry refers to as data-oriented design, so I guess I need to look deeper into that.

I wanted to try to answer your question, but honestly your question is very confusing. I barely understood one sentence. Your terminology might not be what people are used to here, so if no one is responding it might be because of that.

I'm confused. I know that they use a texture atlas to optimize render speed, maybe that's what the image you saw was describing?

I wanted to try to answer your question, but honestly your question is very confusing. I barely understood one sentence. Your terminology might not be what people are used to here, so if no one is responding it might be because of that.

Yeah, that's what I figured, so nvm. The idea is just to:

1) load primary tilesheet (an image of x * y individual tiles, from some spritesheet / tile atlas file);

(for loop:)

2) load image0 from tilesheet (at tilesheet location/size x/y / w/h);
3) put image0 into a secondary tilemap (at tilemap location x/y w/h);
4) destroy image0;

5) Repeat for all images to complete tilemap;

(end of loop)

6) render tilemap;

The idea is to keep X tilemaps stored in memory rather than X tiles.

Anyways, if nobody gets it then I'm probably wrong and need to learn more. Thanks anyways, though. smile.png

that sounds like a texture atlas. I'll refer you to this gamasutra article, since I'm not really a super-modern graphics guy: http://www.gamasutra.com/view/feature/130940/practical_texture_atlases.php

Like phil_t, I couldn't parse your first question, but let's try this one :)

I don't know what your tilemap is. It seems to differ from my idea, I'll try to define mine:

A tilesheet is normally a single image, with several independent graphics, which are displayed at different occasions or locations.

It could be for example all ground tile graphics, or all frames in a single animation.

In a world that is divided in small regular areas (rectangular, or hexagon) each area is called a tile. A tilemap contains information about each tile in the world, typically just enough data to decide what to draw at the tile. (That is, it contains numbers what to draw rather than the graphics themselves.)

Graphics of a tile are almost never for a complete single tile. Commonly, there are several images stacked on top of each other within a single tile. For example, you have graphics for the ground plate (eg one for sand, one for grass, one for forest ground, and so on), one for tracks on the ground, one for a conveyor belt, and one for each type of goods on the conveyor belt (and the latter two being animated, you have a frame-sequence of those), and one for wires above the ground.

The trick with splitting the graphics in layers is that the number of available combinations grows very quickly. It makes you can have a large variety of tiles with only a few graphics.

For drawing, you load all graphics into memory (video card, nowadays). When drawing the world, you examine the tilemap (to decide what you need to draw), and then draw each layer from the bottom, upwards (eg draw wires after the ground, to make the wires visible above the ground).

You can fully draw one tile, and then the next, and the next, ... and so on, or you can do the bottom layer for the entire screen, then the second to bottom layer, and so on. Any combination of these is also possible. In particular when you have graphics that partly overlap in a neighbouring tile, drawing order matters a lot.

"drawing a single graphics" is commonly called "blitting". It's like copying, but since you're copying (part of) an image in 2D, specialized techniques exist to do that efficiently.

This blitting works (sort of) directly from the tilesheet onto the display, you never load new images and destroy them afterwards for a single tile, it's a straight copy.

*snip*

That's not quite what I'm talking about, but thanks for trying. smile.png As said, I'll probably need to read more up on DOD and other things in order to find a better way to word myself. Preferably test it myself and see if it's even computable without being redundant.

To be more clear, that's my way of saying that this thread can be locked.


To be more clear, that's my way of saying that this thread can be locked.

We don't do that on these forums.

It seems like your general question are about how one goes about simulating and rendering a world that is too large to fit in memory? Factorio is probably a bad example, since they explicitly state that the entire discovered world must fit in memory.

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

This topic is closed to new replies.

Advertisement