Sign in to follow this  

Streaming textures

This topic is 4756 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm working on a game where I need to be loading textures as the player comes near stuff he has not yet loaded. Each object has several textures and there is about a .7 second pause when loading. As I see it I have a few options, and just want to get opinions on them. Option 1) Every frame read a limited amount of data. I'm not liking this too much because it is dependent on their frame rate. Option 2) Using threads create a new gl context and share the texture information and use a thread to load, this will not stop the game just slow it. Any suggestions or comments?

Share this post


Link to post
Share on other sites
Where is the delay mostly coming from - loading the data off disk or creating a texture with it? You should be able to upload quite a bit of texture data every frame. It's quite possible to copy the whole framebuffer to a texture, mess with it on the CPU, and put it back and still have a 2-digit FPS (I'm so glad I have a card that supports shaders now...). If it's the disk access, you could have a background thread load the image data, and create a few textures every frame. In fact, you could probably just keep everything in system RAM all the time anyway, unless you have a *big* game world.

One important thing is to anticipate texture loading requirements somewhat. The sooner you start loading data before the object is seen, the more you can spread the process out (of course, you'll also get more false hits where the user never actually gets to the object)

Share this post


Link to post
Share on other sites
Another Thing you could do would be dynamically adjust a texture's priority as the player nears/leaves it
that will tell openGL which texes you want in video memory and which ones yuo dont care about as much, (but you can still access and render with cause they are in the openGL runtime system memory instead
)

Share this post


Link to post
Share on other sites
I have a BIG world with far to many textures to keep loaded at once so I will have to have them load/unload as the players move around.

Share this post


Link to post
Share on other sites
Are any of these textures shared between objects?

For example, say two objects that are exactly the same and use the exact same textures, are you giving each object its own texture, or are they sharing them?

Or in other words, are you loading textures that are allready loaded?

Share this post


Link to post
Share on other sites
No textures are only being loaded once for the first instance of an object, I have a textureManager taking care of that.

Share this post


Link to post
Share on other sites
I did that a year ago with OpenGL, the idea was to let the game start and keep the textures loading (texture loading needed 30sec (~150MB, getting compressed to DXT1 1:6 ->~25 MB), rest of the game data needed only 1-2sec). the 2 main problems were (when using threading): you have to allow the other thread to load and select textures (handle stuff) and the other main problem was the incredible amount of lag in the other game-thread! its not like you only get 5fps or something, the engine did still run with around 100fps. the problem is that you get very bad lag when loading lot of small textures (this means you have maybe 20 frames with each 10ms, but then 1 frame with 150ms, then 15 good again, etc.). There is no way to find out when this lags are happening and this was just getting to mad for me and I skipped the whole idea (everything was loaded at startup and worked fine, just little load time when starting the game).

Good luck with your big world ^^

Share this post


Link to post
Share on other sites
I've seen some games where you get only a short loading time at the beginning (2 or 3 seconds) and then you can visit the whole world without waiting anymore.
I can't believe these games do really load everything at sturtup, so they must be doing it at runtime.
How do you think they menaged to do such a thing?
(just think of "Prince of Persia: sands of time" or "warrior within", or even of an older game like "Soul reaver: legacy of Kain")

Share this post


Link to post
Share on other sites
Store the texture twice - once as 16x16 (or whatever minisize),
once with fullsize (2048x2048 :p ).
Now if you enter a new part of the world, start streaming the
big texture (because it can be expected you'll need it soon).

If the textureManager actually gets a request for this texture
(and streaming didn't yet finish), then do an immediate load on
the miniversion...
After the streaming finishes, just let the manager return the big
version and discard the small one (or keep for caching as long
as you have enough place).

Share this post


Link to post
Share on other sites
Quote:
Original post by Kitt3n
Store the texture twice - once as 16x16 (or whatever minisize),
once with fullsize (2048x2048 :p ).

That's a good idea but I already tried it and I can say it's not so easy to get it running completely well.
The problem is that getting the 16*16 texture needs some processing and this takes a load of time. Halving a 8k*4k texture takes alot of time.
This could be improved by having cached thumbnails but this is not always possible.

As for the need to swap textures in/out of server memory, are we sure we really need that? Some issues with this:
1- I hardly believe the driver doesn't have a way to realize this (and it can move bytes much faster than us).
2- It could be a problem to understand when a texture is useful or not.
3- Considering we all have to develop 'the whole thing' are we really sure to make a super-texture-loading-mechanism when we still need a ton of other features?

Share this post


Link to post
Share on other sites
Hmm,

Metroid Prime has a really nice loading mechanism (or so I find it to be). The world is divided into several areas, each divided into several rooms. When the player enters an area, the game starts loading data - you can hear the disc reader go nuts. When a player is in a room, the game loads the needed data for the rooms connected to the room.

On the other side, the idea has some drawbacks. IE, when aroom is still loading while you try to enter it, the gate won't open until you do. I think they should actually put a small Loading message somewhere near the bottom of the screen to indicate that the game is loading.

You could implement the idea of 'rooms' into an open world too. Divide the map in areas, and when a player enters an area, there are 16x16 texture loaded as recommended before, and the full texture data of the current logical room is loaded. The game starts loading the near 'rooms' in the background, and when the player passes the logical room border, then the new room is entered, and the rooms around the new room start loading. Whenever the player tries to enter a room that's not fully loaded yet, either use a loading message or let the player enter the room with the 16x16 textures and give giant priority to the loading :)

Forgive me if this is absolute nonsense, because I have never wrote such a thing before - it's just a theory.

Share this post


Link to post
Share on other sites
Quote:
Original post by Krohm
The problem is that getting the 16*16 texture needs some processing and this takes a load of time. Halving a 8k*4k texture takes alot of time.

I don't quite understand this part. Halo 2 uses a similar approach (note the texture popping artifacts) and I can't see how pre-processing wouldn't be possible for the mini-textures. You just pre-process the textures at install-time and have them available ad-hoc during load-/runtime. For textures as small as this you won't even need mip-map levels (or maybe just two).
Anything that can be pre-processed should be pre-processed as soon as possible (e.g. for distributed media such as DVDs you can store the thumbnails on the media, for online distribution I'd recommend install-time pre-processing to reduce download size).

Even for dynamically generated content (such as procedural textures) you can still create the small version (maybe even at a lower quality for further speed-up) at a very low performance overhead.

Regards,
Pat.

Share this post


Link to post
Share on other sites

>That's a good idea but I already tried it and I can say it's not so
>easy to get it running completely well.
Hehe, I agree on that - in my implementation I had some nasty
threading problems, but I got them under control ;)


>...and I can't see how pre-processing wouldn't be possible for the mini-

Exactly - I meant to store the texture twice in your archive,
once in full size, once preprocessed as a second file in 16x16
(I didn't mean to load the 8kx8k texture and then convert it
to 16x16 - why would you, after all if the big texture is
available anyway).

>2- It could be a problem to understand when a texture is useful or not.
Well, that's a standard caching problem - if your cache is 10gb, then
let's put evth in it - pretty easy :)
If your cache is 100mb and you have to cache 500mb then decide what is
important (no matter what, you'll always have a speed/visual quality
penalty). If there can be 300mb of textures visible at the same time,
then maybe consider dropping texture-size a bit - or smartly re-use
textures more often...

>3- Considering we all have to develop 'the whole thing' are we really sure
Obviously I don't know how much time you have/are willing to invest in it -
I suggest to make good interfaces (spend a little extra time on this) and
get sth basic running...
if you have time then optimize it (if your interface is good you can change
the underlying implementation without to much problems).

@Sijmen:
Your suggestion makes perfect sense. In his world-case he would divide
it into sectors (rooms) and each sector has 9 surround ones. So take the
closest 3 and if you have cachespace available, start loading it already
(and set priority to low or sth - when you get closer, or if the texture
is actually requested, make the prio higher).

Share this post


Link to post
Share on other sites
I must say I like pretty much how this thread evolved because a serious topic has been issued. This is the interaction between content and code.

Some people have been using console games as example. Console games are examples of how well integration between code and content can raise the whole product quality without significant effort.
Counter-example of the "metroid": say your artist doesn't like rooms. Quake2 has a a thing called "areaportal entity". It has been used to clip geometry. While some people have correctly understood its importance (see SPoG - Ascension) some other artists does not place areaportals at all even now. There are some times in which this approach is pretty hard to apply. For production phases, that's quite easy but when it comes to the wild net, no one knows what will happen.
Counter-example of the "thumbnail": say you have to provide thumbnails or the program. This would be seen by the community as an hack. Historically, I don't remember of a single commercial program wich lived with that.

As a side note, I post this as "line of thinking" ideas, I don't really feel the need for an advanced preview mechanism like this, I have more important components on my schedule.

Share this post


Link to post
Share on other sites

This topic is 4756 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this