Archived

This topic is now archived and is closed to further replies.

pym

Dynamic loading of 3D world

Recommended Posts

Here is the situation : I have a 3D game engine that works quite well (physic engine + graphic engine + sound engine). And I use it in a game... but I wi*ould like to optimise the usage of the memory because it is quite basic for the moment. Here is the axact problem : For the moment fot each level I load the whole level at the begining and then, acording to the situaton, I render the scene. Of course most of the data of the world are useless for the rendering (for exemple the one in my back ). So it would be a good idea to have a DYNAMIC LOAD of the world... it would be really nice ! So does anyone has ever had that problem/idea before ? And do you have any idea of the way to manage such thing : what should be load/unload ? things like that.... Merci d''avance ! (yes I am French ) PYM

Share this post


Link to post
Share on other sites
For the moment no reply... but I have some new data so I post them !

In fact the game looks like... Mid Town Maddness I think... you drive in big world in a car ! And the world is in fact TOO big to be loaded in one piece (for the moment it works like that because the beta levels are quite small) but I can''t stop the game with "now loading" screens all the time !

So the idea is maybe to divide each level in square that could be loaded when it is needed... a kind of streaming loading

Is it realy a good idea ?
When should I load/unload a piece of level ?
How to make a good streamloading ???

Here are the questions.....



PYM

Share this post


Link to post
Share on other sites
I read about this problem in another forum I think. I think that the consensous was that you split your track into "sectors" and load enough ahead, behind and to the sides to allow the game to see everything around it.

When you advance out of one sector to the next, the one "behind" can be freed and the next loaded into memory.


Oli



All the best problems start with C

The Most Boring Website Ever?

Share this post


Link to post
Share on other sites
Thanks for your reply !

What do you mean by "memory mapped file" ?

But I think idea of loading the "near" parts of track is a good start... but if I move fast ? I should load very quickly le new part... maybe I could load all the parts that I could see if I drive has a mad hell''s angel . In fact the maximum speed of my car is perhaps important ?...

What do you think of that ?

PS : "I read about this problem in another forum I think." which ones ? it could help me a lot



PYM

Share this post


Link to post
Share on other sites
[link]http://www.gamedev.net/community/forums/topic.asp?topic_id=78575[/link]

That was one of the posts on here.

A memory mapped file is basically memory... mapped into a.... file

You can toy with it as if it were just standard memory, but it''s physical location is on a disc. If your map (and vetices, etc) is (are) huge then this could help?

I dunno.



Oli



All the best problems start with C

The Most Boring Website Ever?

Share this post


Link to post
Share on other sites
quote:
Original post by Simagery
Actually, a memory mapped file is like a file... mapped into... memory! You can perform standard file operations, but the file actually is stored in memory, not on disc.


Well, that's not entirely true, it's more the other way around. What you say is that a memory mapped file is basically just memory that acts like a file. It's not that simple.

Basically, a memory mapped file is a real file (you actually map an existing file), which can be bigger than the amount of RAM you have on your system. When you call the APIs to map the file, it actually gives you back a memory pointer. So, say you create a memory mapped file of 1GB, but you only have 128MB. You can access that memory pointer as if it's the full 1GB, like "real" memory. If you try to access a part of the memory (file) that is not in memory, the system will automatically discard rarely used parts of the file, and load (swap) in the parts you request (in 4K blocks I believe). It uses buffering, etc, so it is pretty efficient. This is exactly how virtual memory on your system works. There is a physical swap file on your drive.

A memory mapped file is great for vast amounts of data, but you have to make sure you keep things that will be loaded together in the same area of the file, so not to read from too many parts of the file at once, which would be a performance hit. I have used memory mapped files for other things in the past, but not for levels. Eventually I will use them for levels too.

Read up in MSDN on the CreateFileMapping and MapViewOfFile APIs.

SS



Edited by - Axter on February 8, 2002 2:48:49 AM

Share this post


Link to post
Share on other sites
Thanks

Now I have an other problem (that mapped file can''t solve I think) I can''t use swaping because in fact the game will be ported on console (I hope St Developper pray for me). And consoles (except XBOX) have no HD... no HD means no swaping... gasp !

I do not want to do a level management system unportable... so I am looking for some kind of idea...

In fact maybe I can map a file on a cdrom or dvd... that could be an idea... or split my data files into sectors as downgraded said...

Whatdo you think of those 2 idea ?

note : while writing that post I am also reading the things about mapped files... never used it before... but seems really very iinteresting... for many usage...



PYM

Share this post


Link to post
Share on other sites
what do you mean no swapping? so your saying you cant stream off the cd/dvd/gd? this is how its done in most modern games. the trick is to make the sections overlap enough so that while your loading the new data, the player can still see stuff. just look at shenmue2, they had TONS of npcs most with different textures and models (yes even the generic ppl). shenmue''s engine handled this by fading them in. distracting yes, but effective. the world data on the other hand (also quite detailed) never had the fading problem, but you knew the dc was always loading stuff due to the drive noise. though at many points shenmue required load screens (which were not too long). this was acceptable because the chunks were large and the next area ussually used an entirly different texture set and npcs. though i am ranting abit (shenmue2 is such a grand view of the power of the dc).

now on to things to consider. in a racing game, you can do this (gta3 does it) as well, just a bit more complex. in gta3, you will notice the change form high detail from lower detail when you transition through certain blocks. its very difficult to make it completly transparant. you have to deal with the dirve seek time and read speed. also you will have to stream the data while you render. the best way would be to break up the sections by visiblity. try to have obsticales that will hide some of the next section so that while your loading it, the player wont notice. also try to orgainze your sections so that they have the data close to a section transition at the begining. you may have to duplicate your data on the disc for optimal reading. which wont be much of a problem when dealing with dvds on ps2 or gamecube.

rolling yoru own level mangemnet system will probally be faster then relying on an os level one (mainly because you KNOW what data will be needed next and can premptivly start a load when the player gets close). i suggest testing these ideas on a pc with the data on a cdrom or with file functions that artifical mimick the read speed you will have (dont make it a perfect read speed either). this will help work out the bugs.

btw when you split the data you dont have to use seperate files. just ensure that things are ordered nicely. you could theorectically stream the level at all times, and only have some problems if the player does a quick 180 (like in gta3). basically you need some priority for the data. for instance npc models behind the player, that cant harm the player, should be unloaded if the player is moving forward for a certain amount of time. this will help keep down watsed ram on things that dont need to be kept track of. now in gta3, the level never faded in too much (ie from a 180) but at times it was noticable when turing at certain sections. lower quality textures and models may be used to reduce this. ie use the lower quality stuff when you begin caching the level so you progressivly load things like a progressive jpg. low->med->full detail.

Share this post


Link to post
Share on other sites
When I said "swaping" I meant extension of the memory on a disk... just like the swaping system on the PC... What is impossible on PS2 for exemple because there is no diskdrive.

I just tried GTA3... yes it seems to work just as I''d need ! If evezr you have any link about how the load/unload is managed exactly in this game it could help...

I will start to test some few things using that idea !

Thanks for your help



PYM

Share this post


Link to post
Share on other sites
One way to stream the data is to define a system of a world, section, and block locations making up the game world. Your world is an array of the sections, your sections are an array of the blocks, and the blocks contain a reference of the data to be used. The position in the world you are currently occuping determains the sections to load, for example if you''re in section x,y you will load at least the sections which could be directly moved to from your location for 9 sections total if you have full x & y freedom (includes the one you occupy). You set up the engine to load the blocks of each section into memory, swap the memory for the sections and respective blocks as you move across into other sections. The engine takes the blocks located in memory and from them generates a virtual map for the portion of the world that could be visible (ignoring the viewing fustrum for now) based on your location. Now do your T&L, collision detection, etc on the virtual map.

Now one of the prime advantages of this method is you can scale up or down how many vertices you allow to reside in the blocks and how many blocks reside in the sections based on data transfer rates, memory, and anything else you need to account for. Also this system can do many things in abstract to reduce memory overhead, such as collision basic collision detection for things not visible using the blocks occupied to make the determaination and only loading data for NPC that is within x number of blocks from player position and/or visable.

For a more complex engine and even more flexablity you can add progressive loading as suggested by "a person". One way to do that is by having a priority assigned to vertices and textures of the blocks, your engine can fetch the high priority data first to define the basic object and increase the detail as more of the vertices and textures load.

For a basic example; You have a pillar of marble in the distance you are moving toward. As it moves into the barely becoming visable range you load vertices for 2 triangles that define the basic shape, shade them to color as the object is still only between 0.5-8 pixels max on screen (if you need less draw distance I would suggest a slow fade-in). Now once you push past a few pixles into something where texture becomes noticable, you can add simple texture to your triangles. You where loading while the basic shape was on screen right?!?! Now would be the time to flesh-out the object by adding vertices that will sub-divide your polygons for an object with depth, you need only add the ones visable from your direction and I recommend leaving vertices less importain to the look of the object until a little later. We have had time to load the detailed texture so slap it on and load any remaining vertices, as our object passes further in the opposite direction of our travel we can unload data almost in opposite order of what we loaded.

Hope that helps.

Share this post


Link to post
Share on other sites
Yes !!!!!! That exactly the loading management system I was looking for ! I just have now to integrate a dynamic way of managing the memory...

It's cool to know what and when to load something (here sections of map) but do I have enought free mem to store it ? And if I have not what should I unload ? Here is the way I see it :

look at the picture here :
the picture

I want to use a priority map : for each section around the player (here the player is the small arrow) and accordind to his direction (thats why I use an arrow ) I assign a priority for loading stuff...

Of course I have to discreetise the direction of the player (4 main directions) but I think it could be quite nice to do my memory management like that...

what do you think of that ?
Do you see any problem/improvement ?




PYM

Edited by - pym on February 12, 2002 8:50:19 AM

Share this post


Link to post
Share on other sites
I think I have a quite good thing now... in fact I explain it on the LionHead forum... here :
my post in LionHead forum
if you have any suggestions... or remark...

Oh I have a question did anyone before use such quide of "loading priority" tech ? I never read anything about it... I had the idea while reading a book about C++ STL... when I read "priority_queue" I immediatly thought to that algo... It's a kind of magic... but of course the real magicians all the people that accept to read all the looooooooong things I wrote in my 2 posts

PYM

Edited by - pym on February 19, 2002 4:58:15 AM

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Just one comment, if you are going to be streaming stuff in and out, you need to consider dynamic objects in your world that get initialised before they are actually visible, or if your game is such that you will be leaving locations and returning to them, objects that need to correctly react with the world in the interem.

Gibberish explained by example: Say you have an object that does collision detection with the world, and you go and (literally) stream the world out from under it''s feet, is it gonna get stuck somewhere rediculous when the geometry gets streamed back in, or maybe fall through the world while it''s geometry is unavailable for collsion tests?

Not a problem if your game world consists purely of static physical objects, but if you are trying to simulate a slightly more lively world it may be.

Share this post


Link to post
Share on other sites
Yes your right... it is a problem... I (the rest of the team) have some solutions : when a mobile&interacting object is in a unloaded part of the world it switch to a different way of interacting (more simple because we don''t see it )... it is the main idea...

For the moment I just focalise on static&graphic things... but I keep in mind that I will have to integrate mecanics and IA stuff...

But that''s right that I have to think of it now (to avoid doing thing that will block me then). Thanks for reminding it to me !

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Well, we had the problem that objects just ''disappeared'' out of the world, and we realised that this was the problem.. solution, make them inactive while their block is not streamed into mem.

Share this post


Link to post
Share on other sites