Jump to content
  • Advertisement
Sign in to follow this  
rouncED

streaming point cloud

This topic is 3077 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Ive got this streaming point cloud in the form of a large landscape taking up 10's of gigs of hard disk space, right.

Heres a pic of it, the data isnt very interesting at the moment, its just a plane with a few outrusions in the distance and random point colours.
(note the pixels are all drawn with billboards in the dx10 geometry shader.)


Its rastering quite fine, as I pick a correct mip of level of detail, im getting around 30 fps on my 8800gt.

The only problem is its not streaming fast enough, I figure its not the hard disk load, but its the send to the video card ram thats slowing it down, im sending full 24 byte [edit 12 byte sorry] floating point positions for each point, is there any other way around this?

Note, this is similar to sparse voxel octrees, except I think my streaming capability is ruined by the fact I have to send whole floating point positions for each point.

Would there be any way to speed this up? Like at the moment, it is working, but its more of a experiment than a game at this stage, right now ive just formulated some tools to get the level looking cool (like boolean objects and marching cubes), its just whats the point in implementing it when its streaming cant keep up with the camera movement...

[Edited by - rouncED on July 13, 2010 9:16:04 AM]

Share this post


Link to post
Share on other sites
Advertisement
Quote:

I figure its not the hard disk load,

How do you figure? HDDs are SLOW compared to ram, and SLOW compared to the PCI bus. Your looking at the difference between GB/s once the data is loaded, and at most a couple hundred MB/s while reading the disk (way less if you have a mechanical HDD, and are seeking all over the place).

Share this post


Link to post
Share on other sites
Yes, I'm not sure why you're ruling out HDD I/O as the bottleneck. How is the data laid out in the file? Are the points continous (ie grouped together) or are they just randomly spread out all over the file (this will cause massive amounts of seeks).

Also, defrag your hard disk if you haven't already!

Finally, are you compressing the data? This will help with HDD read times immensely.

Share this post


Link to post
Share on other sites
If you haven't, yeah do some profiling! find out exactly where your time is going and then you can start to optimize towards it.

If HD reading really is the slow part, like the other guys said, compressing the data could help since it gets more data per read (at the cost of cpu doing more work on the data to make it useable).

I was working on an open world (streaming) AAA game last year (no longer there) and the big hurdle to a streaming world that we hit was not the disk IO time, but the time spent on handling the data after it was loaded. So that kind of points towards what you were thinking, that it might not be HD speed.

But yeah, you should profile and make sure!

If it is indeed the data sent to the video card maybe you can figure out another format, like for instance instead of sending a bunch of verts individually to the video card, maybe you can do something like send a heightmap image of your data to the video card and a vertex shader can look at the texture for it's geometry (basically just height at that point!). You could cut it down to 1 to 4 bytes per vertex then instead of 24 bytes. should free up some of that data bandwidth!

Also, if it is just in general not possible for you to stream your world fast enough for one reason or another, you can always fudge it by decreasing your draw distance!

You could maybe even do something like have a fog that moves in and out based on the status of streaming.

IE as you sit in one spot and do nothing longer, it streams more and pushes the fog back so that more data can be shown as it's available.

Oh and one last thing, you can have high res and low res versions of the data (possibly more data storage so not sure if you are interested...) and load in the low rest stuff first, then the high res stuff as you can. Modern games do this quite a bit.

If you are worried about the pop from how the low res data looks to how the high res data looks, you can lerp the verts over time from the low res data to the high res so it morphs to the new data, making a pop a lot less noticeable.

Hope theres something useful here for you (:

Share this post


Link to post
Share on other sites
Quote:
Original post by Atrix256
the big hurdle to a streaming world that we hit was not the disk IO time, but the time spent on handling the data after it was loaded.


That's definitely a good point. Modern games bake/preprocess raw source assets into formats that can be loaded directly into memory/pushed directly into VRAM. This, combined with trying to reduce HDD reads/using the reads more intelligently is the key to high performance streaming.

Another good point from Atrix256 is about storing lower detail versions of your geometry - you could do a mip mapping type of thing, and progressively load higher detail "mips".

Share this post


Link to post
Share on other sites
Yes atrix! you are right! I could pull in a positionmap of compressed data as a texture then use the geometry shader to rebuild the full uncompressed positions!

Ill have a hard think about it.

But all you guys that said the hard disk had something to do with it too, im listening ill be making sure thats not the case.

[Edited by - rouncED on July 13, 2010 6:29:40 AM]

Share this post


Link to post
Share on other sites

If you are serious about this get some real data from a referenced site like scan data or similar. Like that people can compare and see if what is done in litterature already solved this 10 times over ;-) If you however prefer to work in the dark without any references and not being able to compare to others, then be my guest.

Edited by jbadams
Restored post contents from history.

Share this post


Link to post
Share on other sites
I can still compare it to other peoples work, and yes, mine does stream slower. The data is irrelevant, but it would be nice for screenshots yes.
Anyway, if the idea Atrix gave me works, I should be able to reduce data sent on the hard disk and video memory send just for good measure... I should have it reduced by 90% (224 bits to just one a8r8g8b8 texture, each texel is voxel point data for a single point) believe it or not, so ill see what it goes like after I modify it. (its a bit of a bad recode tho.)

Mostly thanks to Atrix256 tho, your enthusiasm is catching. ;)

But SpinningCube, I definitely should have a look for some pre made voxel scenes, it definitely would make my life easier.

Share this post


Link to post
Share on other sites
Quote:Original post by rouncED But SpinningCube, I definitely should have a look for some pre made voxel scenes, it definitely would make my life easier.

And improve visual quality ehmmm ;-)

Edited by jbadams
Restored post contents from history.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!