Jump to content

  • Log In with Google      Sign In   
  • Create Account


File for storing meshes


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
11 replies to this topic

#1 Xperience   Members   -  Reputation: 879

Like
0Likes
Like

Posted 18 March 2014 - 09:46 AM

Hi, I have a problem with choosing good file format for storing geometry data. Until now, I've used m3d format(created by Frank Luna), but it is not suitable for using in games(As example to learn is perfect).

I want to find format which can do this:

1. Loading from memory(I have a Resource Cache system in which I load every needed file in single call and then I work with memory)

2. Fast

3. Easy importing from Graphics programs(3DS Max, Maya etc)

I found FBX, but I don't know if it can be loaded from memory. Thank you for your opinions(I'm using C++ and D3D11).



Sponsor:

#2 Buckeye   Crossbones+   -  Reputation: 4593

Like
5Likes
Like

Posted 18 March 2014 - 11:43 AM

If you're looking to store data for later loading into your app from memory, I'd suggest a custom binary format. I.e., create a "file converter" to load from 3DS Max, Maya, or other format, and save to a binary format to be loaded by your app.

 

I have a 2-step process:

 

1. Create a file loader for a particular input format (DirectX, 3DS Max, etc.) and load the data into a custom (used for all input formats) data structure. Either store the custom data to disk for later conversion, or pass the structure to your file saver.

2. Create a file saver for your custom binary format, using the data from the custom data structure.

 

That allows you the flexibility to write file loaders for any new input format you like, requiring only that you load into the custom data structure. Also, knowing ahead of time what data is required by your app, you can check at conversion time to see if the input file has the data you need.

 

That also allows you to revise just the loader if the input format changes. And, if you need to change your binary format, you revise only the saver.

 

With binary formatted data on disk, loading data is much faster, and you don't have to revise your app if you need to revise the loader.


Edited by Buckeye, 18 March 2014 - 11:53 AM.

Please don't PM me with questions. Post them in the forums for everyone's benefit, and I can embarrass myself publicly.


#3 TheChubu   Crossbones+   -  Reputation: 4192

Like
1Likes
Like

Posted 18 March 2014 - 03:10 PM


m3d format(created by Frank Luna)
btw, either he was doing OpenGL in that time and working at Id, or you're mistaken. http://en.wikipedia.org/wiki/MD3_%28file_format%29 It was made by Id for Quake 3 (or better said, for Id Tech 3).

"I AM ZE EMPRAH OPENGL 3.3 THE CORE, I DEMAND FROM THEE ZE SHADERZ AND MATRIXEZ"

 

My journals: dustArtemis ECS framework and Making a Terrain Generator


#4 Buckeye   Crossbones+   -  Reputation: 4593

Like
2Likes
Like

Posted 18 March 2014 - 08:19 PM

You read too quickly: Frank Luna made up the m3d ("model 3d") format for his D3D10/11 books. Id used the MD3 format. blink.png


Edited by Buckeye, 18 March 2014 - 08:25 PM.

Please don't PM me with questions. Post them in the forums for everyone's benefit, and I can embarrass myself publicly.


#5 Xperience   Members   -  Reputation: 879

Like
0Likes
Like

Posted 19 March 2014 - 11:06 AM

No, it isn't my mistake. It is m3d as pointed Buckeye.

Thank you for advice. I'm thought that this is only option, but I didn't know if it is good option(common for  used).

I have a small problem with creating my own binary format(rather with storing and loading strings)

First I tried to save and load whole object at once, but it failed. I found, that error is inside loading string.

 

Now I'm saving and loading strings as:
1. I save a size of string
2. I save a string

But isn't there a better way to do this?



#6 TheChubu   Crossbones+   -  Reputation: 4192

Like
0Likes
Like

Posted 19 March 2014 - 01:37 PM

You read too quickly: Frank Luna made up the m3d ("model 3d") format for his D3D10/11 books. Id used the MD3 format. blink.png

Ohh, thanks for the heads up.


But isn't there a better way to do this?

Save direct values. Not strings. Values don't need to be parsed.

 

Your own format would be defined like:

 

byte position where vertex data starts.

byte position where normal data starts.

vertex data (which is just triplets of floats)

vertex data ends.

normal data starts (again, triplets of floats)

normal data ends.

etc.

 

And the whole format header shenanigans that goes beyond just putting byte positions for sections.

 

So when you read a file, you know the first 4 bytes (an int) are the position where vertex data starts, the following 4 bytes are the position where normal data starts (again, an int), and then you can read until you reach the vertex data position, and you know that for each 4 bytes you got a float, and for each 3 floats you got a vertex, and so on.

 

You could read that link to Id's MD3 format. It might give you some idea how binary formats work, how they store data, how a header looks, etc.


"I AM ZE EMPRAH OPENGL 3.3 THE CORE, I DEMAND FROM THEE ZE SHADERZ AND MATRIXEZ"

 

My journals: dustArtemis ECS framework and Making a Terrain Generator


#7 Buckeye   Crossbones+   -  Reputation: 4593

Like
1Likes
Like

Posted 19 March 2014 - 04:51 PM

Strings are a pain, but sometimes you have to save/load them. However, they are, in fact, similar to other arrays. E.g., you read in an int to get the number of vertices, create a buffer, then read in numVertices*3*sizeof(float). Strings you read in an int to get the number of characters, create a buffer (sized for 1 extra character), then read in that many characters. NOTE, however, if you're using null-ended string, the buffer into which you read a string should leave space for that terminating null. That's the "1 extra character." Don't forget to add that terminating null.


Please don't PM me with questions. Post them in the forums for everyone's benefit, and I can embarrass myself publicly.


#8 spazzarama   Members   -  Reputation: 712

Like
1Likes
Like

Posted 19 March 2014 - 06:12 PM

Depending on your requirements and the flexibility you desire you could use the Visual Studio 2012+ graphics content pipeline .CMO format (which is created at build time from a .FBX etc). I have used this previously with Blender exporting to .FBX.

 

DirectXTK has a reader for the .cmo file format.


Justin Stenning | Blog | Book - Direct3D Rendering Cookbook (using C# and SharpDX)

Projects: Direct3D Hook, EasyHookAfterglow, C#raft

@spazzarama

 

#9 Xperience   Members   -  Reputation: 879

Like
0Likes
Like

Posted 20 March 2014 - 09:43 AM

 

You read too quickly: Frank Luna made up the m3d ("model 3d") format for his D3D10/11 books. Id used the MD3 format. blink.png

Ohh, thanks for the heads up.


But isn't there a better way to do this?

Save direct values. Not strings. Values don't need to be parsed.

 

Your own format would be defined like:

 

byte position where vertex data starts.

byte position where normal data starts.

vertex data (which is just triplets of floats)

vertex data ends.

normal data starts (again, triplets of floats)

normal data ends.

etc.

 

And the whole format header shenanigans that goes beyond just putting byte positions for sections.

 

So when you read a file, you know the first 4 bytes (an int) are the position where vertex data starts, the following 4 bytes are the position where normal data starts (again, an int), and then you can read until you reach the vertex data position, and you know that for each 4 bytes you got a float, and for each 3 floats you got a vertex, and so on.

 

You could read that link to Id's MD3 format. It might give you some idea how binary formats work, how they store data, how a header looks, etc.

 

 

Sorry, but thanks for advice. I probably didnt' explain it enough. I know, how I have to store Vertices and Indices(I know a basic principles about storing data inside binary format).

I mean using strings(saved binary) to indicate which texture want mesh to use, or which effect I can apply on this mesh(or something like that).

 

Strings are a pain, but sometimes you have to save/load them. However, they are, in fact, similar to other arrays. E.g., you read in an int to get the number of vertices, create a buffer, then read in numVertices*3*sizeof(float). Strings you read in an int to get the number of characters, create a buffer (sized for 1 extra character), then read in that many characters. NOTE, however, if you're using null-ended string, the buffer into which you read a string should leave space for that terminating null. That's the "1 extra character." Don't forget to add that terminating null.

 

I do it this way, but I didn't know nothing about null-ended string(Thank you for explain). Is better to use it or miss it? If I saved string like that:

fout.write(str.c_str(), str.size());

is there saved string with null at the end?

 

Depending on your requirements and the flexibility you desire you could use the Visual Studio 2012+ graphics content pipeline .CMO format (which is created at build time from a .FBX etc). I have used this previously with Blender exporting to .FBX.

 

DirectXTK has a reader for the .cmo file format.

 

If I use it, will slow it the build time?


Edited by Xperience, 20 March 2014 - 09:54 AM.


#10 Buckeye   Crossbones+   -  Reputation: 4593

Like
1Likes
Like

Posted 20 March 2014 - 09:57 AM


I do it this way, but I didn't know nothing about null-ended string(Thank you for explain). Is better to use it or miss it? If I saved string like that:

fout.write(str.c_str(), str.size());

is there saved string with null at the end?
 

First: it appears you're writing the string and then the size. Write the size first, so you'll know how many characters to read in. Whenever you write something to be read later, think about how you'll need to read it back in, and what information you'll need to do that.

 

I don't know what your fout function does, so I can't tell you if it stores the terminating null. What does that output look like in the file?

 

You don't have to write out the terminating null. You just need to know whether it's there when you read it in. Probably better to NOT write the terminating null. When you read the string in (read the string length; read in string-length characters), you don't have to worry about whether there's another byte after it or not.


Edited by Buckeye, 20 March 2014 - 10:05 AM.

Please don't PM me with questions. Post them in the forums for everyone's benefit, and I can embarrass myself publicly.


#11 Xperience   Members   -  Reputation: 879

Like
0Likes
Like

Posted 20 March 2014 - 11:24 AM

 

First: it appears you're writing the string and then the size. Write the size first, so you'll know how many characters to read in. Whenever you write something to be read later, think about how you'll need to read it back in, and what information you'll need to do that.

 

 

Now I'm saving and loading strings as:
1. I save a size of string
2. I save a string

But isn't there a better way to do this?

 

 

 

I don't know what your fout function does, so I can't tell you if it stores the terminating null. What does that output look like in the file?

 

 

I forgot add this(it's simply std::ofstream call):

std::ofstream fout(Filename.c_str(), std::ios_base::binary);
fout.write(str.c_str(), str.size());


#12 TheChubu   Crossbones+   -  Reputation: 4192

Like
1Likes
Like

Posted 20 March 2014 - 01:07 PM


I probably didnt' explain it enough. I know, how I have to store Vertices and Indices(I know a basic principles about storing data inside binary format).

I mean using strings(saved binary) to indicate which texture want mesh to use, or which effect I can apply on this mesh(or something like that).
Ohh yeah, well that's a place where I'm still thinking what to do. I (and probably you) have several options:

 

Serializing the objects directly (with material/mesh data). Bethesda Game Studios games (TES, Fallout) for example serialize .nif files, which are entire scene graph nodes, I'm guessing that at load time they just load the .nif file and plug the data directly in their scene graph.

Storing the mesh data as binary and the material data as an additional file (text, other binary, with markup language, etc).

Storing everything in a single YAML/XML file, maybe compressed with zip/gzip.

You can also index the materials so the files only store mesh data + material indices instead of storing the string name of the material (this one would need to be maintained but its probably the fastest to load).

You could also hash the texture file names and store the hash along the mesh data, that way when you load the mesh you could pass the hash to a texture/asset loader, and it will work out which texture path the hash relates to.

 

There are many options and tradeoffs between ease of use and faster processing.


"I AM ZE EMPRAH OPENGL 3.3 THE CORE, I DEMAND FROM THEE ZE SHADERZ AND MATRIXEZ"

 

My journals: dustArtemis ECS framework and Making a Terrain Generator





Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS