Sign in to follow this  
skauert

Custom binary model format

Recommended Posts

Hello guys.

After doing a lot of research on what people tend to do with their model assets and the likes, I decided that what I want to do for my game engine is to create a new tool that allows me to convert certain formats (such as COLLADA and OBJ) to my own binary format that will be used by my engine. This seems to make the most sense for both convenience and performance, however I am more or less in the dark when it comes to this because it's all very new to me. For this reason, I would greatly appreciate if I could get some input here to push me in the right direction.

OBJs have worked so far for me. The only reason I'm doing this now is because I'm looking into doing skeletal animation (from Maya). I'll probably end up going with COLLADA, since MD5 is apparently not very supported by Assimp, which is what I'm using to import model/scene data into my converter tool. While reading this, please assume that I'm looking to do this in one of the simplest ways possible. I do not wish to extend this too much, as it will become very time demanding - which I'm rather short on at the moment. :-(

Anyway, enough ranting.

This is the current format I have thought out for my binary file. At the moment it's very basic, and I have no doubt that it's missing stuff.

[code]
<magic_number>
<uint:meshLocation>
<uint:materialLocation>
<uint:textureLocation>
<uint:skeletonLocation>
<uint:animationLocation>
<data:mesh>
<data:material>
<data:texture>
<data:skeleton>
<data:animation>
[/code]

The location header data would simply be 0 for data that isn't used. When something is non-zero, the importer would go to that location and start reading data. I was thinking of simply putting in tightly packed floating point values, for example "vX,vY,vZ,nX,nY,nZ,tS,tT" for mesh vertex data.

Since I have never done animation before, I do not know if this approach is viable for skeletal and animation data.

Every little piece of advice is helpful! This area of programming has probably bewildered me the most out of all of them until now.

Share this post


Link to post
Share on other sites
There are a few things to [url="http://lspiroengine.com/?p=36"]consider[/url] if you want your files to be as small as possible.
There is certainly no need to store normals with 96 bits, for example.


As for the rest, it is entirely up to you. I have never had a need to create a dictionary for my formats, but it is certainly a valid approach.


L. Spiro

Share this post


Link to post
Share on other sites
Hey! Thanks for the response.

By dictionary, I assume you mean having a header which contains locations of specific data. Is that correct?

I thought a bit about it, and I came up with this way of doing it:

[code]
<magic_number>
<mesh_identifier>
<num_vertices>
<vertex_data>
<material_identifier>
<sub_material_identifier #1>
<material_data>
<sub_material_identifier #2>
<material_data>
...
<texture_identifier>
<texture_dimensions>
<texture_data>
<skeleton_identifier>
...
<animation_identifier>
...
[/code]

I guess it looks a tiny bit like XML. This is not intended, and is only for clarity in this example.

[quote]As for the rest, it is entirely up to you.[/quote]
Yeah, that's the essence of creating your own model format. I am however (in my opinion) enough in the dark when it comes to this that I'm looking for some input on whether I'm actually going in the right direction or not. One of the main issues I'm trying to work around with these binary files is how to design it so that data can optionally be present or not. If all data was required to be there all the time, this wouldn't be a problem, but alas. The other is obviously the fact that I have not done skeletal animation before, so some of the approaches I'm doing with e.g. the mesh might be invalid in this case. :/

I read your article, by the way. Very interesting, and definitely something I will consider in the future. For now, space is not a very big issue, as I'm mainly focusing on getting something -working- for my upcoming projects (which are all PC/Windows projects), which shouldn't be that heavy on models either.

Share this post


Link to post
Share on other sites
By dictionary I mean having offsets in the header pointing to parts of the file.
Again, nothing wrong with it, but I haven’t had the need.
I load my files linearly, and if something is optional, such as a skeleton or embedded textures, I simply write a byte 0 or 1 indicating its absence or presence.

You don’t have to think too hard about it because the format from which you are converting will likely guide you considerably since your format will basically be the same, but only include the data you want to use.
And if you find out you want to store more data, less data, or the same data differently you can always revise your format as you are developing your loader and engine.

One thing you do need in the header, however, is a version number. This will be passed down to your loading routines so that as you upgrade your format, your loader can avoid trying to load data that is absent in older versions of your file.


L. Spiro

Share this post


Link to post
Share on other sites
We split our files up at work, so for example a simple model would have a file for vertex data, a file for index data, a 'meta file' and a material file it would reference; the material file then references textures

The meta file is what glues everything together (basically dependancies + counts) while the vertex and index data can be directly loaded into vram.

(The various files are then bundled up together in one file for loading/dependancy reasons)

You really don't need much more than that.

Share this post


Link to post
Share on other sites
Hey guys.

Sorry for not replying right away. I spent yesterday and all of today working on my converter.

@phantom: Your way sounds very clean. If I were doing a bigger project, I would probably look into something like it. My current approach isn't very optimized at all, but the fact that my project is so small makes it not really that bad. ;o I also feel like I'm learning a bit more by doing it in a very straight-forward way -- but maybe I'm just imagining things.

Right now, I've gotten it to a point where I use Assimp in my converter tool to import any kind of format it supports, and then I export the data into a binary file with the following format:

[code]
<magic_number>
<mesh_identifier> // Can appear multiple times per model
<material_id>
<num_vertices>
<vertex_data>
<vX,vY,vZ>
<nX,nY,nZ>
<tS,tT>
<num_bones>
<bone_data>
<bone_id>
<weight>
<material_identifier> // Can appear multiple times per model
<mat_data>
<id>
<ambient:R,G,B>
<diffuse:R,G,B>
<specular:R,G,B>
<shininess:float>
<texture_strlen>
<texture_string>
<bump_strlen>
<bump_string>
<normal_strlen>
<normal_string>
<skeleton_identifier>
<num_bones>
<bone_data>
<id>
<transform:mat4>
<animation_identifier>
...
[/code]

I've only tested it with a sample OBJ file I downloaded from gfx-3d-model.com ([url="http://www.gfx-3d-model.com/2010/04/3d-character-model-3/#more-1075"]http://www.gfx-3d-mo...el-3/#more-1075[/url]), and it seems to render it just fine. This means that mesh and material data is working as it should. Your input has been very valuable in all of this. Thanks! ^_^

The only thing remaining is skeleton and animation data. This is a project for tomorrow and onwards. As you can see, I've barely started the design of how I want skeleton/animation data to be stored. From what I've learned when reading up on these topics, I feel like the following -should- work (in theory):

- The skeleton will start out with a root node, and each node will have a list of child nodes, as well as a rotation matrix. When animating, the rotation matrices will be modified recursively in respect of their parents' matrix.
- Each vertex will contain a list of bones that they are connected to, with weights affecting exactly how big of an effect it is. In the vertex shader, I will simply multiply gl_Position with the matrix of each bone, in addition to the mvp matrix.
- As for animation... I have to admit I have not read up very much about that yet. :(

It's a little bit off-topic, but if there are some tips in regards of skeletons/animation that you guys can provide, I'd love to hear it! :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this