Archived

This topic is now archived and is closed to further replies.

Dirge

Making your own Model format, idea's, comments, concerns?

Recommended Posts

Dirge    300
Hey, this is kind of an open ended question that I want to have a small discussion about, but what do you guys think is the best model format to use. Now I don''t neccessarily mean 3ds or obj, not format in that way, but the actual structure and hierarchy of a Model for a professional game. For instance, I see a game like Need for Speed(1/2/3/4/5) or Gran Turismo (1/2/3) and we have a basic car, right? Well lets see, we of course had to load vertices and faces as well as texture coordinates and vertex colors. Perhaps we wanted to optimize a bit so we stored all vertex normals of the model in the model file (even better, maybe we used smooth groups). Now at this point we can do a few things extra. We can store which objects or faces use environment mapping to create some nice shininess effects. Add some nice material effects as well (emmisiveness, ambient/diffuse color, specular) including transparency and texture and multi-texture info. All right, now we can divide our entire model into different objects (ala 3ds format), so now we can have a front_window object as well as chasis, left_front_wheel, right_front_wheel and so on. Perhaps we even attach a special matrix and/or quaternion per object, so that when we read in our model, and object, we decide whether something is a wheel or not (by it''s name or ID), and if it is, we can animate the wheel rotation based on some form of input (AI or User). This would also make it convenient for some nice damage and explosion events (where objects fly everywhere). Now these are my thoughts so far of what make a very powerful model format. These things allow for so much flexibility and allow an artist to represent a model as best as you are willing to let them (we still want a nice framerate). So i''ve decided to make my own format following these loose guidelines based upon the 3ds file format (I like the binary chunk data implementation). It should be almost compatible with 3ds but with no plans to make it completely, although the reader of this format (which I will just call the ARM format, my initials then m for model) should be able to read 3ds files. It would expand upon the 3ds framework to add things that game developers would actually need. Perhaps even bounding volume data could be included. I also plan 2 modified versions of this (or perhaps extend this model format to these levels, but the complexity would be too high). The first modified version would allow for character animation through animated/keyframed bones. The second would allow for bsp/portal/octree/bounding volume data to be embedded within the format itself. This would be the world model format. It would contain doors and special triggers which may be activated from within the engine. So please, tell me what you think. I know there are many model formats out there already to choose from but I want to make my own, just for the learning, and because I know more than other people what I need out of my code. Please feel free to discuss the flaws in my design and please offer suggestions. Thanks!

Share this post


Link to post
Share on other sites
Scheermesje    169
I was also thinking about making my own model format and I think you''re idea sounds good

But as you said, it''s going to look like the 3ds format. But what are the big differences with this format? Are the things you want to add realle necesary?(or how you write that :D)

Share this post


Link to post
Share on other sites
Yann L    1802
There is a difference between a good general purpose 3D format (flexible and good data organization, easy to read/parse, large functionality set, etc) and a good data format for a game engine.

A really good game engine model format is one that is perfectly adjusted to the inner structures of your 3D engine. Of course it should be extensible, so a chunk like structure is good. I wouldn''t divide the geometry by object though. I would create ''entities'', means objects that can contain themselves a hierarchy and subsequent attached subobjects. Eg. a car would be an entity, but the wheels, etc would be subobjects. Geometry is explicitely only stored with each entity, in a directly streamable way that is perfectly adjusted to the memory layout you use for your geometry in your engine. I would also separate the actual geometry data from the hierarchy data of each entity: normally (in a game engine) I load the whole scene in memory at once, including all objects (I don''t like on the fly loading of models). That way, you could read in the entire geoemtry data for your level (incl. entities) with a single fread().

For the object hierarchies themselves, I would also lay them out as relocateable hiearchical trees. You could also read them in using a single fread, followed by a relocation loop. You don''t have to process every object individually.

Obviously, such a format concept is primarily optimized for raw loading speed. If you change your memory layout, you have to change your format. So it''s more something you''d do towards the end of your engine development cycle, where data structures don''t change anymore.

Oh and from personal experience: never forget to add 1) a version ID for you format, and 2) some spare unused fields in the header. This will save you lots of headaches afterwards

/ Yann

Share this post


Link to post
Share on other sites
Dirge    300
Interesting. So basically since I save hierarchal information about an entity, I can just push it into my scene graph and the hierarchy is adjusted?

See I hadn''t thought of that, I was just going to use the file to read in the model info, but I neglected the fact that I can store valuable scene information as well.

I''m not sure what you mean by memory layout, but do you mean how I would store vertices, then normals, then whatever in a straight binary pattern?

Also, you say the geometry data is contained just by the entity?
I don''t want to get into algoritms too much but so the tree root would contain hierarchy and model info, which it''s children may reference? Or would you have a ModelEntity class with a tree inside it of SubObjects? Cool approach, I like it! I''m already adjusting what I thought I would need for a good format!

Share this post


Link to post
Share on other sites
S1CA    1418
Yep, I have to agree with Yann''s suggestions (after many years of refining our own file formats on various games).

Storing *optional* hierarchies in your file is a good idea, however it shouldn''t be fixed once loaded - you should be able to "re-wire" the hierarchy at runtime.

Memory formats are a common way of doing things, particularly on consoles where the hardware is fixed. Think of the data types and structures your meshes will need when submitted to the 3D API/hardware for rendering - for example if you were using separate vertex component streams, the data in the file would be in the exact same format as those streams.
A simpler (non model file) example to explain is with texture maps - for development purposes you might use a format like 32bit .TGA or .JPG and then convert the data at *load time* into whatever format the hardware required (e.g. 1555, 565, 4444 etc). For production purposes, if you knew the data was always converted to say 565, and your maps were always 256x256, instead of doing the conversion every load, you could do it once offline and then when you load, just pass the raw pre-formatted data to the API. Same deal for meshes. [I didn''t explain the above too well, but I''m sure you get what I mean]

If your files will be transferred over an internet connection (either directly or in a downloaded demo), you might want to consider compression of data or at least pre-processing to make the vertex data in particular more compressible with general data compressors. This is in contradiction with use of memory file formats - which you choose depends on how the file will be used. I''d probably allow both and have a flag or different chunk type to determine which is used.

Our format is like this:

1. we have an array of "meshes", a mesh is a bunch of vertices and indices which use a single "shader" (i.e. is capable of being rendered with a single API call once states are set up). A mesh knows nothing of other meshes, it doesn''t have a name, any matrices associated with it etc. The shader is just an ID to describe the rendering style. All meshes in the array in the file are (optionally) pre-sorted by shader ID.

2. next is an array of "objects". An object is a group of meshes which has a name and a *single* ObjectToWorld matrix (i.e. all meshes within the object are transformed identically). The object simply has a list of mesh IDs contained within that object. This allows instancing of identical meshes, saving memory & file size.

3. next are "groups", these are similar to 3DS Max groups, and similar to the entities Yann describes. A group is a named list of objects which has its own object to world matrix. A group is identical to an "object" really except its children are objects rather than meshes.

4. the heirarchy information simply contains references to groups and objects.

Using the same example as Yann:

The car would be a "group", the group matrix would be the position/orientation of the car in world ("CAR").
A wheel would be an "object" within the group ("LEFT_FRONT_WHEEL").
A wheel would be made from multiple meshes, one for the tyre using a rubber style shader, one for the hub using a metal style shader etc

For shaders and texture maps, all of those used within the file are collected together into single lists so that names don''t unnecessarily get duplicated.

Some other stuff you might consider for your file format:

- The time & date the file was exported stored internally - can be useful for version control - handy with big teams.

- The name of the person who exported the file - useful for tracking who you need to chase up when things go wrong.

- A CRC (might be part of your higher level format though) to check the integrity of the file format.

- A standalone file info/edit utility - something which produces a standalone human readable list of the stuff in the file and the texture files required can be useful so the artists don''t have to reload the source file in Max just to find something out about the contents of the file.



--
Simon O''Connor
Creative Asylum Ltd
www.creative-asylum.com

Share this post


Link to post
Share on other sites
Void    126
The main problem with your own file format is you need to write exporters from modelling packages like 3DMax, SoftImage, Maya to fully access the parameters from the modelling tool. Each tool has different limitations, so designing a general structure is nothing less than a pain.

SoftImage has tried to do this with the .X file. You should be able to find info on that in Avid website.

Deep Exploration is another doing that. So far, it doesn''t seems to be that well yet.

Note:3DS file format is pretty limited.

Share this post


Link to post
Share on other sites
Dirge    300
Void -
I don''t know if that''s the biggest problem. I kind of look forward to writing a nice exporter. The thing is, I wouldn''t try to add too much off my own functionality within the 3ds max (my preference) environment. Material properties and such are automatic (just do it as if it were a 3ds file), but special hierarchys like Simon (S1CA) mentioned would be a little more diificult. He made a point of showing the similarities between max and his proposed format though, so if we where going to make a group, it would be easy, that''s built into max, but than we have objects, which can either be max objects, or represent a mesh. We lose a level of the hierarchy here (unless we name our group''s correctly, you can do that I believe, but I don''t have max installed on this comp to test). In order to not make it so incredibly complex, we have to use tricks, and unfortunetly, tricks don''t make things easy to understand (I don''t know what modeler would be happy with this system, but if he had too, he could deal with it).

Also, all the other data, such as environment mapping, bounding volumes, shaders, etc... I would create my own little app to manage. You could use this to tweak and add things to a model. Also to quickview it as Simon mentioned (I know what you mean. I hate having to open up max to check the poly count of a model. 3D Exploration is nice but very buggy).

S1CA (Simon) -
woah, that''s awesome! That seems like exactly the kind of abstraction I''ll need from my models. These questions will seem all over the place as I''m reskimming what you wrote and just asking random questions, please bear with me.

1: Can you explain what you mean by "Shader"? Like a Vertex or Pixel Shader. Or would it just say render wireframe/solid, etc... like a render state?

2: So would you load all meshs, objects, and groups as an array for each. Then, each element of an object (for instance) references (through ID) the meshs in the global list, and then groups do the same for objects? And you would do the same for textures and materials right (I do this now with my 3ds loader)?

3: Who keeps material info, a mesh, or object (I''m guessing mesh)?

4: Your rule number 4 (about hierarchys), can you explain a little more? So you load the hierarchy info, but all thats in there are ID''s, right?

5: Do you generate ID''s through a global manager (scene graph) which checks to see which ID''s are available and just throws one back at ya (like how you ask for a texture from OpenGL)?

6: How do you generate a CRC?

Also, what kind of features do you guy''s think are essential for making shnazzy graphics using todays and tomorrows technology. Shaders and Environment mapping (cubic, spherical, whatever) are some nice thought, but are they really needed in the actual model format (as Scheermesje mentioned)?

Amazing info, I wanna get coding but I''m afraid someone else is gonna offer an even better system! Either way I won''t be able to sleep tonight!

Share this post


Link to post
Share on other sites
Deebo    128
For my engine, I am planning on something like this.

1. The actual model will be a skeleton with weighted joints and bones for physics.
2. Next there will be individual meshes that can be applied over different bones and switched at runtime.
3. Textures and texture info, including bump maps and shaders, for the individual meshes. Shaders can be used to make certain textures glow, eyes blink, or whatever else I will think of. These are not vertex or pixel shaders because my engine is a realtime raytracer. This means I can have glass, mirrors, lights, and all that fancy stuff without having to code any special "tricks".
4. Model info which will store script-like info that will let the engine know which mesh to apply to the bones at what time. For example, if hit in the head, replace normal head mesh with the fucked up head mesh. This will also have control over mesh physics so fat jiggles and metal doesn''t.
5. Animations. I haven''t really studied this subject yet but I assume that I can store animation info for indivudal skeleton parts and blend them at run-time while applying correct physics to it.
6. Version info to make things compatible in the future.

Share this post


Link to post
Share on other sites
S1CA    1418
Dirge:

1. Shader has become quite an overloaded term these days . In the context of our file format/engine it means "rendering style". Our shaders are a bit like 3DS Max materials, except they don''t contain any texture maps, only the "how to draw this stuff" information. Texture maps for us are properties of the mesh rather than the shader. A shader is just a black box for rendering meshes - it takes a mesh and a list of texture maps as inputs


2. Yes, that''s essentially what happens (sparing the engineering details). Though there is control of what does and doesn''t get loaded from whoever is asking for the load.


3. Yep. Much of what would traditionally be in a "material" is in our shader. The mesh has a list of "texture map IDs". Any parameters of the shader which might change on a per mesh basis can be optionally stored with the mesh. The details of the actual maps (filenames, format & size info etc) are stored elsewhere in the file. This is intentionally quite similar to how Max does it (for ease of exporter coding + artist familiarity with what the engine is doing).


4. Since the IDs for things are nothing more than indices into arrays within the file, they can be trivially converted to pointers for a tree when loaded (multiply each index by the size of an element of the array and add the chunk memory start address). This is an example of what Yann suggested with relocatable hierarchies.
The tree/graph is stored separately from the actual data so all the elements are the same size and so in the file it could look like this:

root:
2 // there are 2 groups
//group 1:
2 // there are 2 children of group 1
19 // group1 child1 is object 19
3 // group1 child2 is object 3
//group 2:
3 // there are 3 children of group 2
17 // group2 child1 is object 17
22 // group2 child2 is object 22
19 // group2 child3 is object 19



5. At load time?, yes something similar - its a handle based thing, and there is checking and reference counting performed on the (unique) name of the thing being loaded to reduce load times (particularly important with textures). That''s how it looks from the client side anyway - in reality (in the nasty guts of the system), there''s also hardware specific resource IDs - in a multimon system, one texture might be associated with 2 different output devices. The client doesn''t have a concept of how many devices are actually in use - it just loads textures into the engine which may be using multimon (based on user settings).

6. Cyclic Redundancy Check - just search the web for CRC32 and you should be able to find some ready made source and documentation.


Features & formats ?

IMO apart from situations where there is only ever going to be one implementation of something, keep specifics of implementation out of the file format! - this has allowed us for example to use the same format across PC, PS2 and GameCube.

In fact I''m in the process of partially overhauling the shader section of our engine base at the moment. Shaders are going to get a similar status to texture maps so that within the model file format they simply have a name rather than a pre-compiled ID system we use at the moment. This will allow shaders to be stored externally from the code base in text format (think RenderMan shaders meets D3DX Effects & Techniques). This should ease development of new shaders significantly (no need for recompile of code base) and allow a "catalogue" approach to special effects. [Reading ATIs recent paper on this and seeing the new stuff in DX9 was the final shove I needed in the direction of describing shaders externally].


--
Simon O''Connor
Creative Asylum Ltd
www.creative-asylum.com

Share this post


Link to post
Share on other sites
Dirge    300
I think I''m ready to begin work and I am going to go towards the method suggested by S1CA but I have a question. In Direct3D, it''s pretty expensive to keep calling DrawPrimitive() (or even DrawIndexPrimitive()), so how do you keep it efficient even when there are so many meshs contained within an object? Or is there really no way to combat this since each mesh might be very different in texture or renderstate or whatever? Thanks for all the help everyone! You''ve definetly opened up my eye''s to some cool possiblities!

Share this post


Link to post
Share on other sites
Dark Rain    157
I'm doing skeleton animation in my engine and I came to a solution to the problem you're mentionning. It works only if you're using bone animation and works very well with skinning.

Basicaly, I have a Joint (it's a bone) class and a Skeleton class. The skeleton obviously contain all the bones that have pointers to each others in child/sibling order.

Details aside, here are the important parts :

class CJoint
{
public:
std::string m_Name;
CJoint *m_Child;
CJoint *m_Sibling;

std::vector m_IndexList;

sRenderList *m_RenderList;

};

class CSkeleton
{
CJoint *RootNode;
public:
struct sRenderList
{
int *m_IndexList;
float *m_TextureCoordinates;
float *m_VertexCoordinates;
point3 *m_Vertex;
CMaterialUnit m_Material;
};
sRenderList *m_RenderList;
};


So, the idea is to have all the vertex, texture and index data in the skeleton class. It's optimised for OpenGL, so the data is stored in arrays that are ready to be drawn with DrawIndexPrimitive(). Only the vertex data has to be updated each frame, the rest stays the same.

To update the data, I start going throught my tree and I use the index in CJoint to find the corresponding vertex to apply the bone transformation matrix to them. I output the result in my float *m_TextureCoordinates array and it's all ready to render ^_^.

Obviously, I cut animation part and some other stuff for simplification sake so there might be some missing stuff and optimisations.

EDIT : Oh yeah, I forgot to mention that each models can have many textures, that's why there's an array of renderlist and indexlist.


[edited by - Dark Rain on June 11, 2002 2:59:29 AM]

Share this post


Link to post
Share on other sites