I've been working with 3D model file formats for a short while now and I'm honestly tired of using various file formats which are unsuited to computer games, are outdated or do not support the features which I want in a 3D model format. Furthermore, model formats are very difficult to find information on and some of them are proprietary formats which are unusable by small developers. So, I'm going to solve this problem by writing my own standard file format for amateur, independent and student game development and I'm planning on releasing both the model format and any file format converters I make under a GNU license. I will probably also write C++ code for reading these models into OpenGL at least (maybe DirectX too) and release that code under GNU license. By no means do I intend to create an industry standard, I'm no where near influential enough to do that (not yet at least) but I do want to share my solution of this problem (when I get there) with everyone else who comes across the same problem as me (aren't programmers just problem-solvers who use computers after all?). Firstly, I'm looking for advice on two issues: -What to include in my file formats. -Which of the various GNU licenses to release the formats under. My goals for my primary file format are as follows: -The format must be directed towards use in real time computer games, rather than in an animation package and should be optimised as such. (I will refer to this property as being a "Lite" file format from here on) -The format must be as extensible as possible. In other words, this format must be future proof and customisable so you can tailor it to your needs. -The format must support compression of various kinds, including zlib. (Excluding LZ77 or variants due to patent issues) -The format must support skeletal animation. -The format must support all current technology relating to computer games, including pixel/fragment and vertex shaders. My answer to this problem is the (shamelessly named) "Lite Extensible Object" (.leo) file format. It is loosely structured based on the IFF file format, although it is (optionally) random rather than serial access. It also borrows heavily from the 3DS file format, the MS3D file format and the MD2 and MD3 file formats. A .leo file may store the following information: -Vertices and indices stored in arrays for direct transfer into OpenGL vertex and index arrays (These can be easily converted into Direct3D compatible FVF vertices and indices. I am working on solving this problem now) -Edges -Triangles -Triangle strips and fans -Separate meshes -Tags used to join different meshes together (ala MD3) -Bones with masses and quaternion rotations -Bounding spheres, cylinders, OBBs, AABBs and (later when I learn the math) ellipsoids -Materials -Textures including Light/Fog maps, Normal maps, Glow maps, Displacement maps, Ambient Occlusion maps, Gloss maps, Detail textures and Decal textures -Hotpoint rays (for when you want to shoot a ray out of your model) -Splines -Pixel and vertex shaders -Adjacency information -Billboard quads -Traditional key framed animation -Skeletal key frames -Skeletal animation sequences -Animation blending -Morph targets -(Possibly in future) LOD -(Possibly in future) Encryption based on RSA or some such algorithm. (Did I miss anything?) Furthermore, the format will be divided up into random access chunks, so you can disregard information which you don't want to use and skip directly to the information you want to use. Note how it is a "lite" file format, meaning it does not contain any information which is not relavent to computer game development (such as you may find in the 3DS and MS3D file formats), it is also extensible in that anyone can add new types of chunks easily in future to allow for the format to accommodate new technology. Note that simplicity or ease of use is not an aim of this file format. Loading and using the .leo file format is quite involved, being designed around both chunks and random access and stored in binary. I would not expect a beginner at file handling to be able to load a .leo file easily (without a library, and even then any library I write will not account for the future extensions of the file format). Therefore, I have come up with the beginnings of a simpler sister file format which is not based around chunks and is stored sequentially. The "Lite Inextensible Object" (.lio pronounced "lie-oh") file format will be a "beginners" file format containing the simpler features of the .leo file in sequential access without chunks (although still binary because I can't be bothered using text files). The .lio file format will not use quaternions for rotations, but rather Euler angles for ease of use. I hope this doesn't end up as just all talk on my part but I am genuinely busy with the third year of my degree at the moment. As such, I will work on this file format, study-load allowing, over a (probably) lengthy period of time. I will release the file format specification with my first model converter and loading code to whoever will host it (the more the merrier). As I said, I'm just posting here to get advice on my file format, I could easily have kept the format to myself and used it in my own projects but I feel that it would reduce some of the frustration of learning how to program 3D graphics for other people. Maybe it will expose the profession to new talent in the form of students such as myself who are interested in getting straight into 3D programming without the fuss of writing a custom model loader or using an outdated or unsuitable file format. A big thank you if you've bothered reading all that. :) Advice anyone? EDIT: Here they are: http://www.freewebs.com/leomadegames/3dformatsleolio.htm Alternatively: http://sourceforge.net/project/showfiles.php?group_id=164595 P.S. Don't expect me to finish soon, I've not even finalised the format itself yet, let alone gone through the complex task of writing a format converter and loading code. P.P.S. Is there anyone who would be interested in my releasing this format under GNU license? If it doesn't hold any interest to anyone else I will probably just keep it to myself. [Edited by - Leo_E_49 on May 9, 2006 9:41:03 PM]

Cheers, that looks to be an interesting read. However, I'm not interested in making an XML based format. I personally find XML to be not to my taste :/ although I do agree that it is an excellent metalanguage. I am aiming to write this as a binary file so that it can be used directly with C++ in binary format without text handling.

Thanks for the link however, I will read it thoroughly to gain a better understanding. :)

EDIT: Morphing and data validation in COLLADA should be particularly interesting. I may try to include them in my format too.

EDIT: Scene graph information might be a good thing to include too, as well as possibly lights.

EDIT: I'm thinking the format could be extended to allow for keyframed animation too, I'll look into that.

[Edited by - Leo_E_49 on April 3, 2006 1:18:09 AM]

Quote:Original post by Endar
I don't have a whole lot of experience in creating file formats for use in anything, really, but a couple of things struck me as I was reading your list.

Quote:
-Tags used to join different meshes together (ala MD3)
-Bones with masses and quaternion rotations

Doesn't using tags to join meshes together suggest that you're using a model without a skeleton? Wouldn't you select either vertex keyframe animation or skeletal?

Not if you attach a tag onto a bone, which is my proposed method of using tags.

Quote:

Quote:
-Bounding spheres, cylinders, OBBs, AABBs and (later when I learn the math) ellipsoids

You don't mean all of them do you? Wouldn't it be easier to just provide what's needed to calculate whichever one will be used? I can't imagine that a model would need different ways of calculating collisions. Eg. if it's a collision with the level, use the AABB and if it's a collision with something else, use the ellipsoid.

One of the goals of the format is extensibility. I decided upon including the facility for using any of them, rather than deciding upon a specific format. You just pick the bounding volume type you want to use and away you go. Honestly though, because it's going to be a GNU licensed format anyway, you can write your own bounding volumes into it anyway. All I'm going to do is reserve space for the bounding volumes I've listed. You could add bounding tori or bounding cones later if you wanted to.

Furthermore, people who are speculating that this model format will take ages to load, it won't. It's going to be random access, you can skip the data which you don't want to use from the beginning of the file and go directly to the relavent data you want. I know this is a new concept to many people, because most file formats (even IFF) are serial, but I feel that serial access belongs back in the 20th century and I want this to be a model format for the 21st century.

EDIT: A further implication of the format being random access, chunk based and GNU licensed is that you can include custom information in it with relative ease (if you are a halfway decent file programmer). Not only that, files will all be backwards compatible and if your program can't handle a certain piece of data or someone else's custom data, just ignore it.

[Edited by - Leo_E_49 on April 3, 2006 2:23:43 AM]

As I said, it would be easy to convert to Direct3D. In fact, I may just write a DirectXVertex chunk which indexes into the vertex arrays. The problem is that in Direct3D, you specify your own vertex format using either FVF or the newer format specifiers which work with shaders. Therefore, there really is no such thing as a standard vertex format to work with in Direct3D. And while my file format can be converted easily to match your FVF or vertex format; my base set, which I will personally write for use within the base file "library" of chunks will not handle every permutation of these vertex formats.

As to the support:
Normals are supported.
Tangents are not supported but could easily be added.
Binormals, I'm afraid I'm inexperienced with, but could also be added easily.
Blend weights are supported. (Arbitrary number of bones)
Arbitrary numbers of texture coordinates are supported.
Spherical harmonics coefficients are not supported, but again could be added easily.

Currently, vertices store data directly relating to OpenGL vertex arrays and bone weights.

I would appreciate your help in finding more information on these other issues however, so that I can write chunks to include them.

Never the less, if you want a different type of vertex from the one that I supply, you can write your own chunk type and stream it in yourself.

Order is not important in my file format, it's random access like I said (think IFF or 3DS).

Arbitrary collision meshes can be added, yes.

Quote:Can models share the same, separately stored skeletons and related animations? (That is, a change to the walking animation should affect all models immediately, without rebuilding files.)

This is a tricky one and it's the primary flaw in my model format. For my purposes, I want the animations stored in the same file as the model data. Now, this makes it difficult to share animations between model files. There are a number of ways it can be done, however:
-Create a .leo file only containing animation data and the rest of your .leo files with models containing only vertex and index data and then animate the models using the animation .leo.
-Create all of your models in one .leo file and animate them with the animation structure in that file (multiple meshes are supported in my file format).

I would suggest using the former, rather than the latter method.

Points are defined as rays in my file format (to generalise them), however you could add a point chunk if you wanted. Extra non rendered triangles are just as simple to add.

Heck, if you wanted to add in a poetry chunk to compose your own sonnets in the model file, you could. :p

There is a base material type which includes all of the data from a .3DS file, but if you want to replace that, just write your own.

Almost everything in my model format works by indexing other things in my model format, so it's all relative addressing. If you want to use one kind of material instead of another, just start your indices in a different chunk.

P.S. When I say this is a 3D model file format, I actually lie. (That's why it's not called the Lite Extensible Model format. The "Object" was chosen on purpose for its meaning as well as to coincide with my vanity :p) It's an everything object file format (like IFF), I just happen to be using it to describe 3D models. It could be used for levels, sound files, textures, text files, statistical data, spreadsheets, anything. In fact, it theoretically can be used for all the resources in a computer game, but even I won't go there. :p

P.P.S. The .lio file format will not support anything I mentioned above, that's a very simplistic file format intended for beginners.

[Edited by - Leo_E_49 on April 3, 2006 2:50:59 AM]

Quote:Original post by Mike2343
You realize most major formats are well documented and most major 3d art programs allow programmers to export to custom formats that suit their needs? Your format to suit enough peoples needs would likely be bloated and that would turn a lot of folks off (hence why most people don't stay with ASE/3DS/etc formats and go a custom route). Making your own format with exporters/importers is normally takes a few days at best over the life of a project.

Not saying you shouldn't do it, just make sure there is enough interest before you dive in.

Also, want to know about file formats, hit up: http://www.wotsit.org/

Thank you for your advice. :)

I visit wotsit regularly, I'm writing up a BVH file loader at the moment incidentally thanks to that website.

Furthermore, my files will not be bloated at all. As I said before, you can pick and choose what you want to use from my specification and if it's not included in the file format, you just ignore it.

For example: House.leo may include normal mapping and shaders, where as Car.leo may not. They are both .leo format and both can be loaded by any program which supports .leo files. However in programs which support normal mapping and shaders, House.leo will look much better than Car.leo, in which the normal mapping information and shaders are absent.

In other words, the exporter which is used will determine the content of the .leo file, as opposed to the other way around. The format is designed to be flexible and in touch with the needs of it's users.

EDIT: Perhaps I should just release the file format now for your scrutiny to demonstrate that I'm not all talk, although I've not written a loader/converter for it yet. However, I will not do so until I have received advice on which GNU license to release it under (I'm thinking of releasing it under LGPL). Basically, I want to ensure no one person can own this model format, although I want my name stamped on it somewhere.

[Edited by - Leo_E_49 on April 3, 2006 2:14:14 AM]

Quote:Original post by Leo_E_49
This is a tricky one and it's the primary flaw in my model format. For my purposes, I want the animations stored in the same file as the model data. Now, this makes it difficult to share animations between model files. There are a number of ways it can be done, however:
-Create a .leo file only containing animation data and the rest of your .leo files with models containing only vertex and index data and then animate the models using the animation .leo.
-Create all of your models in one .leo file and animate them with the animation structure in that file (multiple meshes are supported in my file format).

Quote:Almost everything in my model format works by indexing other things in my model format, so it's all relative addressing. If you want to use one kind of material instead of another, just start your indices in a different chunk.

Why not solve the problem with animations by allowing indexing to refer to other files? You could store animations in a separate file, but for purposes of the user , you would still only load a single file. This would also solve any future problems where you might want to share some data between leo files.