Sign in to follow this  
Leo_E_49

OpenGL Standard 3D model file format

Recommended Posts

I've been working with 3D model file formats for a short while now and I'm honestly tired of using various file formats which are unsuited to computer games, are outdated or do not support the features which I want in a 3D model format. Furthermore, model formats are very difficult to find information on and some of them are proprietary formats which are unusable by small developers. So, I'm going to solve this problem by writing my own standard file format for amateur, independent and student game development and I'm planning on releasing both the model format and any file format converters I make under a GNU license. I will probably also write C++ code for reading these models into OpenGL at least (maybe DirectX too) and release that code under GNU license. By no means do I intend to create an industry standard, I'm no where near influential enough to do that (not yet at least) but I do want to share my solution of this problem (when I get there) with everyone else who comes across the same problem as me (aren't programmers just problem-solvers who use computers after all?). Firstly, I'm looking for advice on two issues: -What to include in my file formats. -Which of the various GNU licenses to release the formats under. My goals for my primary file format are as follows: -The format must be directed towards use in real time computer games, rather than in an animation package and should be optimised as such. (I will refer to this property as being a "Lite" file format from here on) -The format must be as extensible as possible. In other words, this format must be future proof and customisable so you can tailor it to your needs. -The format must support compression of various kinds, including zlib. (Excluding LZ77 or variants due to patent issues) -The format must support skeletal animation. -The format must support all current technology relating to computer games, including pixel/fragment and vertex shaders. My answer to this problem is the (shamelessly named) "Lite Extensible Object" (.leo) file format. It is loosely structured based on the IFF file format, although it is (optionally) random rather than serial access. It also borrows heavily from the 3DS file format, the MS3D file format and the MD2 and MD3 file formats. A .leo file may store the following information: -Vertices and indices stored in arrays for direct transfer into OpenGL vertex and index arrays (These can be easily converted into Direct3D compatible FVF vertices and indices. I am working on solving this problem now) -Edges -Triangles -Triangle strips and fans -Separate meshes -Tags used to join different meshes together (ala MD3) -Bones with masses and quaternion rotations -Bounding spheres, cylinders, OBBs, AABBs and (later when I learn the math) ellipsoids -Materials -Textures including Light/Fog maps, Normal maps, Glow maps, Displacement maps, Ambient Occlusion maps, Gloss maps, Detail textures and Decal textures -Hotpoint rays (for when you want to shoot a ray out of your model) -Splines -Pixel and vertex shaders -Adjacency information -Billboard quads -Traditional key framed animation -Skeletal key frames -Skeletal animation sequences -Animation blending -Morph targets -(Possibly in future) LOD -(Possibly in future) Encryption based on RSA or some such algorithm. (Did I miss anything?) Furthermore, the format will be divided up into random access chunks, so you can disregard information which you don't want to use and skip directly to the information you want to use. Note how it is a "lite" file format, meaning it does not contain any information which is not relavent to computer game development (such as you may find in the 3DS and MS3D file formats), it is also extensible in that anyone can add new types of chunks easily in future to allow for the format to accommodate new technology. Note that simplicity or ease of use is not an aim of this file format. Loading and using the .leo file format is quite involved, being designed around both chunks and random access and stored in binary. I would not expect a beginner at file handling to be able to load a .leo file easily (without a library, and even then any library I write will not account for the future extensions of the file format). Therefore, I have come up with the beginnings of a simpler sister file format which is not based around chunks and is stored sequentially. The "Lite Inextensible Object" (.lio pronounced "lie-oh") file format will be a "beginners" file format containing the simpler features of the .leo file in sequential access without chunks (although still binary because I can't be bothered using text files). The .lio file format will not use quaternions for rotations, but rather Euler angles for ease of use. I hope this doesn't end up as just all talk on my part but I am genuinely busy with the third year of my degree at the moment. As such, I will work on this file format, study-load allowing, over a (probably) lengthy period of time. I will release the file format specification with my first model converter and loading code to whoever will host it (the more the merrier). As I said, I'm just posting here to get advice on my file format, I could easily have kept the format to myself and used it in my own projects but I feel that it would reduce some of the frustration of learning how to program 3D graphics for other people. Maybe it will expose the profession to new talent in the form of students such as myself who are interested in getting straight into 3D programming without the fuss of writing a custom model loader or using an outdated or unsuitable file format. A big thank you if you've bothered reading all that. :) Advice anyone? EDIT: Here they are: http://www.freewebs.com/leomadegames/3dformatsleolio.htm Alternatively: http://sourceforge.net/project/showfiles.php?group_id=164595 P.S. Don't expect me to finish soon, I've not even finalised the format itself yet, let alone gone through the complex task of writing a format converter and loading code. P.P.S. Is there anyone who would be interested in my releasing this format under GNU license? If it doesn't hold any interest to anyone else I will probably just keep it to myself. [Edited by - Leo_E_49 on May 9, 2006 9:41:03 PM]

Share this post


Link to post
Share on other sites
Cheers, that looks to be an interesting read. However, I'm not interested in making an XML based format. I personally find XML to be not to my taste :/ although I do agree that it is an excellent metalanguage. I am aiming to write this as a binary file so that it can be used directly with C++ in binary format without text handling.

Thanks for the link however, I will read it thoroughly to gain a better understanding. :)

EDIT: Morphing and data validation in COLLADA should be particularly interesting. I may try to include them in my format too.

EDIT: Scene graph information might be a good thing to include too, as well as possibly lights.

EDIT: I'm thinking the format could be extended to allow for keyframed animation too, I'll look into that.

[Edited by - Leo_E_49 on April 3, 2006 1:18:09 AM]

Share this post


Link to post
Share on other sites
I don't have a whole lot of experience in creating file formats for use in anything, really, but a couple of things struck me as I was reading your list.
Quote:

-Tags used to join different meshes together (ala MD3)
-Bones with masses and quaternion rotations

Doesn't using tags to join meshes together suggest that you're using a model without a skeleton? Wouldn't you select either vertex keyframe animation or skeletal?

Quote:

-Bounding spheres, cylinders, OBBs, AABBs and (later when I learn the math) ellipsoids

You don't mean all of them do you? Wouldn't it be easier to just provide what's needed to calculate whichever one will be used? I can't imagine that a model would need different ways of calculating collisions. Eg. if it's a collision with the level, use the AABB and if it's a collision with something else, use the ellipsoid.

Share this post


Link to post
Share on other sites
Quote:
Original post by Endar
I don't have a whole lot of experience in creating file formats for use in anything, really, but a couple of things struck me as I was reading your list.
Quote:

-Tags used to join different meshes together (ala MD3)
-Bones with masses and quaternion rotations

Doesn't using tags to join meshes together suggest that you're using a model without a skeleton? Wouldn't you select either vertex keyframe animation or skeletal?


Not if you attach a tag onto a bone, which is my proposed method of using tags.

Quote:


Quote:

-Bounding spheres, cylinders, OBBs, AABBs and (later when I learn the math) ellipsoids

You don't mean all of them do you? Wouldn't it be easier to just provide what's needed to calculate whichever one will be used? I can't imagine that a model would need different ways of calculating collisions. Eg. if it's a collision with the level, use the AABB and if it's a collision with something else, use the ellipsoid.


One of the goals of the format is extensibility. I decided upon including the facility for using any of them, rather than deciding upon a specific format. You just pick the bounding volume type you want to use and away you go. Honestly though, because it's going to be a GNU licensed format anyway, you can write your own bounding volumes into it anyway. All I'm going to do is reserve space for the bounding volumes I've listed. You could add bounding tori or bounding cones later if you wanted to.

Furthermore, people who are speculating that this model format will take ages to load, it won't. It's going to be random access, you can skip the data which you don't want to use from the beginning of the file and go directly to the relavent data you want. I know this is a new concept to many people, because most file formats (even IFF) are serial, but I feel that serial access belongs back in the 20th century and I want this to be a model format for the 21st century.

EDIT: A further implication of the format being random access, chunk based and GNU licensed is that you can include custom information in it with relative ease (if you are a halfway decent file programmer). Not only that, files will all be backwards compatible and if your program can't handle a certain piece of data or someone else's custom data, just ignore it.

[Edited by - Leo_E_49 on April 3, 2006 2:23:43 AM]

Share this post


Link to post
Share on other sites
You realize most major formats are well documented and most major 3d art programs allow programmers to export to custom formats that suit their needs? Your format to suit enough peoples needs would likely be bloated and that would turn a lot of folks off (hence why most people don't stay with ASE/3DS/etc formats and go a custom route). Making your own format with exporters/importers is normally takes a few days at best over the life of a project.

Not saying you shouldn't do it, just make sure there is enough interest before you dive in.

Also, want to know about file formats, hit up: http://www.wotsit.org/

Share this post


Link to post
Share on other sites
Quote:
Original post by Leo_E_49
I'm honestly tired of using various file formats which are unsuited to computer games
...
-Vertices and indices stored in arrays for direct transfer into OpenGL vertex and index arrays (Sorry Direct3D guys, I'm primarily an OpenGL user so you kind-of got the short end of the stick here.
That'd be a mistake. Although, I suspect that your vertices will work fine for D3D anyway. Building an incompatible format between the two is a difficult stunt to pull off, although building a slow one is easier to do.

Does your format support vertices with:
Normals
Tangents
Binormals
Blend Weights
Arbitrary numbers of tex coord sets of 1-4 dimensions
Spherical harmonics coefficients
etc.
In short, do you support completely and utterly arbitrary combinations, orderings, dimensions, and types of vertex attributes?
Can you store arbitrary collision meshes, rather than just bounding volumes?
Can models share the same, separately stored skeletons and related animations? (That is, a change to the walking animation should affect all models immediately, without rebuilding files.)
Can arbitrary non-geometric "things" be added to model files? Points, extra non-rendered triangles, whatever?
Can material data be completely defined by the engine? That is, can I define a material property called "cheesiness" and save it into the file?

Share this post


Link to post
Share on other sites
As I said, it would be easy to convert to Direct3D. In fact, I may just write a DirectXVertex chunk which indexes into the vertex arrays. The problem is that in Direct3D, you specify your own vertex format using either FVF or the newer format specifiers which work with shaders. Therefore, there really is no such thing as a standard vertex format to work with in Direct3D. And while my file format can be converted easily to match your FVF or vertex format; my base set, which I will personally write for use within the base file "library" of chunks will not handle every permutation of these vertex formats.

As to the support:
Normals are supported.
Tangents are not supported but could easily be added.
Binormals, I'm afraid I'm inexperienced with, but could also be added easily.
Blend weights are supported. (Arbitrary number of bones)
Arbitrary numbers of texture coordinates are supported.
Spherical harmonics coefficients are not supported, but again could be added easily.

Currently, vertices store data directly relating to OpenGL vertex arrays and bone weights.

I would appreciate your help in finding more information on these other issues however, so that I can write chunks to include them.

Never the less, if you want a different type of vertex from the one that I supply, you can write your own chunk type and stream it in yourself.

Order is not important in my file format, it's random access like I said (think IFF or 3DS).

Arbitrary collision meshes can be added, yes.

Quote:
Can models share the same, separately stored skeletons and related animations? (That is, a change to the walking animation should affect all models immediately, without rebuilding files.)


This is a tricky one and it's the primary flaw in my model format. For my purposes, I want the animations stored in the same file as the model data. Now, this makes it difficult to share animations between model files. There are a number of ways it can be done, however:
-Create a .leo file only containing animation data and the rest of your .leo files with models containing only vertex and index data and then animate the models using the animation .leo.
-Create all of your models in one .leo file and animate them with the animation structure in that file (multiple meshes are supported in my file format).

I would suggest using the former, rather than the latter method.

Points are defined as rays in my file format (to generalise them), however you could add a point chunk if you wanted. Extra non rendered triangles are just as simple to add.

Heck, if you wanted to add in a poetry chunk to compose your own sonnets in the model file, you could. :p

There is a base material type which includes all of the data from a .3DS file, but if you want to replace that, just write your own.

Almost everything in my model format works by indexing other things in my model format, so it's all relative addressing. If you want to use one kind of material instead of another, just start your indices in a different chunk.

P.S. When I say this is a 3D model file format, I actually lie. (That's why it's not called the Lite Extensible Model format. The "Object" was chosen on purpose for its meaning as well as to coincide with my vanity :p) It's an everything object file format (like IFF), I just happen to be using it to describe 3D models. It could be used for levels, sound files, textures, text files, statistical data, spreadsheets, anything. In fact, it theoretically can be used for all the resources in a computer game, but even I won't go there. :p

P.P.S. The .lio file format will not support anything I mentioned above, that's a very simplistic file format intended for beginners.

[Edited by - Leo_E_49 on April 3, 2006 2:50:59 AM]

Share this post


Link to post
Share on other sites
Quote:
Original post by Mike2343
You realize most major formats are well documented and most major 3d art programs allow programmers to export to custom formats that suit their needs? Your format to suit enough peoples needs would likely be bloated and that would turn a lot of folks off (hence why most people don't stay with ASE/3DS/etc formats and go a custom route). Making your own format with exporters/importers is normally takes a few days at best over the life of a project.

Not saying you shouldn't do it, just make sure there is enough interest before you dive in.

Also, want to know about file formats, hit up: http://www.wotsit.org/


Thank you for your advice. :)

I visit wotsit regularly, I'm writing up a BVH file loader at the moment incidentally thanks to that website.

Furthermore, my files will not be bloated at all. As I said before, you can pick and choose what you want to use from my specification and if it's not included in the file format, you just ignore it.

For example: House.leo may include normal mapping and shaders, where as Car.leo may not. They are both .leo format and both can be loaded by any program which supports .leo files. However in programs which support normal mapping and shaders, House.leo will look much better than Car.leo, in which the normal mapping information and shaders are absent.

In other words, the exporter which is used will determine the content of the .leo file, as opposed to the other way around. The format is designed to be flexible and in touch with the needs of it's users.

EDIT: Perhaps I should just release the file format now for your scrutiny to demonstrate that I'm not all talk, although I've not written a loader/converter for it yet. However, I will not do so until I have received advice on which GNU license to release it under (I'm thinking of releasing it under LGPL). Basically, I want to ensure no one person can own this model format, although I want my name stamped on it somewhere.

[Edited by - Leo_E_49 on April 3, 2006 2:14:14 AM]

Share this post


Link to post
Share on other sites
Quote:
Original post by Leo_E_49
This is a tricky one and it's the primary flaw in my model format. For my purposes, I want the animations stored in the same file as the model data. Now, this makes it difficult to share animations between model files. There are a number of ways it can be done, however:
-Create a .leo file only containing animation data and the rest of your .leo files with models containing only vertex and index data and then animate the models using the animation .leo.
-Create all of your models in one .leo file and animate them with the animation structure in that file (multiple meshes are supported in my file format).


Quote:
Almost everything in my model format works by indexing other things in my model format, so it's all relative addressing. If you want to use one kind of material instead of another, just start your indices in a different chunk.


Why not solve the problem with animations by allowing indexing to refer to other files? You could store animations in a separate file, but for purposes of the user , you would still only load a single file. This would also solve any future problems where you might want to share some data between leo files.

Share this post


Link to post
Share on other sites
Quote:
Original post by Selkrank
Why not solve the problem with animations by allowing indexing to refer to other files? You could store animations in a separate file, but for purposes of the user , you would still only load a single file. This would also solve any future problems where you might want to share some data between leo files.


Excellent idea! :D I'll add that in right away.

EDIT: Silly me, I forgot that I store bone, keyframe and animation information completely separately from meshes in my format anyway. So, problem solved before I even tried to fix it. :p

EDIT: Traditional key framed animation is now added to the model format.

By the way, does anyone know whether LGPL can apply to a model format?

Share this post


Link to post
Share on other sites
Quote:
Original post by Leo_E_49
Excellent idea! :D I'll add that in right away.

By the way, does anyone know whether LGPL can apply to a model format?


Could the Creative Commons license be used? See http://creativecommons.org/about/licenses/meet-the-licenses, especially the last two options. I'm really not sure which would be the best.

Actually you're solving the problem I'm having at the moment. My current plan is to use COLLADA as the "source format" and since I'm working with Python, use the standard pickle module to create quick-to-load binary files (I've yet to see how fast pickling and unpickling is for this purpose). Eventually it would be nice to support a real binary format though, so what you're doing is interesting.

Share this post


Link to post
Share on other sites
Quote:
Original post by Selkrank
Could the Creative Commons license be used? See http://creativecommons.org/about/licenses/meet-the-licenses, especially the last two options. I'm really not sure which would be the best.


I would probably go with the last one. If I'm really serious about making this a standard, I should release it for commercial use, even if people don't pay me for it. ( Which I'm fine with as long as I get credited. Imagine how it would look on a CV though :D ) I'm going to give it some more thought first though. I don't think there are many people who just jump into making their work free and I'm certainly not one of them.

Quote:
Actually you're solving the problem I'm having at the moment. My current plan is to use COLLADA as the "source format" and since I'm working with Python, use the standard pickle module to create quick-to-load binary files (I've yet to see how fast pickling and unpickling is for this purpose). Eventually it would be nice to support a real binary format though, so what you're doing is interesting.


It's good to see that there is interest. Don't get your hopes up yet though. :/ This is my pet project, but it's not my job, nor even my primary area of study in university. My lecturers would certainly not give me credit for creating this format nor even count it towards my studies. :( In fact, if I developed this on a university machine or in any relation to the university in the UK, I lose the license to it and it becomes the university's intellectual property (by UK law). :( Perhaps I could write it up as a dissertation in my final year of the degree, but it's far too simple an issue for that (not to mention it would no longer be public license). And besides I've got bigger plans. :p

EDIT: Morph targets are now supported by the .leo file format.

[Edited by - Leo_E_49 on April 3, 2006 10:09:35 AM]

Share this post


Link to post
Share on other sites
A while ago, I wrote a long and detailed specification for the DirectMesh2 animation API. You can get it here. I covered a lot of things, so you may want to check some of the sections out. I have since stopped working on it because someone else in animation middleware gave me a better offer that let me concentrate more on rendering.

Share this post


Link to post
Share on other sites
While enthusiasm and "If you build it, they will come" mentality is good, it doesn't account for business reality.

Popular 3D formats evolved not because they would be perfect (or even complete), but because they are instantly useful within a particular engine, or in combination with a particular design software. COLLADA apears to be a well defined format, yet this is the first time I ran across it. The fact that its XML seems to be unpopular with you, despite XML being a completely valid format.

Interest ("that would be cool") does not equal interest ("our next line of 3 AAA games will rely on this format, and its integration it into 3DS max"). Although people express casual interest, you can only hope for adoption, if you get the format used by several commercially succesful and publically exposed games.

The problem with formats is not their specification (this is a matter of a couple of days work), but their suitability for the task. Asking what people would like to see is not necessarily the same as what people need.

In a typical production pipeline, you will want a format that will cover your entire process. Modeling in software X, conventient storage in versioning software, natural capability of change tracking, natural integration into your engine.

XML for example has the advantage over binary formats of being very versioning software friendly, and can be edited without any tools.

Another major hurdle of generic specifications is the learning curve. Even if you provide the API, documentation, examples, etc., people using it will need to learn it. And there will always be a thing your format will not support. Most productions rely on having flexibility, and adaptability to their immediate needs. Writing and defining a storage format is a moderately trivial task when viewed in light of entire production. What severly outweighs "perfect" format is the ease of use, integration with existing tools and widespread experience with it.

And last, how will you guarantee long-term support for your format. Simply saying "I will support it" is not sufficient.

Quote:
Sorry Direct3D guys, I'm primarily an OpenGL user so you kind-of got the short end of the stick here. Don't worry however, it should be easy to convert the vertex and index arrays into vertex and index buffers for use in DirectX.


Agreed, who needs Direct X. Who needs x-box or windows support afterall. DirectX is clearly a non-standard platform, who cares about a few weirdos who use it. What is the point of a standard format that cannot be used with majority of platforms? If that is the case, then label this as OpenGL standard format, not general 3D format.

Will you support importers/exporters for Maya, 3DS, Lightwave, Blender, Unreal editor, Quake editor, Doom3 editor, ... Will it be suitable for streaming for use in online games? Which of these do you consider irrelevant and not worthy of support.

Popular formats have evolved from popular products. Everyone will tell you that in the end it doesn't matter how good or bad something is, it needs to get the job done. And there isn't a specification in existance, that would do that. It is implementations that count, stuff that has been tried and tested, and can be used out of box right here and right now. When you have a release cycle of 9-18 months, you just cannot afford to wait 3 months for update to format, which introduces support for the one and only feature you need (for example cloth simulation nodes)

The reason standardized formats are having hard time in game development is because of extremly varying needs, insane rate of technology advancement and feature scopes, and the almost negligible need for standardization (there's little asset reuse beyond images).

If you're looking for a learning experience then this is great. But before even thinking about standardization, make sure to look WHY formats are the way they are, and HOW they are used.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by Antheus
While enthusiasm and "If you build it, they will come" mentality is good, it doesn't account for business reality.


I'm not in business, nor am I building this format for business purposes. If I was, I'd certainly not release it for free. What I'm saying is that if you're in business and happen to like my format, you're free to use it commercially as long as you keep my name on it.

Quote:
Popular 3D formats evolved not because they would be perfect (or even complete), but because they are instantly useful within a particular engine, or in combination with a particular design software. COLLADA apears to be a well defined format, yet this is the first time I ran across it. The fact that its XML seems to be unpopular with you, despite XML being a completely valid format.


I'm not fond of XML for the simple reason that it has to be parsed as a text file and is difficult to convert into binary. It's great for certain things, for other things it's not as nice.

Quote:
Interest ("that would be cool") does not equal interest ("our next line of 3 AAA games will rely on this format, and its integration it into 3DS max"). Although people express casual interest, you can only hope for adoption, if you get the format used by several commercially succesful and publically exposed games.


I should probably make it clear that this is more of a learning aid to beginning 3D programmers such as myself rather than for use in the actual industry. I make no claim that this could even possibly be used in a commercial game. I would prefer that it only be used by independent developers and smaller companies if used commercially at all, but the license I would release this under does not specify company size.

Quote:
The problem with formats is not their specification (this is a matter of a couple of days work), but their suitability for the task. Asking what people would like to see is not necessarily the same as what people need.


Agreed.

Quote:
In a typical production pipeline, you will want a format that will cover your entire process. Modeling in software X, conventient storage in versioning software, natural capability of change tracking, natural integration into your engine.


Again, I agree.

Quote:
XML for example has the advantage over binary formats of being very versioning software friendly, and can be edited without any tools.


I have yet to encounter someone who edits 3D model files by hand. If you know any, I would like to contact them to find out their opinion on my model format specification.

Quote:
Another major hurdle of generic specifications is the learning curve. Even if you provide the API, documentation, examples, etc., people using it will need to learn it. And there will always be a thing your format will not support. Most productions rely on having flexibility, and adaptability to their immediate needs. Writing and defining a storage format is a moderately trivial task when viewed in light of entire production. What severly outweighs "perfect" format is the ease of use, integration with existing tools and widespread experience with it.


Agreed. My .leo format is not all that easy to use. However, my .lio format, while being limited, will be easier to use.

Quote:
And last, how will you guarantee long-term support for your format. Simply saying "I will support it" is not sufficient.


I will not guarantee support of the format, nor is there any warantee on the format. Such is the nature of open source. However, if there is interest, someone will be upgrading the format for their own needs, and will probably release their version of my format under creative commons too.

Quote:
Agreed, who needs Direct X. Who needs x-box or windows support afterall. DirectX is clearly a non-standard platform, who cares about a few weirdos who use it. What is the point of a standard format that cannot be used with majority of platforms? If that is the case, then label this as OpenGL standard format, not general 3D format.


I should really get rid of that statement. This format will work perfectly with DirectX, it just takes a bit more time and effort to load in, that's all.

Quote:
Will you support importers/exporters for Maya, 3DS, Lightwave, Blender, Unreal editor, Quake editor, Doom3 editor, ... Will it be suitable for streaming for use in online games? Which of these do you consider irrelevant and not worthy of support.


I will write an exporter for 3DS Max, because I need one personally. However, if anyone wants to write exporters for other packages or converters, I invite them to do so. It is open source after all.

Quote:
Popular formats have evolved from popular products. Everyone will tell you that in the end it doesn't matter how good or bad something is, it needs to get the job done. And there isn't a specification in existance, that would do that. It is implementations that count, stuff that has been tried and tested, and can be used out of box right here and right now. When you have a release cycle of 9-18 months, you just cannot afford to wait 3 months for update to format, which introduces support for the one and only feature you need (for example cloth simulation nodes)


Support for future technology (extensibility) is inherent in the file format, I suggest you read my post more carefully. Popular formats are varied and generally not available for use to the general programmer/hobbyist/student. Proprietary formats do not work between modelling packages and do not contain the same feature sets. One of the primary benefits of my format is that it contains the essential information which in my opinion is required for an amateur game developer and is extensible to cover other data.

The purpose of standards is to facilitate ease of communication and openness of format in order to facilitate reduced development and porting times. RAD is not popular in the games industry and I personally believe that is a mistake.

Quote:
The reason standardized formats are having hard time in game development is because of extremly varying needs, insane rate of technology advancement and feature scopes, and the almost negligible need for standardization (there's little asset reuse beyond images).


There are other industries which change rapidly too and they benefit from standardisation. Technological advancement will not affect the usability of my format because it is both backwards compatible and extensible.

Quote:
If you're looking for a learning experience then this is great. But before even thinking about standardization, make sure to look WHY formats are the way they are, and HOW they are used.


Will do. :)

Share this post


Link to post
Share on other sites
Quote:
Original post by Anonymous Poster
Quote:
Popular 3D formats evolved not because they would be perfect (or even complete), but because they are instantly useful within a particular engine, or in combination with a particular design software. COLLADA apears to be a well defined format, yet this is the first time I ran across it. The fact that its XML seems to be unpopular with you, despite XML being a completely valid format.


I'm not fond of XML for the simple reason that it has to be parsed as a text file and is difficult to convert into binary. It's great for certain things, for other things it's not as nice.

The main point being that it's going to take for-freaking-ever to load. Not to mention that the files themselves will be enormous. Imagine parsing a file with arbitrary chunks, hundreds of thousands of vertex elements, and tens of thousands of indexes. It gets ugly for XML very fast.

Share this post


Link to post
Share on other sites
Quote:
Original post by Anonymous Poster
Quote:
In a typical production pipeline, you will want a format that will cover your entire process. Modeling in software X, conventient storage in versioning software, natural capability of change tracking, natural integration into your engine.


Again, I agree.


I forgot to mention, the purpose of a standard model format is to provide compatibility with all kinds of software, be it versioning, modelling packages, engines, etc, without modifying the file or writing exporters (which is a time consuming process). Making this an open source endeavour, rather than a commercial or private one enables people to create their own interfaces into the model format one time only, for use in future. If (fictional character) Bob writes an open source Max exporter for .leo and releases it open source, no one else needs to write one, they just use Bob's exporter, depending on how he chooses to license it. This frees up staff from writing exporters and designing file formats from scratch, reducing the project time and production costs. If there is a requirement for the use of a feature in a game which is not supported by the .leo base "library" data set, then at least the individual in charge of the file format production has an extensible and powerful framework upon which to add more data.

Standards are powerful things if done properly.

BTW, I will recommend that people who create derivative products from my file format and exporters/loaders retain the same license for their exporters which I am using (unless used for a commercial project) in order to preserve the spirit of open source and free software which I am writing this file format with.

Share this post


Link to post
Share on other sites
Quote:
Original post by circlesoft
Quote:
Original post by Anonymous Poster
Quote:
Popular 3D formats evolved not because they would be perfect (or even complete), but because they are instantly useful within a particular engine, or in combination with a particular design software. COLLADA apears to be a well defined format, yet this is the first time I ran across it. The fact that its XML seems to be unpopular with you, despite XML being a completely valid format.


I'm not fond of XML for the simple reason that it has to be parsed as a text file and is difficult to convert into binary. It's great for certain things, for other things it's not as nice.

The main point being that it's going to take for-freaking-ever to load. Not to mention that the files themselves will be enormous. Imagine parsing a file with arbitrary chunks, hundreds of thousands of vertex elements, and tens of thousands of indexes. It gets ugly for XML very fast.
I was just eating my dinner and reading through the COLLADA GDC Presentation... from what I gather from the diagrams at the end, the XML file format is more of an intermediary one - so that all the tools have a common interchange format. There's a step on one slide that indicates it'd (effectively) be compiled down to a binary for the shipped product.

Jack

Share this post


Link to post
Share on other sites
Quote:
I have yet to encounter someone who edits 3D model files by hand. If you know any, I would like to contact them to find out their opinion on my model format specification.


Shaders - text format usually as source. You will not want to be forced to use some editor to edit them.
Future proofing - binary formats are often based on current architecture. Several years ago, it was not at all uncommon to have bit-packed fields, 4,8 and 16 bit indexes and bit-juggling compression schemes to make files small.
Many formats include meta data - You will want your chair model to contain name, description, tags, attributes, all text data. Editing them in a simple notepad will make it considerably easier to toy around with when debugging, testing and modding when you do not want to install whole SDK

The above are just some pro-text points, they are by no means definitive. Just trying to point out, there's other ways to look at data rather than just through engine/3DS point of view.

Quote:
The main point being that it's going to take for-freaking-ever to load. Not to mention that the files themselves will be enormous. Imagine parsing a file with arbitrary chunks, hundreds of thousands of vertex elements, and tens of thousands of indexes. It gets ugly for XML very fast.


Completely true. But this matters at one point only, during run-time. An xml can always be "binarized" for high performance, but it still retains the richness of the original. Think source vs. binary comparison. Or AutoCAD and dxf/dwg. Same content, same structure, different encoding. While I forgot the details, I believe dxf has emphasis on backward AND forward compatibility.

I do agree however, that XML is being way too abused for stuff it never should be.

Quote:
RAD is not popular in the games industry and I personally believe that is a mistake.


Very true. But what is RAD and why do such tools apear? They are used in areas, where topics and domains are simple, repetitive and defined on meta levels. There's tons of DB management RAD tools, tons of CRUD application RAD tools. But RAD tools do not allow going beyond the borders of what they were designed for.
Unreal engine is a RAD tool. Can you make a MMORTS with it? Gamebuilder is a RAD tool. Can you make a web based multi-user card/poker/slots casino with it? And development of RAD tools inevitably responds to the needs of developers, hence it lags behind cutting edge by 6 months or over a year in most cases.

Quote:
I will not guarantee support of the format, nor is there any warantee on the format. Such is the nature of open source. However, if there is interest, someone will be upgrading the format for their own needs, and will probably release their version of my format under creative commons too.


Standards succeed only if they are standardized. Any standardized format can succeed only, and only if it is unambiguosly supported by all parties. As soon as divergence occurs, disasters happen. Just look at HTML and its evolution. And that was even a proper standard.

Quote:
Support for future technology (extensibility) is inherent in the file format, I suggest you read my post more carefully.


Sad reality is, future-proofing is impossible. You may make it either incredibly generic, which makes it too broad for immediate practical use, or settle for a set of requirements, only realizing things change in a way you never imagined. XML was designed as an extensible data format. And look what's going on, you have XML programming languages, meta-meta-relation-abstraction layer frameworks defined in XML and operating over XML for description of relation tables, you have 26 flavors of HTML trying to comply with XML.

The most practical solution you could do (study and all), would be to define a meta format. Examine different formats used, then work out a meta format, which would serve as convenient translation tool between them. This might give you a nice academic topic, and give you broad understanding of formats, their strengths and weaknesses, and their specific uses.

A few years back, I worked on a similar hobby project on a meta language, which used an intermediate language to be able to freely translate between several languages (C, Java, Pascal, Oberon). It was useless for practical purposes, but I learned a great deal about intricate details of various languages, and what makes them unique. The greatest benefit that came from there was why "one size fits all" solution is inevitably doomed, and why finding the right tool for the job is critical.

You're in the exactly same situation here.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
Shaders - text format usually as source. You will not want to be forced to use some editor to edit them.
Future proofing - binary formats are often based on current architecture. Several years ago, it was not at all uncommon to have bit-packed fields, 4,8 and 16 bit indexes and bit-juggling compression schemes to make files small.
Many formats include meta data - You will want your chair model to contain name, description, tags, attributes, all text data. Editing them in a simple notepad will make it considerably easier to toy around with when debugging, testing and modding when you do not want to install whole SDK

The above are just some pro-text points, they are by no means definitive. Just trying to point out, there's other ways to look at data rather than just through engine/3DS point of view.


Excellent points. I'll think about writing a text file format in future.

EDIT: I've decided against writing a text file format. COLLADA seems to fill that niche nicely, I see no reason to reinvent the wheel of text based extensible file formats as it were.

Quote:
Very true. But what is RAD and why do such tools apear? They are used in areas, where topics and domains are simple, repetitive and defined on meta levels. There's tons of DB management RAD tools, tons of CRUD application RAD tools. But RAD tools do not allow going beyond the borders of what they were designed for.
Unreal engine is a RAD tool. Can you make a MMORTS with it? Gamebuilder is a RAD tool. Can you make a web based multi-user card/poker/slots casino with it? And development of RAD tools inevitably responds to the needs of developers, hence it lags behind cutting edge by 6 months or over a year in most cases.


I'm not a fan of RAD tools either (they often live up to their reputation of being poorly programmed, badly thought out and buggy), but I will concede that they can reduce development time considerably in most cases.

Quote:
Standards succeed only if they are standardized. Any standardized format can succeed only, and only if it is unambiguosly supported by all parties. As soon as divergence occurs, disasters happen. Just look at HTML and its evolution. And that was even a proper standard.


I would cite for you the development of Unix and other open source programs. I will not go into arguing the validity and modification of open source code. I will however point out that it is beyond my financial means to develop and support a real standard. However, if a company wants to develop a real industry standard based on the .leo file format and invest the time and money to provide support for it, by all means.

Quote:
Sad reality is, future-proofing is impossible. You may make it either incredibly generic, which makes it too broad for immediate practical use, or settle for a set of requirements, only realizing things change in a way you never imagined. XML was designed as an extensible data format. And look what's going on, you have XML programming languages, meta-meta-relation-abstraction layer frameworks defined in XML and operating over XML for description of relation tables, you have 26 flavors of HTML trying to comply with XML.


I would cite the IFF file format. It was developed ages ago by EA and is still being used today in the form of AIFF, WAV, etc files. What I'm going to provide is a set base functionality for the file data. This can be thought of as similar to OpenGL, in that there are standard functions in OpenGL and extensions provide extra functionality on top of the base function set.

Quote:
The most practical solution you could do (study and all), would be to define a meta format. Examine different formats used, then work out a meta format, which would serve as convenient translation tool between them. This might give you a nice academic topic, and give you broad understanding of formats, their strengths and weaknesses, and their specific uses.


.leo is similar to a meta-format, however, it provides a base set of functionality. You'll see what I mean when I release the file format specification soon.

Quote:
A few years back, I worked on a similar hobby project on a meta language, which used an intermediate language to be able to freely translate between several languages (C, Java, Pascal, Oberon). It was useless for practical purposes, but I learned a great deal about intricate details of various languages, and what makes them unique. The greatest benefit that came from there was why "one size fits all" solution is inevitably doomed, and why finding the right tool for the job is critical.

You're in the exactly same situation here.


I certainly hope not. :)

EDIT: How many of you would be bothered if I used British English instead of American English for this file format?

EDIT: I'm wondering about little-endian and big-endian systems. Could they pose a problem for my file format? (there is a little bit of bit flagging in the format) If so, how could I deal with these problems?

[Edited by - Leo_E_49 on April 4, 2006 1:14:19 AM]

Share this post


Link to post
Share on other sites
Quote:
I'm wondering about little-endian and big-endian systems. Could they pose a problem for my file format? (there is a little bit of bit flagging in the format) If so, how could I deal with these problems?


The endinaness must be defined, unless you specify which platform it is intended for.

Just a thought, why is the format not specified as an extension to IFF. It has same structure, it just specializes individual chunks. The 8-byte size seems the only difference from that, and I would hope that no section of 3D model would never need more than 4Gb (4-byte) for individual section.

File formats shouldn't normally use (type)* notation, that's a matter of memory representation. Putting (type)* into file format specification can create a slightly ambiguous situation, since you're in a sense stating that the value in file is value of a pointer, not an array of elements.

The "arbitrary" size should define how much that is. While it can be deciphered in most cases, several fields can be confusing with regard to which field defines the value.
Also, how is length of strings represented. What is the format of strings (codepage, UNICODE, c strings, pascal strings?)

I also believe that LZ77 (or its update LZW/LZ78 version) patent has expired in 2003/2004 (depending on region).

Share this post


Link to post
Share on other sites
Quote:
Original post by AntheusThe endinaness must be defined, unless you specify which platform it is intended for.


I defined it as little endian somewhere in both of the documents.

Quote:
Just a thought, why is the format not specified as an extension to IFF. It has same structure, it just specializes individual chunks. The 8-byte size seems the only difference from that, and I would hope that no section of 3D model would never need more than 4Gb (4-byte) for individual section.


It's similar, however the headers are different from IFF headers, they have an indexing structure and encryption and compression flags in them. The indexing structure means that it's easier in a .leo to find the chunk you want than it is in a .IFF if you don't know the format of the file, this should theoretically lead to faster loading times. It wouldn't be right to call it IFF.

I'm wondering about the long referencing myself, perhaps I'll cut them. I was trying to future-proof it, but now I think that's being a bit silly.

EDIT: I'm keeping the long referencing, because the long variable is also used for the entire file size. I can imagine the file size of a single .leo file being up to 4 Gb in future, especially if it's used for more than just model data (levels, etc). However, I will change the .lio file format to use integers.

Quote:
File formats shouldn't normally use (type)* notation, that's a matter of memory representation. Putting (type)* into file format specification can create a slightly ambiguous situation, since you're in a sense stating that the value in file is value of a pointer, not an array of elements.


Well the array length is dynamic, typically I use pointers to reference dynamic arrays. However, perhaps a type[] format would be better?

Quote:
The "arbitrary" size should define how much that is. While it can be deciphered in most cases, several fields can be confusing with regard to which field defines the value.


It's hard to define arbitrary, the length of the elements are loaded in by the unsigned int variables above them and the structure is declared by the left column.

For example, say we have 24 vertices. On the left hand column it defines the coordinates as x,y,z. The right hand column says float *.

We'll have vertices loaded in using:
x0,y0,z0,x1,y1,z1,x2,y2,z2,etc
where each of these is a floating point number.

Perhaps I should put <Arbitrary 4,4,4> to describe the size of each of the sub-elements, even though "float *" indicates that each of the elements x,y,z are 4 byte?

Quote:
Also, how is length of strings represented. What is the format of strings (codepage, UNICODE, c strings, pascal strings?)


Yes, I should specify that they are null terminated ASCII character arrays.

EDIT: What's the difference between a c string and a pascal string when stored in a binary file?

Quote:
I also believe that LZ77 (or its update LZW/LZ78 version) patent has expired in 2003/2004 (depending on region).


I'm going to avoid any possibility of complications by steering clear of it at all.

Edit: If someone wants to add in LZ77 they can easily do so.

Thanks for your input, you've been very helpful. :)

[Edited by - Leo_E_49 on April 4, 2006 4:55:00 PM]

Share this post


Link to post
Share on other sites
Quote:
Yes, I should specify that they are null terminated ASCII character arrays.

EDIT: What's the difference between a c string and a pascal string when stored in a binary file?


The naming is a bit dated, but it's still valid.

C-style strings are null terminated.

Pascal style uses length designation first, then the actual string arrays. How length is specified tends to vary, it could be 8, 16 or even 32 bit field, which also determines the maximum string length.

Quote:
Perhaps I should put <Arbitrary 4,4,4> to describe the size of each of the sub-elements, even though "float *" indicates that each of the elements x,y,z are 4 byte?


A proposal:

Number of vertices (N_VERT) | int | int
Vertices | N_VERT * Vertex | Vertex[N_VERT]

This might be better. The size is very exactly specified.

Usually you'd define the types before, so instead of this, you'd have "Vertex" defined at start as "Vertex, 12, float[3]", and then in the table you'd just specify "Vertex * N_VERT" or something along those lines. There's nothing wrong with defining custom data types, especially vectors that aprear quite often.

You can also safely use int/short/long, etc for sizes, but define them beforehand. It tends to work better, since you're explaining the contents, not only physical layout. And once you define int as 4 bytes and little endian, the structure in the file is implied.

This would in some extent also make it unecessary to specify both file and C++ representation, since one implies the other.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this