Sign in to follow this  
skwee

File formats for games.

Recommended Posts

Hello there! How are you people? I was thinking, there are a lot of different formats for maps/areas and players for example md5, md3, md2, smd, mdl and etc, and maps like bsp is the only one I know. Also some engines like unreal engine use a self made extensions like utx (for textures) and more. Every single extensions designed specially for the specific game like bsp of quake will have the quake "magic number", quake lump and will be totally different (or maybe not, depends on map i think) from half-life's "magic number" and number of lumps. The perfect thing to do for a beginner game programmer IMHO, is to take some of these extensions learn their format and render them (as i did), but again IMHO its not suitable for a commercial (or not) game that you are going to develop. So I want to ask what do you prefer for a game that you develop and think that it will have potential to be published and to be known by some amount of people (don't must to be best seller or etc) what file format they should use? Should they use already developed formats? Or develop their own? Are there some tutorials or explanation how to design data files like map storage? player storage and animation? I mean IMHO a project that have potential to be published should develop their own format (unless they use some ready engine, again like unreal). What do you think? Well I hate when I cant describe my question in small amount of lines -.- Hope you got me, have a good weekend :)

Share this post


Link to post
Share on other sites
This isn't a direct answer to your question, but it is hopefully something to think about. From where I am standing, there are 3 requirements for each file format that I use:

  • It must be easily created.
    I don't want to spend all my time writing and debugging exporters and conversion tools, so it is best if it is an existing (and widely used) format.

  • It must be easily loaded.
    I don't want to spend all my time writing loaders either, so it must either be a simple, well documented format, or there must be a library that already loads this for me.

  • It must store data efficiently.
    Disk space, and even more importantly, download size, is at a premium. I could store all my 3d models in xml (like Collada), but the files will be at least an order of magnitude larger than an efficient binary format.


However, there is another aspect to this. For instance, the Collada format passes #1 and #2 with flying colours, but fails miserably at #3 - which indicates that I should use Collada for my asset pipeline, but I will need to convert it to a binary format before shipping a product.

Share this post


Link to post
Share on other sites
I actually started my app to create and load my custom data file format. Here's what I shot for when writing my format specification.

Make it flexible
I made a stand alone tool to create my game data files. It has a class for each file type that handles loading the original input files (currently only .ac, .bmp and .wav) and also saving the data in the new format. If I want to add another file format, I just add another class.

Make it fast
Store the data in a format that is closest to how it is stored in memory. Strive to be able to stream the data directly from the file to the object in memory that you're going to use. The less processing after loading the data, the better. In my opinion, drive space is cheap and no one likes long load times.

I also preprocess the data and then make a header that has a list of the object ids, types, and exactly where the object starts in the file so I can jump to any object in the file without sifting through everything before it.

And like swiftcoder said, make a nice easy to read and easy to update specification for the format.

I'm no expert but those are my opinions. [smile]

Share this post


Link to post
Share on other sites
Ok I see.

You two told that you made your own formats. May I ask how? I mean I see one possibility, to model the character/level in 3ds max for example and then write your own exporter that will export it to your format.
If its possible is there any tutorials how to create custom formats? Or things that will help me? I mean I don't need an algorithm like, "take this code, compile, run and you done", just things that will explain me about data storage maybe some compression algorithms, or ways of data organization connected to storage of 3D models, I don't need compression algorithms to compress music for example, not yet :). Thanks a lot.

Share this post


Link to post
Share on other sites
hey

I think designing a file format comes naturally when you understand the data and routines used to processs/use the data.

So firstly you should learn and understand how to render a model with all the features you would like. Well maybe you can start off with something simple, and gradually make things more advanced.

So for starters, maybe learn to render just a model without textures or material properties, and then design a format to suite these requirements. Well for testing your routines and such you may want to make a model loader using something basic/simple, such as the WaveFront .OBJ format.

If your arn't sure about this, then you may want to design your routines based on an existing format. Hmm, or even just looking at other file format descriptions/specifications may help.

Anyhows I advise you to lookup the WaveFront .OBJ file format description/specifications (just google such a term), and learn from them :). It's a very basic ASCII format, so you can just open the file up in a text editor and see the data :).

Otherwise it might even be better to start off by making your own texture loader and/or format, and learn to draw this on a quad or whatever.

Anyhows later on you may want to look into more advanced formats, and get to understand the data and such involved. The DirectX .X format is quite nice, as it supports alot of things (such as skeletal animation), plus it's also an ASCII format. Hmm, if you like I could probably dig up some docs for ya, well I've posted links to such docs in other threads on these forums :).

BTW, I think there is a binary version of the DirectX .X format, but I don't think it's very well supported.

Quote:

You two told that you made your own formats. May I ask how? I mean I see one possibility, to model the character/level in 3ds max for example and then write your own exporter that will export it to your format.


You missed another possibility. That is to write a converter from one format (which should be quite well supported, and support whatever things you need) to your own format ;).

The advantage of this is that you can get to your model format or whatever from many different applications. Well I think most 3d software now days can import/export to many different formats anyhows, so this may not be such a big issue.

Also, this is actually what I originally done.

Well I started doing such things when I designed my own image format, which had a little compression scheme :). I think I converted to it from the Bitmap .BMP format.

Now days in the world of 3d, I use my own texture, model, skeleton, and animation formats. I convert to my texture format from the Truevision .TGA format, and I export (from blender) to my model/skeleton/animation formats. I originally converted to my model/skeleton/animation formats from the DirectX .X format.

Well actually, before doing any of these I had made loaders for the WaveFront .OBJ, and Autodesk .3DS formats. Anyhows I think that's about it. Hmm, somewhere along the line I also figured out the THPS4/THUG/THUG2 (Tony Hawks Pro Skater) image/texture/model formats, and made some converters and such.

Anyhows I think the first 3d format I ever looked into was the WaveFront .OBJ format :), which I would recommend you to firstly look into also.

cya

Share this post


Link to post
Share on other sites
Quote:
Original post by PsychoPumpkin
Make it fast
Store the data in a format that is closest to how it is stored in memory. Strive to be able to stream the data directly from the file to the object in memory that you're going to use. The less processing after loading the data, the better. In my opinion, drive space is cheap and no one likes long load times.
Agreed that you should make it fast, but depending on what you mean by 'fast', you should take different routes. Real-time loading of resources would likely benefit from the method you stated, being something you could just dump straight into use. Typical 'loading screen' type of loading though could often times benefit from some compression, since disk reads are time consuming and your processor would be otherwise useless. Performing real-time loading means you're actively using the processor for more pressing matters, and the heavy processing load associated with unraveling a tight compression would be problematic. Also, you can just throw the disk read into another thread and let the huge read block for who cares how many frames.

If you have files with a very large read latency [large files, or files that will be transfered over a network on demand], then you might want to try to make them useful earlier in the loading process if you're shooting for real-time loading. For example, storing meshes as a number of levels of detail, so loading can be started when the object is far away [and can be rendered in the lower detail], and shift to the higher detail as loading completes, which will be needed as the object moves closer. Seems silly, but makes real time loading of complex data much more tolerable in certain instances where you can't easily predict what resources will be required.

Ask yourself : When do you intend to load your resources.
How soon will you need a resource to be used from when you identify it will be needed.
What information will you want to be included that isn't 'typically' included in a given type of file [typically only really important with animating files, since everything else will likely be either fully standard or fully proprietary].

Also, as far as exporting your resources, it is often times easier to just write a quick converter from an easy to use but insufficient format into your own format than it is to write an actual export script... just something to consider.

Share this post


Link to post
Share on other sites
I'm not sure which is the most hazardous and painstaking pastime: Making video games, making stable nitroglycerine, and making homemade lasers. One of the main problems is that you either have to take off the shelf components and make the rest of your creation fit them, or you have to make your own, whereby they will be bulkier, harder to transport from A to B safely, and not work as efficiently.

For images, I use .tga format. Not perfect, took a while to find a decent library to load it, but well supported. I can photoshop my textures, they support RGBA colour, and I now have libraries to load them.
For static geometry I used the proprietry file format from the application AC3D, which has a public domain file format, and is easier to read. I then convert it into octrees in memory at load time.

If my game was going commercial I would convert the code to use my own file format for textures and models. But its not, so I

For GUI stuff I considered using CEGUI, or Guichan but they both would have increased my applications dependencies, and were bloated for my purposes. (worms type control panel, and the inventory for a sandbox game). So, I'm now spending a month writing my own. (nearly finished.) Because I wrote it to fit the application, it fits perfectly. I'm using STL containers, and my development time has actually been far shorter than I was warned about.

So here is the point of decision: If you take an off the shelf component will you reduce development time/cost/hair loss or will you be better off making your own component that fits the bill perfectly. The ability to make this decision applies to most disciplines, not only software development.

Share this post


Link to post
Share on other sites
You could always do what was done in a few of the commercial games i've edited use, which is explained (hopefully with examples below);

eg. .tga, .dds, .jpg, .jpeg just simply with a different file extension at the end, to edit them, its simply, change them back to the respective type then save it and change the extension again. Or even more simple, just use those file extensions, makes it really easy. (i use jpeg's and tga's usually).

Maps are just matrices of values so a text document like a .txt should be sufficient.

Share this post


Link to post
Share on other sites
yosh64
Thanks for a detailed explanation.
As I already told I loaded some formats so I do have experience with various formats.

Drigovas
Totally agree.

speciesUnknown
Well many people tell that better to take a ready product and to use it, and its true, the cost of development decreases, also the time of development, time of debugging (cause well you will use a ready loader of gif/tga/jpeg or whatever that probably has been checked and debugged), and well it also help you to focus on your goal (a game? a rocket control system? or whatever) instead of wasting time on rewriting algorithms that already exists in x,y,z.

dmandn
Don't see point in it. Its a possibility but I have no idea why to do it.

Thanks a lot to all who posted :)
Other opinions welcome as well :)

Edit:
I thought about it, why big companies develop their own format for every game? I mean its not that they want to "have experience in compression algorithms or data structures", like me for example, why not to use a ready formats?

[Edited by - s.kwee on February 4, 2008 7:13:47 AM]

Share this post


Link to post
Share on other sites
Quote:
Original post by s.kwee
Well many people tell that better to take a ready product and to use it, and its true, the cost of development decreases, also the time of development, time of debugging (cause well you will use a ready loader of gif/tga/jpeg or whatever that probably has been checked and debugged), and well it also help you to focus on your goal (a game? a rocket control system? or whatever) instead of wasting time on rewriting algorithms that already exists in x,y,z.


It depends a bit. Basically you want to save time and money. If making your own format is quicker/cheaper then learning to use someone elses format and tools then you are better off making your own.

Basically the more complex things become, the more likely it is that you'll save time by using an existing solution.

Another factor to consider is, how well existing solutions fit to your problem. If they are overly complex and you need only 10% of the features or if they are too limited and you cannot expand them, you might want to make your own instead.

For the simple things, sometimes you are able to do them in the time you even need to search and evaluate existing solutions. But there isn't much that is that simple.

Quote:
Original post by s.kwee
Edit:
I thought about it, why big companies develop their own format for every game? I mean its not that they want to "have experience in compression algorithms or data structures", like me for example, why not to use a ready formats?


Obscurity could be a reason. If they don't want that others easily can inspect or reverse engineer their things, self made formats can help to raise the level of skill the other party needs to reverse engineer their properties.

Some companies have spend years of work on in house tools which are used in many of their projects. If their workers are used to those tools, they are better off to develop them further and use them all over than to look for new tools for each new project.

Also for bigger projects it becomes more difficult to find tools that cover 100% of the projects needs, even worse if the projects are on the bleeding edge of technological advancement or are highly specialized in one or more aspects.

Maybe control is also a reason. Particularly if you use closed source 3rd party tools, they are often not in your control. If the vendor goes bankrupt, or stops support, you're out of luck. If you have formats and tools under your control you are less dependant on others.

Share this post


Link to post
Share on other sites
My short answer (w/o drunk comments about lasers and bombs)

sometimes you can save time developing your own component rather than learning to use somebody elses, and making it work. (e.g. my homemade GUI system)

Sometimes you are better using an existing component (e.g. I use .tga for my textures)

Sometimes you can use a combination of the two (e.g. my levels are a pre-built file format with a separate file, my own design, for the entity locations and properties)

Sometimes you can't decide. If in doubt, use an off the shelf component. (no need to make you own texture format, for example).

$0.02

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this