Jump to content

  • Log In with Google      Sign In   
  • Create Account

Awesome job so far everyone! Please give us your feedback on how our article efforts are going. We still need more finished articles for our May contest theme: Remake the Classics

#ActualHodgman

Posted 03 October 2012 - 08:20 AM

In my current engine, I use in-place binary formats wherever possible. Most of these don't require any kind of parsing step, or OnLoad/Init type functions whatsoever. You just read the file from disk into memory, cast the binary blob to some specific type of structure, and your game is ready to go immediately. Loading times are completely bound by I/O speed.

The main thing that OnLoad/Parse functions do is patch up pointer values, because you obviously can't serialize pointers to disk directly -- so most of my resource structures don't use pointers, which means they can be read/written to disk as-is, without any kind of serialization layer. Instead of pointers, I use offsets and integer addresses.

e.g. Some structures from my shader file format look like:
struct CBuffer
{
	u32 index;
	StringOffset name;
	Offset<CBufferDefaults> defaults;
	Offset<List<VariableInfo>> variables;
};

struct Sampler
{
	u32 index;
	StringOffset name;
	Offset<SamplerState> samplerState;
};

struct Technique
{
	u32 passMask;
	Offset<List<Pass>>	passes;
	Offset<List<CBuffer>> cbuffers;
	Offset<List<Sampler>> samplers;
};
struct ShaderPackBlob
{
	...
	u32						 numTechniques;
	ArrayOffset<Technique>	  techniques;
	ArrayOffset<StringOffset>   techniqueNames;
};
The shader compiler tool takes .hlsl/.cg files, parses/compiles them, and uses a C# binary writer to write out the above structures. The C++ engine can then just load the whole binary file into memory, and cast it to a ShaderPackBlob, and the game can access any of the sub-structures without having to parse the file.
[edit] Just realised you're asking for a C# engine...
I'm not sure how to implement in-place loading of binary structures like this in C#, but it would probably involve unsafe and StructLayout(LayoutKind.Explicit)...

#1Hodgman

Posted 03 October 2012 - 08:08 AM

In my current engine, I use in-place binary formats wherever possible. Most of these don't require any kind of parsing step, or OnLoad/Init type functions whatsoever. You just read the file from disk into memory, cast the binary blob to some specific type of structure, and your game is ready to go immediately. Loading times are completely bound by I/O speed.

The main thing that OnLoad/Parse functions do is patch up pointer values, because you obviously can't serialize pointers to disk directly -- so most of my resource structures don't use pointers, which means they can be read/written to disk as-is, without any kind of serialization layer. Instead of pointers, I use offsets and integer addresses.

e.g. Some structures from my shader file format look like:
struct CBuffer
{
	u32 index;
	StringOffset name;
	Offset<CBufferDefaults> defaults;
	Offset<List<VariableInfo>> variables;
};

struct Sampler
{
	u32 index;
	StringOffset name;
	Offset<SamplerState> samplerState;
};

struct Technique
{
	u32 passMask;
	Offset<List<Pass>>    passes;
	Offset<List<CBuffer>> cbuffers;
	Offset<List<Sampler>> samplers;
};
struct ShaderPackBlob
{
	...
	u32                         numTechniques;
	ArrayOffset<Technique>      techniques;
	ArrayOffset<StringOffset>   techniqueNames;
};
The shader compiler tool takes .hlsl/.cg files, parses/compiles them, and uses a C# binary writer to write out the above structures. The C++ engine can then just load the whole binary file into memory, and cast it to a "ShaderPackBlob", and the game can access any of the sub-structures without having to parse the file.

PARTNERS