Jump to content
  • Advertisement
Sign in to follow this  
NekoCode

Optimization Asset storage

This topic is 374 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hey everyone! Currently I am making my engine and I got one thing I am worried about. I am using text data format to store my assets (e.g. JSON objects or kinda "human-readable"

formats), it requires to register every field I want to serialize and read it manually from file data:

void Resource::Save(IFile* file) {
	file->Serialize("myField", myFieldValue);
}
void Resource::Load(IFile* file) {
	file->Deserialize("myField", &myFieldValue)
	.. and so on, manually, nothing else!
}

But I can't breathe calmly since I saw UE4 serialization/asset storage system, it uses rtti, it's MUCH easier to serialize objects and now I am unsure which method I should use: should I give all responsibility to rtti system(with lots of code hidden) or load everything I need manually just like first code example? I know I can code rtti that way so it will output "human-readable" files, but is it good thing to use?

Edited by NekoCode

Share this post


Link to post
Share on other sites
Advertisement

Having recently (mostly) done something like prefabs loading myself, what I can say is: only you can decide. I'm assuming C++ because well, pointers (and I'd advise using references if the functions cannot work if they are null. One less NULL check, a smaller chance of headache seg fault).

Personally, I'd say YAGNI. A RTTI system in c++ is overly complex, will rely on templates or out-of-compiler code management, or special compiler rules, or everything together in a painful mix that'll take ages to finish and have it working fine. You probably won't have the time, and this can kill your motivation. I'd stick - as I did - with serializing and deserializing functions.

There's a chance you'll need or want something more flexible or with less boiler plate, later on. Only when you are sure that something will have a real, consistent, strong return of investment of your time, only then you refactor. This day, after all, might never come.

But anyway, you can make your life easier. Use composition, inheritance or any other technique to isolate the serial code so that it'll be changeable with less effort if you ever choose to. If you never choose to change, you'll still have clean and testable code.

On a personal note, having Save and Load methods for serialization can become a problem. If you want it to go both ways (read and write), I'd say to use only one function for both reading and writing. For example, you can have AbstractSerializer, which is implemented by JsonReadSerializer and JsonWriteSerializer. Only one function is half the effort, and it mitigates the risk of your Write and Read operations somehow getting out of sync about the order or size of data.

Share this post


Link to post
Share on other sites
On 1.11.2017 at 12:35 AM, Kirlim said:

A RTTI system in c++ is overly complex, will rely on templates or out-of-compiler code management, or special compiler rules, or everything together in a painful mix that'll take ages to finish and have it working fine

I do not agree to this as I did an RTTI system some weeks ago on my own from scratch and it took more time about reading about how other people have done it as it took to roll my own one. Yes you will rely mostly on templates and some compiler tricks depending on the complexity but mostly it will be stupid code production work. I'm currently in progress to automate the process (as for the fact I already have my own build tool written in C#) so I added some special comment parsing to it to behave like C# attributes because it is part of a framework and should so be flexible to extend for types.

//$[[Attribute][(Parameter)]] <-- will be parsed to certain attribute with or without fields set

...
  
//$[Serializable]
class A
{
   public:
      //$[Constructor]
      inline A() : myField() { }
  
      //$[Property(Set)]
      inline void MyField(MyType const& value) { myField = value; }
      //$[Property(Get)]
      inline MyType MyField() const { return myField; }
  
  	  //$[Function]
      inline void DoSomething(int N)
      {
         ... 
      }
  
   protected:
      //$[Field]
      //$[Protected]
      MyType myField;
};

will create a C++ header file like this

class TypeInfo<A> : public Type
{
	public:
		inline TypeInfo()
		{
			properties[0] = Property(&ReflectionContext<void (MyType const&)>::ConstClassContext<A, &A::MyField>, &ReflectionContext<MyType ()>::ConstClassContext<A, &A::MyField>, Meta::GetType<MyType>(), "MyField");

			//DoSomething
			#if defined(USE_EXTENDED_RTTI_INFOS) && USE_EXTENDED_RTTI_INFOS == 1
			doSomethingFunctionArgs[0] = Parameter("N", 0, Meta::GetType<int>());
			functions[0] = Function(&ReflectionContext<void (int)>::ClassContext<A, &A::DoSomething>, Meta::GetType<void>(), 1, doSomethingFunctionArgs, "DoSomething");
			#else
			doSomethingFunctionArgs[0] = Meta::GetType<void>();
			doSomethingFunctionArgs[1] = Meta::GetType<int>();
			functions[0] = Function(&ReflectionContext<void (int)>::ClassContext<A, &A::DoSomething>, Meta::GetType<void>(), 2, doSomethingFunctionArgs, "DoSomething");
			#endif

			//ctor~1
			#if defined(USE_EXTENDED_RTTI_INFOS) && USE_EXTENDED_RTTI_INFOS == 1
			constructors[0] = Function(&ReflectionContext<A* ()>::StaticContext<&Create1>, Meta::GetType<void*>(), 0, 0, 0);
			#endif
  
               ...
		}
		inline ~TypeInfo()
		{ }

		inline virtual uint32 TypeId() const { return Meta::GetTypeId<A>(); }

		inline virtual const char* Name() const { return Meta::GetTypeName<A>::Name(); }
		inline virtual const char* FullName() const { return Meta::GetTypeName<A>::FullName(); }

		inline virtual ArrayBuffer<Type*> Composites() { return ArrayBuffer<Type*>(0, 0); }
		inline virtual ArrayBuffer<Type*> Generics() { return ArrayBuffer<Type*>(0, 0); }

		inline virtual ArrayBuffer<Field> Fields() { return ArrayBuffer<Field>(1, fields); }
		inline virtual ArrayBuffer<Property> Properties() { return ArrayBuffer<Property>(1, properties); }
		inline virtual ArrayBuffer<Function> Functions() { return ArrayBuffer<Function>(1, functions); }

		#if defined(USE_EXTENDED_RTTI_INFOS) && USE_EXTENDED_RTTI_INFOS == 1
		inline virtual ArrayBuffer<Function> Constructors() { return ArrayBuffer<Function>(1, constructors); }
		#endif

		inline virtual uint32 Size() const { return sizeof(A); }
		inline virtual uint32 Flags() const { return (Type::SerializableFlag); }

		inline virtual void* Instantiate(IAllocator& allocator, ArrayBuffer<void*>& args) const { return Meta::Creator<A>::Instantiate(allocator, args); }
		inline virtual void* Instantiate(byte* buffer, ArrayBuffer<void*>& args) const { return Meta::Creator<A>::Instantiate(buffer, args); }

	private:
        Field fields[1];
		Property properties[1];

		Function functions[1];
		#if defined(USE_EXTENDED_RTTI_INFOS) && USE_EXTENDED_RTTI_INFOS == 1
		Parameter doSomethingFunctionArgs[2];
		#else
		Type* doSomethingFunctionArgs[3];
		#endif

		#if defined(USE_EXTENDED_RTTI_INFOS) && USE_EXTENDED_RTTI_INFOS == 1
		Function constructors[1];
	
		inline static A* Create1() //proxy function because you cannot get a pointer to constructor
		{
			return MainAllocator::Allocator().Allocate<A>();
		}
		#endif
};

I then include this as A.Meta.h at the bottom into my A.h header file. This is something for convenience rather than really a compiler trick ;)

Anything else is a set of simple template functions for the RTTI informations I want to have including a type's name, id and struct like above that contains its composition.

template <typename T> struct GetTypeName
{
	public:
		static inline const char* Name() { ... }
		static inline const char* FullName() { ... }
};
template <typename T> inline uint32 GetTypeId()
{
	return GetHash(GetTypeName<T>::FullName());  
}
  
template<> inline const char* GetTypeName<void>::FullName() { return "void"; }
template<> inline const char* GetTypeName<void*>::FullName() { return "void"; }
template<> inline uint32 GetTypeId<void>() { return 0; }
template<> inline uint32 GetTypeId<void*>() { return 0; }
  
#define REFLECTABLE_TYPE_NAME(T, NAME) template<> inline const char* GetTypeName<T>::FullName() { return NAME; }
REFLECTABLE_TYPE_NAME(byte, "byte");
REFLECTABLE_TYPE_NAME(bool, "bool");
REFLECTABLE_TYPE_NAME(char, "char");
REFLECTABLE_TYPE_NAME(int8, "sbyte");
REFLECTABLE_TYPE_NAME(int16, "int16");
REFLECTABLE_TYPE_NAME(uint16, "uint16");
REFLECTABLE_TYPE_NAME(int32, "int32");
REFLECTABLE_TYPE_NAME(uint32, "uint32");
REFLECTABLE_TYPE_NAME(long, "long");
REFLECTABLE_TYPE_NAME(unsigned long, "ulong");
REFLECTABLE_TYPE_NAME(int64, "int64");
REFLECTABLE_TYPE_NAME(uint64, "uint64");
REFLECTABLE_TYPE_NAME(float, "float");
REFLECTABLE_TYPE_NAME(double, "double");
REFLECTABLE_TYPE_NAME(const char*, "cstring");
  
class Type
{
	public:
		enum TypeFlags
		{
			SerializableFlag = 0x1,
		};

		virtual uint32 TypeId() const = 0;

		virtual const char* Name() const = 0;
		virtual const char* FullName() const = 0;

		#if defined(USE_EXTENDED_RTTI_INFOS) && USE_EXTENDED_RTTI_INFOS == 1
		virtual ArrayBuffer<Type*> Composites() = 0;
		inline Type* Parent() const 
		{
			if(Composites().Size() == 0) return 0;
			else return Composites()[0];  
		};
		#else
		inline virtual Type* Parent() const { return 0; }
		#endif
		virtual ArrayBuffer<Type*> Generics() = 0;

		virtual ArrayBuffer<Field> Fields() = 0;
		virtual ArrayBuffer<Property> Properties() = 0;
		virtual ArrayBuffer<Function> Functions() = 0;

		#if defined(USE_EXTENDED_RTTI_INFOS) && USE_EXTENDED_RTTI_INFOS == 1
		virtual ArrayBuffer<Function> Constructors() = 0;
		#endif

		virtual uint32 Size() const = 0;
	    virtual uint32 Flags() const = 0;
			
		inline bool IsSerializable() const { return HasFlag(Flags(), SerializableFlag); }

		inline bool operator ==(Type const& other) const { return (TypeId() == other.TypeId()); }
		inline bool operator !=(Type const& other) const { return (TypeId() == other.TypeId()); }

		virtual void* Instantiate(IAllocator& allocator, ArrayBuffer<void*>& args) const = 0;
		virtual void* Instantiate(byte* buffer, ArrayBuffer<void*>& args) const = 0;
};
template<typename T> class TypeInfo : public Type
{
	public:
		inline virtual uint32 TypeId() const { return Meta::GetTypeId<T>(); }

		inline virtual const char* Name() const { return Meta::GetTypeName<T>::Name(); }
		inline virtual const char* FullName() const { return Meta::GetTypeName<T>::FullName(); }

		#if defined(USE_EXTENDED_RTTI_INFOS) && USE_EXTENDED_RTTI_INFOS == 1
		inline virtual ArrayBuffer<Type*> Composites() { return ArrayBuffer<Type*>(0, 0); }
		#endif
		inline virtual ArrayBuffer<Type*> Generics() { return ArrayBuffer<Type*>(0, 0); }

		inline virtual ArrayBuffer<Field> Fields() { return ArrayBuffer<Field>(0, 0); }
		inline virtual ArrayBuffer<Property> Properties() { return ArrayBuffer<Property>(0, 0); }
		inline virtual ArrayBuffer<Function> Functions() { return ArrayBuffer<Function>(0, 0); }

		#if defined(USE_EXTENDED_RTTI_INFOS) && USE_EXTENDED_RTTI_INFOS == 1
		inline virtual ArrayBuffer<Function> Constructors() { return ArrayBuffer<Function>(0, 0); }
		#endif

		inline virtual uint32 Size() const { return sizeof(T); }
		inline virtual uint32 Flags() const { return 0; }

		inline virtual void* Instantiate(IAllocator& allocator, ArrayBuffer<void*>& args) const { return Meta::Creator<T>::Instantiate(allocator, args); }
		inline virtual void* Instantiate(byte* buffer, ArrayBuffer<void*>& args) const { return Meta::Creator<T>::Instantiate(buffer, args); }
};
template<> inline uint32 TypeInfo<void>::Size() const { return 0; }

template <typename T> inline Type* GetType()
{
	static TypeInfo<T> type;
	return &type;
}

As you can see A.Meta.h contains a specialization TypeInfo<A> of template TypeInfo that lets the compiler choose this one instead of the default one. I did this trick because I wanted to have RTTI information for EVERY possible type without compiler errors on template parameters that do not have an own TypeInfo struct yet. Same is true for Name and FullName of GetTypeName struct template. This utilizes the __FUNCSIG__ macro compiler trick that benefits from compile time compiler dependent constant replacement for the complete function signature (using a template function const char* GetName<T>() will then be replaced to "char* const GetName< Namespace::TypeName >(void)") so you could get the full name from your compiler without specializing the template.

I however specialized it for often used types and will specialize it for types with a TypeInfo struct to get a cross platform compatible type id (as I wrote above the __FUNCSIG__ macro trick is compiler dependent and results in different signatures for different compilers) to be used in serialization.

This was the compile time constant stuff, now a small but important runtime code. To utilize a full deserialize you will need a creator function that is capable to map type id to the associated creator. My TypeConverter needs a little runtime initialization to fill a hash table with type id -> function pointer pairs.

An automated Serialize function could then look like follows

void Serialize(IDataWriter& stream, Type const& type, void* instance)
{
	stream.Write((byte*)bigEndian_Cast(type.TypeId), 4);

	const uint8 fieldCount = (uint8)type.Fields.Size();
	stream.Put(fieldCount);
	for(uint8 i = 0; i < fieldCount; i++)
	{
    	const Type& fieldType = type.Fields[i].GetType();
        
        if(fieldType.IsPrimitive()) //simply write a single byte type code and field value
        else
        {
        	const void* fieldValue = type.Fields[i].Get(instance);                    
            if(fieldValue)
            {
            	stream.Write((byte*)bigEndian_Cast(fieldType.TypeId), 4);
            	Serialize(stream, fieldType, type.Fields[i].Get(instance));
            }
            else stream.Put(TypeCodes::Null);
        }
    }
}

or you do the same as for TypeInfo struct and specialize your Serialize functions for each type you want to be serializable. Keep in mind to prepare for versioning of your serializer so you might need to add some extra information (like a chunk id) to group fields for preventing backwards incompatibility.

class A
{
	int X;
    float Y;
};

--> ATypeCode[4]X[4]Y[4]
  
class A
{
	int X;
    float Y;
    double Z;
};

--> ATypeCode[4]X[4]Y[4]Z[8]
  
Deserialize v1 with v2 --> Oops! 8 byte missing
Deserialize v2 with v1 --> Oops! Found a type of 8 bytes that should not be there

 

Share this post


Link to post
Share on other sites

Well, I already have rtti implementation, it does not need out-of-compiler scripts (like @Shaarigan's implementation). But until this moment I didn't know should I utilize rtti-style or not. Now I think I'll mostly use simple serialization/deserialization, but keep rtti system, may be one day it could be useful. Thanks for replies, opened my eyes a bit :)

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!