Resource Manager problem
Author
Hello there,
I have a resource management problem of sorts. I have this large amount of resources that may be duplicated, this resources have various types: textures, 3d models, sounds, w/e. The problem is that sometimes I may load a resource which uses another already stored resource, for instance, a 3d model that uses some texture that is common to another 3d model, I want to look up the resources and, if the texture is already in there, point to that texture (instead of reloading it). The problem is in the resource identifier. For textures it COULD be the texture name (assuming that I don't have 2 textures with the same name) but for other things (models,sounds,render targets, blah blah) that's more complicated.
Is there any bitwise hash algorithm that I can use on the data itself to generate unique IDs based on the data itself?
Someone suggested
while(i < datasize)
{
id = id XOR data
i++;
}
Is that "safe" ? is there any other algorithm for doing it, notice that it doesn't have to be VERY secure I can afford 1 or 2 missed resources, but I want a generic way to get IDs
Thanks alot
Hi,
I'm not sure if you mean an identifier that never changes or only an ID that is unique during runtime.
Just use a counter and assign every texture a unique number?
Or use pointers in code and forget about identifiers completely? For human-generated data (e.g. textures, models) it should be no problem to come up with unique names.
I'm not sure if you mean an identifier that never changes or only an ID that is unique during runtime.
Just use a counter and assign every texture a unique number?
Or use pointers in code and forget about identifiers completely? For human-generated data (e.g. textures, models) it should be no problem to come up with unique names.
Quote:For textures it COULD be the texture name (assuming that I don't have 2 textures with the same name)
First off, the asset name should cover 99% of the cases that you will encounter. Because full paths to an asset are unique to a particular model or texture.
Quote:Is there any bitwise hash algorithm that I can use on the data itself to generate unique IDs based on the data itself?
Look up CRC32 or SHA-128 or SHA-256. CRC32 is easy to implement, and there are a few libraries that are collections of hashing algorithms.
The only reason you would need to do this is if you can't trust the assets to be different (ie someone copies "metal.tga" into the tank/textures/ and car/textures/ but it is the same texture).
Author
Thanks, I'll look into those, and in fact, I don't want to care about names, paths and stuff, I see a lot of games with replicate texture files (exactly like your metal example...) I want my resource manager to deal with that automatically, thanks for the ideas.
What are the arguments you pass to your resource manager to load the resource? Those arguments are your key/ID as they should uniquely identify what ever it is you're loading (else how do you know what to load?).
One problem with the "generating an Id from the data itself" approach is that you will need to load the resource anyways, because you cannot know the Id beforehand (unless you only use that Id to load resources).
But I too think that using the resource's filename is sufficient enough.
But I too think that using the resource's filename is sufficient enough.
Engines should be able to go as fast as they possibly can. This means a couple things:
Do not load the same resources twice, wasted memory isnt effecient.
Don't do any processing at runtime you can do at save time (within memory/speed tradeoffs)
It is unwise to process all of your data upon loading it for the mere hope you catch someone doing something silly. And as pointed out you would have to load up the second copy all of the time anyways to determine if they had duplicate data.
In an ideal world each and every asset would have a unique identifier that is quick to process. For simplification lets assume this is an integer. Based on this integer everyone would be able to reference that asset and the resource system would be able to find the asset.
This ideal world isn't always human readable, but sometimes it doesn't have to be. Depending on your toolset you can allow the user to work with human readable names and then always write the Ids into the assets themselves. Unfortunately for truly unique ids you frequently need a database to track things which adds a lot of extra work into your toolset. Though the rewards pay handsomely. With an id based system you will either have to have your asset filenames in a fixed folder with their unique id as their filename, or you will need a zip file like system where the id is the id of the file within that system. This also saves you a lot of memory later on if you don't need to keep any strings around for the path/asset name, although this isn't always feasable if you have a script system and want to keep everything "human readable" in the scripts as well. Some engines make the scripters work with the ids, some resolve the names into ids at runtime (requiring the storing of the names for id lookup) and some may have tools to resolve the names to ids at a preprocessing time when preparing content for the engine (although this can easily mess up debugging scripts in the engine if not careful)
A more natural unique identifier tends to be the file path as mentioned, on the downside strings are expensive to compare (at least compared to an integer) and a '/' instead of a '\' can mess up all comparsions usually requiring some form of path normalization before comparison. A less unique but much faster way is to normalize the path once then hash it. Use the hash on the path (not the data) from then on. On the plus side the path works well for knowing where the file is. The larger the game is though the larger your chances of colliding hashes is. A system used to detect hash collisions (also offline) is pretty important.
If you really want to detect any duplicate data entries this is best left for an offline process where you crc all of your assets and compare the crcs against eachother finding unnecessary duplicates and either fixing up things then or printing out a report for a user to fix them up later.
Where and when you want to put all of your processing is up to you, as mentioned the most effecient methods at runtime require a lot of preprocessing and tool work to keep things working nicely. How you want to spend your time will affect where you put the processing as well.
Do not load the same resources twice, wasted memory isnt effecient.
Don't do any processing at runtime you can do at save time (within memory/speed tradeoffs)
It is unwise to process all of your data upon loading it for the mere hope you catch someone doing something silly. And as pointed out you would have to load up the second copy all of the time anyways to determine if they had duplicate data.
In an ideal world each and every asset would have a unique identifier that is quick to process. For simplification lets assume this is an integer. Based on this integer everyone would be able to reference that asset and the resource system would be able to find the asset.
This ideal world isn't always human readable, but sometimes it doesn't have to be. Depending on your toolset you can allow the user to work with human readable names and then always write the Ids into the assets themselves. Unfortunately for truly unique ids you frequently need a database to track things which adds a lot of extra work into your toolset. Though the rewards pay handsomely. With an id based system you will either have to have your asset filenames in a fixed folder with their unique id as their filename, or you will need a zip file like system where the id is the id of the file within that system. This also saves you a lot of memory later on if you don't need to keep any strings around for the path/asset name, although this isn't always feasable if you have a script system and want to keep everything "human readable" in the scripts as well. Some engines make the scripters work with the ids, some resolve the names into ids at runtime (requiring the storing of the names for id lookup) and some may have tools to resolve the names to ids at a preprocessing time when preparing content for the engine (although this can easily mess up debugging scripts in the engine if not careful)
A more natural unique identifier tends to be the file path as mentioned, on the downside strings are expensive to compare (at least compared to an integer) and a '/' instead of a '\' can mess up all comparsions usually requiring some form of path normalization before comparison. A less unique but much faster way is to normalize the path once then hash it. Use the hash on the path (not the data) from then on. On the plus side the path works well for knowing where the file is. The larger the game is though the larger your chances of colliding hashes is. A system used to detect hash collisions (also offline) is pretty important.
If you really want to detect any duplicate data entries this is best left for an offline process where you crc all of your assets and compare the crcs against eachother finding unnecessary duplicates and either fixing up things then or printing out a report for a user to fix them up later.
Where and when you want to put all of your processing is up to you, as mentioned the most effecient methods at runtime require a lot of preprocessing and tool work to keep things working nicely. How you want to spend your time will affect where you put the processing as well.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
