Hello,

I am trying to implement a user-friendly way to label and identify my in-game assets and entities. I have been using an enumeration system where each class of entity or asset will have an enumeration entry, making the program easier to understand. For me, this works. However, I have read about string hashing and how this can be used to give designers and level editors (i.e. myself in the future) the ability to search for and create entities and assets using strings, but still maintains the run-time performance of integer identification.

However, I have not yet been able to find a suitable example of how (and when) hashing could be used in a video-game case to provide this functionality. I assume that the hashing would have to be done at compile time to make the system as fast as integer identification, but then, how would that enable the level editor and other offline tools to use strings?

If someone could provide a concise example of how string hashing works with regard to asset and entity identification I would be greatly appreciative.

Thank you as always.

- Dave Ottley

Basically, these kinds of systems boil down to:int foo = TurnStringIntoIdentifier("foo");
int bar = TurnStringIntoIdentifier("bar");
int other = TurnStringIntoIdentifier(string("f") + "oo");
bool isFoo = other == foo;
bool isBar = other == bar;It doesn't have to be done at compile time, just as long as you're not constantly calling [font=courier new,courier,monospace]TurnStringIntoIdentifier[/font]. As long as you cache the result in an integer variable, then the code's gonna be faster than using string comparisons everywhere.

Aside from hashing, another way to make one of these systems is via string interning, which boils down to something like:container<string> strings;
int TurnStringIntoIdentifier( string input )
{
if strings does not contain input then
add input to strings
endif
return index of input within strings
}

For a basic version using hashing, the [font=courier new,courier,monospace]TurnStringIntoIdentifier[/font] is just a hash function. I often use the Fnv-1a hash, but there's plenty to choose from.
The problem with this basic version is that it's possible for two strings to return the same hash, which means when you compare the integers you can get false-positives. e.g. above, [font=courier new,courier,monospace]isFoo[/font] and [font=courier new,courier,monospace]isBar[/font] could both be true!

One way to deal with this is to still perform regular string comparisons, but only when the hash compare succeeds. If the hash comparison is false, you can skip the string comparison.

Another way is to simply assume that this event will never occur
In my engine, in development builds, whenever I hash a string, I add it to a global map, with the hash as the key and the string as the value. If that key is already present in the map, then I assert that the existing value matches the string that I'm currently hashing. If this assertion fails, it means I've ended up in the unfortunate situation where two strings have generated the same hash. When this occurs, you can change the "salt" constant used in your hashing algorithm until the problem goes away.
In shipping builds, I skip this assertion checking, and assume that no strings will generate the same hash.

You probably don't want a 'real' hash function for this - just associate each string with an integer. For example you could start at zero increment for each string added, putting them as key and value in an std::map. Things get more complicated if you want to remove strings and reuse integers, but you probably won't need to.

Edit: turns out this is called string interning. Thanks Hodgman!

Hodgeman and MrBastard,

Is there any advantage to a hash function over interning? It would seem that interning is always faster. Is it just the memory usage for the string container?

Best,

Dave

The memory usage isn't that bad - you're not likely to have megabytes of identifiers in your game.

Interestingly, many 'map' structures are 'hash tables' (the C++ one is a 'red black tree', but anyway), so when you check if a string is present in your container, the container my be accelerating that lookup by hashing the string.
So if you implement interning, you might also be using hashing somewhat ;)

As for speed, the cost of generating the hash of a string is relative to its length, whereas looking up a string in a sorted container is relative (logarithmically) to the number of items already in the container and the length of the string. This of course varies from container to container, e.g. Lookup in a hash table will be comparable to the cost of the hash!
In any case, the interning/hashing is ok to be expensive, as long as it results in cheap integer comparisons afterwards.

One issue with interning is that it generates different IDs for the same string depending on the order for which you requested your IDs. This means it's hard to make use of these IDs in save games, for example.
Also, if your interned string collection is created at runtime, then you don't have the option of generating IDs at build time. E.g. My asset names are hashed when the files are created, and assets that link to other assets refer to them by these hashes only. At runtime, filenames don't exist.
If a script wants to load an asset by name, it can run the same hashing algorithm at runtime, and get the same ID that was generated at build time.

I've used the Dan Bernstein hash for stuff like this in the past, but as pointed out, the number of bits you use will dictate how frequent a collision occurs. If you get too many collisions you lose the O(1) access time and revert to O(n) plus the cost of doing the hash function.

If you use STL, you could try something like this:

std::map< std::string, DWORD > identMap;

This will essentially hash a std::string into a DWORD and vice versa.

Is there any reason not to use std::hash<std::string> ? Is it slower than the other hash functions mentioned above?

std::hash will skip characters of the string if the string is longer then 10 characters, at least the VS2010 implementation does this. Other then that it won't be any slower or faster then your own implementation, the actual implementation looks like FNV anyway.


template<>
class hash<_STD string>
: public unary_function<_STD string, size_t>
{ // hash functor
public:
typedef _STD string _Kty;
size_t operator()(const _Kty& _Keyval) const
{ // hash _Keyval to size_t value by pseudorandomizing transform
size_t _Val = 2166136261U;
size_t _First = 0;
size_t _Last = _Keyval.size();
size_t _Stride = 1 + _Last / 10;
for(; _First < _Last; _First += _Stride)
_Val = 16777619U * _Val ^ (size_t)_Keyval[_First];
return (_Val);
}
};