So then for example, if you had 10 textures they would have the IDs {0,1,2,3,4,5,6,7,8,9}, and if you deleted the last 5 it would look like: {0,1,2,3,4}, then if you added another texture... am I right that it would be assigned ID 10 and 5-9 would never be used again?
Sortof, yes. Since I check at load-time for the highest ID, it'd actually start counting from 5 again. But if you had {0,1,2,3,4,5,6,7,8,9}, and deleted all of them except '9', it'd continue counting at 10.
This isn't a big deal, because I'm using a map not an array, so the "wasted" IDs take up literally zero space. (If I was using a vector, then there would be wasted "empty" spots, but not with maps). And I don't care if I waste a couple ID numbers, because I use uint16_t's as my IDs (unsigned shorts - 2 bytes), giving me up to 65,535 IDs, which is plenty for my needs. If I needed more, I'd use uint32_t's (unsigned ints - 4 bytes), which would give me >4 billion IDs.
But if it became a big deal, it's equally easy to write code to have the editor find an available empty ID, reusing spots that have already been needed. It's just not necessary in my case.
If we take the xml example, I assume the original .xml data was created using the editor itself?
Yes, but they could be hand-created if desired. An editor isn't necessary.
So something like: Open editor, load texture, save as .xml with generated ID, then it saves the map in .dat form also?
So in this case, do the .xml files only exist as a second copy of the data, so as to be human-readable?
Yes, sortof. You don't actually need to save the .dat for smaller projects.
In my engine, when running the editor, the editor saves/writes the XML files. However, as an optimization, when the editor closes, it writes all the texture details into a single .dat file so the actual game doesn't have to go crawling through the directory looking for files, reading and parsing thousands of text human-readable files,
This is a premature optimization on my part, just to make the real game's start-up occur faster for end users. It's premature because it wasn't actually running slow, and I just was guessing by the time I get all my >10,000 different files added to the game, that it might run slow. This is why I say for smaller projects this probably isn't necessary. Or rather, for situations where the number of resources are fewer than a couple thousand.
In my case, the editor automatically scans the directory for textures, and also scans the directory for human-readable files (YAML, in my case).
If it finds textures but no matching YAML, then it auto-adds that texture to the list (but marks it as "New" so I can visually see it in the editor, so I can add any details I need to).
If it finds a YAML but no matching texture file, then it marks that texture's details as "Missing"
Here's my actual code, copy+pasted: (thankfully it's one of the better commented areas of my project)
//Crawls the editor resource directories looking for Yaml files describing what resources exist,
//as well as looking for new resources.
void Resources::CrawlDirectoryForEditorResources()
{
ERROR_CONTEXT("When crawling the resource directories for YAML files...", "");
//Load the tag redirects.
this->Load_TagRedirectMap_ForEditor();
this->Load_MaterialDetails_ForEditor();
//==========================================================
//Load the texture details and animation details.
//---------------------------------------
//Phase 1: Find all the image files.
StringList missingTextures;
StringList imageFilepaths = GetFilesInDirectory(this->Paths.Editor.TextureDirectory, SV_MatchesWildcard({"*.png", "*.jpg", "*.jpeg"}));
//---------------------------------------
//Phase 2: Load all the Yaml files.
StringList yamlFilepaths = GetFilesInDirectory(this->Paths.Editor.TextureDirectory, SV_MatchesWildcard({"*.yaml"}));
for(std::string &filepath : yamlFilepaths)
{
//Load the yaml filepath.
this->priv_loadTextureAnimationDetailsFile(filepath);
//Remove ".yaml" from the filepath, so we can compare the strings later.
if(filepath.size() > 5)
{
//The image filepath is the same as the yaml filepath, except without ".yaml" as an extension.
std::string imageFilepath = filepath.substr(0, (filepath.size() - 5));
//Remove the image filepath from all the images we found when we crawled the directory.
if(!RemoveAll(imageFilepaths, imageFilepath, PathsAreEquivilent))
{
//If we can't find this image filepath in the files we've just crawled, mark it as missing.
missingTextures.push_back(imageFilepath);
}
}
}
//---------------------------------------
//Phase 3: Process our missing and new textures.
//Any remaining filepaths in 'imageFilepaths' were images we aren't yet aware of.
StringList newTextures = imageFilepaths;
//For every new texture, create a new TextureDetails.
for(const std::string &filepath : newTextures)
{
this->CreateTextureAnimationDetailsFile(filepath);
}
//For every missing texture, modify the TextureDetails to mark it as missing.
for(std::string &filepath : missingTextures)
{
Engine::TextureID textureID = MapGetKeyFromValue(this->Details.TextureDetails, Engine::TextureDetails(filepath), Engine::InvalidTextureID);
if(this->Details.TextureDetails.count(textureID) > 0)
{
this->Details.TextureDetails[textureID].fileStatus = TextureDetails::FileStatus::Missing;
}
}
//==========================================================
}
//Creates a new details for the image located at 'textureFilepath' (assumes there is no existing details for the image).
void Resources::CreateTextureAnimationDetailsFile(const std::string &textureFilepath)
{
//Get an available ID.
Engine::TextureID newTextureID = (this->Details.nextTextureID++);
//Fill in the details.
Engine::TextureDetails newTextureDetails;
newTextureDetails.filepath = textureFilepath;
newTextureDetails.tagSuggestions = this->GenerateTagsFromFilepath(textureFilepath);
newTextureDetails.fileStatus = TextureDetails::FileStatus::New;
//Add it to the details map.
this->Details.TextureDetails[newTextureID] = newTextureDetails;
//Save the details.
this->Save_TextureDetail_ForEditor(newTextureID);
}
//Loads the details for the image located at 'textureFilepath', and returns the texture's ID.
Engine::TextureID Resources::priv_loadTextureAnimationDetailsFile(const std::string &yamlFilepath)
{
ERROR_CONTEXT("When loading a TextureDetails/AnimationDetails editor-friendly file", yamlFilepath);
//...yaml crap...
}
//============================================
typedef std::vector<std::string> StringList; //In some header somewhere
//============================================
//In some other file somewhere: (feel free to take!)
#include <boost/filesystem.hpp>
/*
Returns a list of every file in the folder and (if desired) the files in each sub-folder.
If 'timeThreshold' is specified, only files that have been written to *after* that threshold are returned.
If 'fileFilterFunc' is specified, this filters out all file names (not the entire path) that don't pass the filter.
If 'folderFilterFunc' is specified, entire subdirectories are filtered out.
If 'pathPrefix' is specified, this determines what prefixes the returned paths.
Suggestions for 'pathPrefix' are: Nothing ("FolderA"), "./" ("./FolderA") for relative paths, or passing in baseFolder for absolute paths.
Example:
"./File1.ext"
"./File2.ext"
"./FolderB/File3.ext"
"./FolderB/File4.ext"
"./FolderB/Folder2/File5.ext"
"./File6.ext"
*/
StringList GetFilesInDirectory(const std::string &baseFolder, SubFoldersEnum subFoldersEnum, const std::string &pathPrefix, std::time_t timeThreshold)
{
//Call the original function, but with filters that accept everything.
return GetFilesInDirectory(baseFolder, IsAnything, IsAnything, subFoldersEnum, pathPrefix, timeThreshold);
}
StringList GetFilesInDirectory(const std::string &baseFolder, StringValidatorFunc fileFilterFunc, SubFoldersEnum subFoldersEnum, const std::string &pathPrefix, std::time_t timeThreshold)
{
//Call the original function, but with a directory filter that accepts everything.
return GetFilesInDirectory(baseFolder, fileFilterFunc, IsAnything, subFoldersEnum, pathPrefix, timeThreshold);
}
StringList GetFilesInDirectory(const std::string &baseFolder, StringValidatorFunc fileFilterFunc, StringValidatorFunc folderFilterFunc,
SubFoldersEnum subFoldersEnum, const std::string &pathPrefix, std::time_t timeThreshold)
{
StringList listOfFiles;
if(!DirectoryExists(baseFolder))
return listOfFiles;
boost::filesystem::directory_iterator directoryIterator(baseFolder);
boost::filesystem::directory_iterator endOfDirectory; //An unitialized iterator is the end iterator.
//Loop through everything in this directory.
for(; directoryIterator != endOfDirectory; directoryIterator++)
{
//Make sure the item found is a file and not a folder.
if(boost::filesystem::is_regular_file(directoryIterator->path()))
{
//Get the name of the folder.
std::string filename = GetFilenameFromPath(directoryIterator->path().generic_string());
//Check if the filename passes the filter.
if(fileFilterFunc(filename))
{
//Check if it was written to after 'timeThreshold'.
if(timeThreshold == 0 || timeThreshold < boost::filesystem::last_write_time(directoryIterator->path()))
{
//Add it to our list.
listOfFiles.push_back(ConvertWinPathToUnix(pathPrefix + '/' + filename));
}
}
}
else if(boost::filesystem::is_directory(directoryIterator->path()))
{
//Check to make sure we want to crawl subfolders.
if(subFoldersEnum == IncludeSubFolders)
{
//Get the name of the folder.
std::string directoryName = GetFinalDirectoryFromPath(directoryIterator->path().generic_string());
//Check if the directory name passes the filter.
if(folderFilterFunc(directoryName))
{
//Get all subfolders and add them to the list.
listOfFiles += GetFilesInDirectory(baseFolder + "/" + directoryName, fileFilterFunc, folderFilterFunc, IncludeSubFolders, (pathPrefix + '/' + directoryName + '/'), timeThreshold);
}
}
}
}
return listOfFiles;
}
In this case, I'm just showing what I'm doing, so you can get an idea and more food for thought, and am not necessarily advising you do the same. Since you're talking about items, not textures, you don't need to keep track of textures AND yaml files, you can just have your item XML files.
If you don't use an editor, but edit the files by hand, you can do something like: Create an XML file manually, but don't create an <ID> field. When the game starts up, if it finds an XML file without an ID field, it assigns the next available ID and re-saves the file, so the file now has a permanent unique ID. Any missing fields could be initialized to defaults and re-saved, with the 'default' ID being a unique one.
I also wonder about how to refer to the resources by ID. In the first example, if the IDs are assigned by the editor and saved to a .dat, how do I know the IDs of each Item so I can e.g. give an item to the player at some specific point in the story?
In my specific case with the textures, the IDs are displayed by the editor, so I can see that #5742 = blah. In the small ORPG I mentioned previously, we just had our scripts do: AddItemToInventory(playerID, itemID). And really, for the several hundred items we had, this worked fine, because we had our editor open while we edited scripts (a different editor, different project, written by different programmer). Yea, we had to do some scrolling through lists of items to find what the ID was, but that really wasn't a problem.
In my new editor, I wouldn't even have to scroll, since the new editor can search lists of [textures, in my case] using wildcards (e.g. "potion" displaying "Red potion","Potion of healing", "HiPotion", and so on), to filter the lists by tags and display names, and filters instantly while typing in the search box.
Though you could also have your scripts lookup an item by name if you want to, but again you might accidentally later rename the item, or more than one item with the same name or typos in the name (though you can typo numbers too!).
This might look something like this:
AddItemToInventory(LookupItemByName("Bob's Quest Item"))
(Where 'LookupItemByName' would emit a descriptive error if the item isn't found, and a different descriptive error if more than one item with the same name exists).
Or do we read them from inside the editor once the data has been loaded?
Ah yes, this one.
If you don't have a fancy editor, and don't want to waste the development time making one (smart choice!), you could do this very easy hack:
When your game loads up, it reads the directory looking for all the item XML files (and fills in any missing IDs), it could then immediately output (while the game is still running) a very simple, very basic, CSV (comma-seperated-value) text file that contains just the display name and the ID of every item.
1,Potion
2,Sword of Pwning
17,Fluffy Kitten
18,Midnight Armor
142,Bob's Quest Item
This file wouldn't be used by the game at all, just auto-generated by the game at startup for your own use during development.
What you do then is just open it up in Excel or Google Docs or whatever (which you can have open in another window/tab while you work) - both of which can read CSV files, and whenever you want to know the ID of an item for your scripts or events or whatever, you just do a Ctrl+F search for the display name of the item.
(CSV is a real simple format. Feel free to hack apart my CSV outputting function to get started. My function has a few weird dependencies on my own libraries, but you can cut those out easily enough, especially since you don't need all those features)