IO approach in engine/framework

Started by
12 comments, last by Hodgman 6 years, 6 months ago

Hi,

During the journey of creating my 2nd 3D engine, I'm basing quite some of the approaches on the Game Engine Architecture book (Jason Gregory). I've now arrived on the topic: IO.
In short, the target platforms for me are PC, XBone and PS4.

With this main assumption I thought of the following logics/ guidelines to follow:

- I can assume file/folder structures will work on all 3 platforms, when I use '/' as a folder separator
-- I will define 1 global with the base folder for the application, the rest will 'inherit' from there
(which can be anything, independent of the 'mount' or drive)
-- for now I'll create 1 subfolder with data/files that might need write access, so later on I only have to worry about 1 subfolder (settings, configs etc.).
- file extensions can be longer than 3 characters (in Linux based FreeBSD on PS4)
- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

My questions:
- what do you think about this approach?
- regarding the last point, which of the 2 options would you apply?
-- option 2 might work fine but feels a bit 'static', not very flexible (on the other hand, would you actually need flexibility here?)

Any input is appreciated, as always.

Crealysm game & engine development: http://www.crealysm.com

Looking for a passionate, disciplined and structured producer? PM me

Advertisement
52 minutes ago, cozzie said:

- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

Alternatively, you pass relative file-paths to functions, but use something like SetCurrentDirectory/chdir/... before to "fixup" the filepath. I've personally found that approach pretty viable at least for a lot of operations (ie. the asset-loader will change the current dir before scanning the content-folder for assets). You just have to make sure the change the directory back after the fact, which I decided to use a stack-based solution:


const WorkingDirectory dir(L"../Game"); // change  current directory from APPLICATION/Bin to APPLICATION/Game

{
  const WorkingDirectory dir(L"Scenes"); // change to Game/Scenes
  sceneIO.LoadSceneDeclarations();
} // back to Game/

{
	const WorkingDirectory dir(L"Assets"); // change to Game/Assets
	assetIO.LoadAssets();
} // back to Game/ again

Which is probably part of the reason why it works well for me in the first place. So this way, I seldomly have to pass a full filepath or a folder-path at all (even if I do I usually just create a WorkingDirectory-struct at the header of the function if it involves multiple file-operations, seems way cleaner to me then to string-construct all filepaths to the full-path).

"I will define 1 global with the base folder for the application, the rest will 'inherit' from there" - it's not safe to assume that you can store all your read/write files in the same place as your read only files. Usually there is a different location for "program data" to that of "user data", not only because there could be more than one user accessing the same program, but for security reasons.

"file extensions can be longer than 3 characters (in Linux based FreeBSD on PS4)" - file extensions are irrelevant.

"I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.) " - you probably do want a system like this of "well-known" locations, just don't assume they can all be located under one top level directory.

Consider Unity: it offers Application.dataPath (application data, considered read only), Application.persistentDataPath (per-user data), and Application.temporaryCachePath (scratch pad, not intended to persist). You would probably have your game assets relative to the dataPath, but configuration and settings relative to the persistentDataPath.

Assuming you always want to load a whole file,instead of some part of it could be tricky, in particular if it also involves decompressing the file while reading.

(A plain archive format like tar that simply concatenates files is simpler, as you can simply seek to the correct spot, assuming the systems support file seeking.)

Thanks all, this really helps.

I'm going to look into better separation of read/write and read data and maybe using the change cwd solution.

Crealysm game & engine development: http://www.crealysm.com

Looking for a passionate, disciplined and structured producer? PM me

On 9/24/2017 at 1:33 AM, cozzie said:

- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

Consider passing around streams instead of filenames and/or directories.

Most code only cares about accessing its data, not where it's stored or how it's organized with relation to other data, and a stream abstracts away those details. You can easily attach a stream to a disk file, a network source, a memory location, etc., or add in functionality like compression, without requiring any changes to code that only sees an I/O stream. This approach will also make archives (as @Alberth mentioned), which are fairly common in games, trivial to support.

When working on console platforms PS4/XB1 your are anyways stuck to certain location you are allowed to read from and potentially another one you are allowed to read/write to for security reasons and the system model. Most engines try to go on those platforms in an async reading way and use memory mapped files to speedup I/O for vendor guideline and commit rule reasons so working with filenames is way more complicated here. Instead a stream interface working on a memory mapped page may make more sense

You should not only avoid assuming all files exist within a common root, but also support a choice of multiple possible locations for various groups of writable or temporary files (e.g. saved games in the official recommended location in the user profile or in an arbitrary user-specified path on Windows, downloaded updates in a path with plenty of space and no backup) and a hierarchy of places to load read-only files from (e.g. levels from arbitrary user-selected archives, overriding user-installed packages of assets, overriding patch archives of changed and added core assets, overriding the originally shipped archive of game assets).

Note that some of these locations might not be proper files (e.g. some console-specific API for saved games and downloaded content, or virtual file systems from inside an archive); separate abstract locations and maybe names for various resource types from the lower-level concern of actual access to a file system. 

Omae Wa Mou Shindeiru

Thanks for the additional thoughts. Will also dig into file streams

Crealysm game & engine development: http://www.crealysm.com

Looking for a passionate, disciplined and structured producer? PM me

On 9/26/2017 at 3:53 PM, Zipster said:

Consider passing around streams instead of filenames and/or directories.

Most code only cares about accessing its data, not where it's stored or how it's organized with relation to other data, and a stream abstracts away those details. You can easily attach a stream to a disk file, a network source, a memory location, etc., or add in functionality like compression, without requiring any changes to code that only sees an I/O stream. This approach will also make archives (as @Alberth mentioned), which are fairly common in games, trivial to support.

One thing to be aware of with this approach is that even if the code itself doesn't care where the data is coming from, you as the programmer debugging the code probably will at some point. Debugging problems that appear only for specific assets is incredibly frustrating when you have no way to trace a particular piece of data back to its source asset. You may want to consider having at least some kind of debugging-only system that lets you do that.

This topic is closed to new replies.

Advertisement