# C++ IO approach in engine/framework

This topic is 434 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hi,

During the journey of creating my 2nd 3D engine, I'm basing quite some of the approaches on the Game Engine Architecture book (Jason Gregory). I've now arrived on the topic: IO.
In short, the target platforms for me are PC, XBone and PS4.

With this main assumption I thought of the following logics/ guidelines to follow:

- I can assume file/folder structures will work on all 3 platforms, when I use '/' as a folder separator
-- I will define 1 global with the base folder for the application, the rest will 'inherit' from there
(which can be anything, independent of the 'mount' or drive)
-- for now I'll create 1 subfolder with data/files that might need write access, so later on I only have to worry about 1 subfolder (settings, configs etc.).
- file extensions can be longer than 3 characters (in Linux based FreeBSD on PS4)
- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

My questions:
- regarding the last point, which of the 2 options would you apply?
-- option 2 might work fine but feels a bit 'static', not very flexible (on the other hand, would you actually need flexibility here?)

Any input is appreciated, as always.

##### Share on other sites
52 minutes ago, cozzie said:

- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

Alternatively, you pass relative file-paths to functions, but use something like SetCurrentDirectory/chdir/... before to "fixup" the filepath. I've personally found that approach pretty viable at least for a lot of operations (ie. the asset-loader will change the current dir before scanning the content-folder for assets). You just have to make sure the change the directory back after the fact, which I decided to use a stack-based solution:

const WorkingDirectory dir(L"../Game"); // change  current directory from APPLICATION/Bin to APPLICATION/Game

{
const WorkingDirectory dir(L"Scenes"); // change to Game/Scenes
} // back to Game/

{
const WorkingDirectory dir(L"Assets"); // change to Game/Assets
} // back to Game/ again

Which is probably part of the reason why it works well for me in the first place. So this way, I seldomly have to pass a full filepath or a folder-path at all (even if I do I usually just create a WorkingDirectory-struct at the header of the function if it involves multiple file-operations, seems way cleaner to me then to string-construct all filepaths to the full-path).

##### Share on other sites

"I will define 1 global with the base folder for the application, the rest will 'inherit' from there" - it's not safe to assume that you can store all your read/write files in the same place as your read only files. Usually there is a different location for "program data" to that of "user data", not only because there could be more than one user accessing the same program, but for security reasons.

"file extensions can be longer than 3 characters (in Linux based FreeBSD on PS4)" - file extensions are irrelevant.

"I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.) " - you probably do want a system like this of "well-known" locations, just don't assume they can all be located under one top level directory.

Consider Unity: it offers Application.dataPath (application data, considered read only), Application.persistentDataPath (per-user data), and Application.temporaryCachePath (scratch pad, not intended to persist). You would probably have your game assets relative to the dataPath, but configuration and settings relative to the persistentDataPath.

##### Share on other sites

Assuming you always want to load a whole file,instead of some part of it could be tricky, in particular if it also involves decompressing the file while reading.

(A plain archive format like tar that simply concatenates files is simpler, as you can simply seek to the correct spot, assuming the systems support file seeking.)

##### Share on other sites

Thanks all, this really helps.

I'm going to look into better separation of read/write and read data and maybe using the change cwd solution.

##### Share on other sites
On 9/24/2017 at 1:33 AM, cozzie said:

- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

Consider passing around streams instead of filenames and/or directories.

Most code only cares about accessing its data, not where it's stored or how it's organized with relation to other data, and a stream abstracts away those details. You can easily attach a stream to a disk file, a network source, a memory location, etc., or add in functionality like compression, without requiring any changes to code that only sees an I/O stream. This approach will also make archives (as @Alberth mentioned), which are fairly common in games, trivial to support.

##### Share on other sites

When working on console platforms PS4/XB1 your are anyways stuck to certain location you are allowed to read from and potentially another one you are allowed to read/write to for security reasons and the system model. Most engines try to go on those platforms in an async reading way and use memory mapped files to speedup I/O for vendor guideline and commit rule reasons so working with filenames is way more complicated here. Instead a stream interface working on a memory mapped page may make more sense

##### Share on other sites

You should not only avoid assuming all files exist within a common root, but also support a choice of multiple possible locations for various groups of writable or temporary files (e.g. saved games in the official recommended location in the user profile or in an arbitrary user-specified path on Windows, downloaded updates in a path with plenty of space and no backup) and a hierarchy of places to load read-only files from (e.g. levels from arbitrary user-selected archives, overriding user-installed packages of assets, overriding patch archives of changed and added core assets, overriding the originally shipped archive of game assets).

Note that some of these locations might not be proper files (e.g. some console-specific API for saved games and downloaded content, or virtual file systems from inside an archive); separate abstract locations and maybe names for various resource types from the lower-level concern of actual access to a file system.

##### Share on other sites

Thanks for the additional thoughts. Will also dig into file streams

##### Share on other sites
On 9/26/2017 at 3:53 PM, Zipster said:

Consider passing around streams instead of filenames and/or directories.

Most code only cares about accessing its data, not where it's stored or how it's organized with relation to other data, and a stream abstracts away those details. You can easily attach a stream to a disk file, a network source, a memory location, etc., or add in functionality like compression, without requiring any changes to code that only sees an I/O stream. This approach will also make archives (as @Alberth mentioned), which are fairly common in games, trivial to support.

One thing to be aware of with this approach is that even if the code itself doesn't care where the data is coming from, you as the programmer debugging the code probably will at some point. Debugging problems that appear only for specific assets is incredibly frustrating when you have no way to trace a particular piece of data back to its source asset. You may want to consider having at least some kind of debugging-only system that lets you do that.

1. 1
2. 2
3. 3
Rutin
23
4. 4
5. 5
khawk
14

• 9
• 11
• 11
• 23
• 10
• ### Forum Statistics

• Total Topics
633653
• Total Posts
3013153
×