Sign in to follow this  

C++ IO approach in engine/framework

Recommended Posts

Hi,

During the journey of creating my 2nd 3D engine, I'm basing quite some of the approaches on the Game Engine Architecture book (Jason Gregory). I've now arrived on the topic: IO.
In short, the target platforms for me are PC, XBone and PS4.

With this main assumption I thought of the following logics/ guidelines to follow:

- I can assume file/folder structures will work on all 3 platforms, when I use '/' as a folder separator
-- I will define 1 global with the base folder for the application, the rest will 'inherit' from there
(which can be anything, independent of the 'mount' or drive)
-- for now I'll create 1 subfolder with data/files that might need write access, so later on I only have to worry about 1 subfolder (settings, configs etc.).
- file extensions can be longer than 3 characters (in Linux based FreeBSD on PS4)
- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

My questions:
- what do you think about this approach?
- regarding the last point, which of the 2 options would you apply?
-- option 2 might work fine but feels a bit 'static', not very flexible (on the other hand, would you actually need flexibility here?)

Any input is appreciated, as always.

Share this post


Link to post
Share on other sites
52 minutes ago, cozzie said:

- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

Alternatively, you pass relative file-paths to functions, but use something like SetCurrentDirectory/chdir/... before to "fixup" the filepath. I've personally found that approach pretty viable at least for a lot of operations (ie. the asset-loader will change the current dir before scanning the content-folder for assets). You just have to make sure the change the directory back after the fact, which I decided to use a stack-based solution:

const WorkingDirectory dir(L"../Game"); // change  current directory from APPLICATION/Bin to APPLICATION/Game

{
  const WorkingDirectory dir(L"Scenes"); // change to Game/Scenes
  sceneIO.LoadSceneDeclarations();
} // back to Game/

{
	const WorkingDirectory dir(L"Assets"); // change to Game/Assets
	assetIO.LoadAssets();
} // back to Game/ again

Which is probably part of the reason why it works well for me in the first place. So this way, I seldomly have to pass a full filepath or a folder-path at all (even if I do I usually just create a WorkingDirectory-struct at the header of the function if it involves multiple file-operations, seems way cleaner to me then to string-construct all filepaths to the full-path).

Share this post


Link to post
Share on other sites

"I will define 1 global with the base folder for the application, the rest will 'inherit' from there" - it's not safe to assume that you can store all your read/write files in the same place as your read only files. Usually there is a different location for "program data" to that of "user data", not only because there could be more than one user accessing the same program, but for security reasons.

"file extensions can be longer than 3 characters (in Linux based FreeBSD on PS4)" - file extensions are irrelevant.

"I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.) " - you probably do want a system like this of "well-known" locations, just don't assume they can all be located under one top level directory.

Consider Unity: it offers Application.dataPath (application data, considered read only), Application.persistentDataPath (per-user data), and Application.temporaryCachePath (scratch pad, not intended to persist). You would probably have your game assets relative to the dataPath, but configuration and settings relative to the persistentDataPath.

Share this post


Link to post
Share on other sites

Assuming you always want to load a whole file,instead of some part of it could be tricky, in particular if it also involves decompressing the file while reading.

(A plain archive format like tar that simply concatenates files is simpler, as you can simply seek to the correct spot, assuming the systems support file seeking.)

Share this post


Link to post
Share on other sites
On 9/24/2017 at 1:33 AM, cozzie said:

- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

Consider passing around streams instead of filenames and/or directories.

Most code only cares about accessing its data, not where it's stored or how it's organized with relation to other data, and a stream abstracts away those details. You can easily attach a stream to a disk file, a network source, a memory location, etc., or add in functionality like compression, without requiring any changes to code that only sees an I/O stream. This approach will also make archives (as @Alberth mentioned), which are fairly common in games, trivial to support.

Share this post


Link to post
Share on other sites

When working on console platforms PS4/XB1 your are anyways stuck to certain location you are allowed to read from and potentially another one you are allowed to read/write to for security reasons and the system model. Most engines try to go on those platforms in an async reading way and use memory mapped files to speedup I/O for vendor guideline and commit rule reasons so working with filenames is way more complicated here. Instead a stream interface working on a memory mapped page may make more sense

Share this post


Link to post
Share on other sites

You should not only avoid assuming all files exist within a common root, but also support a choice of multiple possible locations for various groups of writable or temporary files (e.g. saved games in the official recommended location in the user profile or in an arbitrary user-specified path on Windows, downloaded updates in a path with plenty of space and no backup) and a hierarchy of places to load read-only files from (e.g. levels from arbitrary user-selected archives, overriding user-installed packages of assets, overriding patch archives of changed and added core assets, overriding the originally shipped archive of game assets).

Note that some of these locations might not be proper files (e.g. some console-specific API for saved games and downloaded content, or virtual file systems from inside an archive); separate abstract locations and maybe names for various resource types from the lower-level concern of actual access to a file system. 

Share this post


Link to post
Share on other sites
On 9/26/2017 at 3:53 PM, Zipster said:

Consider passing around streams instead of filenames and/or directories.

Most code only cares about accessing its data, not where it's stored or how it's organized with relation to other data, and a stream abstracts away those details. You can easily attach a stream to a disk file, a network source, a memory location, etc., or add in functionality like compression, without requiring any changes to code that only sees an I/O stream. This approach will also make archives (as @Alberth mentioned), which are fairly common in games, trivial to support.

One thing to be aware of with this approach is that even if the code itself doesn't care where the data is coming from, you as the programmer debugging the code probably will at some point. Debugging problems that appear only for specific assets is incredibly frustrating when you have no way to trace a particular piece of data back to its source asset. You may want to consider having at least some kind of debugging-only system that lets you do that.

Share this post


Link to post
Share on other sites

For game assets (textures, levels, etc), I have an abstract "blob loader" API, which you provide with a request -- consisting of an abstract "Asset name" and a bunch of callbacks -- and then it figures out the asset size, calls your allocation-callback with that size to get an address of where to stream the data to, streams the data into that allocation, and then calls your on-completion-callback. On most platforms, the implementation will be streaming sections of one large archive file into memory (and probably also decompressing them). In development builds we also have an implementation that reads an individual files for each asset, so that you don't have to keep running the archive-build process :)

For user profile / save-game data, that's a completely different abstract "user profile data" API that acts like a transactional key-value store. On most platforms, this will write files into the specific directory where the OS expects apps to write their user-specific files to.

Share this post


Link to post
Share on other sites
4 hours ago, Oberon_Command said:

One thing to be aware of with this approach is that even if the code itself doesn't care where the data is coming from, you as the programmer debugging the code probably will at some point. Debugging problems that appear only for specific assets is incredibly frustrating when you have no way to trace a particular piece of data back to its source asset. You may want to consider having at least some kind of debugging-only system that lets you do that.

You can easily have the stream implementations store additional private information in debug build, such as filenames or network addresses. It will be readily available to the debugger if you ever need it.

Share this post


Link to post
Share on other sites
10 hours ago, Zipster said:

You can easily have the stream implementations store additional private information in debug build, such as filenames or network addresses. It will be readily available to the debugger if you ever need it.

This is important :)

Originally my callback-based API was quite minimal for the sake of "efficiency", but it became apparent quite quickly that I needed to pass "useless" data around to the callbacks to assist in debugging, so file parsing callbacks could assist debugging-programmers in tracking the loading of specific files and tell exactly which files they were parsing at which times. 

The "overhead" of tracking a few extra bytes per file request is kind of non-existent anyway... :D

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this  

  • Forum Statistics

    • Total Topics
      628707
    • Total Posts
      2984310
  • Similar Content

    • By Benjamin Shefte
      Hey there,  I have this old code im trying to compile using GCC and am running into a few issues..
      im trying to figure out how to convert these functions to gcc
      static __int64 MyQueryPerformanceFrequency() { static __int64 aFreq = 0; if(aFreq!=0) return aFreq; LARGE_INTEGER s1, e1, f1; __int64 s2, e2, f2; QueryPerformanceCounter(&s1); s2 = MyQueryPerformanceCounter(); Sleep(50); e2 = MyQueryPerformanceCounter(); QueryPerformanceCounter(&e1); QueryPerformanceFrequency(&f1); double aTime = (double)(e1.QuadPart - s1.QuadPart)/f1.QuadPart; f2 = (e2 - s2)/aTime; aFreq = f2; return aFreq; } void PerfTimer::GlobalStart(const char *theName) { gPerfTimerStarted = true; gPerfTotalTime = 0; gPerfTimerStartCount = 0; gPerfElapsedTime = 0; LARGE_INTEGER anInt; QueryPerformanceCounter(&anInt); gPerfResetTick = anInt.QuadPart; } /////////////////////////////////////////////////////////////////////////////// /////////////////////////////////////////////////////////////////////////////// void PerfTimer::GlobalStop(const char *theName) { LARGE_INTEGER anInt; QueryPerformanceCounter(&anInt); LARGE_INTEGER aFreq; QueryPerformanceFrequency(&aFreq); gPerfElapsedTime = (double)(anInt.QuadPart - gPerfResetTick)/aFreq.QuadPart*1000.0; gPerfTimerStarted = false; }  
      I also tried converting this function (original function is the first function below and my converted for gcc function is under that) is this correct?:
      #if defined(WIN32) static __int64 MyQueryPerformanceCounter() { // LARGE_INTEGER anInt; // QueryPerformanceCounter(&anInt); // return anInt.QuadPart; #if defined(WIN32) unsigned long x,y; _asm { rdtsc mov x, eax mov y, edx } __int64 result = y; result<<=32; result|=x; return result; } #else static __int64 MyQueryPerformanceCounter() { struct timeval t1, t2; double elapsedTime; // start timer gettimeofday(&t1, NULL); Sleep(50); // stop timer gettimeofday(&t2, NULL); // compute and print the elapsed time in millisec elapsedTime = (t2.tv_sec - t1.tv_sec) * 1000.0; // sec to ms elapsedTime += (t2.tv_usec - t1.tv_usec) / 1000.0; // us to ms return elapsedTime; } #endif Any help would be appreciated, Thank you!
    • By mister345
      Hi, I'm building a game engine using DirectX11 in c++.
      I need a basic physics engine to handle collisions and motion, and no time to write my own.
      What is the easiest solution for this? Bullet and PhysX both seem too complicated and would still require writing my own wrapper classes, it seems. 
      I found this thing called PAL - physics abstraction layer that can support bullet, physx, etc, but it's so old and no info on how to download or install it.
      The simpler the better. Please let me know, thanks!
    • By lawnjelly
      It comes that time again when I try and get my PC build working on Android via Android Studio. All was going swimmingly, it ran in the emulator fine, but on my first actual test device (Google Nexus 7 2012 tablet (32 bit ARM Cortex-A9, ARM v7A architecture)) I was getting a 'SIGBUS illegal alignment' crash.
      My little research has indicated that while x86 is fine with loading 16 / 32 / 64 bit values from any byte address in memory, the earlier ARM chips may need data to be aligned to the data size. This isn't a massive problem, and I see the reason for it (probably faster, like SIMD aligned loads, and simpler for the CPU). I probably have quite a few of these, particular in my own byte packed file formats. I can adjust the exporter / formats so that they are using the required alignment.
      Just to confirm, if anyone knows this, is it all 16 / 32 / 64 bit accesses that need to be data size aligned on early android devices? Or e.g. just 64 bit size access? 
      And is there any easy way to get the compiler to spit out some kind of useful information as to the alignment of each member of a struct / class, so I can quickly pin down the culprits?
      The ARM docs (http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka15414.html) suggest another alternative is using a __packed qualifier. Anyone used this, is this practical?
    • By Josheir
      In the following code:

       
      Point p = a[1]; center of rotation for (int i = 0; I<4; i++) { int x = a[i].x - p.x; int y = a[i].y - p.y; a[i].x = y + p.x; a[i].y = - x + p.y; }  
      I am understanding that a 90 degree shift results in a change like:   
      xNew = -y
      yNew = x
       
      Could someone please explain how the two additions and subtractions of the p.x and p.y works?
       
      Thank you,
      Josheir
    • By alex1997
      Hey, I've a minor problem that prevents me from moving forward with development and looking to find a way that could solve it. Overall, I'm having a sf::VertexArray object and looking to reander a shader inside its area. The problem is that the shader takes the window as canvas and only becomes visible in the object range which is not what I'm looking for.. 
      Here's a stackoverflow links that shows the expected behaviour image. Any tips or help is really appreciated. I would have accepted that answer, but currently it does not work with #version 330 ...
  • Popular Now