Sign in to follow this  
cozzie

C++ IO approach in engine/framework

Recommended Posts

Hi,

During the journey of creating my 2nd 3D engine, I'm basing quite some of the approaches on the Game Engine Architecture book (Jason Gregory). I've now arrived on the topic: IO.
In short, the target platforms for me are PC, XBone and PS4.

With this main assumption I thought of the following logics/ guidelines to follow:

- I can assume file/folder structures will work on all 3 platforms, when I use '/' as a folder separator
-- I will define 1 global with the base folder for the application, the rest will 'inherit' from there
(which can be anything, independent of the 'mount' or drive)
-- for now I'll create 1 subfolder with data/files that might need write access, so later on I only have to worry about 1 subfolder (settings, configs etc.).
- file extensions can be longer than 3 characters (in Linux based FreeBSD on PS4)
- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

My questions:
- what do you think about this approach?
- regarding the last point, which of the 2 options would you apply?
-- option 2 might work fine but feels a bit 'static', not very flexible (on the other hand, would you actually need flexibility here?)

Any input is appreciated, as always.

Share this post


Link to post
Share on other sites
52 minutes ago, cozzie said:

- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

Alternatively, you pass relative file-paths to functions, but use something like SetCurrentDirectory/chdir/... before to "fixup" the filepath. I've personally found that approach pretty viable at least for a lot of operations (ie. the asset-loader will change the current dir before scanning the content-folder for assets). You just have to make sure the change the directory back after the fact, which I decided to use a stack-based solution:

const WorkingDirectory dir(L"../Game"); // change  current directory from APPLICATION/Bin to APPLICATION/Game

{
  const WorkingDirectory dir(L"Scenes"); // change to Game/Scenes
  sceneIO.LoadSceneDeclarations();
} // back to Game/

{
	const WorkingDirectory dir(L"Assets"); // change to Game/Assets
	assetIO.LoadAssets();
} // back to Game/ again

Which is probably part of the reason why it works well for me in the first place. So this way, I seldomly have to pass a full filepath or a folder-path at all (even if I do I usually just create a WorkingDirectory-struct at the header of the function if it involves multiple file-operations, seems way cleaner to me then to string-construct all filepaths to the full-path).

Share this post


Link to post
Share on other sites

"I will define 1 global with the base folder for the application, the rest will 'inherit' from there" - it's not safe to assume that you can store all your read/write files in the same place as your read only files. Usually there is a different location for "program data" to that of "user data", not only because there could be more than one user accessing the same program, but for security reasons.

"file extensions can be longer than 3 characters (in Linux based FreeBSD on PS4)" - file extensions are irrelevant.

"I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.) " - you probably do want a system like this of "well-known" locations, just don't assume they can all be located under one top level directory.

Consider Unity: it offers Application.dataPath (application data, considered read only), Application.persistentDataPath (per-user data), and Application.temporaryCachePath (scratch pad, not intended to persist). You would probably have your game assets relative to the dataPath, but configuration and settings relative to the persistentDataPath.

Share this post


Link to post
Share on other sites

Assuming you always want to load a whole file,instead of some part of it could be tricky, in particular if it also involves decompressing the file while reading.

(A plain archive format like tar that simply concatenates files is simpler, as you can simply seek to the correct spot, assuming the systems support file seeking.)

Share this post


Link to post
Share on other sites

Thanks all, this really helps.

I'm going to look into better separation of read/write and read data and maybe using the change cwd solution.

Share this post


Link to post
Share on other sites
On 9/24/2017 at 1:33 AM, cozzie said:

- all class members functions needing to load a file, shouldn't have to now about the file structure of the logical application, so they will all take a full filename string including path
(combining root + subfolder + filename and separators is then the responsibility of the caller/ calling code)
- some functions will need to be passed a folder, because contents in that folder need to be read OR I can simply define a small list of defined subfolders (under root/ base), because it won't be more then 5 to 10 folders in total (data/shaders, data/textures, data/objects, data/sound etc.)

Consider passing around streams instead of filenames and/or directories.

Most code only cares about accessing its data, not where it's stored or how it's organized with relation to other data, and a stream abstracts away those details. You can easily attach a stream to a disk file, a network source, a memory location, etc., or add in functionality like compression, without requiring any changes to code that only sees an I/O stream. This approach will also make archives (as @Alberth mentioned), which are fairly common in games, trivial to support.

Share this post


Link to post
Share on other sites

When working on console platforms PS4/XB1 your are anyways stuck to certain location you are allowed to read from and potentially another one you are allowed to read/write to for security reasons and the system model. Most engines try to go on those platforms in an async reading way and use memory mapped files to speedup I/O for vendor guideline and commit rule reasons so working with filenames is way more complicated here. Instead a stream interface working on a memory mapped page may make more sense

Share this post


Link to post
Share on other sites

You should not only avoid assuming all files exist within a common root, but also support a choice of multiple possible locations for various groups of writable or temporary files (e.g. saved games in the official recommended location in the user profile or in an arbitrary user-specified path on Windows, downloaded updates in a path with plenty of space and no backup) and a hierarchy of places to load read-only files from (e.g. levels from arbitrary user-selected archives, overriding user-installed packages of assets, overriding patch archives of changed and added core assets, overriding the originally shipped archive of game assets).

Note that some of these locations might not be proper files (e.g. some console-specific API for saved games and downloaded content, or virtual file systems from inside an archive); separate abstract locations and maybe names for various resource types from the lower-level concern of actual access to a file system. 

Share this post


Link to post
Share on other sites
On 9/26/2017 at 3:53 PM, Zipster said:

Consider passing around streams instead of filenames and/or directories.

Most code only cares about accessing its data, not where it's stored or how it's organized with relation to other data, and a stream abstracts away those details. You can easily attach a stream to a disk file, a network source, a memory location, etc., or add in functionality like compression, without requiring any changes to code that only sees an I/O stream. This approach will also make archives (as @Alberth mentioned), which are fairly common in games, trivial to support.

One thing to be aware of with this approach is that even if the code itself doesn't care where the data is coming from, you as the programmer debugging the code probably will at some point. Debugging problems that appear only for specific assets is incredibly frustrating when you have no way to trace a particular piece of data back to its source asset. You may want to consider having at least some kind of debugging-only system that lets you do that.

Share this post


Link to post
Share on other sites

For game assets (textures, levels, etc), I have an abstract "blob loader" API, which you provide with a request -- consisting of an abstract "Asset name" and a bunch of callbacks -- and then it figures out the asset size, calls your allocation-callback with that size to get an address of where to stream the data to, streams the data into that allocation, and then calls your on-completion-callback. On most platforms, the implementation will be streaming sections of one large archive file into memory (and probably also decompressing them). In development builds we also have an implementation that reads an individual files for each asset, so that you don't have to keep running the archive-build process :)

For user profile / save-game data, that's a completely different abstract "user profile data" API that acts like a transactional key-value store. On most platforms, this will write files into the specific directory where the OS expects apps to write their user-specific files to.

Share this post


Link to post
Share on other sites
4 hours ago, Oberon_Command said:

One thing to be aware of with this approach is that even if the code itself doesn't care where the data is coming from, you as the programmer debugging the code probably will at some point. Debugging problems that appear only for specific assets is incredibly frustrating when you have no way to trace a particular piece of data back to its source asset. You may want to consider having at least some kind of debugging-only system that lets you do that.

You can easily have the stream implementations store additional private information in debug build, such as filenames or network addresses. It will be readily available to the debugger if you ever need it.

Share this post


Link to post
Share on other sites
10 hours ago, Zipster said:

You can easily have the stream implementations store additional private information in debug build, such as filenames or network addresses. It will be readily available to the debugger if you ever need it.

This is important :)

Originally my callback-based API was quite minimal for the sake of "efficiency", but it became apparent quite quickly that I needed to pass "useless" data around to the callbacks to assist in debugging, so file parsing callbacks could assist debugging-programmers in tracking the loading of specific files and tell exactly which files they were parsing at which times. 

The "overhead" of tracking a few extra bytes per file request is kind of non-existent anyway... :D

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this  

  • Forum Statistics

    • Total Topics
      627719
    • Total Posts
      2978790
  • Similar Content

    • By SR D
      I've been learning how to do vertex buffers plus index buffers using Ogre, but I believe this is mostly the same across several engines. I have this question about using vertex buffers + index buffers.
      Using DynamicGeometryGameState (from Ogre) as an example, I noticed that when drawing the cubes, they were programmatically drawn in order within the createIndexBuffer() function like so ...
       
      const Ogre::uint16 c_indexData[3 * 2 * 6] = { 0, 1, 2, 2, 3, 0, //Front face 6, 5, 4, 4, 7, 6, //Back face 3, 2, 6, 6, 7, 3, //Top face 5, 1, 0, 0, 4, 5, //Bottom face 4, 0, 3, 3, 7, 4, //Left face 6, 2, 1, 1, 5, 6, //Right face };
      From the above, the front face is drawn using the vertices 0, 1, 2, 2, 3, 0. But when reading in thousands of vertices from a file, one obviously doesn't code an array specifying which vertices make up a face.
      So how is this done when working with a large number of vertices?
    • By Josheir
      I am working on a SFML c++ program that uses two rendering windows passed from main to the function drawmessage in a class textmessage.  I was using the second window for displaying debug information that is displayed because I could not get the appropriate information from the SFML object.
      With that said, here is the part of that function that works the first time through and does not on the second usage.  I really have changed the code to try and get it working.   For example I created the two objects locally here for testing.  I am sorry about the extra commented statements they help convey the message too.
      There is the same problem though, the statement :     string test =     message_holder10.getString(); is working and shows "asty" on every run.  On the first run of the program there is a display of the text correctly however on the second call there is no display of it.  (I am stepping through until the display command.)
      I feel like I have exhausted my tries so I am asking for help please.
      If it is the font I will just die, I really don't think it is.
       
       
       
                  sf::Text message_holder10;
                  sf::RenderWindow windowtype3(sf::VideoMode(700, 1000), "a");

                  if ((space_is_used && on_last_line) || (space_is_used && ((line_number) == (total_lines - 2))))
                  {

                      //string temp_string = message::Get_Out_Bound_String();
                      //int length_of_string = temp_string.length();
                      sf::Font Fontforscore;
                      if (gflag == 0)
                      {
                          gflag = 1;
                          
                          if (!Fontforscore.loadFromFile("ARIALBD.ttf"))
                          {
                              exit(1);
                          }

                          message_holder10.setFont(Fontforscore);
                          message_holder10.setCharacterSize(100);
                          message_holder10.setFillColor(sf::Color::Red);
                          message_holder10.setOrigin(0, 0);
                          message_holder10.setPosition(0, 0);
                          windowtype2.close();
                      }
                      message_holder10.setString("asty");
                          
                          //int y_for_space = display_y_setting + (total_lines - 2) * each_vertical_offset_is;
                          //int this_width = 0;
                          
                          //float x = message_holder.getLocalBounds().width;

                          
                          
                          //message_holder.setPosition( ( (first_width - x )/2), y_for_space);
                          
                  
                          //windowtype2.close();

                          string test =     message_holder10.getString();
                          
                          windowtype3.clear();
                          windowtype3.draw(message_holder10);
                          windowtype3.display();
                          
                          
                           
       
                          //windowtype.display();
                      

                       Wait_For_Space_Press();
                      
       
      /////////////////////////
       
      Before, the :      windowtype3.display()  without the clear was drawing other text in this call, just not this one particular text message with it!
       
      Thank you so much I am wondering what it can be,
       
      Josheir
    • By Tispe
      Hi
      I want to test out a polymorphic entity component system where the idea is that the components of an entity are "compositioned" using templated multiple inheritance. But I am running into an issue because I am stacking a bunch of methods with the same names inside a class (but they have different signatures). I want these methods to be overloaded by the template type but my compiler says the access is ambiguous. I have issues making them unambiguous with the using declaration because the paramter pack expansion causes a syntax error.
      Can anyone here give me some advice on this?
       
      template <class T> class component { T m_data; protected: component() {}; ~component() {}; public: void set(const T& data) { m_data = data; }; }; template <class ...Ts> class entity : public component<Ts>... { public: entity() {}; ~entity() {}; //using component<Ts>::set...; // syntax error }; struct position { float x{}; float y{}; float z{}; }; struct velocity { float x{}; float y{}; float z{}; }; int main() { entity<position, velocity> myEntity; position pos = { 1.0f, 1.0f, 1.0f }; velocity vel = { 2.0f, 2.0f, 2.0f }; myEntity.set(pos); // error C2385: ambiguous access of 'set' //myEntity.set(vel); return 0; }  
    • By Baemz
      Hello,
      I've been working on some culling-techniques for a project. We've built our own engine so pretty much everything is built from scratch. I've set up a frustum with the following code, assuming that the FOV is 90 degrees.
      float angle = CU::ToRadians(45.f); Plane<float> nearPlane(Vector3<float>(0, 0, aNear), Vector3<float>(0, 0, -1)); Plane<float> farPlane(Vector3<float>(0, 0, aFar), Vector3<float>(0, 0, 1)); Plane<float> right(Vector3<float>(0, 0, 0), Vector3<float>(angle, 0, -angle)); Plane<float> left(Vector3<float>(0, 0, 0), Vector3<float>(-angle, 0, -angle)); Plane<float> up(Vector3<float>(0, 0, 0), Vector3<float>(0, angle, -angle)); Plane<float> down(Vector3<float>(0, 0, 0), Vector3<float>(0, -angle, -angle)); myVolume.AddPlane(nearPlane); myVolume.AddPlane(farPlane); myVolume.AddPlane(right); myVolume.AddPlane(left); myVolume.AddPlane(up); myVolume.AddPlane(down); When checking the intersections I am using a BoundingSphere of my models, which is calculated by taking the average position of all vertices and then choosing the furthest distance to a vertex for radius. The actual intersection test looks like this, where the "myFrustum90" is the actual frustum described above.
      The orientationInverse is the viewMatrix in this case.
      bool CFrustum::Intersects(const SFrustumCollider& aCollider) { CU::Vector4<float> position = CU::Vector4<float>(aCollider.myCenter.x, aCollider.myCenter.y, aCollider.myCenter.z, 1.f) * myOrientationInverse; return myFrustum90.Inside({ position.x, position.y, position.z }, aCollider.myRadius); } The Inside() function looks like this.
      template <typename T> bool PlaneVolume<T>::Inside(Vector3<T> aPosition, T aRadius) const { for (unsigned short i = 0; i < myPlaneList.size(); ++i) { if (myPlaneList[i].ClassifySpherePlane(aPosition, aRadius) > 0) { return false; } } return true; } And this is the ClassifySpherePlane() function. (The plane is defined as a Vector4 called myABCD, where ABC is the normal)
      template <typename T> inline int Plane<T>::ClassifySpherePlane(Vector3<T> aSpherePosition, float aSphereRadius) const { float distance = (aSpherePosition.Dot(myNormal)) - myABCD.w; // completely on the front side if (distance >= aSphereRadius) { return 1; } // completely on the backside (aka "inside") if (distance <= -aSphereRadius) { return -1; } //sphere intersects the plane return 0; }  
      Please bare in mind that this code is not optimized nor well-written by any means. I am just looking to get it working.
      The result of this culling is that the models seem to be culled a bit "too early", so that the culling is visible and the models pops away.
      How do I get the culling to work properly?
      I have tried different techniques but haven't gotten any of them to work.
      If you need more code or explanations feel free to ask for it.

      Thanks.
       
    • By AyeRonTarpas
      A friend of mine and I are making a 2D game engine as a learning experience and to hopefully build upon the experience in the long run.

      -What I'm using:
          C++;. Since im learning this language while in college and its one of the popular language to make games with why not.     Visual Studios; Im using a windows so yea.     SDL or GLFW; was thinking about SDL since i do some research on it where it is catching my interest but i hear SDL is a huge package compared to GLFW, so i may do GLFW to start with as learning since i may get overwhelmed with SDL.  
      -Questions
      Knowing what we want in the engine what should our main focus be in terms of learning. File managements, with headers, functions ect. How can i properly manage files with out confusing myself and my friend when sharing code. Alternative to Visual studios: My friend has a mac and cant properly use Vis studios, is there another alternative to it?  
  • Popular Now