Jump to content

  • Log In with Google      Sign In   
  • Create Account

We're offering banner ads on our site from just $5!

1. Details HERE. 2. GDNet+ Subscriptions HERE. 3. Ad upload HERE.


jmakitalo

Member Since 05 Apr 2006
Offline Last Active Dec 11 2014 08:37 AM

Topics I've Started

Resource manager for open world game

27 November 2014 - 02:28 PM

I'm making an open world (outdoors) cross-platform 3D game with C++ and trying to rework some of the fundamental parts of the code. For loading resources, I have thus far had pretty specialized routines for different type of resources. I would like to unify and improve this part of the code.
 
As I've learned, there are many approached to implementing resource systems and each game has its own requirements. Below I try to list my requirements and issues.
 
TYPES OF RESOURCES
 
The types of resources my game will have:
- texture
- shader
- vertex array object (VAO)
- sound
- skeletal animation
- particle system
 
Those are pretty self-contained resources. But some resources
will depend on others, such as:
- material -> texture,shader
- mesh -> material, VAO
 
I also get the feeling that VAO is a special type of resource in that it is not referenced by name and not declared outside the execution of the game. Perhaps it deserves a dedicated manager.
 
WHEN TO LOAD
 
The world can become quite large, so it would be essential to support loading and discarding resources during gameplay. I think that mostly this applies to textures, as they usually take the most memory. Controlling which resources are loaded and when may not necessarily be the responsibility of resource managers, but they would need no provide an interface for it.
 
I'm thinking of dividing the world into imaginary rectangular sectors. Only resources required by, say, 9 sectors closest to camera need to be in memory. The engine could keep track of this and ask managers to discard/load resources as camera crosses sector boundaries.
 
SMART POINTERS OR HANDLES
 
I've been learning C++11 and am interested in leveraging its functionalities. I've given thought for smart pointers, but their use for resource managers gives mixed feelings. I think that usually resource managers do not take ownership of resources, but only delegate them to users. So one approach would be to store weak pointers in the manager and deliver shared pointers. However, it is not clear whether a resource should be destroyed immediately after it loses its last user. For example, one could define quotas for resource memory use and only after the quota is reached, are some resources destroyed. I guess that handles would prove more flexible for this type of system, but with handles, the manager usually owns the data (not that this is a drawback by itself).
 
Another advantage of handles is that if a resource is requested, it may not be immediately available or may not even be found in some error situation. It would be nice to be able to handle this gracefully, i.e., the handle would temporarily correspond to some dummy resource.
 
RESOURCE HEADERS
 
All resources (except VAO) are defined in text files (XML). Each resource is an XML element with name attribute. The element may have child elements that describe properties of the resource, such as filename, sound volume, animation frame rate, etc. I would think that this data could be read into memory at startup for all resources. A given resource would then have its header in memory and its status would be "not loaded", until the resource is requested to be loaded (a handle is requested).
 
I also want to be able to modify resource parameters in-game. Thus I have written a template class CVar, which is given some type and it can be easily read from XML and can be fetched for an in-game menu system for inspection and modification. A group of CVar objects (CVarGroup) is then given for each resource.
 
I could just store resources of particular type in a vector, which is fully populated at init from the XML file. Then an index to the resource would remain well-defined over the course of the game. A handle would then contain this index, which could point to some dummy data if the resource is not yet loaded.
 
TYPE SAFE STORAGE
 
A major problem I have not figured out yet is how to store the various types of resources in a convenient way. I could make a templated resource manager, that stores a vector of given type. But then I would need to have a separate instance of the manager for each type of resource. Also, as shown above, some resources need to invoke other resource managers, which could then be difficult (loading mesh entails loading material, which then loads textures).
 
On the other hand, I could write a resource manager, which only stores pointers to some resource base class. But in this approach, it seems that dynamic typecasts would have to be performed in hot rendering code. However, this approach could give rise to a really nice interface, such as
 
Handle<Mesh> mesh = manager.get<Mesh>("house");
 
where only a single manager object is used to get all types of resources.
 
A related matter is cache utilization. Is it worth the trouble to try to store the resources contiguously in memory or just as pointers? I would think that, e.g., when rendering meshes in game, it is not likely that mesh data required to draw visible mesh objects would be within a cache line anyway in a complex world.
 
SORTING BY RESOURCES
 
The resource should store some integer that can be use to contruct a sorting key for rendering. For example, meshes should be sorted by shader, material and VAO, so each resource should provide an integer. In the best case, these integer types would be only a few bits long so that they could be combined into one integer type. This should not be a major problem, though. If resources are stored in vector, then the vector index may serve as such integer.
 
MY ATTEMPT
 
I have written some preliminary code, which is based on handles and the approach that each resource has its own manager. Some methods are just there without implementation to sketch it out. I tried to solve the resource manager interdependency by dependency injection, which is illustrated with barebones implementations of material and mesh resources. Please find the code at
 
 
I'm always trying to avoid the singleton pattern so that the managers are objects in some world class.
 
I'm interested in any kind of feedback to improve my approach. Hopefully my goal is not too vaguely stated.

Entity component system: data locality vs. templates

04 November 2014 - 04:41 AM

I'm trying to improve my game/engine design by the use of some clearly defined structuring of objects. My main project is an open world FPS with a lot of static objects, such as trees, rocks etc. In the meanwhile, I figured I could test some new programming ideas on a simple separate platformer game, trying to keep in mind how the ideas would work in the main project.

 

The ECS seems to be quite popular nowdays, but after the praising introductory, many issues seem to arise.

 

I have read

http://www.gamedev.net/page/resources/_/technical/game-programming/understanding-component-entity-systems-r3013

http://gameprogrammingpatterns.com/component.html

which give good, but simplified descriptions. I don't want to over-engineer my approach, but I don't want to end up writing a lot of repeated code either.

 

At the moment, I'm struggling with mainly two things:

 

1) Data locality: store components in contiguous arrays, so that entities only store indices to these arrays. Alternatively, use base component class and templates and store all component base object pointers in a single array.

 

2) Implementation of systems: write directly as methods to the class containing the components so that systems can have access to basicaly everything. Alternatively, implement systems as classes and store system base object pointers in some array.

 

At this point I add some simplified code that hopefully utilizes data locality:

// Components.

struct CTransform
{
 vec3 pos;
 float scale;
};

struct CBoundingShape
{
 vec3 boxSize;
 float radius;
};

struct CMesh
{
 // Refers to some array where mesh data is stored.
 int meshDataIndex;
};

// End of components.

enum EComponentType{
 compTransform,
 compBoundingShape,
 compMesh,
 numComp
};

struct CEntity
{
 // Maps component type to index to component array.
 // Index -1: entity doesn't have the component.
 int componentIndices[numComp];
 
 // For simple entity property tagging.
 long int flags;
};

class CEngine
{
 // Store components in contiguous arrays.
 vector<CTransform> transforms;
 vector<CBoundingShape> boundingShapes;
 vector<CMesh> meshes;
 
 vector<CEntity> entities;
 
 // Add indices to entities here. Only requires transform and bounding shape info.
 CQuadtree *quadtree;

 // Component "getters". 
 CTransform *getTransform(int entityIndex);
 // for each component type.
 
 // Create component and return index to vector.
 int createTransform();
 // for each component type.
 
 void drawMeshes()
 {
  vector<int> indicesToVisible = quadtree->frustumCull();
  
  // Render visible entities that have transform and mesh components.
  // These components are not necessarily contiguous in memory...
  for(size_t i=0; i<indicesToVisible.size(); i++){
   const CEntity &ent = entities[indicesToVisible[i]];
   
   if(!getTransform(ent) || !getMesh(ent))
    continue;
   
   drawMesh(getTransform(ent), getMesh(ent));
  }
 }
};

There are several issues here.

a) Adding new type of components is a hassle and prone to error: for each new type, one has to implement createSomeComponent and getSomeComponent.

b) For drawing meshes, any realistic game will have some spatial partitioning. In my usual implementation, the quadtree/some other returns a vector of indices to visible entities. When using these indices, the entities and their components that are actually processed may not be contiguous anymore.

c) Here drawMeshes() can be seen as a system operating on certain components. By keeping up this way, the CEngine class will soon be bloated with system methods. However, drawing meshes needs access to a variety of the engine's internals, such as textureManager, shaderManager, vertexArrayManager etc. (not shown above). Thus separating systems into individual class objects could still lead to these objects being very tightly coupled with the engine class.

 

I guess one thing that affects whether (a) will be an issue, is the granularity of components. But I think that in order to take advantage of ECS, one should give a component just one type of thing to store.

 

I'm thinking if I should have more than one type of ECS. The engine could implement something similar to the above, trying to achieve data locality with the expense of more difficult component management. The game could implement its own ECS, which would be based on template based design, allowing the creation of game specific components easily (here performance might not be so critical). The motivation is that I would think that usually a game world may consist of thousands of entities, but only few of them would be equipped with actual game related components, whereas most would have a transform and a mesh component.


Threads with SDL within C++ classes

24 September 2014 - 07:26 AM

I'm planning on trying out SDL 1.2 threads for my cross-platform game to run some specific tasks. At first a terrain data streamer (even though there may be better ways to do this directly). I'm trying first to figure out how to make a C++ class based design so that the required mutexes would be contained within the class and it would be safe to use the class interface.

 

Below I tried to envision a class for processing some data (here just an array of integers) by a separate thread. The idea is that the main thread creates an instance of the class and then requests work to be done by method requestWorkToBeDone(). The main thread can then ask for the status of the work by getStatus() and acquire the processed data by getData(). The class should be responsible for appropriate mutexes.

enum EStatus {
 statusIdle,
 statusWorking,
 statusDone
};

class MyClass
{
public:
 ~MyClass();
 bool init(); // Create thread and mutex, allocate data.

 static int threadFuncMediator(void *p) // called by SDL_CreateThread to run threadFunc() in separate thread.
 {
  return (static_cast<MyClass*>p)->threadFunc();
 }

 void getStatus(EStatus &_status) const
 {
  SDL_mutexP(mutex);
  _status = status;
  SDL_mutexV(mutex);
 }

 bool requestWorkToBeDone(int someParameter)
 {
  SDL_mutexP(mutex);
  if(status!=statusIdle){
   SDL_mutexV(mutex);
   return false;
  }

  status = statusWorking;

  // Use someParameter to specify work ...

  SDL_mutexV(mutex);

  return true;
 }

 bool getData(int *_data)
 {
  SDL_mutexP(mutex);
  if(status!=statusDone){
   SDL_mutexV(mutex);
   return false;
  }
  memcpy(_data, data, sizeof(int)*numData);
  status = statusIdle;
  SDL_mutexV(mutex);

  retrurn true;
 }
private:
 SDL_Thread *thread;
 SDL_mutex *mutex;

 EStatus status;

 int numData;
 int *data;

 // This runs in a thread created by init() via threadFuncMediator().
 // Calls getStatus() to see if work should be done.
 // When work is done, calls setStatus(statusDone);
 int threadFunc();
};

The thing I'm worried is that threadFunc() also has to call getStatus() to see if it should do work or not. But the instance of the class is owned by the main thread, so is this function call itself safe, eventhough data handling within the function is handled with mutexes?


Advanced heightfields in Bullet physics

16 September 2014 - 01:19 AM

At the moment I'm using Bullet for physics and collision detection with a basic heightfield terrain. The heightmap is 2048*2048 and so cannot yield very high resolution for my roughly 8 km x 8 km terrain.

 

I'm using a vertex shader for some portions of the terrain, which locally masks another heightmap, which is repeated 100 times, over the terrain. The heightmaps and mask are read in vertex shader to displace the terrain grid. The grid itself can be quite dense, so that the additional information from the masked heightmap is visibly utilized. This way I can achieve nice bumpy appearance for, e.g., forests and fields.

 

The problem is communicating this masked heightfield to the physics part of the game. My quick hack was to make a higher resolution map, say 4096x4096, where the two heightmaps are combined, and pass it to the Bullet physics. Of course this has still insufficient resolution in some cases and it wastes memory.

 

I started wondering if it would be easy to modify Bullet's btHeightfieldTerrainShape.cpp module to allow for such additional masked heightmap. A quick look gave me the idea that Bullet just samples the given heightmap on demand and doesn't do any costly pre-processing or caching. This suggests that I could add my masked heightmap easily by modifying getRawHeightFieldValue function and storing the other heightmap and mask in btHeightfieldTerrainShape. Can anyone confirm this?

 

Another related topic is that in the future, I may want to divide my terrain into, say 1 km x 1 km blocks that have their own heightmaps. The heightmaps would then be stored at varying resolutions and streamed from the disk. I would need to be able to update the Bullet heightfield quickly after a stream is complete and this requires that Bullet does not do any heavy processing on heightfield updates. Has anyone experience with anything similar?

 

Thank you in advance.


Efficient instancing in OpenGL

30 June 2014 - 06:32 AM

The game I'm working on should be able to render dense forests with many trees and detailed foliage. I have been using instancing for drawing pretty much everything, but even so, I have lately hit some performance issues.

 

My implementation is based on storing instance data in uniforms. I restrict the object transformations so that only translation, uniform scale and rotation along one axis are allowed. For the rotation part, I pass sin(angle) and cos(angle) as uniforms. Thus 6 floats are passed per instance. This way, I can easily draw 256 instances at once by invoking glUniform4fv, glUniform2fv and glDrawElementsInstancedBaseVertex per batch. The particular draw command is used, because I use large VBO:s that store multiple meshes.

 

Lately I have noticed, that the performance is too low for my purposes. I used gDebugger in an attempt to finding the bottleneck. The FPS count was initially roughly 40. Lowering texture resolution had no effect. Disabling raster commands had negligible effect. Disabling draw commands boosted FPS to over 100. Thus I guess the conclusion is that the excecution is not CPU nor raster operation bound, but has to do with vertex processing.

 

I'm also using impostors for the trees, and level of detail for the meshes, but I have the feeling that I should be able to draw more instances of the meshed trees than what I'm currently able to. I actually had quite ok FPS of 80 with just the trees in place, but adding the foliage (a lot of instances of small low poly meshes) dropped the FPS to 40. Disabling either the trees or the foliage increases the FPS significantly. Disabling the terrain, which uses a lot of polygons, has no effect, so I think the issue is not being just bound by polygon count.

 

Could it be that uploading the uniform data is the limiting factor?

 

For some of the instanced object types, such as the trees, the transformation data is static and is stored in the leaf nodes of a bounding volume hierarchy (BVH) in proper arrays, so that glUniform* can be called without further assembly of data. It would then make sense to actually store these arrays in video memory. What is the best way to do this these days? I think that VBO:s are used in conjuction with glVertexAttribDivisor. To me this does not seem very neat approach, as "vertex attributes" are used for something that are clearly of "uniform" nature. But anyway, I could then make a huge VBO for the entire BVH and store a base instance value and number of instances for each leaf node. To render a leaf node, I would then use glDrawElementsInstancedBaseVertexBaseInstance. This is GL 4.2 core spec. which might be a bit too high. Are there better options? I also have objects (the foliage), for which the transformation data is dynamic (updated occasionally), as they are only placed around the camera. What would be the best way to store/transfer the transformation data in this case?

 

Thank you in advance.


PARTNERS