Hello everyone, just returning to gamedev after a long break.

At the moment I am trying to understand at a " high level " how drawing is performed in games.

To be honest its been a a struggle ... I am finding it hard to grasp the " big picture " and key , salient points as there are very many pieces that integrate and interact in many various ways.

I understand the " low level " details to a certain extent : VBOs , render targets , shaders , uniforms etc. etc.

The big uncertain area to me is how the " game " interfaces with the " drawing. "

Basically how is drawing organized , the division of responsibilities , the coupling between " game logic " and the " logical update loop " in general with the " draw loop. "

I have tried searching the forums and while there has been much discussion about these topics, I am personally having a hard time understanding the various approaches.

From searching the forums I found a nice post from " haegarr " in a nice thread by " Ansharus. "

Basically " haegarr " describes the " draw loop " as being more or less an extension of the " game update loop " consisting of the following steps :

- Iterate over all meaningful objects in the scene of interest while apply a filter to select which of these objects to draw and in what manner ( e.g. compute LOD , compute current transformation matrices , update particle positions etc. )

- For each object selected to be drawn , insert a " draw task or job " into a queue ( a simplification ), where a " draw task or job " is effectively a complete description of the salient information required to draw this object ( e.g. VBO IDs , texture IDs , etc. ).

- Sort each list of " draw tasks or jobs " to minimize draw state changes.

- Process each newly sorted list of " draw tasks or jobs " ... basically (A) issue OpenGL commands to set all state changes as explicitly indicated in the " draw task or job " (B) activate VBO(s) (C) activate shader program (D) issue OpenGL draw command(s) to draw the geometry.

Now the above is a simplification, certain things may be more complex. For example, there may be multiple lists of " draw tasks or jobs " ( e.g. one for " opaque " objects , one for a certain " draw pass " , etc. ).

I actually understand this description of a " draw loop " .... but certain things are unclear to me.

The biggest area I find difficult to understand is how does the " game " communicates with the " draw loop " as described above.

From searching the forums, the " standard " approach seems to be to maintain a " scene manager " as a sort of database. And then have this database be a kind of central and shared " blackboard " data structure between the " game logic update loop " and the " draw loop. " Basically at the simplest level, the " game logic update loop " writes information to the " scene manager " and the " draw loop " reads this same information from the " scene manager. " Or at least that's what I think

From certain descriptions the " scene manager " sounds very much like a unsorted list. In other descriptions the " scene manager " sounds like some complicated set of data structures ... a quad-tree , unsorted list of everything, sorted lists of specialized things etc. etc.

I understand that a " scene manager " is very much defined by the specifics of the game. But I would be really grateful if someone could describe examples of " scene managers " just for illustration.

For example, if you were to develop a city roaming game like " GTA5 " what kind of data structure(s) would you use for a " scene manager " , what types of objects might be maintained in it , what would add objects to and remove objects from it , how would game objects be added to and removed from it and at what times in the " update loop. " That sort of stuff

Basically the " execution flow " from start of a scene to the shut down of a scene from the perspective of communication or interactions between the " game logic update loop " and the " draw loop " through the shared " scene manager. "

But that's just the " scene " ... what about the objects themselves....

One particularly confusing thing to me is how do the game objects control in some way how they themselves are drawn.

Lets say we have a " spaceship " with very many meshes , particle effects , and other great things. How does this spaceship get instantiated in the scene as a complete drawable object ? Where is its various drawing -related data stored ? What components are responsible for executing what drawing -related work ? How do these components executing drawing -related work communicate with each other ?

From searching around the Internet, most game engines like Unity3D , have the " game logic " as it executes explicitly add instantiate and configure objects that are drawing -related and then change internal member variables of these drawing -related objects as desired over time.

So in our spaceship example, the spaceship " game object " would instantiate and configure a " particle effect " object, insert it into the " scene manager " and then change its properties over time to change the drawable representation of this same spaceship " game object. "

But again, where is this " drawing -related " data stored ... from searching the forums it appears that the " scene manager " only contains some of the data ... mostly the shared data that allows for communication between the " game logic update loop " and the " draw loop. " The more " drawing specific " data such as VBOs is maintained within the " drawing system. "

So effectively there is a division of data between what is maintained in the " game logic " ( mostly " game logic centric " ) and that maintained in the " drawing system " ( mostly " drawing centric " ).

That sounds pretty reasonable .... but what about the division of work ?

The " game logic " should not be " updating " the positions of particles ... or should it ?

From mulling things abit, I think the above description of the " draw loop " suggests that its a more " gradual " transition .... the " game logic " performs more " higher level " work to set internal state for the following " drawing -related " work  , and then the " drawing -related " work performs successive " levels " or " layers " of more " mechanical " drawing -related work until we have " draw tasks or jobs " that can be sent to be drawn by a " minimal " OpenGL or whatever " interface " layer.

So in our example ,  the spaceship " game object " would instantiate and configure a " particle effect " object, insert it into the " scene manager " and then change its properties over time to change the drawable representation of this same spaceship " game object " ... but the " drawing loop " would be responsible for " updating " particle positions , performing collision detection of particles , performing " culling " of particles and doing other more " low level " and " more mechanical " particle drawing -related work in some regular order one-by-one until we have " draw tasks or jobs. "

Does this sound right ?

Well that's it ... sorry for the basic questions and the length of this post. I am very much a newbie to game development.

I would really appreciate it if someone could possibly explain to me how games do their drawing from the " game logic " down to the " insert draw tasks or jobs in draw queue " in a very newbie friendly terms   I sort of understand the parts after the " insert draw tasks or jobs in draw queue " ... its the stuff that occurs before that is confusing for me

Your OP targets the architectural view, and there is of course not a single valid answer. And yes, diving into details will need further discussion in probably more threads. That said, here is an overview of how I manage the stuff. Several other solutions exist as well.

You've already cited a post where the game loop is described as an ordered sequence of updates on sub-systems with the (graphical) rendering being the last step in the loop. That is still the case. However, the term "sub-system" is a bit like the term "manager": It's often too broad. So, let's say that any sub-system can have tasks and services. Such a task is needed to be invoked from the game loop, i.e. essentially one kind of update(...); I say "one kind" because a sub-system may have more than one update to be called, e.g. the animation sub-system comes to mind. The services, on the other hand, are routines that are provided for the use by other sub-systems.

Now with tasks being executed one after one in order of the game loop, we can safely say that any task earlier in the loop is already completed when a later task is executed. Hence the later task can access the results of all earlier tasks by using the appropriate services. This makes a blackboard concept on the given level needless, because we clearly define execution instead of letting sub-systems self look-up whether they can do something. Regarding data ownership, the tasks define what sub-system owns what data and hence provides what service to allow access. However, chances are that data need to be duplicated. This may happen e.g. if a third party API (like physics) is used by a sub-system, and the API has its own idea of data organization.

When I build the game loop, all services of participating sub-systems are registered, and the registry is passed to sub-system factories when a new instance is requested. Therefore sub-systems can look for services they have a dependency on, and store the resulting pointer.

Now coming to the process of instantiation of game objects. First of, because there are several sub-systems involved for every single game object, instantiation is not an ad-hoc process. Remember that a task relies on earlier tasks having done their jobs, an instantiation has to began with the next run through the loop, or else some tasks may become out of sync. Therefore, if a sub-system wants a game object to be instantiated, it calls the (let's say) EntityService::scheduleCreation(Model const*) with the model resource as argument. (We need to shortly discuss what "model" means here in a later section.) As the service's routine name suggests, the service just generates a job and enqueues this in a job list. The same proceeding is done when a game object should be deleted by invoking EntityService::scheduleDeletion(EntityId). Well, the entity management sub-system also has an update task, and hence is integrated into the game loop. Because it manages entities, it must be placed very early in the game loop. The update then removes all active jobs and activates all scheduled jobs. That's all. Later on down the game loop, when a task of another sub-system that manages a component of game objects is updated, it first uses the entity service to get access to all of the active deletion jobs, and handles them accordingly to the purpose of the service (or perhaps sub-system). It then does the same with the creation jobs. Because all tasks depending on are already up-to-date, the task can safely access data of the other sub-systems at that moment. (Of course, in reality things are a bit more complex when considering features like resource prefetching and asynchronous loading.)

Game objects are defined as entities with components, and the belonging data is managed by the sub-systems. So this is a CES - Component Entity System - way of handling game objects. A difference exists when this CES is compared to many of the existing ones: I use a model (which is just a container object) as recipe of how a game object (or "entity") has to be instantiated. That means also that the components are not used directly as they are attached to the Model, but are interpreted by the belonging sub-systems when those instantiate the game object. Hence the internal structure of components may (and often does) differ from the structure seen from the outside. The services then grants read-only access to the internal structure or an image of that.

With this description in mind, a Model instance may provide a ShapeComponent with a mesh inside, a ParticleEmitterComponent, of course a PlacementComponent with positioning and orientation in the world, perhaps an AnimationComponent, a GraphicTechniqueComponent, ... and so forth. The different components are "consumed" by the belonging sub-systems.

Phew, let me take a break here...

Thanks haegarr for taking the time and effort to explain things much further

Your description of " sub-systems " pretty much answers most questions I had. So thank-you for that

Now the bit about game objects is kinda hard for me to understand

Ok so if understand correctly ...

If and at some point in the execution of a game sub-system ( either during a call to a service or during a periodic task i.e. a kind of " update() " ) this same game sub-system wants to instantiate a new game object instance, it registers a asynchronous request to be fulfilled in the next tick or iteration of the " game update loop. "

The reason for this is that for consistency reasons the current iteration of the " game update loop " assumes that no existing game objects are removed in whole or in part and no new game objects are added in whole or in part.

That much I think I understand, thank-you for such clear and thorough explanations

Now the confusing part to me is this bit here ....

Quote

Well, the entity management sub-system also has an update task, and hence is integrated into the game loop. Because it manages entities, it must be placed very early in the game loop. The update then removes all active jobs and activates all scheduled jobs. That's all. Later on down the game loop, when a task of another sub-system that manages a component of game objects is updated, it first uses the entity service to get access to all of the active deletion jobs, and handles them accordingly to the purpose of the service (or perhaps sub-system). It then does the same with the creation jobs. Because all tasks depending on are already up-to-date, the task can safely access data of the other sub-systems at that moment. (Of course, in reality things are a bit more complex when considering features like resource prefetching and asynchronous loading

So game objects are maintained in a " entity management sub-system " which being a type of " sub-system " has an internal state and services it may provide and tasks to be executed at some one or more points in the " game update loop. "

One particular task of the " entity management sub-system " is to be called " very early " ( I imagine at the very start ) of the " game update loop. " This task (A) first removes all existing game objects that were requested to be removed in the preceding tick or iteration of the " game update loop " (B) and then adds all new game objects that were requested to be added in the preceding tick or iteration of the " game update loop. "

I think sounds about right, I am sort of confused because your describe this as " removes all active jobs and activates all scheduled jobs."

The rest I don't really understand very well

Shouldn't it be the responsibility of the " entity management sub-system " to ensure that all new game objects are added and existing game objects removed as requested ( in the preceding iteration of the " game update loop " ) ?

Basically I imagine that the " entity management sub-system " should (A) first remove all existing game objects that were requested to be removed in the preceding tick or iteration of the " game update loop " by removing relevant data in its own internal state and then synchronously calling a relevant routine of each other arbitrary " sub-system " to remove relevant data from the internal state of this same arbitrary " sub-system " ; " sub-system " by " sub-system " in some known and regular order (B) and then adds all new game objects that were requested to be added in the preceding tick or iteration of the " game update loop " by adding relevant data in its own internal state and then synchronously calling a relevant routine of each other arbitrary " sub-system " to add relevant data into the internal state of this same arbitrary " sub-system " ; sub-system " by " sub-system " in some known and regular order.

But from your description it seems that the work to remove existing game objects and add new game objects is performed " lazily " ... first the " entity management sub-system " does its work to remove existing game objects and then add new game objects , then as each arbitrary other " sub-system " has its own task executed it first performs state synchronization with the " entity management sub-system " by querying it for existing game objects that were removed and new game objects that were added and then removing and adding arbitrary relevant data to its own internal state before actually executing its meaningful work.

So basically a single iteration of the " game update loop " looks something like this :

at the very start of this current iteration of this " game update loop " the " entity management sub-system " removes existing game objects as requested to be removed in the  previous iteration of this " game update loop" " entity management sub-system ... and then secondly adds new game objects as requested to be added in the previous iteration of this "game update loop"

at some relevant point in this current iteration of this " game update loop " the " sub-system " named " X " is requested to " update() " whereupon it first queries the " entity management sub-system " to retrieve a listing of all existing game objects that were removed and a listing of all game objects that were added in this current iteration of this " game update loop " as requested to be removed or added, respectively, in the previous iteration of this " game update loop " ... and then secondly processes these lists by removing arbitrary relevant data in its own internal state and adding relevant data in its own internal state ... and then thirdly goes on to perform its arbitrary meaningful " update() " work.

I think that's how it goes

One question I have is why have the " sub-systems " asynchronously perform the removal and addition of data in their internal state in their respective " update() " before performing their arbitrary meaningful work ; in turn " one-by-one " over the course of the current iteration of the " game update loop " and not have the " entity management sub-system " just synchronously " force " this to be done at the very start of the current iteration of the " game update loop " ?

Do you have some links to where I can read more information about " component entity systems " similar to how you have implemented yours ?

I have heard about " component entity systems " , well discussed topic. But there seems to be many " styles " of " component entity system " and very many implementation details that are not well described or at least " glossed over. "

This post is pretty long, I am going to make a new one below with further questions I have

Thank-you so much haegarr for taking the the time and effort and writing so very clearly about these complex topics It really is appreciated

Ok so I am back

So if I understand things correctly, there is a sort of " loose coupling " between the game object representation and the various representations maintained by the various " sub-systems. "

We have " game objects " as containers of " components. " And the whole set of " game objects " being maintained in the " entity management sub-system. "

We then have each arbitrary " sub-system " then maintaining its own internal representations of this same " game object. "

Typically this being some representation of one or more " components " of this same " game object. "

In your example, you might have a " game object " of type " Model " with a " component " of type " ShapeComponent. "

The " Model " contains the " ShapeComponent " and the entire assembly is maintained in the " entity management sub-system. "

Now if and at some point in the execution of a " sub-system " ( either during a call to a service or during a periodic task i.e. a kind of " update() " of ), this same " sub-system " wants to instantiate a new instance of " Model " game object, it registers a asynchronous request to be fulfilled in the next tick or iteration of the " game update loop. "

We can then amend single iteration of the " game update loop " to look something like this :

At the very start of this current iteration of this " game update loop " the " entity management sub-system " removes existing game objects as requested to be removed in the  previous iteration of this " game update loop" " entity management sub-system ... and then secondly adds new game objects as requested to be added in the previous iteration of this "game update loop."

In our example this means that the " entity management sub-system " would process our request to instantiate a new instance of " Model " game object, by instantiating a " game object " container and then configuring it with a new instance of " ShapeComponent " component and then adding this composite object to its own internal data structure(s), or in other words its own internal state.

At some relevant point in this current iteration of this " game update loop " the " sub-system " named " X " is requested to " update() " whereupon it first queries the " entity management sub-system " to retrieve a listing of all existing game objects that were removed and a listing of all game objects that were added in this current iteration of this " game update loop " as requested to be removed or added, respectively, in the previous iteration of this " game update loop " ... and then secondly processes these lists by removing arbitrary relevant data in its own internal state and adding relevant data in its own internal state ... and then thirdly goes on to perform its arbitrary meaningful " update() " work.

In our running example this means that the, for example, " Mesh Drawing sub-system " would receive, from querying the "entity management sub-system," a listing of new game object instances that would include our new instance of " Mesh " game object. It would then detect the presence of the " ShapeComponent " component on this new instance of " Mesh " game object and then instantiate and configure and insert one or more, for example, static geometry meshes in its own internal data structure(s), or in other words its own internal state.

Similar work would be performed on removing game objects.

So in your case, the " game objects " as container objects effectively are configuration data that " data-drive " the substantiation and configuration of purpose-specific objects within relevant " sub-systems. "

When you say this ....

Quote

 Hence the internal structure of components may (and often does) differ from the structure seen from the outside.

I take it that you mean that " game objects " are effectively configuration data that may and generally do not actually reflect the actual " underlying " data contained in and structure of the corresponding object(s) contained in the various " sub-systems. "

In our example, the " Model " game object with a " ShapeComponent " component is more or less a set of simple primitive data e.g. a file path to a polygonal mesh to " load. " This " ShapeComponent " is " interpreted " by the relevant " sub-system(s) " and used to instantiate and configure, for example, a VBO object.

Here the " ShapeComponent " component is a very simple data object... but the " sub-system(s) " then " translate " or " interpret " this simple data to some potentially very complex data.

The following sentence I don't really understand

Quote

The services then grants read-only access to the internal structure or an image of that.

Thinking this over, I think what you mean is that a " sub-system " exposes a " public API " ( excluding the task(s) or in other words specialized " update()s " that are called by the " game update loop " proper ) that allows later ordered " sub-systems " in the " game update loop " to query and obtain read-only data.

The idea being that, as you stated in a earlier post, later ordered " sub-systems " in the " game update loop " read " up-to-date " data from earlier ordered " sub-systems. "

But what if a " sub-system " wants to write to arbitrary data of another " sub-system " ?

Is this done " synchronously " i.e. enqueue a request within a target " sub-system " to be processed in the relevant task or in other words  relevant specialized " update() " of this same target " sub-system " of the next tick or iteration of the " game update loop " ?

This is probably the biggest question I have from your explanation.

Ok another long post, so sorry haegarr for writing so much and asking so many things. I really appreciate you taking the time and effort to explain things to me

Just a few more questions haegarr 

Basically three topics (A) the internal state of " sub-systems " (B) the information flow or communication between " sub-systems " (C) the execution flow of the " game update loop + draw loop. "

From the sounds of things, it looks like there is no real big " scene manager " object of some kind.

It seems that the " entity management sub-system " maintains the authoritative state of game objects ... as containers of components.

The more specialized data being stored and maintained in individual " sub-systems. "

The sorts of things that should be placed in the " game objects " and thus within the internal state of the " entity management sub-system " seem to be configuration data... and pretty much only configuration data.

The more interesting data sounds like it belongs in relevant " sub-systems. " Things like spatial query acceleration data structures, bounding volumes, drawing -related data etc.

In your first reply you mentioned that there is often significant duplication in data across the " sub-systems. "

I take that means that there might be, for example, two quad-trees maintained each in two " sub-systems " in the " game update loop. "

But what about efficiency ? Updating multiple duplicate or near duplicate data structures sounds inefficient. But I suppose it makes for more flexibility. And if things do get slow or memory consumption becomes too high, you can always " combine " common data structures into a single data structure maintained by and in a single " sub-system " and have other " sub-systems " query this " sub-system " as appropriate.

Which brings me to the information flow or communication between " sub-systems. "

The " read-only " thing about services provided by " sub-systems " really is interesting.

My interpretation is something like the following .... please correct me if am wrong

There are basic four things we might want to do :

- Add new game objects

- Delete existing game objects

- Read properties of existing game objects

- Modify or change properties of existing game objects

Looking at the above list, the first two actions are relatively straightforward.... we (A) register a add or delete request ( as appropriate ) to be processed by the " entity management sub-system " in the next iteration of the " game update loop " (B) and then at the very start of the next iteration of the " game update loop, " have the " entity management sub-system " in its " update() " add a new game object instance to its internal state or delete a existing game object instance from its internal state ( a " game object " being a container with components ) (C) and then as each " sub-system " is " updated() " in this same next iteration of the " game update loop " it first adds or deletes data from its internal state to reflect the " delta " state of the " entity management sub-system " ( i.e. added or deleted game objects ).

So we have a asynchronous addition and removal process of " game objects. " A request is registered in " N " iteration of the " game update loop " and this request is processed in the " N + 1 " iteration of the " game update loop. "

The last two actions are more interesting.

From the sound of things, the " game update loop " seems to follow a " linear information flow. "

Basically in a single iteration of the " game update loop " a later ordered task of some " sub-system " can access and read from the internal state of some other " sub-system " that completed its execution earlier in this same iteration of the " game update loop. "

So effectively the " public API " or in other words services or routines exposed by a " sub-system " are " read only. "

This implies that the order of execution of the " sub-systems " comprising the " game update loop " is critical.

I suppose this is where dividing the work of a " sub-system " into multiple tasks or in other words specialized ' update()s " is useful.

This allows us to linearly order tasks more finely so that, for example, " task A " of " sub-system #1 " can execute after " task C " of " sub-system #9 " but before " task B " of this same " sub-system #1. "

But what about modifying data across " sub-systems " ?

In order to guarantee consistency, write operations would need to be asynchronous as well ... just like the addition and removal of game objects.

So that means in the current iteration of the " game update loop " if some task of some " sub-system " wanted to write to the internal state of the " entity management sub-system " or some other " sub-system " it would need to (A) insert or register a request or message into a internal message queue of this same other " sub-system " (B) this same other " sub-system " would then during some " task " in the next ( or in other words N + 1 ) iteration of the " game update loop " would process this request or message and then insert or register a response or message in the internal message queue of this same " sub-system " (C) this same " sub-system " would then during some " task " in the next to next ( or in other words N + 2 ) iteration of the " game update loop " would process this response or message.

So we use message passing, write request sent in " N ' iteration of the " game update loop " is processed in " N + 1 " iteration of the " game update loop " and a response ( as required ) is received and processed in the " N + 2 " iteration of the " game update loop. "

Well anyways that's my questions for now. Sorry for writing so much. Newbie trying to understand these things

10 hours ago, Basket said:

One particular task of the " entity management sub-system " is to be called " very early " ( I imagine at the very start ) of the " game update loop. " This task (A) first removes all existing game objects that were requested to be removed in the preceding tick or iteration of the " game update loop " (B) and then adds all new game objects that were requested to be added in the preceding tick or iteration of the " game update loop. "

I think sounds about right, I am sort of confused because your describe this as " removes all active jobs and activates all scheduled jobs."

...

The purpose of the entity manager (let's name the sub-system so) is to allocate and deallocate entity identifiers and to manage entity creation and deletion jobs. For this it has 4 job queues:

* a queue where currently active creation jobs are linked
* a queue where currently active deletion jobs are linked
* a queue where currently scheduled creation jobs are linked
* a queue where currently scheduled deletion jobs are linked

Whenever a sub-system requests an entity creation or deletion, the entity manager instantiates a corresponding job and enqueues it into the corresponding scheduled jobs queue (a.k.a. it "schedules the job"). Whenever a sub-system asks for the current creation or deletion jobs, it gets access to the corresponding active jobs queue.The entity manager has a task that is to be integrated into the game loop in front of each task of any other sub-system that deals with entity components. When this task's update() is executed, it destroys all jobs enqueued in the both active jobs queues. This is because the previous run through the game loop had given all sub-systems the possibility to perform their part on creation/deletion of entities, so the currently active jobs are now rather deprecated. Then the currently pending jobs queues are made the new currently active jobs queues, so that ATM there are no longer any pending jobs.

Notice that the entity manager itself does not really create or delete entities. It just organizes creation/deletion in a way that the other sub-systems can do creation/deletion w.r.t. their respective participation without ever becoming out-of-order. This is because wherever in the run through the game loop a sub-system means that an entity should be created or deleted, the actual creation/deletion process is deferred until the beginning of the next run, and the sub-systems get involved in the order of dependency.

You may have noticed that the above mechanism works if and only if a sub-system's task actually does not cancel its part due to an inability. A typical reason for cancelation would be a missing resource. To overcome this problem, the entity manager actually deals with a fifth list:

* a list where currently pending creation jobs are linked

I said earlier that an incoming creation request creates a hob in the scheduled creation jobs queue. That is not exactly the case. Instead, the Model instance for which a creation is requested has a kind of BOM attached, i.e. a "bill or resources". Whenever the entity manager is requested for a creation, it first invokes the resource manager (which is the front-end of the resource sub-system) with the said BOM. On return the entity manager is notified whether
a) all resources that are listed in the BOM are available; or else
b) all resources that are listed and tagged as mandatory are available, but other are not yet; or else
c) at least one resource tagged as mandatory is in the load process; or else
d) at least one resource tagged as mandatory is finally not available.

Then, only if a) or b) is the case, the entity manager enqueues the job into the scheduled creation jobs queue; if c) is true, the job is linked to the pending creation jobs list; and finally, if d) is true the job is discarded and an error is logged. Jobs in the pending jobs queue will eventually become scheduled in one of the following runs through the loop, whenever the resource management will notify that all mandatory resources are finally available.

So the process of an entity creation is like this:
* in run N a sub-system requests the creation of an entity
* the entity manager immediately invokes the resource manager with the BOM
* in this simple example, the resource manager returns a "all resources are available"
* the entity manager schedules the creation job
* all sub-sequent sub-system will not see the job, because it is not active yet
* in run N+1 the entity manager's task's update() makes the previously scheduled job become an active job
* subsequently in run N+1, the tasks of other sub-systems cause their part of creation to happen
* at the end of run N+1, the entity is totally build and rendered the first time
 

6 hours ago, Basket said:

So if I understand things correctly, there is a sort of " loose coupling " between the game object representation and the various representations maintained by the various " sub-systems. "

We have " game objects " as containers of " components. " And the whole set of " game objects " being maintained in the " entity management sub-system. "

We then have each arbitrary " sub-system " then maintaining its own internal representations of this same " game object. "

Typically this being some representation of one or more " components " of this same " game object. "

In your example, you might have a " game object " of type " Model " with a " component " of type " ShapeComponent. "

The " Model " contains the " ShapeComponent " and the entire assembly is maintained in the " entity management sub-system. "

Mostly true, but there are some misunderstandings. 

The Model is a container class with a list of components. The sum of all the specific types of the components together with the therein stored parametrization constitutes an entity (or game object; I'm using these both terms mostly equivalent). The Model instance is used only during the entity creation process. It is a static resource, so it will not be altered at any time. Hence I wrote that its role is being a recipe, because it just allows the set of belonging sub-systems to determine what they have to do when creating or deleting an entity that matches the Model.

The entity management sub-system does not know how to deal with particular components. It just knows that other sub-systems need to investigate the Model's components during the creation and deletion process, and that those sub-systems will generate identifiers for the respective inner structures that will result from components.

Each sub-system that itself deals with a component of a Model has some kind of internal structure that is initialized accordingly to the parameters of the component. This inner structure will further be the part that is altered during each run through the game loop. Hence this inner structure is a part of the active state of an entity.

Notice that this is a bit different to some other ECS implementations. Here we have a Model and its components, and we have an entity with its - well, so to say - components. There is some semantic coupling between both kinds of components, but that's already all coupling that exists.

So the entity manager just knows how many entities are in the world, how many entities will be created or deleted soon, and which identifiers are attached to them. Even if the Model is remembered, the entity manager has no understanding of what any of its components means.

EDIT: Well, having so much posts in sequence makes answering complicated. I've the feeling that some answers I've given here are already formulated by yourself in one of the other posts...

3 hours ago, Basket said:

From the sound of things, the " game update loop " seems to follow a " linear information flow. "

Basically in a single iteration of the " game update loop " a later ordered task of some " sub-system " can access and read from the internal state of some other " sub-system " that completed its execution earlier in this same iteration of the " game update loop. "

So effectively the " public API " or in other words services or routines exposed by a " sub-system " are " read only. "

This implies that the order of execution of the " sub-systems " comprising the " game update loop " is critical.

...

Absolutely, although I would not say that services are read-only per se, but they are mostly read-only.

Notice please that the S in ECS stands for "system" (or sub-system in this manner). This makes it distinct from component based entity implementations without (sub-)systems. The purpose of such systems is to deal with a specific more-or-less small aspect of an entity, which is given by one or at most a small number of what is called the "components". The (sub-)systems do this in a bulk operation, i.e. they work on the respective aspect for all managed entities in sequence. If this is done then we have an increment of the total state change done for all entities, and this is the basis for the next sub-sequent sub-system to stack up its own increment.

You're right: This of course works if and only if the sub-systems are run in a defined order. That is the reason for the described structure of the game loop. Well, having a defined order is not bad. A counter-example: When a placement of an entity is updated, running a collision detection immediately is not necessarily okay, because that collision detection may use some other entities with already updated placements and some with not already updated placements. The result would be somewhat incomplete. You may want to read this Book Excerpt: Game Engine Architecture I'm used to cite at moments like this. 

However, this high level architectural decision does not avoid e.g. message passing at some lower level. When message passing is beneficial at some point, let it be the tool of choice.

So sorry for writing so many posts, I just wanted to add a bit more organization, so it woulnd't be a large wall of text.

Ok so I think I understand your concept of " entity management system " ... its a bit complicated, but the basic idea is that (A) game object substantiation and deletion is deferred to next iteration of the " game update loop " so as to allow each " sub-system " to process the request (B) game object substantiation and deletion must be made atomic ( that is " reversable " on error ). That's basically it

Your concept of " Model " is interesting, it basically amounts to a static configuration dataset used to " data-drive " the instantiation and configuration of purpose-specific objects within each " sub-system. "

So I take it that there is no " shared and common " game object dataset maintained somewhere.

Basically game objects are unique IDs : instantiating a new game object causes the " entity management sub-system " to allocate and maintain a unique ID which acts as the global identifier for all state associated with this game object maintained across the " sub-systems. "

To obtain specific data about a game object we must query or other words read the internal state of the responsible " sub-system " for all objects with the unique ID assigned to this same game object.

Something like " join " processing in databases.

When you say that ...

Quote

Absolutely, although I would not say that services are read-only per se, but they are mostly read-only.

Could you please describe this a bit further ? I just want to have some idea of what services might not be " read-only. "

Quote

Notice please that the S in ECS stands for "system" (or sub-system in this manner). This makes it distinct from component based entity implementations without (sub-)systems.

Yes your " entity component system " sounds very much like that described by Adam at t-machine.org.... back in 2007 yeesh how time flies.

I actually did not understand this particular architecture when I first read about it, but your description makes things alot more understandable. So thank-you

This is the key insight ...

Quote

The (sub-)systems do this in a bulk operation, i.e. they work on the respective aspect for all managed entities in sequence. If this is done then we have an increment of the total state change done for all entities, and this is the basis for the next sub-sequent sub-system to stack up its own increment.

Ok so I think I understand the basics enough that I can research for more information as required. Thank-you

So what this means is that the " game update loop " is just a linear " pipeline " of " sub-systems " that progress from executing " game logic " -related tasks towards executing " drawing " -related tasks. There is no clear division, just a linear flow.

So for example, a " fire simulation " " sub-system" task could execute and the later a " draw fire " " sub-system " task could execute, inspect the new state of the " fire simulation "" sub-system " and generate and emit one or more " draw jobs or tasks " to be enqueued in a queue for sorting and then drawing.

So that's the " draw loop "

7 hours ago, Basket said:

So sorry for writing so many posts, I just wanted to add a bit more organization, so it woulnd't be a large wall of text.

Well, history shows that the forum is frown upon both walls of text and multiple subsequent posts of the same poster. If in doubt one can split off a thread, so that an aspect can be discussed in greater detail in a companion thread.

7 hours ago, Basket said:

Ok so I think I understand your concept of " entity management system " ... its a bit complicated, but the basic idea is that (A) game object substantiation and deletion is deferred to next iteration of the " game update loop " so as to allow each " sub-system " to process the request (B) game object substantiation and deletion must be made atomic ( that is " reversable " on error ). That's basically it

It seems a bit complicated, but w.r.t. ECS one has the need to synchronize things over sub-system borders anyway. Sending messages in an unorganized way would not work, and notifying listeners in order ist just the push data flow variant where my way is the pull data flow variant.

Regarding reversibility: Yep, I'm trying to detect error conditions early just to avoid hazardous situations. In the given case the aspect of asynchronous resource loading comes as addition into play. That resource loading means that resources may become available in the future, and of course you have to handle this in a way so that the sub-system can still work somehow.

7 hours ago, Basket said:

Your concept of " Model " is interesting, it basically amounts to a static configuration dataset used to " data-drive " the instantiation and configuration of purpose-specific objects within each " sub-system. "

That is a well formulated description. If it isn't copyrighted, I'll tend to use it in future posts ... 

7 hours ago, Basket said:

Could you please describe this a bit further ? I just want to have some idea of what services might not be " read-only. "

Already requesting creation/deletion jobs is kind of a modification. The main occurrences of non-read-only are when parts of internal components are enabled or disabled. For example, the SpatialServices manages Placement for the game objects. Those are the global positions and orientations. As such a Placement may be constraint (by mechanisms like Parenting, Targeting, Aiming, ...). While the execution of the constraint is driven by a task, the enabling is driven by service invocation.

This ...

8 hours ago, Basket said:

So what this means is that the " game update loop " is just a linear " pipeline " of " sub-systems " that progress from executing " game logic " -related tasks towards executing " drawing " -related tasks. There is no clear division, just a linear flow.

... is it. The coarse flow is from input disposal to entity management to simulation stuff to graphical rendering. Things like input catching, resource loading, and sound rendering run concurrently to the game loop.

However, this ...

8 hours ago, Basket said:

So for example, a " fire simulation " " sub-system" task could execute and the later a " draw fire " " sub-system " task could execute, inspect the new state of the " fire simulation "" sub-system " and generate and emit one or more " draw jobs or tasks " to be enqueued in a queue for sorting and then drawing.

... perhaps needs some more discussion. Depending on how fire is simulated, it may or may not be implemented in an own sub-system. Sub-systems should provide generally useable services. If a fireplace is simulated by texture clip swapping, then it is nothing special and will be handled by the generic animation sub-system. If the fire is simulated by e.g. two particle systems then also a generic sub-system is used. Only if there is a special - say - physically based simulation or an extending forest fire, then a specific sub-system is useful.

Think of duck typing: A game object made from a Model with a placement (of course), a mesh (looking like pieces of firewood), an animated billboard (fire), a particle emitter (with sparkle like particles), a second particle emitter (with smoke cloud like particles) ... gives you a fireplace. Nothing but the look of mesh and textures is specific to fire.