Modifying entities and sending updates

Started by
7 comments, last by Telanor 9 years, 7 months ago
I've run into an issue where I'm now creating entities on a different thread than the one that handles sending out entity network updates, which leads to concurrency issues, and I'm not sure what direction to take to fix it. Previously only the main thread ever created entities and it was also the thread that handled sending the updates out. But now the terrain generation is creating entities (trees, in this case) and the terrain generation runs on separate threads. What happens is that the update system tries to enumerate the components of the entity while the entity is still being constructed. I'm using an entity-component system by the way.

I could set up a system of locking an entity while it's being created/modified, but since creating an entity isn't a 1 step process, every place in the code where "new Entity" appears (meaning every entity "class"), I'd have to throw in some kind of a "entity.StartUpdate/EndUpdate" pair. If anyone ever forgets to do that... well that's an ugly bug just waiting to happen. I'd prefer a more centralized solution that isn't so easy to screw up.

The other option I see is to require entity creation only occur on the main thread. I'm not sure if a restriction like that is a good idea or not. I couldn't find anything useful on google, so I've no idea what the usual design is for such a system.
Advertisement

What happens is that the update system tries to enumerate the components of the entity while the entity is still being constructed. I'm using an entity-component system by the way.


This sounds to me like the real problem is that a reference to the entity under construction is available to other parts of your process before that entity is actually ready to be accessed.
I would recommend changing the entity creation to first create the object, not inserted to the world or visible to anything else, and then, once the object is fully created and configured, insert it into the world and make it visible to the rest of the process.

"Entity created" events can then come from the world itself, when an entity is inserted into it.
enum Bool { True, False, FileNotFound };
I suppose that would solve the issue during creation time. There'd still have to be an extra call at the end of creating the entity but at least forgetting that would just result in a useless entity rather than a crashing bug, so that's a plus. What about entity modification though? Suppose I want to add a component to an entity after it's already in the world.

What about entity modification though? Suppose I want to add a component to an entity after it's already in the world.


The way I would solve that would be to not allow component add/remove while in the world; instead, I'd require you to remove the object from the world, re-configure components, and re-add it to the world.

Also, the "generate state dump for network update" and "add object to world" operations would probably use the same atomic unit (lock, or thread.) Just slurping the data out of the object is really lightweight, so putting that on the main thread wouldn't likely be a performance problem; any formatting and packetizing (and visibility management etc) could be done in another network-related thread if you want.
enum Bool { True, False, FileNotFound };

There'd still have to be an extra call at the end of creating the entity but at least forgetting that would just result in a useless entity rather than a crashing bug, so that's a plus.


You can easily remove the need for extra calls with a little function composition. In C++, you could write something like:

SpawnGameObject(blueprintName, [&](GameObject* obj)
{
  obj->SetPosition(4, 5);
  obj->GetComponent("a")->SetFoo(1);
  obj->GetComponent("b")->SetFoo(2);
});
So the function you pass in does all the post-creation pre-replication configuration work. The implementation is super simple:

void SpawnGameObject(std::string parent, std::function<void(GameObject*)> configure) {
  GameObject* object = CreateObject(parent);

  if (configure)
    configure(object);

  object->FinishInitialization();
}
You can use that pattern whenever you have chunks of code you need wrapped in other chunks of code. The RAII pattern also works but is clunkier for use cases like this.

What about entity modification though? Suppose I want to add a component to an entity after it's already in the world.


First, do you really need to do that? I've worked on several big component-based games now that never had any use case for adding/removing components at runtime, and most engines I'm familiar with don't even allow it. For many cases you can't enable/disable the component, add or remove message listeners, or register/deregister the component with external systems. The object component structure itself stays unchanged.

Second, you can make the network layer resistant to object structure changing. There are a billion articles online about concurrency-safe data structures, including a concurrent arrays. I don't think you need or want this, though.

What I usually do is have the network thread collect update requests from the logic thread (so the logic thread can decide when to send updates when it's safe to do so) to send and receives update packets and just queues them up on the logic thread. This is important for gameplay, really, if you think about.

You only want to sent consistent snapshots of components over the network. E.g. if two component variables need to be in a particular combination, you don't want the network thread reading those variables right in the middle of you changing them. Position and rotation are good examples of these; you don't want to send half of a physics update! If instead your game logic is responsible for sending an event to the network thread with the updated variables only when it's ready, the component can be sure that both values are updated in sync before sending it over.

Likewise for receiving updates, you don't want incoming changes to be applied to objects in the middle of your logic. Physics is still a decent example - if the code just did some work with its current position but then gets a new position+rotation right before it uses the rotation, you're going to see wonky physics behavior. It's much better to be sure that the network thread is only telling the logic thread that updates are available and then letting the logic thread apply them when appropriate.

Generally, this is true for almost all threading. Just don't have multiple thread modify the same data, period. Don't have multiple threads read the same data unless it's constant/immutable. Make threads communicate over thread-safe message queues. This fundamental approach to threading is what makes working in highly-concurrent languages like Eiffel or Go so much easier, safer, and often even more performant than the willy-nilly threading seen in C and C++ apps.

Sean Middleditch – Game Systems Engineer – Join my team!

Simply have a logic thread system that runs which collects dirty/changed notifications for game objects. It queues those up on the logic thread and then at a sync point, this system will synchronize with the network thread, swapping the contents of the logic thread's send queue for the network thread's receive queue. The system can then decode the network packets that were received, translate them into commands and dispatch commands to the logic thread's command queue which will be inspected and interpolated in the current/next frame depending on where you place your network logic system update pass (end of loop vs top of loop); i prefer top right after "input".

After looking through my code I found that I'm not actually doing any addition/removal of components after the initial setup phase, so perhaps I don't need to allow that. I suppose if I did need to allow that, I could pass a lambda function to a ModifyEntity function which the main thread could run at an appropriate time.

The function composition is a good idea Sean, shame I didn't set it up that way from the start. Refactoring that is probably going to be quite a pain, more-so because I'm using c# and lambda functions that use capture end up causing allocations, so I'll have to rearrange things to avoid any capturing.

I think this'll probably solve the issue though, so thanks for the input guys.
Which C# environment are you using? The Microsoft CLR is very good, and if you capture a few variables per frame, there will be no measurable GC penalty for this -- it will never "stop and collect" for a long time.
If you're using Mono, well, I feel your pain :-)
enum Bool { True, False, FileNotFound };
Microsoft, but the physics library already puts enough pressure on the GC to cause occasional pauses, so I try to keep the GC pressure to a minimum where possible. I admit I'm not hugely familiar with the inner workings of the GC, so maybe pressure from short lived objects doesn't matter, but it can't hurt to try to keep it to minimum

This topic is closed to new replies.

Advertisement