I'm finally restructuring my base actor class and want to minimize the overhead of data synchronization between threads. In particular, below is the scheme I'm currently considering. It has a couple of issues:

1) I'd like to constrain synchronization to atomics only, but I don't really see a way

2) handling prediction in the render thread

3) extensibility and future-proofness

4) clarity and modularity (this approach isn't modular at all, since synchronization is tied to one central class, IGame)

5) I want to come up with a system so that each thread has its own copy of the data it needs, but no more. The question becomes where and how to store this data. Don't really want to bog down IActor, but don't really want to create a disjoint cache either.

Here's an example of what I have right now. It's a hand-written example, so it might contain some mistakes and does contain omissions for clarity.

struct ISceneComponent {
  private:
    friend class IGame;
    friend class IPhysicsEnv;

    TIMESTAMP timestamp;

    IWorldTransform transform;
    ...

    std::mutex _mutex;
};

struct IRenderable { 
  private:
    friend class IGame; 

    IWorldTransform transform;
    TIMESTAMP timestamp;

    IWorldTransform transformCurFrame;

    ...

  public:

    //NOTE: does not block in case transformCurFrame is being update in IGame::UpdateActorRenderable()
    const IWorldTransform& GetTransform() const { return transformCurFrame; }
};

struct IActor {
  ISceneComponent scene;
  IRenderable renderable;
};

//#physics thread
void IPhysicsEnv::MoveActor(IActor& actor)
{
//do heavy work
actor.scene._mutex.lock();
actor.scene.transform = newTransform;
...
actor.scene.timestamp = GetTimeStamp();
actor.scene._mutex.unlock();
}

void IGame::UpdateActorRenderable(IActor& actor)
{
   if(actor.scene.timestamp != actor.renderable.timestamp)
      {
      //need to use mutex here, because don't want to miss any updates in IPhysicsEnv::MoveActor,
      //although render thread can live with old data a bit longer
      if(actor.scene._mutex.try_lock())
        {
        actor.renderable.transform = actor.scene.transform;
        ...
        actor.scene._mutex.unlock();
        }
      }

   //due to using try_lock, updates can be missed (although this should be very rare) and fraction can become > 1
   real fraction = GetCurFractionOfPhysicsUpdate(GetTimeStamp());
   //currently IActor::renderable is accessible from any thread, but the copy is local, which is get and set without a lock; want to change this
   actor.renderable.transformCurFrame = TRANSFORM_Move(actor.renderable.transform, actor.renderable.orientation * actor.renderable.velocity * fraction);
}

//#render thread
void RenderScene()
{
  game->UpdateActorRenderable(player);
  DrawSmallPenis(player.renderable.GetTransform());
}

The good part about this is that it works. The part about it that I don't necessarily like is the fact that it requires each actor to have its own lock per sync pair and that the physics thread can be blocked. Also, it works well with just two modules communicating with each other, but I'm not sure if it's the best solution for, say, x-way communication when I add in AI, pathfinding and whatever other modules that all run on different threads. Handling contention (eg choosing the latest timestamp) is fairly trivial this way, but then again offloading synchronization to some form of external manager system would be more extensible further down the line.

I guess it boils down to where and how do I keep and update local copies. The problem stems from the fact that I don't really know what I'm doing here (as far as writing a game engine from scratch for the first time goes) so I don't know quite how to best prepare for problems I can't foresee. You know - it's that uneasy feeling that goes something like "this is nice, but somehow I feel it's not quite what it's supposed to be".

That being said, suggestion would be welcome!

You could try a singular synchronization point within your game loop where all actors have their state sync'd, instead of having each actor be responsible for syncing its own state whenever it's ready. Something like this:

while(true)
{
   // run physics frame
   physicsThread.doFrame();

   // render until physics frame is done
   do
   {
      doRenderFrame();
   } while (!physicsThread.done());

   // sync physics state to render state
   syncState();
}

Whenever new state from frame N is synced to a renderable, you cache off the old state from frame N-1, and interpolate between the two during the next frame. In this way the renderer doesn't need to do any prediction, since it already has two known states, but you still have the option to if physics falls behind.

As for where the data lives, I think you're doing fine with each component storing the relevant data used by its respective system. The real issue is who is accessing that data and when. If you add more systems that run on separate threads, you'll have to start double-buffering shared state to eliminate contention.