Sign in to follow this  

Engine distributed across several machine adivces

Recommended Posts


I have this project of mine that i am trying to put on it's feet. I want to make a video mapping engine distributed across several machine, or if you prefer a interactive installation engin were several machine has different role.

in example:
we have
0..N GUI connected,
1..N 3d player (machine connected to a projector),
maybe N or just one server and
1 data database.

All those process have their role and have to communicate over network.
- Gui are really just a user interface to do live interaction, or to assemble and integrate an installation.
- 3d player are just rendering and receiving the new updated data.
- Server update the scene and send the changes to GUI and 3d player.
- Data database is where the object serialized would live and ensure no 2 gui modify data at the same time (I am not sure yet about this process)

Now where i need some advice are of the design of the communication and serializing the changes onto the network. Also I am thinking of using an entity component system because i think this way i could distribute the actual update of the scene. An other question I had is the actual gain I can get from an paralelnized entity component system, for example I have a thread that update the TransformationComponent, but this component trigger more thread for independent braches of the tree, to me that look like potentially have a problem of too many thread. I would also ideally go a step further and not just paralelnize by "task" or component but by independent data.

So to recapitulate I need advice on how to takle the communication, do a simple RPC would suffice, I think i'll need a combination of remote function call and sending data.
How to distribute load on different thread and process.

I know those are not easy subject, but i think that is why it is interesting to talk about them so don't be shy and ask me question if you don't understand or post you advice

thanks alot


p.s. and if it is not interesting enough I have an other question how to work with live video where it need to get to all player and server potentially and being process for effect :)

Share this post

Link to post
Share on other sites
Yes I can, but I don't have a concret installation concept, but I do know what I want to be able to do at minimum with the engin.

Ok If we keep it simple, lets take for example video mapping. Lets do a projection on a car and suppose you need 2 projectors to cover the car. So you need 2 "3dPlayer" machine, a "Server" machine, and a "GUI" machine.
You would construct the 3d scene with the GUI, add 1080p video to be mapped on the cars, you add and manipulate the 3d models of the cars in the 3d scene to setup the video mapping, and add post process and correct for keystone and blending between the 2 projectors and lets say that you also add a particle emitter on the car. So the GUI setup and control the 3d scene.
Then all those 3d objects have to me updated, particle simulated..... That is the role of the server, it will process, update the scene.
Then to show an image with the projector we need to render the scene that is the job of the 3d player.

All those machine have to communicate and that is a part of what i need advice on. How to design a communication protocol for this kind of distributed/cloud processing.

For example:
With the gui you create a particle system. The gui will send the Particle System creation command to the server. The server will process it, a particle system will be added to the scene graph. Then since the scene graph have been modified, the gui will receive a msg or the object serialize, so that it can updated it's views on the data. The 3d players will also be notified and he will create the 3d representation of a particle system. Then the server will update the particle system in parallel, maybe with other particle system or each particles will be updated by several thread, (but not 1 thread per particle ;) ) So the 3d player need to receive the update of the position for each of the particles.

Anybody have idea how to takle this problem?

I have a reflection system that I will use for serializing to disk the data, I tough of using it also to send the updates on the network, so that the implementor of the class can "tag" the field to be serialized on the network.

I hope that clarify and hopefully i'll get some advices.

thanks all

Share this post

Link to post
Share on other sites
This isn't much different than how networked 3D FPS games work, they support N clients all looking at a single coherent 3D world were the server usually a dedicated machine connects and synchronizes all the actors. There are some generic network API which touch upon these concepts but they are horrible bloated and not really suitable for realtime work which you need. I've not used them myself but if you want to research more you can visit this link..


[EDIT: also there are a few opensource projects touching upon your needs such as ]

All FPS games roll their own network infrastructure, you can peruse open sourced games to get an idea of how they do it. Essentially like a FPS game you have to design a protocol for object creation / update / destruction. Objects states are lazily synchronized based upon game logic and bandwidth availability. The most common network topology is just a simple star, where the server is the center node. There are more complex network topology if you want to support say upwards of 32+ clients since the server eventually becomes output bound unless you have massive bandwidth..

For more fancy implementation you can implement a virtual file service where files are shared between clients on this VFS and they get grab a handle and start streaming data from it, to them it looks like a file handle or a handle to some resource like a video stream..This has several benefits it abstracts away local / remote file, making the program simpler and the client doesn't have to handle the low level socket layer complexities..

Many AAA games have remote connect and edit abilities, because they run on a console using these concepts. So it should work for u too i think..

Good Luck!


Share this post

Link to post
Share on other sites
Thanks for the link i'll give it a look.

There is a difference between a FPS and what i want to do. Usually a Client would update it's own scene and get approximation of data and extrapolate for it. For a FPS that can be enough, but not for my application. And I want to push it a bit further, were the "client" would only render and the servers on different machine would update and send the data to the clients. It is alot of data and it need to be efficient and have a way to know what to send.

Still I think to look at how FPS does it is a start at to know the design and architecture that exist. I would need some idea of technologies or design pattern or keyword to help me look in to this. Or like how to takle a problem where the position of a player is updated on one machine but the ai is updated on an other and so that they share the result of the updates?

thanks again

Share this post

Link to post
Share on other sites
It's not that much data really, it's more hardware. You can do realtime compression on the server side, setup a GPU farm to service multiple clients (they will also encode and stream it to the clients). This is what cloud gaming services like Onlive and Gakai do.. There are major complexity issues with trying to run a distributed simulation and trying to maintain coherence between the nodes. RTS approach the problem by creating completely deterministic simulation and explicitly controlling the inputs ( using a multistage update process where clients send all their inputs to the server who then sends each user the complete set of inputs that update step). That is how a game like StarCraft can synchronize an unlimited number of variables real time against 4 players. This won't scale well but it will allow you to synchronize massively complex simulations. There is alot of smoke and mirrors ofcoruse to hide the inherent latency. But in theory the update time steps are only limited to the speed and quality of your clients connection so maybe wi-fi u can run it really fast?

Good Luck!


Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this