I'm designing a top-down twitchy space shooter a la Subspace/Continuum, and am in the process of doing research to plan my server architecture (built in C# with lidgren-gen3, communicating to a Unity Web Client for the game client -- very slick stuff). I'm primarily basing it off of the FPS model, with UDP packets sent at regular intervals to players (20-30 Hz). I have a number of questions regarding these packets.
In particular, I studied David Aldridge's Halo Reach GDC talk, where he talks about flow control and the server model, and I want to adopt something a little simpler to begin with. My simpler system goes a little something like this:
Client:
- Sends control information to the server
- Control information is immediately realized locally
- Other entities in the world are Interpolate based on past received state unless new state contradicts (dead reckoning, etc.)
- (Possible expansion: Receive other players' control input and use that for prediction)
Server:
- Receives control information from clients and modifies the world accordingly
- Each tick creates a new immutable state from the last, to avoid errors caused by update order
- Dispatches delta-compressed state updates to each client regularly (based on their last acknowledged received state)
- If bandwidth allows, dispatches events to justify change in state (weapon fired, bomb exploded, etc.)
- (Possible expansion: Distribute other players' control input to clients for prediction)
I plan on running at least two threads. These will be:
1 Communication thread:
- Handles sending data, and dispatching received data to wherever it needs to go
1..n Arena threads:
- Simulate a single arena's world model and state, receiving information from a communication thread
The arena threads might be multiple threads to manage different sectors of an arena, but let's assume that one arena is one thread.
There's the relevant details, now we're getting to my question. The Halo Reach server paradigm has a Flow Controller, which tells the game logic manager "I have this much room for a packet to send to player X, how would you like to fill it?" then the game logic manager prioritizes state updates and events, and returns a packing of that packet space based on what information is most important and relevant for player X. If I'm understanding that incorrectly, stop me here.
This is complicated with threads, since I'm guessing all of my communication will be through C# lockless queues. Receiving messages and telling the Arena thread about them is fine. The problem is, how do I say the following things (let's pretend this is a chat log between the Flow Controller and the Arena thread):
---
<Flow_Controller>: Arena thread, here's a packet for you of size N, time to fill it.
(i.e. How do I tell the Arena thread that it's time to send out messages, OR should the Arena thread constantly be sending out messages by priority?)
---
<Arena_Thread>: Flow controller, remember that low-priority state update I sent you a minute ago that you haven't sent over the network yet? Well, it isn't relevant now, so forget about it.
(i.e. Once a state update is already in the queue, how can the Arena thread invalidate it so it won't be sent out?)
---
<Arena_Thread>: Flow controller, remember that low-priority state update I sent you a minute ago that you haven't sent over the network yet? Well, here's a new state update about that same entity that's now really important, so forget about the original update and send this one.
(i.e. A state update that was previously enqueued as very low priority message is now really important and should be sent out ASAP.)
---
There's two ways of approaching this, as I see it:
1) 20-30 times a second, the Flow Controller tells the Arena thread that it's time to send a packet, and we can send x bytes for player X, and y bytes for player Y, etc. Once the Arena thread gets that message, it pauses simulating, picks the best messages to send to each player based on those size restrictions, enqueues them for the Flow Controller, and goes back to simulating.
Problem: There will be a delay between when the Flow Controller asks for the packet, and the Arena thread gets around to filling it. The Arena thread also has to stop for a period of time (that grows with each connected player).
2) The Arena thread is constantly, asynchronously outputting messages with priorities, maintaining, updating, and deleting old messages to that at any time the Flow Controller wants it, there's a nice up-to-date list of messages to send out with correct priorities based on the current (or very recent) world state.
Problem: Standard mutual exclusion nightmare. How does the Arena thread constantly update and maintain this list while the Flow controller is reading it and consuming messages it sent out? This sounds like a lot of locks, and performance compromise. The Arena thread will also have to maintain one of these lists for each player.
So, am I completely off-track here? I really like the idea of prioritizing messages and selecting the highest-priority messages to send to each player based on bandwidth. I'd like to implement it if possible, but if this sounds completely crazy, I guess I could just try the Quake3/Valve model. Any ideas on how to do this packet-packing problem with inter-thread communication would be quite wonderful. Thanks!
Show differencesHistory of post edits
#2sufficientreason
Posted 02 February 2012 - 11:47 PM
I'm designing a top-down twitchy space shooter a la Subspace/Continuum, and am in the process of doing research to plan my server architecture (built in C# with lidgren-gen3, communicating to a Unity Web Client for the game client -- very slick stuff). I'm primarily basing it off of the FPS model, with UDP packets sent at regular intervals to players (20-30 Hz). I have a number of questions regarding these packets.
In particular, I watched David Aldridge's Halo Reach GDC talk, where he talks about flow control and the server model, and I want to adopt something a little simpler to begin with. My simpler system goes a little something like this:
Client:
- Sends control information to the server
- Control information is immediately realized locally
- Other entities in the world are Interpolate based on past received state unless new state contradicts (dead reckoning, etc.)
- (Possible expansion: Receive other players' control input and use that for prediction)
Server:
- Receives control information from clients and modifies the world accordingly
- Each tick creates a new immutable state from the last, to avoid errors caused by update order
- Dispatches delta-compressed state updates to each client regularly (based on their last acknowledged received state)
- If bandwidth allows, dispatches events to justify change in state (weapon fired, bomb exploded, etc.)
- (Possible expansion: Distribute other players' control input to clients for prediction)
I plan on running at least two threads. These will be:
1 Communication thread:
- Handles sending data, and dispatching received data to wherever it needs to go
1..n Arena threads:
- Simulate a single arena's world model and state, receiving information from a communication thread
The arena threads might be multiple threads to manage different sectors of an arena, but let's assume that one arena is one thread.
There's the relevant details, now we're getting to my question. The Halo Reach server paradigm has a Flow Controller, which tells the game logic manager "I have this much room for a packet to send to player X, how would you like to fill it?" then the game logic manager prioritizes state updates and events, and returns a packing of that packet space based on what information is most important and relevant for player X. If I'm understanding that incorrectly, stop me here.
This is complicated with threads, since I'm guessing all of my communication will be through C# lockless queues. Receiving messages and telling the Arena thread about them is fine. The problem is, how do I say the following things (let's pretend this is a chat log between the Flow Controller and the Arena thread):
---
<Flow_Controller>: Arena thread, here's a packet for you of size N, time to fill it.
(i.e. How do I tell the Arena thread that it's time to send out messages, OR should the Arena thread constantly be sending out messages by priority?)
---
<Arena_Thread>: Flow controller, remember that low-priority state update I sent you a minute ago that you haven't sent over the network yet? Well, it isn't relevant now, so forget about it.
(i.e. Once a state update is already in the queue, how can the Arena thread invalidate it so it won't be sent out?)
---
<Arena_Thread>: Flow controller, remember that low-priority state update I sent you a minute ago that you haven't sent over the network yet? Well, here's a new state update about that same entity that's now really important, so forget about the original update and send this one.
(i.e. A state update that was previously enqueued as very low priority message is now really important and should be sent out ASAP.)
---
There's two ways of approaching this, as I see it:
1) 20-30 times a second, the Flow Controller tells the Arena thread that it's time to send a packet, and we can send x bytes for player X, and y bytes for player Y, etc. Once the Arena thread gets that message, it pauses simulating, picks the best messages to send to each player based on those size restrictions, enqueues them for the Flow Controller, and goes back to simulating.
Problem: There will be a delay between when the Flow Controller asks for the packet, and the Arena thread gets around to filling it. The Arena thread also has to stop for a period of time (that grows with each connected player).
2) The Arena thread is constantly, asynchronously outputting messages with priorities, maintaining, updating, and deleting old messages to that at any time the Flow Controller wants it, there's a nice up-to-date list of messages to send out with correct priorities based on the current (or very recent) world state.
Problem: Standard mutual exclusion nightmare. How does the Arena thread constantly update and maintain this list while the Flow controller is reading it and consuming messages it sent out? This sounds like a lot of locks, and performance compromise. The Arena thread will also have to maintain one of these lists for each player.
So, am I completely off-track here? I really like the idea of prioritizing messages and selecting the highest-priority messages to send to each player based on bandwidth. I'd like to implement it if possible, but if this sounds completely crazy, I guess I could just try the Quake3/Valve model. Any ideas on how to do this packet-packing problem with inter-thread communication would be quite wonderful. Thanks!
In particular, I watched David Aldridge's Halo Reach GDC talk, where he talks about flow control and the server model, and I want to adopt something a little simpler to begin with. My simpler system goes a little something like this:
Client:
- Sends control information to the server
- Control information is immediately realized locally
- Other entities in the world are Interpolate based on past received state unless new state contradicts (dead reckoning, etc.)
- (Possible expansion: Receive other players' control input and use that for prediction)
Server:
- Receives control information from clients and modifies the world accordingly
- Each tick creates a new immutable state from the last, to avoid errors caused by update order
- Dispatches delta-compressed state updates to each client regularly (based on their last acknowledged received state)
- If bandwidth allows, dispatches events to justify change in state (weapon fired, bomb exploded, etc.)
- (Possible expansion: Distribute other players' control input to clients for prediction)
I plan on running at least two threads. These will be:
1 Communication thread:
- Handles sending data, and dispatching received data to wherever it needs to go
1..n Arena threads:
- Simulate a single arena's world model and state, receiving information from a communication thread
The arena threads might be multiple threads to manage different sectors of an arena, but let's assume that one arena is one thread.
There's the relevant details, now we're getting to my question. The Halo Reach server paradigm has a Flow Controller, which tells the game logic manager "I have this much room for a packet to send to player X, how would you like to fill it?" then the game logic manager prioritizes state updates and events, and returns a packing of that packet space based on what information is most important and relevant for player X. If I'm understanding that incorrectly, stop me here.
This is complicated with threads, since I'm guessing all of my communication will be through C# lockless queues. Receiving messages and telling the Arena thread about them is fine. The problem is, how do I say the following things (let's pretend this is a chat log between the Flow Controller and the Arena thread):
---
<Flow_Controller>: Arena thread, here's a packet for you of size N, time to fill it.
(i.e. How do I tell the Arena thread that it's time to send out messages, OR should the Arena thread constantly be sending out messages by priority?)
---
<Arena_Thread>: Flow controller, remember that low-priority state update I sent you a minute ago that you haven't sent over the network yet? Well, it isn't relevant now, so forget about it.
(i.e. Once a state update is already in the queue, how can the Arena thread invalidate it so it won't be sent out?)
---
<Arena_Thread>: Flow controller, remember that low-priority state update I sent you a minute ago that you haven't sent over the network yet? Well, here's a new state update about that same entity that's now really important, so forget about the original update and send this one.
(i.e. A state update that was previously enqueued as very low priority message is now really important and should be sent out ASAP.)
---
There's two ways of approaching this, as I see it:
1) 20-30 times a second, the Flow Controller tells the Arena thread that it's time to send a packet, and we can send x bytes for player X, and y bytes for player Y, etc. Once the Arena thread gets that message, it pauses simulating, picks the best messages to send to each player based on those size restrictions, enqueues them for the Flow Controller, and goes back to simulating.
Problem: There will be a delay between when the Flow Controller asks for the packet, and the Arena thread gets around to filling it. The Arena thread also has to stop for a period of time (that grows with each connected player).
2) The Arena thread is constantly, asynchronously outputting messages with priorities, maintaining, updating, and deleting old messages to that at any time the Flow Controller wants it, there's a nice up-to-date list of messages to send out with correct priorities based on the current (or very recent) world state.
Problem: Standard mutual exclusion nightmare. How does the Arena thread constantly update and maintain this list while the Flow controller is reading it and consuming messages it sent out? This sounds like a lot of locks, and performance compromise. The Arena thread will also have to maintain one of these lists for each player.
So, am I completely off-track here? I really like the idea of prioritizing messages and selecting the highest-priority messages to send to each player based on bandwidth. I'd like to implement it if possible, but if this sounds completely crazy, I guess I could just try the Quake3/Valve model. Any ideas on how to do this packet-packing problem with inter-thread communication would be quite wonderful. Thanks!
#1sufficientreason
Posted 02 February 2012 - 11:34 PM
I'm designing a top-down twitchy space shooter a la Subspace/Continuum, and am in the process of doing research to plan my server architecture (built in C# with lidgren-gen3, communicating to a Unity Web Client for the game client -- very slick stuff). I'm primarily basing it off of the FPS model, with UDP packets sent at regular intervals to players (20-30 Hz). I have a number of questions regarding these packets.
In particular, I watched David Aldridge's Halo Reach GDC talk, where he talks about flow control and the server model, and I want to adopt something a little simpler to begin with. My simpler system goes a little something like this:
Client:
- Sends control information to the server
- Control information is immediately realized locally
- Other entities in the world are Interpolate based on past received state unless new state contradicts (dead reckoning, etc.)
- (Possible expansion: Receive other players' control input and use that for prediction)
Server:
- Receives control information from clients and modifies the world accordingly
- Each tick creates a new immutable state from the last, to avoid errors caused by update order
- Dispatches delta-compressed state updates to each client regularly (based on their last acknowledged received state)
- If bandwidth allows, dispatches events to justify change in state (weapon fired, bomb exploded, etc.)
- (Possible expansion: Distribute other players' control input to clients for prediction)
I plan on running at least two threads. These will be:
1 Communication thread:
- Handles sending data, and dispatching received data to wherever it needs to go
1..n Arena threads:
- Simulate a single arena's world model and state, receiving information from a communication thread
The arena threads might be multiple threads to manage different sectors of an arena, but let's assume that one arena is one thread.
There's the relevant details, now we're getting to my question. The Halo Reach server paradigm has a Flow Controller, which tells the game logic manager "I have this much room for a packet to send to player X, how would you like to fill it?" then the game logic manager prioritizes state updates and events, and returns a packing of that packet space based on what information is most important and relevant for player X. If I'm understanding that incorrectly, stop me here.
This is complicated with threads, since I'm guessing all of my communication will be through C# lockless queues. Receiving messages and telling the Arena thread about them is fine. The problem is, how do I say the following things (let's pretend this is a chat log between the Flow Controller and the Arena thread):
---
<Flow_Controller>: Arena thread, here's a packet for you of size N, time to fill it.
(i.e. How do I tell the Arena thread that it's time to send out messages, OR should the Arena thread constantly be sending out messages by priority?)
---
<Arena_Thread>: Flow controller, remember that low-priority state update I sent you a minute ago that you haven't sent over the network yet? Well, it isn't relevant now, so forget about it.
(i.e. Once a state update is already in the queue, how can the Arena thread invalidate it so it won't be sent out?)
---
<Arena_Thread>: Flow controller, remember that low-priority state update I sent you a minute ago that you haven't sent over the network yet? Well, here's a new state update about that same entity that's now really important, so forget about the original update and send this one.
(i.e. A state update that was previously enqueued as very low priority message is now really important and should be sent out ASAP.)
---
There's two ways of approaching this, as I see it:
1) 20-30 times a second, the Flow Controller tells the Arena thread that it's time to send a packet, and we can send x bytes for player X, and y bytes for player Y, etc. Once the Arena thread gets that message, it pauses simulating, picks the best messages to send to each player based on those size restrictions, enqueues them for the Flow Controller, and goes back to simulating.
Problem: There will be a delay between when the Flow Controller asks for the packet, and the Arena thread gets around to filling it. The Arena thread also has to stop for a period of time (that grows with each connected player).
2) The Arena thread is constantly, asynchronously outputting messages with priorities, maintaining, updating, and deleting old messages to that at any time the Flow Controller wants it, there's a nice up-to-date list of messages to send out with correct priorities based on the current (or very recent) world state.
Problem: Standard mutual exclusion nightmare. How does the Arena thread constantly update and maintain this list while the Flow controller is reading it and consuming messages it sent out? This sounds like a lot of locks, and performance compromise. The Arena thread will also have to maintain one of these lists for each player.
So, am I completely off-track here? I really like the idea of prioritizing messages and selecting the highest-priority messages to send to each player based on bandwidth. I'd like to implement it if possible, but if this sounds completely crazy, I guess I could just try the Quake3/Valve model. Any ideas on how to do this packet-packing problem with inter-thread communication would be quite wonderful. Thanks!
In particular, I watched David Aldridge's Halo Reach GDC talk, where he talks about flow control and the server model, and I want to adopt something a little simpler to begin with. My simpler system goes a little something like this:
Client:
- Sends control information to the server
- Control information is immediately realized locally
- Other entities in the world are Interpolate based on past received state unless new state contradicts (dead reckoning, etc.)
- (Possible expansion: Receive other players' control input and use that for prediction)
Server:
- Receives control information from clients and modifies the world accordingly
- Each tick creates a new immutable state from the last, to avoid errors caused by update order
- Dispatches delta-compressed state updates to each client regularly (based on their last acknowledged received state)
- If bandwidth allows, dispatches events to justify change in state (weapon fired, bomb exploded, etc.)
- (Possible expansion: Distribute other players' control input to clients for prediction)
I plan on running at least two threads. These will be:
1 Communication thread:
- Handles sending data, and dispatching received data to wherever it needs to go
1..n Arena threads:
- Simulate a single arena's world model and state, receiving information from a communication thread
The arena threads might be multiple threads to manage different sectors of an arena, but let's assume that one arena is one thread.
There's the relevant details, now we're getting to my question. The Halo Reach server paradigm has a Flow Controller, which tells the game logic manager "I have this much room for a packet to send to player X, how would you like to fill it?" then the game logic manager prioritizes state updates and events, and returns a packing of that packet space based on what information is most important and relevant for player X. If I'm understanding that incorrectly, stop me here.
This is complicated with threads, since I'm guessing all of my communication will be through C# lockless queues. Receiving messages and telling the Arena thread about them is fine. The problem is, how do I say the following things (let's pretend this is a chat log between the Flow Controller and the Arena thread):
---
<Flow_Controller>: Arena thread, here's a packet for you of size N, time to fill it.
(i.e. How do I tell the Arena thread that it's time to send out messages, OR should the Arena thread constantly be sending out messages by priority?)
---
<Arena_Thread>: Flow controller, remember that low-priority state update I sent you a minute ago that you haven't sent over the network yet? Well, it isn't relevant now, so forget about it.
(i.e. Once a state update is already in the queue, how can the Arena thread invalidate it so it won't be sent out?)
---
<Arena_Thread>: Flow controller, remember that low-priority state update I sent you a minute ago that you haven't sent over the network yet? Well, here's a new state update about that same entity that's now really important, so forget about the original update and send this one.
(i.e. A state update that was previously enqueued as very low priority message is now really important and should be sent out ASAP.)
---
There's two ways of approaching this, as I see it:
1) 20-30 times a second, the Flow Controller tells the Arena thread that it's time to send a packet, and we can send x bytes for player X, and y bytes for player Y, etc. Once the Arena thread gets that message, it pauses simulating, picks the best messages to send to each player based on those size restrictions, enqueues them for the Flow Controller, and goes back to simulating.
Problem: There will be a delay between when the Flow Controller asks for the packet, and the Arena thread gets around to filling it. The Arena thread also has to stop for a period of time (that grows with each connected player).
2) The Arena thread is constantly, asynchronously outputting messages with priorities, maintaining, updating, and deleting old messages to that at any time the Flow Controller wants it, there's a nice up-to-date list of messages to send out with correct priorities based on the current (or very recent) world state.
Problem: Standard mutual exclusion nightmare. How does the Arena thread constantly update and maintain this list while the Flow controller is reading it and consuming messages it sent out? This sounds like a lot of locks, and performance compromise. The Arena thread will also have to maintain one of these lists for each player.
So, am I completely off-track here? I really like the idea of prioritizing messages and selecting the highest-priority messages to send to each player based on bandwidth. I'd like to implement it if possible, but if this sounds completely crazy, I guess I could just try the Quake3/Valve model. Any ideas on how to do this packet-packing problem with inter-thread communication would be quite wonderful. Thanks!